Python’s logging ecosystem offers powerful tools for managing application logs effectively. Let’s explore five essential libraries that enhance logging capabilities.
Loguru stands out as a modern logging solution that eliminates common setup hurdles. Its intuitive API provides immediate value with minimal configuration:
from loguru import logger
logger.add("app.log", rotation="500 MB", compression="zip")
logger.info("Starting application")
try:
result = 1 / 0
except Exception as e:
logger.exception("An error occurred")
The standard logging module remains fundamental to Python applications. It provides granular control over log handling:
import logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
filename='application.log'
)
logger = logging.getLogger(__name__)
logger.info("Processing started")
logger.error("Failed to connect to database")
Python-json-logger transforms logs into JSON format, making them ideal for modern log aggregation systems:
from pythonjsonlogger import jsonlogger
import logging
logger = logging.getLogger()
logHandler = logging.StreamHandler()
formatter = jsonlogger.JsonFormatter()
logHandler.setFormatter(formatter)
logger.addHandler(logHandler)
logger.info("User login", extra={"user_id": 123, "ip": "192.168.1.1"})
Structlog brings structured logging to Python, making logs more consistent and analyzable:
import structlog
logger = structlog.get_logger()
logger.info("order_processed",
order_id=12345,
price=99.99,
status="completed"
)
Eliot excels at tracking complex operations across system components:
from eliot import start_action, to_file
to_file(open("eliot.log", "w"))
with start_action(action_type="process_order", order_id=123):
with start_action(action_type="validate_payment"):
# Payment validation logic
pass
with start_action(action_type="update_inventory"):
# Inventory update logic
pass
Each library serves specific logging needs. Loguru suits developers seeking simplicity and immediate functionality. The standard logging module provides extensive customization options for complex applications. Python-json-logger integrates seamlessly with modern logging infrastructure. Structlog ensures consistent log formatting across large applications. Eliot helps track complex workflows through detailed action chains.
Advanced logging patterns enhance debugging and monitoring capabilities:
# Contextual logging with structlog
import structlog
from typing import Any, Dict
def get_logger(context: Dict[str, Any] = None):
logger = structlog.get_logger()
if context:
return logger.bind(**context)
return logger
# Usage
order_logger = get_logger({"service": "order_processing"})
order_logger.info("new_order", order_id=12345)
Rotation and retention policies prevent log files from consuming excessive storage:
from loguru import logger
import sys
# Configure multiple outputs with different retention policies
logger.add("errors.log", rotation="100 MB", retention="10 days", level="ERROR")
logger.add("debug.log", rotation="12:00", compression="zip", level="DEBUG")
logger.add(sys.stderr, format="{time} {level} {message}", filter="my_module")
Structured error handling improves debugging efficiency:
import structlog
from typing import Optional
class CustomError(Exception):
def __init__(self, message: str, context: Optional[dict] = None):
super().__init__(message)
self.context = context or {}
logger = structlog.get_logger()
def process_data(data_id: int):
try:
# Processing logic
raise ValueError("Invalid data format")
except Exception as e:
logger.error("data_processing_failed",
error_type=type(e).__name__,
error_message=str(e),
data_id=data_id
)
raise CustomError("Data processing failed", {
"data_id": data_id,
"original_error": str(e)
})
Performance optimization through asynchronous logging:
import asyncio
from loguru import logger
import threading
from queue import Queue
class AsyncLogger:
def __init__(self):
self.queue = Queue()
self.thread = threading.Thread(target=self._worker, daemon=True)
self.thread.start()
def _worker(self):
while True:
log_entry = self.queue.get()
if log_entry is None:
break
logger.info(log_entry)
def log(self, message: str):
self.queue.put(message)
def shutdown(self):
self.queue.put(None)
self.thread.join()
# Usage
async_logger = AsyncLogger()
async_logger.log("Processing started")
Integration with external monitoring systems:
import logging
import json
from typing import Any, Dict
from elasticsearch import Elasticsearch
class ElasticsearchHandler(logging.Handler):
def __init__(self, host: str, index: str):
super().__init__()
self.es = Elasticsearch([host])
self.index = index
def emit(self, record: logging.LogRecord):
try:
log_entry = self.format(record)
doc: Dict[str, Any] = {
"timestamp": record.created,
"level": record.levelname,
"message": record.getMessage(),
"logger": record.name
}
if hasattr(record, "extra"):
doc.update(record.extra)
self.es.index(index=self.index, document=doc)
except Exception as e:
self.handleError(record)
# Usage
logger = logging.getLogger(__name__)
handler = ElasticsearchHandler("localhost:9200", "application-logs")
logger.addHandler(handler)
These logging libraries and patterns form the foundation of robust application monitoring. They enable detailed tracking of application behavior, simplified debugging, and integration with modern monitoring infrastructure. The choice of library depends on specific requirements: Loguru for simplicity, standard logging for flexibility, python-json-logger for structured output, structlog for consistent formatting, and Eliot for complex operation tracking.
The combination of these tools creates comprehensive logging solutions that scale with application complexity. Implementation best practices include consistent formatting, appropriate log levels, contextual information, and efficient storage management. This ensures logs serve their primary purpose: providing clear insights into application behavior and facilitating quick problem resolution.
