Python logging with Structlog

 In Python, logging refers to the process of capturing and recording runtime information that can be helpful in debugging, tracking events, and monitoring applications. Logging is essential for understanding the behavior of an application, especially in production environments where problems may not be immediately visible.

Python's built-in logging module is designed to provide a flexible framework for logging information. However, when building complex systems or working in large codebases, developers often face challenges in managing and structuring logs. This is where Structured Logging comes into play, providing a more organized and machine-readable format for logs.

In this article, we will dive deep into structured logging in Python using the structlog library. We will explore how structured logging improves the readability and utility of logs, how to set it up in Python, and how to use it for better monitoring and debugging.

---

What is Structured Logging?

Structured logging involves logging data in a way that allows the log entries to be more easily parsed and queried by machines. Rather than just writing plain text messages (as is common in traditional logging), structured logs capture key-value pairs that provide more detailed and useful information.

For example, a traditional log entry might look like this:

2025-01-02 10:00:00 INFO Request processed successfully

A structured log entry, however, might look like this:

{
  "timestamp": "2025-01-02T10:00:00",
  "level": "INFO",
  "message": "Request processed successfully",
  "user_id": "12345",
  "request_id": "abc123",
  "response_time_ms": 200
}

The difference is that structured logs are more machine-readable and allow for easier searching, filtering, and aggregation, which is crucial when handling large-scale applications or systems with complex interactions.

---

Why Use Structured Logging?

1. Machine Readable: Structured logs, typically in formats like JSON, are easy for machines to parse, analyze, and process. This is helpful for aggregating logs from multiple sources, querying logs for specific events, and integrating with log management tools like ELK (Elasticsearch, Logstash, Kibana) or Splunk.

2. Consistency: Structured logging provides a standardized format for all log entries, making them easier to interpret, both for humans and machines. It helps ensure that logs contain consistent data points, like timestamps, log levels, request IDs, user IDs, etc.

3. Improved Search and Filtering: Since structured logs contain key-value pairs, it becomes easier to search and filter logs based on specific fields. For example, you can query logs for all entries related to a specific user or search for logs with a particular error code.

4. Better Context: Structured logging allows you to add contextual information like request IDs, user IDs, or any other relevant data that can be crucial for debugging and troubleshooting.

---

Overview of structlog

structlog is a Python library designed to make structured logging easy to use. It builds on top of Python's built-in logging module and offers additional features to make logging more flexible and informative. With structlog, you can create structured logs that are consistent, easy to read, and machine-friendly.

Key Features of structlog:

• Structured Logs: It allows logging key-value pairs, making logs machine-readable.
• Rich Context: You can add additional context to each log entry without polluting the log message.
• Flexible Handlers: structlog supports various output formats, including JSON, plain text, or custom formats.
• Log Pipeline: structlog allows you to create a pipeline that processes logs, adding or modifying data as they flow through.
• Integration with logging: It seamlessly integrates with Python’s standard logging module, making it easier to adopt without requiring a major overhaul of existing code.

---

Setting Up structlog in Python

To begin using structlog, we need to install it first. This can be done with pip:

pip install structlog

Once installed, we can start setting up structlog for our application. Below is an example of how to configure structlog to work with Python’s standard logging system.

Basic Configuration:

import logging
import structlog

# Set up basic configuration for Python logging
logging.basicConfig(level=logging.INFO, format="%(message)s")
logger = structlog.get_logger()

# A simple log statement
logger.info("Request processed successfully", user_id=12345, request_id="abc123")

This code sets up a basic logger with structlog. The logging.basicConfig sets up Python's built-in logging system with a minimal configuration. We then use structlog.get_logger() to obtain a logger instance, which can be used to log structured messages.

The output for this code will look something like this:

{"event": "Request processed successfully", "user_id": 12345, "request_id": "abc123"}

As you can see, the log entry is now in a structured JSON-like format, which is much more useful for automated log processing.

---

Advanced Configuration and Pipeline Setup

structlog allows you to set up a pipeline to process logs, which can modify log entries, format them, or add extra context. This is especially useful for larger applications where logs need to be enriched or processed before being output.

Example of a Basic Pipeline Setup:

import logging
import structlog

# Define a custom processor to enrich logs with additional context
def add_context(_, __, event_dict):
    event_dict["application"] = "my_app"
    return event_dict

# Create a processor pipeline
processors = [
    structlog.processors.add_log_level,  # Adds the log level to each log entry
    structlog.processors.TimeStamper(fmt="iso"),  # Adds timestamp in ISO format
    add_context,  # Our custom processor
    structlog.processors.JSONRenderer()  # Output as JSON
]

# Set up the logger with the pipeline
logging.basicConfig(level=logging.INFO, format="%(message)s")
logger = structlog.get_logger()

# Apply the pipeline to the logger
logger = structlog.wrap_logger(logger, processors=processors)

# Logging with the enriched pipeline
logger.info("Request processed successfully", user_id=12345, request_id="abc123")

Output:

{
  "timestamp": "2025-01-02T10:00:00",
  "level": "info",
  "event": "Request processed successfully",
  "user_id": 12345,
  "request_id": "abc123",
  "application": "my_app"
}

In this example:

• We’ve added a custom processor add_context, which adds the "application" field to every log entry.
• We’ve used TimeStamper(fmt="iso") to automatically include a timestamp in ISO format.
• The final output is formatted as a JSON string, making it easy to ingest into log management systems.

---

Log Levels in structlog

Just like the built-in Python logging module, structlog supports different log levels: DEBUG, INFO, WARNING, ERROR, and CRITICAL. You can specify the log level for each log entry using the level argument.

For example:

logger.debug("Debugging the request", request_id="xyz123")
logger.info("Request processed successfully", user_id=67890)
logger.error("Failed to process request", request_id="xyz123", error="Timeout")

In a structured log, these levels will be captured along with the other fields, such as event messages, timestamps, and additional context.

---

Enriching Logs with Contextual Information

Structured logs in structlog shine when you add additional context, such as request IDs, user information, or any other details that can help during debugging or analyzing logs. Contextual logging is crucial in distributed systems, microservices architectures, and web applications.

Example with Contextual Information:

import structlog

logger = structlog.get_logger()

# Enrich logs with contextual information
logger = logger.bind(request_id="abc123", user_id=456)

# Log events with additional context
logger.info("Request processed successfully")
logger.error("Request failed", error_code=500)

Output:

{
  "event": "Request processed successfully",
  "request_id": "abc123",
  "user_id": 456
}

Here, we’ve used the bind() method to enrich the logger with context. The logger will automatically attach the request_id and user_id to all subsequent log entries, which is very helpful for tracing requests and user actions.

---

Handling Exceptions with structlog

structlog also allows you to capture and log exceptions in a structured format. This is particularly useful when debugging errors in production systems.

Example of Logging an Exception:

import structlog
import logging

logger = structlog.get_logger()

try:
    1 / 0  # This will raise a ZeroDivisionError
except ZeroDivisionError as e:
    logger.exception("An error occurred while processing the request", error=str(e))

Output:

{
  "event": "An error occurred while processing the request",
  "error": "division by zero",
  "exception": "ZeroDivisionError"
}

In this example, structlog automatically captures the exception and includes the exception type and message in the structured log. This makes it easy to spot and diagnose errors.

---

Integrating structlog with Log Aggregators

Structured logs are particularly useful when integrating with log aggregation tools like ELK stack (Elasticsearch, Logstash, Kibana), Splunk, or cloud-based services like AWS CloudWatch or Google Stackdriver.

When logs are structured as JSON, they can be easily ingested by these tools for search, filtering, and visualization. This is incredibly helpful for monitoring applications in real-time and analyzing logs to spot issues before they impact users.

---

Conclusion

structlog is a powerful library that transforms Python's logging capabilities into a structured and highly flexible system. Structured logging with structlog improves the readability, queryability, and traceability of logs, especially in complex applications and distributed systems. By enriching logs with context, timestamping them, and formatting them in a machine-readable way (like JSON), structlog helps teams debug faster, monitor applications more effectively, and ensure better insights into their systems.

This article has covered the key features of structlog, its setup, and how to use it in Python applications. We discussed how to set up structured logging pipelines, enrich logs with contextual information, handle exceptions, and integrate with log aggregation systems. By incorporating structured logging into your Python applications, you will significantly improve your ability to track events and diagnose issues in your codebase.