Boston Logger

Smaht logging solutions for Django applications.

Basic Usage

Add the RequestResponseLoggerMiddleware middleware in your Django project's settings file to log all request/responses that Django handles.

Import boston_logger.requests_monkey_patch (or set the ENABLE_REQUESTS_LOGGING flag to True) to log all requests made through the requests library.

For Django settings file

If you want to configure in settings.py start with this:

BOSTON_LOGGER = {
    "ENABLE_OUTBOUND_REQUEST_LOGGING": True,
    "ENABLE_LOGGING_MIDDLEWARE": True,
    "ENABLE_SENSITIVE_PATHS_PROCESSOR": False,
    "ENABLE_REQUESTS_LOGGING": True,  # Enable the requests monkey patch
    "MAX_VERBOSE_OUTPUT_LENGTH": 500,
    "MAX_JSON_DATA_TO_LOG": 0,  # Do not limit json output size, by default
    "MIDDLEWARE_BLOCKLIST": ["admin:index", "swagger-docs"],
    "LOGGER_NAME": "boston_logger",
    "LOG_RESPONSE_CONTENT": False,
    "PREFER_TEXT_FALLBACK_MASKING": False,
    "SHOW_NESTED_KEYS_IN_SENSITIVE_PATHS": False,
}

LOGGING = {
    'disable_existing_loggers': True,
    'version': 1,
    'formatters': {
        'json_formatter': {
            '()': 'boston_logger.logger.JsonFormatter',
            'default_extra': {
                '_logging_metadata': {
                    'category': '/app/environment',
                },
            },
        },
        'smart_formatter': {
            '()': 'boston_logger.logger.SmartFormatter',
        },
    },
    'filters': {
        'request_edge': {
            '()': 'boston_logger.logger.RequestEdgeEndFilter',
        },
    },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'level': 'DEBUG',
            'formatter': 'smart_formatter',
            'filters': ['request_edge'],
        },
        'json': {
            'class': 'logging.StreamHandler',
            'level': 'DEBUG',
            'formatter': 'json_formatter',
            'filters': ['request_edge'],
        },
    },
    'loggers': {
        '': {
            'handlers': ['console'],
            'level': os.environ.get('DJANGO_LOG_LEVEL', 'INFO'),
        },
        'boston_logger': {
            'handlers': ['json'],
            'level': os.environ.get('DJANGO_LOG_LEVEL', 'DEBUG'),
            'propagate': False,
        },
    },
}

All config options, with defaults

ENABLE_OUTBOUND_REQUEST_LOGGING: True - Requests lib requests will be captured (Only if requests has been patched by the monkey patch, or ENABLE_REQUESTS_LOGGING settings).
ENABLE_LOGGING_MIDDLEWARE: True - The django middleware will log START/END events for incoming requests.
ENABLE_SENSITIVE_PATHS_PROCESSOR: False - sensitive_paths.SensitivePaths objects will mask data
ENABLE_REQUESTS_LOGGING: False - Monkey patches requests lib at startup
MAX_VERBOSE_OUTPUT_LENGTH: 500 - Character length for request, header, and response data in SmartFormatter (console logs).
MAX_JSON_DATA_TO_LOG: 0 - If greater than zero, truncate json payloads to this size
MIDDLEWARE_BLOCKLIST: [admin:index, swagger-docs] - Middleware will not log requests that match these named URLs, must be a list.
LOGGER_NAME: boston_logger - Default name of the logger that all request logs will be sent to.
LOG_RESPONSE_CONTENT: False - Log the json responses the site is sending.
PREFER_TEXT_FALLBACK_MASKING: False - If parsing text data to sanitize as a query string fails, mask the whole value.
SHOW_NESTED_KEYS_IN_SENSITIVE_PATHS: False - When True, the objects in a sensitve path will show keys, but all values will be masked. When False, the entire object will be replaced with the masked string.

Filtering Sensitive Data

ENABLE_SENSITIVE_PATHS_PROCESSOR is set to False by default. If enabled, you must define filtering rules and activate them.

Example:

import requests
from django.http import JsonResponse

from boston_logger.context_managers import  SensitivePathContext
from boston_logger.sensitive_paths import SensitivePaths, add_mask_processor

# Global processors are applied to all log messages
# A SensitivePaths instance is the only implemented processor at this time.
# It will mask json and querystring data that matches the given paths.
add_mask_processor(
    "GlobalRuleName", SensitivePaths("obj1/key1", "list1"), is_global=True
)

# Non Global rules must be activated per log message
add_mask_processor(
    "SpecialRule", SensitivePaths("specific_obj1/key1"), is_global=False
)

with SensitivePathContext("SpecialRule"):
    # SpecialRule will apply to logging triggered for this request
    requests.get("https://example.com")
# But not this one
requests.get("https://example.com")

def django_view(request):
    response = JsonResponse({})
    # SpecialRule will apply to logging triggered for this response
    response._apply_mask_processors = ["SpecialRule"]
    return response

# "ALL" is a built-in rule which masks the entire payload
with SensitivePathContext("ALL"):
    # All request data, response data, and headers will be masked.
    requests.get("https://example.com")

Reducing Log Size

MAX_JSON_DATA_TO_LOG tries to ensure that json log messages don't get beyond a certain size. If it is set to a non-zero value, we use it as a message length limit. If the message is too long, we truncate the response data.

Middleware

If you're using the RequestResponseMiddleware in your Django application, you can override BOSTON_LOGGER.MIDDLEWARE_BLOCKLIST in your settings.py if you don't want to generate logs for given URLs. The argument expected is a list of URLs names. By default it won't log admin and swagger requests.

Log JSONResponses

LOG_RESPONSE_CONTENT is set to False by default. Setting it to True will add JSON data to the log responses of django.JsonResponse.

Adding Notes

Boston Logger allows you to add custom notes data to log messages. This can be any type of data that is JSON serializable (recommended to be a string or simple dictionary).

To add notes to an INCOMING request (i.e. one handled by the middleware), simply add an attribute to the WSGI request object called _request_notes; the JsonFormatter will include it in the JSON log output.

Note that if you are using Django REST Framework, the incoming request object in your ViewSet or view method(s) will be an abstraction of the original WSGI request. You'll need to set the attribute on request._request, e.g.:

setattr(request._request, '_request_notes', 'Some extra log data here.')

To add notes to OUTGOING requests (i.e. you're using the requests library to send an external request to another service/system), it's recommend you leverage the requests_monkey_patch functionality described above. This will allow you to specify a notes keyword argument which will attach your notes metadata onto the OUTGOING log message emitted by the JsonFormatter:

requests.post(url, data, notes='Some extra log data here.')

Also, remember that you can always use the RequestLogContext and attach your notes that way:

with RequestLogContext(method='post', notes='Some extra log data here.') as log_context:
    response = requests.post(url, data)
    request = response.request

    log_context.request = request
    log_context.response = response

JBSinc/boston-logger