/structlog-sentry-logger

A multi-purpose, pre-configured, performance-optimized structlog logger with (optional) Sentry integration via structlog-sentry

Primary LanguagePythonApache License 2.0Apache-2.0

Structlog-Sentry-Logger

CI codecov Documentation Status License PyPI - Python Version PyPI pre-commit Ruff Code style: black powered by semgrep

Documentation: https://structlog-sentry-logger.readthedocs.io

Source Code: https://github.com/TeoZosa/structlog-sentry-logger

🧑‍🏫 Overview

A multi-purpose, pre-configured, performance-optimized structlog logger with (optional) Sentry integration via structlog-sentry.

✨ Features

  1. Makes logging as easy as using print statements, but prettier and easier to capture and filter!
  2. Highly opinionated! There are only two (2) distinct configurations.
  3. Structured logs in JSON format means they are ready to be ingested by many of your favorite log analysis tools!
  4. Cloud Logging compatible!

basic logging example for user documentation

🎊 What You Get

💪 Powerful Automatic Context Fields

The pre-configured options include:

  1. ⌚ Timestamps
    • UTC time in ISO 8601 format (YYYY-MM-DDTHH:MM:SS.ffffffZ)
  2. 🚦 Log levels
    • Added to the JSON context for filtering and categorization
  3. 🔍 Logger names
    • Automatically assigned to namespaced versions of the initializing python modules (.py files), relative to your project directory.
      • e.g., the logger in docs_src/sentry_integration.py is named docs_src.sentry_integration
  4. 🔎 Function names and line numbers where logging calls were made

🔥 Tip
For easier at-a-glance analysis, you can also sort log fields by key by exporting the STRUCTLOG_SENTRY_LOGGER_KEY_SORTING_ON environment variable. Note, however, that this has a substantial (~1.6x) performance penalty.

⚡ Performance

structlog-sentry-logger is C-compiled and fully-tuned, leveraging orjson as the JSON serializer for lightning-fast logging (more than a 4x speedup over Python's built-in JSON library1; see here for sample performance benchmarks). Don't let your obligate cross-cutting concerns cripple performance any longer!

For further reference, see:

🤖 Built-in Sentry Integration (Optional)

Automatically add much richer context to your Sentry reports.

  • Your entire logging context is sent as a Sentry event when the structlog-sentry-logger log level is error or higher.
    • i.e., logger.error(""), logger.exception("")
  • See structlog-sentry for more details.

Table of Contents

🎉 Installation

pip install structlog-sentry-logger

Optionally, install Sentry integration with

pip install "structlog-sentry-logger[sentry]"

🚀 Usage

🔊 Pure structlog Logging (Without Sentry)

Simply import and instantiate the logger:

import structlog_sentry_logger

LOGGER = structlog_sentry_logger.get_logger()

Now you can start adding logs as easily as print statements:

LOGGER.info("Your log message", extra_field="extra_value")

📝 Note
All the regular Python logging levels are supported.

Which automatically produces this:

{
    "event": "Your log message",
    "extra_field": "extra_value",
    "funcName": "<module>",
    "level": "info",
    "lineno": 5,
    "logger": "docs_src.pure_structlog_logging_without_sentry",
    "timestamp": "2022-01-11T07:05:37.164744Z"
}

📤 Log Custom Context Directly to Sentry (optional)

If you installed the library with the optional Sentry integration you can incorporate custom messages in your exception handling which will automatically be reported to Sentry (thanks to the structlog-sentry module). To enable this behavior, export the STRUCTLOG_SENTRY_LOGGER_CLOUD_SENTRY_INTEGRATION_MODE_ON environment variable.

An easy way to do this is to put it into a local .env file2:

echo "STRUCTLOG_SENTRY_LOGGER_CLOUD_SENTRY_INTEGRATION_MODE_ON=" >> .env

📝 ️Note
By default, only logs at error-level or above are sent to Sentry. If you want to set a different minimum log level, you can specify a valid Python log level via the STRUCTLOG_SENTRY_LOGGER_SENTRY_LOG_LEVEL environment variable.

For example, to send all logs at warning-level or above to Sentry you would simply set STRUCTLOG_SENTRY_LOGGER_SENTRY_LOG_LEVEL=WARNING

For a concrete example, given the following Python code:

import uuid

import structlog_sentry_logger

LOGGER = structlog_sentry_logger.get_logger()

curr_user_logger = LOGGER.bind(uuid=uuid.uuid4().hex)  # LOGGER instance with bound UUID
try:
    curr_user_logger.warn("A dummy error for testing purposes is about to be thrown!")
    x = 1 / 0
except ZeroDivisionError as err:
    ERR_MSG = (
        "I threw an error on purpose for this example!\n"
        "Now throwing another that explicitly chains from that one!"
    )
    curr_user_logger.exception(ERR_MSG)
    raise RuntimeError(ERR_MSG) from err

We would get the following output:

{
    "event": "A dummy error for testing purposes is about to be thrown!\n",
    "funcName": "<module>",
    "level": "warning",
    "lineno": 12,
    "logger": "docs_src.sentry_integration",
    "sentry": "skipped",
    "timestamp": "2022-01-06T04:50:07.627633Z",
    "uuid": "fe2bdcbe2ed74432a87bc76bcdc9def4"
}
{
    "event": "I threw an error on purpose for this example!\nNow throwing another that explicitly chains from that one!\n",
    "exc_info": true,
    "funcName": "<module>",
    "level": "error",
    "lineno": 19,
    "logger": "docs_src.sentry_integration",
    "sentry": "sent",
    "sentry_id": null,
    "timestamp": "2022-01-06T04:50:07.628316Z",
    "uuid": "fe2bdcbe2ed74432a87bc76bcdc9def4"
}
Traceback (most recent call last):
  File "/app/structlog-sentry-logger/docs_src/sentry_integration.py", line 10, in <module>
    x = 1 / 0
ZeroDivisionError: division by zero

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/structlog-sentry-logger/docs_src/sentry_integration.py", line 17, in <module>
    raise RuntimeError(ERR_MSG) from err
RuntimeError: I threw an error on purpose for this example!
Now throwing another that explicitly chains from that one!

☁️ Cloud Logging Compatibility

The logger will attempt to infer if an application is running in a cloud environment by inspecting for the presence of environment variables that may be automatically injected by cloud providers (namely, KUBERNETES_SERVICE_HOST, GCP_PROJECT, and GOOGLE_CLOUD_PROJECT).

If any of these environment variables are detected, log levels will be duplicated to a reserved severity key in the emitted logs to enable parsing of the log level and the remaining log context (as jsonPayload) by Cloud Logging (see: Cloud Logging: Structured logging).

📝 ️Note
This behavior can also be manually enabled by adding the STRUCTLOG_SENTRY_LOGGER_CLOUD_LOGGING_COMPATIBILITY_MODE_ON variable to your environment, e.g., via a .env file2.

⚠️️ Warning
If a user manually specifies a value for the severity key, it will be overwritten! Avoid using this key if possible to preempt any future issues.

📉 Output: Formatting & Storage

The default behavior is to stream JSON logs directly to the standard output stream like a proper 12 Factor App.

For local development, it often helps to prettify logging to stdout and save JSON logs to a .logs folder at the root of your project directory for later debugging. To enable this behavior, export the STRUCTLOG_SENTRY_LOGGER_LOCAL_DEVELOPMENT_LOGGING_MODE_ON environment variable, e.g., in your local .env file2:

echo "STRUCTLOG_SENTRY_LOGGER_LOCAL_DEVELOPMENT_LOGGING_MODE_ON=" >> .env

In doing so, with our previous exception handling example we would get:

Output_Formatting_example_0 Output_Formatting_example_1

📋 Summary

That's it. Now no excuses. Get out there and program with pride knowing no one will laugh at you in production! For not logging properly, that is. You're on your own for that other observability stuff.

📚 Further Reading

1️⃣ structlog: Structured Logging for Python

2️⃣ Sentry: Monitor and fix crashes in realtime

3️⃣ structlog-sentry: Provides the structlog integration for Sentry

🔧 Development

For convenience, implementation details of the below processes are abstracted away and encapsulated in single Make targets.

🔥 Tip
Invoking make without any arguments will display auto-generated documentation on available commands.

🏗️ Package and Dependencies Installation

Make sure you have Python 3.9+ and poetry installed and configured.

To install the package and all dev dependencies, run:

make provision-environment

🔥 Tip
Invoking the above without poetry installed will emit a helpful error message letting you know how you can install poetry.

📦 Python Module to C-Extension Compilation

The projects's build.py file specifies which modules to package.

For manual per-module compilation, see: Mypyc Documentation: Getting started - Compiling and running

✅ Testing

We use tox and pytest for our test automation and testing frameworks, respectively.

To invoke the tests, run:

make test

Run mutation tests to validate test suite robustness (Optional):

make test-mutations

ℹ️ Technical Details
Test time scales with the complexity of the codebase. Results are cached in .mutmut-cache, so once you get past the initial cold start problem, subsequent mutation test runs will be much faster; new mutations will only be applied to modified code paths.

🚨 Code Quality

We use pre-commit for our static analysis automation and management framework.

To invoke the analyses and auto-formatting over all version-controlled files, run:

make lint

🚨 Danger
CI will fail if either testing or code quality fail, so it is recommended to automatically run the above locally prior to every commit that is pushed.

🔄 Automate via Git Pre-Commit Hooks

To automatically run code quality validation on every commit (over to-be-committed files), run:

make install-pre-commit-hooks

⚠️️ Warning
This will prevent commits if any single pre-commit hook fails (unless it is allowed to fail) or a file is modified by an auto-formatting job; in the latter case, you may simply repeat the commit and it should pass.

📝 Documentation

make docs-clean docs-html

🔥 Tip
For faster feedback loops, this will attempt to automatically open the newly built documentation static HTML in your browser.

🧑‍⚖️ Legal

📄 License

Structlog-Sentry-Logger is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

👥 Credits

This project was generated from @TeoZosa's cookiecutter-cruft-poetry-tox-pre-commit-ci-cd template.

Footnotes

  1. Source: Choosing a faster JSON library for Python: Benchmarking ↩

  2. This library uses python-dotenv to automatically populate your environment with this variable (if it exists) from the local .env file. Alternatively, you may use direnv and a .envrc file. A sample .envrc file (with all features enabled) has been provided at the root of the repository (.envrc.sample). If direnv is already installed, it's as simple as copying .envrc.sample to the root of your project, editing it to reflect your desired configurations, renaming it to .envrc, and running direnv allow 🎉 ↩ ↩2 ↩3