getsentry/sentry-python

Python SDK's `capture_exception` not working within a Celery worker

Closed this issue · 4 comments

markbaydoun commented

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.33.2

Steps to Reproduce

For reporting errors gracefully, we do the following in our Python application:

import sentry_sdk
from mysettings import settings

def init_logger_platform():
    sentry_sdk.init(
        dsn=settings.SENTRY_DSN,
        traces_sample_rate=1.0,
        profiles_sample_rate=1.0,
    )

def report_exception(
    e: BaseException | None = None,
    *,
    extras: Dict[str, Any] | None = None,
    tags: Dict[str, Any] | None = None,
):
    if settings.SENTRY_DSN:
        extras = extras or {}
        tags = tags or {}
        kwargs = extras | tags

        sentry_sdk.capture_exception(e, scope=None, **kwargs)

In a Celery task, we run a function similar to this, where we do some processing associated to an entity, and report any errors:

from platform.logging import report_exception

# Mocked function for testing purposes
class SomeClass:
    def process(self, entity_uuid: str, correlation_id: str):
        try:
            # Mock raise Exception
            raise ValueError("Kaboom!")

            entity = self.fetch_entity(entity_uuid)
            self.process(entity)
            self.update_status(entity_uuid, "ok")
        except Exception as e:
            report_exception(
                e,
                extras={ "process_type": "process_entity", "entity_uuid": entity_uuid, "correlation_id": correlation_id },
            )
            self.update_status(entity_uuid, "failed")  # Handle gracefully and set status

I have added prints and logs in both places to ensure SENTRY_DSN is set, and that the functions are called. When running this same thing from within an IPython REPL, the error is reported correctly. Fore more context, this is how we initialise sentry within the workers:

from celery import Celery, Task, signals
import sentry_sdk
from sentry_sdk.integrations.celery import CeleryIntegration
from platform.logging import logger, report_exception
from platform.settings import settings

if settings.SENTRY_DSN:
    @signals.celeryd_init.connect
    def init_sentry(**_):
        sentry_sdk.init(
            dsn=settings.SENTRY_DSN,
            integrations=[CeleryIntegration(monitor_beat_tasks=True)],
        )

    @signals.task_failure.connect
    def handle_task_failure(**kw: Any) -> None:
        kw = kw or dict()
        kw["Origin"] = "Workers"
        report_exception()

I've also verified that the initialisation works.

Expected Result

The issue should appear in the Sentry dashboard as it was reported.

Actual Result

The issue, when it happens within the Celery task, never shows up in the sentry dashboard. I've added logs everywhere to verify the capture_exception() function runs, and that the DSN is the correct one. Haven't found a way to make this work. A similar issue is #1468.

alexander-alderman-webb commented

Hi @markbaydoun,

Thank you for the issue and code snippets. I was able to reproduce the behavior you are seeing, that no exception is sent to Sentry.

I found that the capture_exception() call fails because unexpected keyword arguments are passed to sentry_sdk.capture_exception(). The keyword arguments end up here:

sentry-python/sentry_sdk/scope.py

Lines 1600 to 1608 in f702ec9

def update_from_kwargs(
self,
user=None, # type: Optional[Any]
level=None, # type: Optional[LogLevelStr]
extras=None, # type: Optional[Dict[str, Any]]
contexts=None, # type: Optional[Dict[str, Dict[str, Any]]]
tags=None, # type: Optional[Dict[str, str]]
fingerprint=None, # type: Optional[List[str]]
):

Could you amend report_exception as follows?

def report_exception(
    e: BaseException | None = None,
    *,
    extras: Dict[str, Any] | None = None,
    tags: Dict[str, Any] | None = None,
):
    if settings.SENTRY_DSN:
        extras = extras or {}
        tags = tags or {}
-       kwargs = extras | tags
+       kwargs = { "extras": extras, "tags": tags }

        sentry_sdk.capture_exception(e, scope=None, **kwargs)

If the problem remains turn on debug mode on locally to collect more information, and reach out again.

markbaydoun commented

Hi @markbaydoun,

Thank you for the issue and code snippets. I was able to reproduce the behavior you are seeing, that no exception is sent to Sentry.

I found that the capture_exception() call fails because unexpected keyword arguments are passed to sentry_sdk.capture_exception(). The keyword arguments end up here:

sentry-python/sentry_sdk/scope.py

Lines 1600 to 1608 in f702ec9

def update_from_kwargs(
self,
user=None, # type: Optional[Any]
level=None, # type: Optional[LogLevelStr]
extras=None, # type: Optional[Dict[str, Any]]
contexts=None, # type: Optional[Dict[str, Dict[str, Any]]]
tags=None, # type: Optional[Dict[str, str]]
fingerprint=None, # type: Optional[List[str]]
):
Could you amend report_exception as follows?

def report_exception(
e: BaseException | None = None,
*,
extras: Dict[str, Any] | None = None,
tags: Dict[str, Any] | None = None,
):
if settings.SENTRY_DSN:
extras = extras or {}
tags = tags or {}

  •   kwargs = extras | tags

  •   kwargs = { "extras": extras, "tags": tags }

  sentry_sdk.capture_exception(e, scope=None, **kwargs)

If the problem remains turn on debug mode on locally to collect more information, and reach out again.

That was exactly it, thank you so much. Without the debug mode there was no way of knowing.

alexander-alderman-webb commented

Thank you for the reply! I am closing the issue as I believe this is resolved.