Substra/substra-backend

Worker: missing information in logs

Closed this issue · 0 comments

Follow-up to #124

Currently, some important information isn't displayed in the logs. Not having access this information makes it harder to troubleshoot errors/bugs. In particular, some of the algo exceptions on prod are being swallowed (every exception until the final retry), so we potentially lose some critical troubleshooting data.

Not available from the logs:

  • Task id
  • Retry attempts (can be inferred, but not explicitly mentioned)
  • Exception that caused each retry (only the exception that caused the final failure is displayed)

Solutions:

Option 1

Restore celery logs: #213

Option 2

Maybe celery logging gives out too much information. In that case, we could explore writing log messages ourselves (e.g. logger.info(f'Starting task {task_id}') etc.)

Option 3

(your idea here?)