Azure/azure-event-hubs-spark

Emit time lag metrics to track how old is the data being processed and avoid data loss

felipepessoto opened this issue · 1 comments

Feature Requests:

  • What issue are you trying to solve?
    Be able to track if the time lag is increasing and react before it reaches the retention time, what would cause data loss.
  • How do you want to solve it?
    Emit metrics containing the time lag, so we can monitor it and create alerts.
  • What is your use case for this feature?
    This is important for production application

The metric values would be: UTC Now - EventData.SystemProperties.EnqueuedTimeUtc

Hi. Do you have any update on this?

I believe this metric is critical for production workloads that can't lose any events.
Would like to hear what you folks think. If this is something we could expect for the next releases?

Thanks