Alerting via custom events and telemetry
embano1 opened this issue · 2 comments
Is your feature request related to a problem? Please describe.
As a VSphereSource administrator,
if VSphereSources (temporarily) cannot connect to its upstream source (vCenter)
I want to get notified via a (custom) event.
Describe the solution you'd like
Emit custom (domain) events on critical conditions, e.g. vCenter connection lost in conjunction with telemetry (see below) to avoid dependencies on Knative components, e.g. broker.
Describe alternatives you've considered
Use telemetry, e.g. OpenTracing or Prometheus to indicate errors (stats) and/or filter on k/v trace labels.
Additional context
This might be useful beyond just VSphereSources so we should spend some time on whether this is applicable for any Knative Source and if we should define a generic schema for such alerts and use cases.
@embano1 yeah, this might be useful for Sources in general. I have been participating in a SIG from the CD Foundation where we created something similar for Tekton. So instead of the sources emitting events, a separate controller is introduced to look at resources and statuses and then emit the corresponding events. If we use a common vocabulary like the one provided by the CDF (http://github.com/cdfoundation/sig-events/) we can provide interoperability with other tools that also understand those events. I wanted to start a controller for Serving but I haven't had the time to bootstrap it yet, it shouldn't be that difficult to do and there are examples like the one that we created for Tekton here: https://github.com/tektoncd/experimental/tree/main/cloudevents
This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.