Send application error logs to Slack channel
sunank200 opened this issue · 5 comments
Describe the feature
We need to consolidate the errors in Ask-Astro application log and show on slack.
It’s pretty standard to measure how many errors you see in the logs or requests and set a threshold (ie if I get more than a few errors in 5 mins, alert me)
example: https://astronomer.slack.com/archives/C055NCV1154/p1701307006879729
[FIRING:1]
High HTTP error rate on 'registry-api' responses
Labels:
• alertname: RegistryAPIHighHTTPErrorRate
• environment: dev
• owner: devex
@sunank200 The example mentioned above is for monitoring and alerts set in Chronosphere.
Do we forward application logs from Ask Astro to Chronosphere currently?
It's usually done via forwarding events to ancillary systems like Chronosphere/Prometheus etc. and setting alerts in those systems and forwarding those alerts to Slack or other mediums. We do not seem to be forwarding events/log to such a system currently.
I am exploring more here and looks like we can set alerts in GCP also for error logs.
I have set an alert for error logs here https://console.cloud.google.com/monitoring/alerting?project=ask-astro
which forwards to my Astronomer Email ID and a Slack channel in my test workspace.
I cannot add a Slack channel from Astronomer workspace as I don’t have permissions.
It will likely need permissions from IT to install #ask-astro-alert as a notification channel for sending alerts for our GCP Ask Astro Channel
I was able to setup from GCP to the Slack channel. Sample notification alert here: https://astronomer.slack.com/archives/C063391HTGA/p1703936599695989
Closing as per above comment