maxwo/snmp_notifier

Extra fields doesn't do templating on resolved alerts

nbzero opened this issue · 0 comments

What did you do?
I've been trying to match output of SNMP Notifier with a 3rd party tool that listen for SNMP trap, almost all of the progress seems great except that we need resolved alert to do templating so we can distinguish the source of that trap and resolve it on 3rd party tool end.

My use case is that there is one alert about Prometheus target down. I've been using extra field to tell 3rd party tool to know that the source can be get from specific oid generated from extra field.

Firing alerts seems to do templating perfectly, but when it is resolved alert all extra fields seems to be empty. The only place I can get data is the label oid which the 3rd party tool doesn't expect that will be how we resolve our alert, so if we can adjust SNMP Notifier to support templating on alert resolved that could be great.

For now I'll open an issue first, I'm not a Go expert but I could take a look at the code and make some contribution but it could take awhile. If there are anyone that can make this adjustment I really appreciate it.

What did you expect to see?
I expect resolved alert to do templating on extra variables.
Here are the example I got from tcpdump on my lab.

# Firing alert
C="demo" V2Trap(612)
E:98789.0.1032.1="1.3.6.1.4.1.98789.0.1032[alertname=NodeExporterDown,alerter=ruler,group=Demo,hostname=demo01,instance=10.153.73.18:9100,job=node_exporter,oid=1.3.6.1.4.1.98789.0.1032,severity=critical,site=demo,system=demoSystem]"
E:98789.0.1032.2="critical" 
E:98789.0.1032.3="demo01/10.153.73.18:9100: Node Exporter down" 
E:98789.0.1032.7="NodeExporterDown" 
E:98789.0.1032.8="2021-06-10 11:30:51.209908121 +0000 UTC" 
E:98789.0.1032.9="critical" 
E:98789.0.1032.4="demo01" 
E:98789.0.1032.5="10.153.73.18:9100" 
E:98789.0.1032.6="demoSystem"

# Resolved alert
C="demo" V2Trap(489)
E:98789.0.1032.1="1.3.6.1.4.1.98789.0.1032[alertname=NodeExporterDown,alerter=ruler,group=Demo,hostname=demo01,instance=10.153.73.18:9100,job=node_exporter,oid=1.3.6.1.4.1.98789.0.1032,severity=critical,site=demo,system=demoSystem]" 
E:98789.0.1032.2="info" 
E:98789.0.1032.3="Status: OK" 
E:98789.0.1032.9="" 
E:98789.0.1032.4=""
E:98789.0.1032.5=""
E:98789.0.1032.6=""
E:98789.0.1032.7=""
E:98789.0.1032.8=""

What did you see instead? Under which circumstances?
Resolved trap seems to not do any templating for extra variables at all.

I'm running on SNMP Notifier 1.1.0.
I see that version 1.1.1 only change is to force SNMPv3 to have username in config so I didn't upgrade yet.

If I'm missing any info please tell me.