Using varbinds with common OID

[falk-w]

Hello, just wondering if there is a way to create all alarms with the SAME OID and send the alarm details in the varbinds.

e.g. with the OID defined in the prometheusrules we get:

Alarm Number one OID 1.3.6.1.4.1.47984.2.3.1.2.1

iso.3.6.1.4.1.47984.2.3.1.2.1.1 = STRING: "1.3.6.1.4.1.47984.2.3.1.2.1[]"
iso.3.6.1.4.1.47984.2.3.1.2.1.2 = STRING: "warning"
iso.3.6.1.4.1.47984.2.3.1.2.1.3 = STRING: "Status: warning

Alarm Number two OID 1.3.6.1.4.1.47984.2.3.1.2.2

iso.3.6.1.4.1.47984.2.3.1.2.2.1 = STRING: "1.3.6.1.4.1.47984.2.3.1.2.2[]"
iso.3.6.1.4.1.47984.2.3.1.2.2.2 = STRING: "warning"
iso.3.6.1.4.1.47984.2.3.1.2.2.3 = STRING: "Status: warning

As we see the OID sent matches the OID in varbind #1

If a monitoring system is parsing alarms manually based on the OID it would be helpful to have a common OID (so not every OID requires an extra parsing rule)

e.g.

Alarm Number one OID 1.3.6.1.4.1.32804.1.6.2.2.1

iso.3.6.1.4.1.32804.1.6.2.1 = STRING: "1.3.6.1.4.1.32804.1.6.2.2.1[]"
iso.3.6.1.4.1.32804.1.6.2.2 = STRING: "warning"
iso.3.6.1.4.1.32804.1.6.2.3 = STRING: "Status: warning

Alarm Number two OID 1.3.6.1.4.1.32804.1.6.2.2.2

iso.3.6.1.4.1.32804.1.6.2.1 = STRING: "1.3.6.1.4.1.32804.1.6.2.2.2[]"
iso.3.6.1.4.1.32804.1.6.2.2 = STRING: "warning"
iso.3.6.1.4.1.32804.1.6.2.3 = STRING: "Status: warning

Is there something like this available by default?

We currently solved it in a kind of hacky was by changing: trapsender/trap_sender.go

trapOid, _ := snmpgo.NewOid(alertGroup.OID)
snmpTrapParam := ".1.3.6.1.4.1.32804.1.6.2"
varBinds = addUpTime(varBinds)
varBinds = append(varBinds, snmpgo.NewVarBind(snmpgo.OidSnmpTrap, trapOid))
varBinds = addStringSubOid(varBinds, snmpTrapParam,, "1", trapUniqueID)
varBinds = addStringSubOid(varBinds, snmpTrapParam,, "2", alertGroup.Severity)
varBinds = addStringSubOid(varBinds, snmpTrapParam,, "3", *descriptions)

[falk-w]

e.g. alarms with a common OID would look like this:

2022-04-05 12:37:16 243.35.132.34.bc.googleusercontent.com [UDP: [34.132.35.243]:1120->[10.128.15.202]:162]:
iso.3.6.1.2.1.1.3.0 = Timeticks: (291400) 0:48:34.00
iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.4.1.47984.2.3.1.2.2
iso.3.6.1.4.1.47984.2.3.2.1 = STRING: "1.3.6.1.4.1.47984.2.3.1.2.2[]"
iso.3.6.1.4.1.47984.2.3.2.2 = STRING: "warning"
iso.3.6.1.4.1.47984.2.3.2.3 = STRING: "Status: warning

• Alert: PodCpuLoadHigh
  Summary: Pod CPU usage high in nginx-588fb797d9-bhhls
  Description: Pod CPU usage above threshold detected
  VALUE = 6.833466648648442\n LABELS: map[pod:nginx-588fb797d9-bhhls]"

[maxwo]

Hi,
It is already possible to share the same OID among several traps.
You can use the --snmp.trap-default-oid to set the OID for good, then, thanks to grouping rules on the alertmanager side, you can share as many traps for the same OID as there are groups (hence the [] in the iso.3.6.1.4.1.47984.2.3.2.1 = STRING: "1.3.6.1.4.1.47984.2.3.1.2.2[]" VarBinds, which will be completed with the labels of the grouping rule.

Your first example would become:

Alarm Number one OID 1.3.6.1.4.1.32804.1.6.2.2.1

iso.3.6.1.4.1.32804.1.6.2.1 = STRING: "1.3.6.1.4.1.32804.1.6.2.2.1[group=one]"
iso.3.6.1.4.1.32804.1.6.2.2 = STRING: "warning"
iso.3.6.1.4.1.32804.1.6.2.3 = STRING: "Status: warning

Alarm Number two OID 1.3.6.1.4.1.32804.1.6.2.2.2

iso.3.6.1.4.1.32804.1.6.2.1 = STRING: "1.3.6.1.4.1.32804.1.6.2.2.2[group=two]"
iso.3.6.1.4.1.32804.1.6.2.2 = STRING: "warning"
iso.3.6.1.4.1.32804.1.6.2.3 = STRING: "Status: warning

[falk-w]

@maxwo can you please share how you mapped the custom OID into this:

iso.3.6.1.4.1.32804.1.6.2.1 = STRING: "1.3.6.1.4.1.32804.1.6.2.2.1[group=one]"

I spent some time analyzing your code and it seems it automatically uses the OID if it is found in a label and ignored the default OID.

Our prometheusrules look like this:

spec:
  groups:
  - name: node-exporter-rules
    rules:
    - alert: nodeMemoryUsageHigh
      annotations:
        description: |
          Node memory usage above threshold detected
          VALUE = {{ $value }}\n  LABELS: {{ $labels }}
        summary: Node memory usage high in {{ $labels.instance }}
      expr: |
        100 - ((node_memory_MemAvailable_bytes * 100) / node_memory_MemTotal_bytes) > 80
      for: 2m
      labels:
        oid: 1.3.6.1.4.1.47984.2.3.1.1.1
        severity: warning

As we have the OID label that is always used as the OID.

As I need to set "trapUniqueID":

	varBinds = addTrapSubObject(varBinds, alertGroup.OID, "1", trapUniqueID)
	varBinds = addTrapSubObject(varBinds, alertGroup.OID, "2", alertGroup.Severity)
	varBinds = addTrapSubObject(varBinds, alertGroup.OID, "3", *description)

Which is build here:
trapUniqueID := strings.Join([]string{alertGroup.OID, "[", alertGroup.GroupID, "]"}, "")

I am wondering how I can set the groupID? If I can set that one I can use "--snmp.trap-default-oid=1.3.6.1.4.1.47984.2.3.1" to build the common part and use the groupID to build trapUniqueID.