/Dynatrace-SNMP

Dynatrace SNMP monitoring ActiveGate extension

Primary LanguagePython

Dynatrace SNMP Polling Extension

A Dynatrace ActiveGate SNMP Polling extension.
Used for monitoring basic health of network devices and other appliances, the ActiveGate will poll for these metrics every minute. Metrics can then be visualised and charted on Dynatrace dashboards and overview pages.
The goal is to be as device compatible and general as possible.

This extension enables Dynatrace to monitor basic device health metrics.

  • HOST (CISCO-MIB, F5-MIB or Host-RESOURCE-MIB)
    • CPU utilisation
    • Memory utilisation
    • Disk Utilisation
  • NETWORK (IF-MIB)
    • Network Traffic - Incoming/Outgoing
    • Network packets - Incoming/Outgoing
    • Errors - Incoming/Outgoing
    • Discarded packets - Incoming/Outgoing

and properties:

  • SNMPv2-MIB
    • sysDescr
    • sysObjectID
    • sysUpTime
    • sysContact
    • sysName
    • sysLocation
    • sysServices
    • sysORLastChange

It makes use of the following MIBs:

You can fork and adapt this to poll for any SNMP exposed metrics

To learn more about Dynatrace ActiveGate extensions see - ActiveGate Plugins
This extension consumes custom metrics, for an understanding of how custom metrics are licensed and consumed in Dynatrace, see: Calculate Monitoring Consumption

Images

See the Wiki for example dashboard, OOTB and configuration screens

Usage

Download the custom.remote.python.snmp-base.zip and upload to your Dynatrace Environment via Settings - Monitoring - Monitored technologies. You will also need to copy and unzip this to a Linux ActiveGate with a remote plugin module installed at path /opt/dynatrace/remotepluginmodule/plugin_deployment

You can restart the Dynatrace remote plugin module via:

  • RedHat - systemctl restart remotepluginmodule
  • Ubuntu - service remotepluginmodule restart
    Note: This will not yet run on Windows ActiveGates - See Known Issues for details.

Once this has been uploaded succesfully you can start to configure monitoring of devices listening for SNMP V2 and V3, using the configuration screen in the Dynatrace UI:

  • Endpoint name - A display name for the config page - recommended to use hostname.
  • Type of device - Used for grouping similar devices together, e.g. Cisco Switch, Datapower
  • Hostname/IP of device - SNMP connection endpoint : e.g. mydevice.domain:161
  • SNMP Version - Version of SNMP to use 2 or 3
  • SNMP User - The Auth user (V3) or the community name (V2)
  • SNMP v3 Auth protocol - MD5 or SHA
  • SNMP v3 Auth Key - The device auth password
  • SNMP v3 Priv Protocol - AES or DES
  • SNMP v3 Priv Key - The device priv password
  • Name of the group - Used for grouping devices belonging to a particular business area, domain...
  • ActiveGate - The ActiveGate to run this extension polling instance from.

Once configured, you should see an 'Ok' status in the configuration UI and will start to see your device in the Technology overview and metrics availabile for custom charting.

Development

Dependencies

  • Python3.6
  • pysnmp >= 4.4.6

Installation

Note: the Dynatrace oneagent comes bundled with Python3.6, so make sure to use Python3.6 when running oneagent_build_plugin (Plugin SDK) as it will compile the correct native cryptodome modules.

Setup

python3.6 -m venv dtsnmp
source dtsnmp/bin/activate
pip install -r requirements.txt
Download the Dynatrace plugin SDK from your Dynatrace Environment
pip install plugin_sdk-1.157.168.20181127.170441-py3-none-any.whl

Building

  1. sudo .../dtsnmp/bin/oneagent_build_plugin - sdk build

Run the SDK command: sudo .../dtsnmp/bin/oneagent_build_plugin
This will fetch dependencies in plugin.json, compile and bundle everything together under: /opt/dynatrace/remotepluginmodule/plugin_deployment/custom.remote.python.snmp-base.zip - Linux

Adding another MIB

To add support for another MIB and to poll for additional SNMP metrics, add and import an additional class under the dtsnmp module. It is expected to implement the poll_metrics function. Each metric is expected to have the following properties:

  • value: The value of the metric (must be convertible to float)
  • dimension: A dictionary containing the dimension name and value (Strings): name: value. If there is no dimension to split the metric on, then explicitly set this to None as that is handled as default by the SDK.
  • is_absolute_number: A boolean, whether or not the metric is absolute or relative
    • Absolute: Independent, stand alone
    • Relative: Dependent on previous value e.g. Counter

e.g.

class CustomMIB():
	def __init__(self, device, authentication):
		self.poller = Poller(device, authentication)

	def poll_metrics(self):
		metrics = {
			'metric1': [
				{
					'dimension': None
					'is_absolute_number': True,
					'value: value'
				}
			],
			'metric2': [
				{
					'dimension': {'Name': 'value'}
					'is_absolute_number': False,
					'value: value'
				},
				{
					'dimension': {'Name': 'value'}
					'is_absolute_number': False,
					'value: value'
				}
			]
		}

		return metrics

Once defined, you can simply add it to the mib_list in custom_snmp_base_plugin_remote.py e.g.

# Custom MIB
custom_mib = CustomMib(device, authentication)
mib_list.append(custom_mib)

e.g. The metric format for Host-Resources-MIB and IF-MIB is:

#HOST_RESOURCE_MIB 
{
	'cpu_utilisation': [
		{
			'dimension': None,
			'is_absolute_number': True,
			'value': 11.0
		}
	],
	'disk_utilisation': [
		{
			'dimension': {'Storage': '/'}, 
			'is_absolute_number': True,
			'value': 77.06483354871978
		},
		{
			'dimension': {'Storage': '/usr/local'},
			'is_absolute_number': True,
			'value': 79.05778271288935
		},
		...
	]
	'memory_utilisation': [
		{
			'dimension': {'Storage': 'Physical memory'},
 			'is_absolute_number': True,
 			'value': 93.7571919497131
		},
		{
			'dimension': {'Storage': 'Virtual memory'},
			'is_absolute_number': True,
			'value': 49.03610213914279
		},
		...
	]
}

# IF_MIB
{
	'incoming_traffic': [
		{
			'dimension': {'Interface': '1'},
			'is_absolute_number': False,
			'value': 415496409580240.0
		},
		{
			'dimension': {'Interface': '2'},
			'is_absolute_number': False,
			'value': 321564641040573.0,
		}
	],
	'outgoing_traffic': [
		{
			'dimension': {'Interface': '1'},
			'is_absolute_number': False,
			'value': 427663219773201.0,
		}
		{
			'dimension': {'Interface': '2'},
			'is_absolute_number': False,
			'value': 295719899494783.0
		}
	],
	...
}

You can Create an instance of the Poller class to handle SNMP connections and call process_metrics in processing.py with a custom function for metric extraction and calculations.

You must also register your custom metrics by adding them to plugin.json under "metrics". e.g.

"metrics":[
   {
      "entity": "CUSTOM_DEVICE",
      "timeseries":{
         "key":"cpu_utilisation",
         "unit":"Percent",
         "displayname":"CPU utilisation"
      }
   },
   ...
   {
      "entity": "CUSTOM_DEVICE",
      "timeseries":{
         "key":"incoming_traffic",
         "unit":"Byte",
         "displayname":"Incoming traffic",
         "dimensions": ["Interface"]
      }
   },
   ...
]

Testing

To run against the oneagent simulator:
.../dtsnmp/bin/oneagent_sim

To print the SNMP MIB class metrics collected
python test.py

You can test against public endpoints with snmplabs

properties.json is already configured to hit demo.snmplabs.com with SNMP version 2
Testing snmp access and support via the snmpwalk command
e.g. snmpwalk -v3 -l authPriv -u <USER> -a SHA -A <AUTHKEY> -x AES -X <PRIVKEY> demo.snmplabs.com
Check IF-MIB support:
snmpwalk -v3 -l authPriv -u <USER> -a SHA -A <AUTHKEY> -x AES -X <PRIVKEY> <HOST> 1.3.6.1.2.1.31.1.1.1.1
Check HOST-RESOURCES-MIB support:
snmpwal -v3 -l authPriv -u <USER> -a SHA -A <AUTHKEY> -x AES -X <PRIVKEY> <HOST> 1.3.6.1.2.1.25.2.3.1.2

Known Issues

  • Does not yet work for Windows ActiveGates - Error(Cannot load native module 'Cryptodome.Cipher._raw_ecb': Trying '_raw_ecb.cp36-win_amd64.pyd': [WinError 126] The specified module could not be found, Trying '_raw_ecb.pyd': [WinError 126] The specified module could not be found) The modules exist and are compiled correctly... it may just not be using the correct Cryptodome\Util\Cipher\ path
  • Thread exception handling - When a thread enounters an exception it will just log (in most cases) and return an empty Dictionary to main. This should be handled more gracefully.
  • Consumes a lot of custom metrics - I will be adding configuration features to refine Disk and interface dimensions so that we don't just pull back metrics for 'everything'.

Contributors

  • Vu Pham - Wrote the first custom script for this on site using memcached as ActiveGate extensions were not yet available. See Branch
  • Martin Ribot De Bressac - Reviewed the original codebase, implemented additional SNMP MIB monitoring for printers.