SNMP not working for any device on network, nothing blocking access and credentials correct.

Question

SNMP not working for any device on network, nothing blocking access and credentials correct.

Closed this issue 9 months ago · 5 comments

Using Netdisco's latest version on an EC2 instance. All traffic is permitted to/from this instance thru port 161, 162, 22, and 5000. Netdisco is running as expected, not showing any errors in logs, and I can connect to the web interface just fine.

Attempting command ~netdisco/bin/netdisco-do discover -d 10.39.25.253 -DIQ where 10.39.25.253 is a Cisco Switch that the instance can successfully ping and SSH into. The EC2 instance's IP address has been listed as an SNMP collector on the switch. When running the command, I receive the following output: (It is worth noting that it hangs for about 10s before timing out, and if any auth is incorrect it immediately times out)

[netdisco@uvaadmmapp01ndc logs]$ ~netdisco/bin/netdisco-do discover -d 10.39.25.253 -DIQ
[13238] 2023-02-13 18:26:44 info App::Netdisco version 2.060003 loaded.
SELECT me.version, me.installed
FROM dbix_class_schema_versions me
WHERE 1 = 0
SELECT me.version
FROM dbix_class_schema_versions me
ORDER BY installed DESC
LIMIT '1'
SELECT me.ip, me.alias, me.subnet, me.port, me.dns, me.creation
FROM device_ip me
WHERE me.alias = '10.39.25.253' AND me.ip = '10.39.25.253'
SELECT me.ip, me.alias, me.subnet, me.port, me.dns, me.creation
FROM device_ip me
WHERE alias = '10.39.25.253'
SELECT me.ip, me.creation, me.dns, me.description, me.uptime, me.contact, me.name, me.location, me.layers, me.num_ports, me.mac, me.serial, me.chassis_id, me.model, me.ps1_type, me.ps2_type, me.ps1_status, me.ps2_status, me.fan, me.slots, me.vendor, me.os, me.os_ver, me.log, me.snmp_ver, me.snmp_comm, me.snmp_class, me.snmp_engineid, me.vtp_domain, me.last_discover, me.last_macsuck, me.last_arpnip, me.is_pseudo, me.pae_is_enabled, me.custom_fields, to_char( me.creation, 'YYYY-MM-DD HH24:MI' ), to_char( me.last_arpnip, 'YYYY-MM-DD HH24:MI' ), to_char( me.last_discover, 'YYYY-MM-DD HH24:MI' ), to_char( me.last_macsuck, 'YYYY-MM-DD HH24:MI' ), extract( epoch
FROM age( now( ), me.creation ) ), extract( epoch
FROM age( now( ), me.last_arpnip ) ), extract( epoch
FROM age( now( ), me.last_discover ) ), extract( epoch
FROM age( now( ), me.last_macsuck ) ), replace( age( timestamp 'epoch' + me.uptime / 100 * interval '1 second', timestamp '1970-01-01 00:00:00-00' ) ::text, 'mon', 'month' )
FROM device me
WHERE me.ip = '10.39.25.253'
[13238] 2023-02-13 18:26:45 info discover: [10.39.25.253] started at Mon Feb 13 13:26:45 2023
[13238] 2023-02-13 18:26:45 debug discover: running with timeout 600s
[13238] 2023-02-13 18:26:45 debug => running workers for phase: check
[13238] 2023-02-13 18:26:45 debug -> run worker check/0 "discover"
[13238] 2023-02-13 18:26:45 debug Discover is able to run.
[13238] 2023-02-13 18:26:45 debug => running workers for phase: early
[13238] 2023-02-13 18:26:45 debug -> run worker early/100 "discover::properties"
[13238] 2023-02-13 18:26:45 debug snmp reader cache warm: [10.39.25.253]
SELECT me.ip, me.snmp_comm_rw, me.snmp_auth_tag_read, me.snmp_auth_tag_write
FROM community me
WHERE me.ip = '10.39.25.253'
SELECT me.ip, me.snmp_comm_rw, me.snmp_auth_tag_read, me.snmp_auth_tag_write
FROM community me
WHERE me.ip = '10.39.25.253'
[13238] 2023-02-13 18:26:45 debug [10.39.25.253:161] try_connect with ver: 3, class: SNMP::Info, comm:
SNMP::Info::_global uptime : DISMAN-EVENT-MIB::sysUpTimeInstance : .1.3.6.1.2.1.1.3.0
SNMP::Info::_global(uptime) Timeout at /home/netdisco/perl5/lib/perl5/App/Netdisco/Transport/SNMP.pm line 263.
SNMP::Info::_global hrSystemUptime : HOST-RESOURCES-MIB::hrSystemUptime.0 : .1.3.6.1.2.1.25.1.1.0
SNMP::Info::_global(hrSystemUptime) Timeout at /home/netdisco/perl5/lib/perl5/App/Netdisco/Transport/SNMP.pm line 263.
SNMP::Info::_global sysUpTime : DISMAN-EVENT-MIB::sysUpTimeInstance : .1.3.6.1.2.1.1.3.0
SNMP::Info::_global(sysUpTime) Timeout at /home/netdisco/perl5/lib/perl5/App/Netdisco/Transport/SNMP.pm line 263.
[13238] 2023-02-13 18:27:12 debug discover failed: could not SNMP connect to 10.39.25.253
[13238] 2023-02-13 18:27:12 debug -> run worker early/100 "discover::properties"
[13238] 2023-02-13 18:27:12 debug -> run worker early/100 "discover::properties"
[13238] 2023-02-13 18:27:12 debug -> run worker early/100 "discover::properties"
[13238] 2023-02-13 18:27:12 debug -> run worker early/100 "discover::properties"
[13238] 2023-02-13 18:27:12 debug => running workers for phase: main
[13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::canonicalip"
[13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::entities"
[13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::neighbors"
[13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::neighbors::docsis"
[13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::neighbors::routed"
[13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::portpower"
[13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::portproperties"
[13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::portproperties::portaccessentity"
[13238] 2023-02-13 18:27:12 debug pae failed: could not SNMP connect to 10.39.25.253
[13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::vlans"
[13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::wireless"
[13238] 2023-02-13 18:27:12 debug -> run worker main/0 "discover::withnodes"
[13238] 2023-02-13 18:27:12 debug => running workers for phase: late
[13238] 2023-02-13 18:27:12 debug -> run worker late/0 "discover::hooks"
[13238] 2023-02-13 18:27:12 debug [10.39.25.253] hooks - 0 queued
[13238] 2023-02-13 18:27:12 info discover: finished at Mon Feb 13 13:27:12 2023
[13238] 2023-02-13 18:27:12 info discover: status defer: discover failed: could not SNMP connect to 10.39.25.253

The switch is using SNMPv3 and auth should be configured correctly in deployment.yml, as follows:

device_auth:

tag: snmpv3_sw
user: myUser
auth:
pass: myAuthPass
proto: SHA
priv:
pass: myPrivPass
proto: AES256

This issue recurs when connecting to ANY network device.

Expected Behavior

SNMP Connection successful and netdisco scans

Possible Solution

I have tried configuring the device as a psuedo device to workaround the SNMP issue without success, being told it does not have an offline cache.

Answer 1 · 2024-02-11T00:10:44.000Z

interesting... wanted to update our docs for snmpv3 and came across this.

let's cover the basics first. going on the output & description i don't see an issue, but let's check these just to be sure.

you say

This issue recurs when connecting to ANY network device.

for clarification: does this happen on different vendors / devices / os versions?
what does netdisco-do show -d 10.39.25.253 -e specify say?

Caution

if you paste that here be aware that it contains cleartext credentials , you might want to remove those first.

to verify, when you say

All traffic is permitted to/from this instance thru port 161, 162, 22, and 5000

does this includes udp/161 (not just tcp)? i'm guessing it does since you mention that incorrect credentials get rejected immediately.
the example device_auth you pasted, can you paste that again but using triple backticks example here, formatting is relevant for the config.

you do mention it's a cisco device & you are using aes-256. i don't have that specific config available for testing atm. i did run into this however: netdisco/netdisco#962 (comment) . tldr: try changing aes256 to aes256c.

dunno what access you have on that aws instance, but if you have cli access testing snmpwalk could also give insights (used values from your example):
snmpwalk -v 3 -x AES256 -X myPrivPass -a SHA -A myAuthPass -u myUser -l authPriv 10.39.25.253
and
snmpwalk -v 3 -x AES256C -X myPrivPass -a SHA -A myAuthPass -u myUser -l authPriv 10.39.25.253

if aes256 and/or aes256c aren't supported you will get the help blurb, if the device rejects it for whatever reason you get snmpwalk: Authentication failure (incorrect password, community or key), if it works - well - you get actual info :)

Answer 2 · 2024-02-11T00:24:36.000Z

I did manage to resolve this, it was a few months ago but if I recall correctly the issue was that netdisco wasn't properly using snmpwalk or that snmpwalk wasn't working with snmpv3. I wound up reinstalling the library snmpwalk is from and installing a different version then reconfiguring that for snmpv3 and it wound up working.

…

On Sat, Feb 10, 2024, 7:10 PM nick n. ***@***.***> wrote: interesting... wanted to update our docs for snmpv3 and came across this. let's cover the basics first. going on the output & description i don't see an issue, but let's check these just to be sure. you say This issue recurs when connecting to ANY network device. 1. for clarification: does this happen on different vendors / devices / os versions? 2. what does netdisco-do show -d 10.39.25.253 -e specify say? Caution if you paste that here be aware that it contains cleartext credentials , you might want to remove those first. to verify, when you say All traffic is permitted to/from this instance thru port 161, 162, 22, and 5000 1. does this includes udp/161 (not just tcp)? i'm guessing it does since you mention that incorrect credentials get rejected immediately. 2. the example device_auth you pasted, can you paste that again but using triple backticks example here <https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#quoting-code>, formatting is relevant for the config. you do mention it's a cisco device & you are using aes-256. i don't have that specific config available for testing atm. i did run into this however: netdisco/netdisco#962 (comment) <netdisco/netdisco#962 (comment)> . tldr: try changing aes256 to aes256c. dunno what access you have on that aws instance, but if you have cli access testing snmpwalk could also give insights (used values from your example): snmpwalk -v 3 -x AES256 -X myPrivPass -a SHA -A myAuthPass -u myUser -l authPriv 10.39.25.253 and snmpwalk -v 3 -x AES256C -X myPrivPass -a SHA -A myAuthPass -u myUser -l authPriv 10.39.25.253 if aes256 and/or aes256c aren't supported you will get the help blurb, if the device rejects it for whatever reason you get snmpwalk: Authentication failure (incorrect password, community or key), if it works - well - you get actual info :) — Reply to this email directly, view it on GitHub <#480 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALGG5ZOCCPJPYGPBPJFV2A3YTAEBBAVCNFSM6AAAAAAU2TQA4WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZXGM3DGNZQHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

Answer 3 · 2024-02-11T01:41:49.000Z

the issue was that netdisco wasn't properly using snmpwalk or that snmpwalk wasn't working with snmpv3

yeah, i was starting to update the snmpv3 documentation for snmp::info & netdisco for newer hashing algo's and quickly ran into all the bits that need to be in sync to make it work. troubleshooting my own setup wasn't that hard, but writing a generalized guide is a challenge.

do you perhaps recall the steps you took to fix it?

what i'm most curious about is the net-snmp version. specifically: was it 5.9.1 or newer, or was it an older version?

Answer 4 · 2024-02-11T02:35:15.000Z

I believe the issue was related specifically to AES256. I used ChatGPT to recreate the issue and troubleshoot so you can find all the info here: https://chat.openai.com/share/d8686d8d-1ebe-4e23-97a5-dac7d384f8af Hopefully that's helpful and let me know if you have any other questions!

…

On Sat, Feb 10, 2024, 8:42 PM nick n. ***@***.***> wrote: the issue was that netdisco wasn't properly using snmpwalk or that snmpwalk wasn't working with snmpv3 yeah, i was starting to update the snmpv3 documentation for snmp::info & netdisco for newer hashing algo's and quickly ran into all the bits that need to be in sync to make it work. troubleshooting my own setup wasn't that hard, but writing a generalized guide is a challenge. do you perhaps recall the steps you took to fix it? what i'm most curious about is the net-snmp version. specifically: was it 5.9.1 or newer, or was it an older version? — Reply to this email directly, view it on GitHub <#480 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALGG5ZLJAA6G5JL3QZDNSVTYTAOWRAVCNFSM6AAAAAAU2TQA4WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZXGM4DOMBZG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

Answer 5 · 2024-02-16T20:37:20.000Z

i've opened #515 based on your troubleshooting workflow and my own observations.

it summarizes everything i think needs better documentation/handling. if there's something you think that should still be added just make a note in that ticket.

(i did already update the netdisco wiki with the basics.)