Cisco NXOS ansible module cannot run commands on NXOS Version 10.3

Question

Cisco NXOS ansible module cannot run commands on NXOS Version 10.3

arpitmidha opened this issue 2 years ago · 9 comments

SUMMARY

We have cisco devices with NXOS version 10.3 , we are trying to run multiple commands via ansible module cisco.nxos 2.7.1 and getting repeated timeout errors , we are able to manually ssh into the devices and run those commands,
One of the commands we are facing issue is " show ip prefix-list"

ISSUE TYPE

Bug Report

COMPONENT NAME

Cisco.nxos.nxos_command

ANSIBLE VERSION

ansible [core 2.11.12]

cisco.nxos version 2.7.1

Answer 1 · 2023-04-11T05:27:12.000Z

@arpitmidha Could you upgrade the cisco.nxos collection to latest (v4.2.0) and possibly the ansible-core version too?
If you still continue to face the same issue, please enable persistent_log_messages, run the playbook and share the logs with us. The steps to do that can be found here - https://docs.ansible.com/ansible/latest/network/user_guide/network_debug_troubleshooting.html#enabling-networking-device-interaction-logging

Thanks.

Answer 2 · 2023-04-20T17:32:24.000Z

@NilashishC Thank you for the quick response. We will try testing this with cisco.nxos v4.2.0 and give an update here.

To add some info to the original post - here is the full error message we are receiving for any commands on NXOS 10.3

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: See the timeout setting options in the Network Debug and Troubleshooting Guide.
fatal: [clf01-r02]: FAILED! => {"attempts": 3, "changed": false, "module_stderr": "Traceback (most recent call last):\
  File \\"/var/lib/awx/.ansible/tmp/ansible-local-490019ncxy5i46/ansible-tmp-1680687251.670244-492510-227161731429098/AnsiballZ_nxos_command.py\\", line 107, in <module>\
    _ansiballz_main()\
  File \\"/var/lib/awx/.ansible/tmp/ansible-local-490019ncxy5i46/ansible-tmp-1680687251.670244-492510-227161731429098/AnsiballZ_nxos_command.py\\", line 99, in _ansiballz_main\
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\
  File \\"/var/lib/awx/.ansible/tmp/ansible-local-490019ncxy5i46/ansible-tmp-1680687251.670244-492510-227161731429098/AnsiballZ_nxos_command.py\\", line 47, in invoke_module\
    runpy.run_module(mod_name='ansible_collections.cisco.nxos.plugins.modules.nxos_command', init_globals=dict(_module_fqn='ansible_collections.cisco.nxos.plugins.modules.nxos_command', _modlib_path=modlib_path),\
  File \\"/usr/lib/python3.9/runpy.py\\", line 225, in run_module\
    return _run_module_code(code, init_globals, run_name, mod_spec)\
  File \\"/usr/lib/python3.9/runpy.py\\", line 97, in _run_module_code\
    _run_code(code, mod_globals, init_globals,\
  File \\"/usr/lib/python3.9/runpy.py\\", line 87, in _run_code\
    exec(code, run_globals)\
  File \\"/tmp/ansible_nxos_command_payload_8j17gllr/ansible_nxos_command_payload.zip/ansible_collections/cisco/nxos/plugins/modules/nxos_command.py\\", line 237, in <module>\
  File \\"/tmp/ansible_nxos_command_payload_8j17gllr/ansible_nxos_command_payload.zip/ansible_collections/cisco/nxos/plugins/modules/nxos_command.py\\", line 206, in main\
  File \\"/tmp/ansible_nxos_command_payload_8j17gllr/ansible_nxos_command_payload.zip/ansible_collections/cisco/nxos/plugins/module_utils/network/nxos/nxos.py\\", line 1370, in run_commands\
  File \\"/tmp/ansible_nxos_command_payload_8j17gllr/ansible_nxos_command_payload.zip/ansible_collections/cisco/nxos/plugins/module_utils/network/nxos/nxos.py\\", line 129, in get_connection\
  File \\"/tmp/ansible_nxos_command_payload_8j17gllr/ansible_nxos_command_payload.zip/ansible/module_utils/connection.py\\", line 200, in __rpc__\
ansible.module_utils.connection.ConnectionError: command timeout triggered, timeout value is 600 secs.\
See the timeout setting options in the Network Debug and Troubleshooting Guide.\
", "module_stdout": "", "msg": "MODULE FAILURE\
See stdout/stderr for the exact error", "rc": 1}

Answer 3 · 2023-04-24T18:01:06.000Z

@nathan815-msft The traceback doesn't have much information on what is causing the timeout. For that, we would require the device interaction logs. Steps to enable that are shared in my previous comment. Please share that when you get a chance, so that we can debug this further. Thank you!

Answer 4 · 2023-04-25T08:54:41.000Z

@NilashishC
We enabled the network device interaction logs and got below output

2023-04-24 20:29:29,671 p=6600 u=root | paramiko [172.16.2.43] userauth is OK
2023-04-24 20:29:29,899 p=6600 u=root | paramiko [172.16.2.43] Auth banner: b'User Access Verification\n'
2023-04-24 20:29:29,899 p=6600 u=root | paramiko [172.16.2.43] Authentication (password) successful!
2023-04-24 20:29:29,900 p=6600 u=root | paramiko [172.16.2.43] [chan 0] Max packet in: 32768 bytes
2023-04-24 20:29:29,976 p=6600 u=root | paramiko [172.16.2.43] Received global request "hostkeys-00@openssh.com"
2023-04-24 20:29:29,976 p=6600 u=root | paramiko [172.16.2.43] Rejecting "hostkeys-00@openssh.com" global request from server.
2023-04-24 20:29:30,017 p=6600 u=root | paramiko [172.16.2.43] [chan 0] Max packet out: 32768 bytes
2023-04-24 20:29:30,017 p=6600 u=root | paramiko [172.16.2.43] Secsh channel 0 opened.
2023-04-24 20:29:30,020 p=6600 u=root | paramiko [172.16.2.43] [chan 0] Sesch channel 0 request ok
2023-04-24 20:29:30,027 p=6600 u=root | paramiko [172.16.2.43] [chan 0] Sesch channel 0 request ok
2023-04-24 20:29:31,261 p=6495 u=root |  network_os is set to nxos
2023-04-24 20:29:31,303 p=6620 u=root | paramiko [172.16.2.43] starting thread (client mode): 0xaa211c18
2023-04-24 20:29:31,304 p=6620 u=root | paramiko [172.16.2.43] Local version/idstring: SSH-2.0-paramiko_3.1.0
2023-04-24 20:29:31,843 p=6620 u=root | paramiko [172.16.2.43] Remote version/idstring: SSH-2.0-OpenSSH_8.3 PKIX[12.5.1]
2023-04-24 20:29:31,843 p=6620 u=root | paramiko [172.16.2.43] Connected (version 2.0, client OpenSSH_8.3)
2023-04-24 20:29:31,850 p=6620 u=root | paramiko [172.16.2.43] === Key exchange possibilities ===
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] kex algos: curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group16-sha512, diffie-hellman-group14-sha1, diffie-hellman-group14-sha256
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] server key: ssh-rsa, rsa-sha2-256
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] client encrypt: aes128-ctr, aes256-ctr, aes256-gcm@openssh.com, aes128-gcm@openssh.com, chacha20-poly1305@openssh.com
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] server encrypt: aes128-ctr, aes256-ctr, aes256-gcm@openssh.com, aes128-gcm@openssh.com, chacha20-poly1305@openssh.com
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] client mac: hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] server mac: hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] client compress: none, zlib@openssh.com
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] server compress: none, zlib@openssh.com
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] client lang: <none>
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] server lang: <none>
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] kex follows: False
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] === Key exchange agreements ===
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] Kex: curve25519-sha256@libssh.org
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] HostKey: ssh-rsa
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] Cipher: aes128-ctr
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] MAC: hmac-sha2-256
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] Compression: none
2023-04-24 20:29:31,851 p=6620 u=root | paramiko [172.16.2.43] === End of kex handshake ===
2023-04-24 20:29:31,878 p=6620 u=root | paramiko [172.16.2.43] kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
2023-04-24 20:29:31,879 p=6620 u=root | paramiko [172.16.2.43] Switch to new keys ...
2023-04-24 20:29:31,880 p=6620 u=root | paramiko [172.16.2.43] Got EXT_INFO: {'publickey-algorithms@roumenpetrov.info': b'x509v3-sign-rsa,x509v3-sign-dss,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512', 'server-sig-algs': b'x509v3-sign-rsa,x509v3-sign-dss,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512'}
2023-04-24 20:29:31,980 p=6620 u=root | paramiko [172.16.2.43] userauth is OK
2023-04-24 20:29:32,209 p=6620 u=root | paramiko [172.16.2.43] Auth banner: b'User Access Verification\n'
2023-04-24 20:29:32,209 p=6620 u=root | paramiko [172.16.2.43] Authentication (password) successful!
2023-04-24 20:29:32,209 p=6620 u=root | paramiko [172.16.2.43] [chan 0] Max packet in: 32768 bytes
2023-04-24 20:29:32,266 p=6620 u=root | paramiko [172.16.2.43] Received global request "hostkeys-00@openssh.com"
2023-04-24 20:29:32,266 p=6620 u=root | paramiko [172.16.2.43] Rejecting "hostkeys-00@openssh.com" global request from server.
2023-04-24 20:29:32,313 p=6620 u=root | paramiko [172.16.2.43] [chan 0] Max packet out: 32768 bytes
2023-04-24 20:29:32,313 p=6620 u=root | paramiko [172.16.2.43] Secsh channel 0 opened.
2023-04-24 20:29:32,318 p=6620 u=root | paramiko [172.16.2.43] [chan 0] Sesch channel 0 request ok
2023-04-24 20:29:32,325 p=6620 u=root | paramiko [172.16.2.43] [chan 0] Sesch channel 0 request ok
2023-04-24 20:29:33,651 p=6498 u=root |  network_os is set to nxos
2023-04-24 20:29:33,694 p=6640 u=root | paramiko [172.16.2.43] starting thread (client mode): 0xae6f9c18
2023-04-24 20:29:33,695 p=6640 u=root | paramiko [172.16.2.43] Local version/idstring: SSH-2.0-paramiko_3.1.0
2023-04-24 20:29:34,296 p=6640 u=root | paramiko [172.16.2.43] Remote version/idstring: SSH-2.0-OpenSSH_8.3 PKIX[12.5.1]
2023-04-24 20:29:34,296 p=6640 u=root | paramiko [172.16.2.43] Connected (version 2.0, client OpenSSH_8.3)
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] === Key exchange possibilities ===
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] kex algos: curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group16-sha512, diffie-hellman-group14-sha1, diffie-hellman-group14-sha256
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] server key: ssh-rsa, rsa-sha2-256
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] client encrypt: aes128-ctr, aes256-ctr, aes256-gcm@openssh.com, aes128-gcm@openssh.com, chacha20-poly1305@openssh.com
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] server encrypt: aes128-ctr, aes256-ctr, aes256-gcm@openssh.com, aes128-gcm@openssh.com, chacha20-poly1305@openssh.com
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] client mac: hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] server mac: hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] client compress: none, zlib@openssh.com
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] server compress: none, zlib@openssh.com
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] client lang: <none>
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] server lang: <none>
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] kex follows: False
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] === Key exchange agreements ===
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] Kex: curve25519-sha256@libssh.org
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] HostKey: ssh-rsa
2023-04-24 20:29:34,302 p=6640 u=root | paramiko [172.16.2.43] Cipher: aes128-ctr
2023-04-24 20:29:34,303 p=6640 u=root | paramiko [172.16.2.43] MAC: hmac-sha2-256
2023-04-24 20:29:34,303 p=6640 u=root | paramiko [172.16.2.43] Compression: none
2023-04-24 20:29:34,303 p=6640 u=root | paramiko [172.16.2.43] === End of kex handshake ===
2023-04-24 20:29:34,318 p=6640 u=root | paramiko [172.16.2.43] kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
2023-04-24 20:29:34,319 p=6640 u=root | paramiko [172.16.2.43] Switch to new keys ...
2023-04-24 20:29:34,320 p=6640 u=root | paramiko [172.16.2.43] Got EXT_INFO: {'publickey-algorithms@roumenpetrov.info': b'x509v3-sign-rsa,x509v3-sign-dss,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512', 'server-sig-algs': b'x509v3-sign-rsa,x509v3-sign-dss,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512'}
2023-04-24 20:29:34,449 p=6640 u=root | paramiko [172.16.2.43] userauth is OK
2023-04-24 20:29:34,677 p=6640 u=root | paramiko [172.16.2.43] Auth banner: b'User Access Verification\n'
2023-04-24 20:29:34,677 p=6640 u=root | paramiko [172.16.2.43] Authentication (password) successful!
2023-04-24 20:29:34,678 p=6640 u=root | paramiko [172.16.2.43] [chan 0] Max packet in: 32768 bytes
2023-04-24 20:29:34,735 p=6640 u=root | paramiko [172.16.2.43] Received global request "hostkeys-00@openssh.com"
2023-04-24 20:29:34,735 p=6640 u=root | paramiko [172.16.2.43] Rejecting "hostkeys-00@openssh.com" global request from server.
2023-04-24 20:29:34,777 p=6640 u=root | paramiko [172.16.2.43] [chan 0] Max packet out: 32768 bytes
2023-04-24 20:29:34,777 p=6640 u=root | paramiko [172.16.2.43] Secsh channel 0 opened.
2023-04-24 20:29:34,780 p=6640 u=root | paramiko [172.16.2.43] [chan 0] Sesch channel 0 request ok
2023-04-24 20:29:34,786 p=6640 u=root | paramiko [172.16.2.43] [chan 0] Sesch channel 0 request ok
2023-04-24 20:29:35,875 p=6502 u=root |  network_os is set to nxos
2023-04-24 20:29:35,917 p=6660 u=root | paramiko [172.16.2.43] starting thread (client mode): 0x84a45c50
2023-04-24 20:29:35,917 p=6660 u=root | paramiko [172.16.2.43] Local version/idstring: SSH-2.0-paramiko_3.1.0
2023-04-24 20:29:36,231 p=6660 u=root | paramiko [172.16.2.43] Remote version/idstring: SSH-2.0-OpenSSH_8.3 PKIX[12.5.1]
2023-04-24 20:29:36,231 p=6660 u=root | paramiko [172.16.2.43] Connected (version 2.0, client OpenSSH_8.3)
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] === Key exchange possibilities ===
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] kex algos: curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group16-sha512, diffie-hellman-group14-sha1, diffie-hellman-group14-sha256
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] server key: ssh-rsa, rsa-sha2-256
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] client encrypt: aes128-ctr, aes256-ctr, aes256-gcm@openssh.com, aes128-gcm@openssh.com, chacha20-poly1305@openssh.com
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] server encrypt: aes128-ctr, aes256-ctr, aes256-gcm@openssh.com, aes128-gcm@openssh.com, chacha20-poly1305@openssh.com
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] client mac: hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] server mac: hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] client compress: none, zlib@openssh.com
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] server compress: none, zlib@openssh.com
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] client lang: <none>
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] server lang: <none>
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] kex follows: False
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] === Key exchange agreements ===
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] Kex: curve25519-sha256@libssh.org
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] HostKey: ssh-rsa
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] Cipher: aes128-ctr
2023-04-24 20:29:36,236 p=6660 u=root | paramiko [172.16.2.43] MAC: hmac-sha2-256
2023-04-24 20:29:36,237 p=6660 u=root | paramiko [172.16.2.43] Compression: none
2023-04-24 20:29:36,237 p=6660 u=root | paramiko [172.16.2.43] === End of kex handshake ===
2023-04-24 20:29:36,248 p=6660 u=root | paramiko [172.16.2.43] kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
2023-04-24 20:29:36,249 p=6660 u=root | paramiko [172.16.2.43] Switch to new keys ...
2023-04-24 20:29:36,249 p=6660 u=root | paramiko [172.16.2.43] Got EXT_INFO: {'publickey-algorithms@roumenpetrov.info': b'x509v3-sign-rsa,x509v3-sign-dss,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512', 'server-sig-algs': b'x509v3-sign-rsa,x509v3-sign-dss,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512'}
2023-04-24 20:29:36,349 p=6660 u=root | paramiko [172.16.2.43] userauth is OK
2023-04-24 20:29:36,577 p=6660 u=root | paramiko [172.16.2.43] Auth banner: b'User Access Verification\n'
2023-04-24 20:29:36,577 p=6660 u=root | paramiko [172.16.2.43] Authentication (password) successful!
2023-04-24 20:29:36,577 p=6660 u=root | paramiko [172.16.2.43] [chan 0] Max packet in: 32768 bytes
2023-04-24 20:29:36,649 p=6660 u=root | paramiko [172.16.2.43] Received global request "hostkeys-00@openssh.com"
2023-04-24 20:29:36,649 p=6660 u=root | paramiko [172.16.2.43] Rejecting "hostkeys-00@openssh.com" global request from server.
2023-04-24 20:29:36,693 p=6660 u=root | paramiko [172.16.2.43] [chan 0] Max packet out: 32768 bytes
2023-04-24 20:29:36,693 p=6660 u=root | paramiko [172.16.2.43] Secsh channel 0 opened.
2023-04-24 20:29:36,696 p=6660 u=root | paramiko [172.16.2.43] [chan 0] Sesch channel 0 request ok
2023-04-24 20:29:36,701 p=6660 u=root | paramiko [172.16.2.43] [chan 0] Sesch channel 0 request ok
2023-04-24 20:29:37,569 p=6494 u=root |  network_os is set to nxos
2023-04-24 20:29:37,612 p=6680 u=root | paramiko [172.16.2.43] starting thread (client mode): 0xd35c18
2023-04-24 20:29:37,612 p=6680 u=root | paramiko [172.16.2.43] Local version/idstring: SSH-2.0-paramiko_3.1.0
2023-04-24 20:29:37,942 p=6680 u=root | paramiko [172.16.2.43] Remote version/idstring: SSH-2.0-OpenSSH_8.3 PKIX[12.5.1]
2023-04-24 20:29:37,942 p=6680 u=root | paramiko [172.16.2.43] Connected (version 2.0, client OpenSSH_8.3)
2023-04-24 20:29:37,948 p=6680 u=root | paramiko [172.16.2.43] === Key exchange possibilities ===
2023-04-24 20:29:37,948 p=6680 u=root | paramiko [172.16.2.43] kex algos: curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group16-sha512, diffie-hellman-group14-sha1, diffie-hellman-group14-sha256
2023-04-24 20:29:37,948 p=6680 u=root | paramiko [172.16.2.43] server key: ssh-rsa, rsa-sha2-256
2023-04-24 20:29:37,948 p=6680 u=root | paramiko [172.16.2.43] client encrypt: aes128-ctr, aes256-ctr, aes256-gcm@openssh.com, aes128-gcm@openssh.com, chacha20-poly1305@openssh.com
2023-04-24 20:29:37,948 p=6680 u=root | paramiko [172.16.2.43] server encrypt: aes128-ctr, aes256-ctr, aes256-gcm@openssh.com, aes128-gcm@openssh.com, chacha20-poly1305@openssh.com
2023-04-24 20:29:37,948 p=6680 u=root | paramiko [172.16.2.43] client mac: hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2023-04-24 20:29:37,948 p=6680 u=root | paramiko [172.16.2.43] server mac: hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2023-04-24 20:29:37,948 p=6680 u=root | paramiko [172.16.2.43] client compress: none, zlib@openssh.com
2023-04-24 20:29:37,948 p=6680 u=root | paramiko [172.16.2.43] server compress: none, zlib@openssh.com
2023-04-24 20:29:37,948 p=6680 u=root | paramiko [172.16.2.43] client lang: <none>
2023-04-24 20:29:37,948 p=6680 u=root | paramiko [172.16.2.43] server lang: <none>
2023-04-24 20:29:37,949 p=6680 u=root | paramiko [172.16.2.43] kex follows: False
2023-04-24 20:29:37,949 p=6680 u=root | paramiko [172.16.2.43] === Key exchange agreements ===
2023-04-24 20:29:37,949 p=6680 u=root | paramiko [172.16.2.43] Kex: curve25519-sha256@libssh.org
2023-04-24 20:29:37,949 p=6680 u=root | paramiko [172.16.2.43] HostKey: ssh-rsa
2023-04-24 20:29:37,949 p=6680 u=root | paramiko [172.16.2.43] Cipher: aes128-ctr
2023-04-24 20:29:37,949 p=6680 u=root | paramiko [172.16.2.43] MAC: hmac-sha2-256
2023-04-24 20:29:37,949 p=6680 u=root | paramiko [172.16.2.43] Compression: none
2023-04-24 20:29:37,949 p=6680 u=root | paramiko [172.16.2.43] === End of kex handshake ===
2023-04-24 20:29:37,962 p=6680 u=root | paramiko [172.16.2.43] kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
2023-04-24 20:29:37,962 p=6680 u=root | paramiko [172.16.2.43] Switch to new keys ...
2023-04-24 20:29:37,963 p=6680 u=root | paramiko [172.16.2.43] Got EXT_INFO: {'publickey-algorithms@roumenpetrov.info': b'x509v3-sign-rsa,x509v3-sign-dss,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512', 'server-sig-algs': b'x509v3-sign-rsa,x509v3-sign-dss,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512'}
2023-04-24 20:29:38,065 p=6680 u=root | paramiko [172.16.2.43] userauth is OK
2023-04-24 20:29:38,293 p=6680 u=root | paramiko [172.16.2.43] Auth banner: b'User Access Verification\n'
2023-04-24 20:29:38,293 p=6680 u=root | paramiko [172.16.2.43] Authentication (password) successful!
2023-04-24 20:29:38,294 p=6680 u=root | paramiko [172.16.2.43] [chan 0] Max packet in: 32768 bytes
2023-04-24 20:29:38,384 p=6680 u=root | paramiko [172.16.2.43] Received global request "hostkeys-00@openssh.com"
2023-04-24 20:29:38,384 p=6680 u=root | paramiko [172.16.2.43] Rejecting "hostkeys-00@openssh.com" global request from server.
2023-04-24 20:29:38,425 p=6680 u=root | paramiko [172.16.2.43] [chan 0] Max packet out: 32768 bytes
2023-04-24 20:29:38,425 p=6680 u=root | paramiko [172.16.2.43] Secsh channel 0 opened.
2023-04-24 20:29:38,428 p=6680 u=root | paramiko [172.16.2.43] [chan 0] Sesch channel 0 request ok
2023-04-24 20:29:38,433 p=6680 u=root | paramiko [172.16.2.43] [chan 0] Sesch channel 0 request ok
2023-04-24 20:29:39,404 p=6501 u=root |  network_os is set to nxos
2023-04-24 20:29:39,445 p=6700 u=root | paramiko [172.16.2.43] starting thread (client mode): 0x877a4ba8
2023-04-24 20:29:39,445 p=6700 u=root | paramiko [172.16.2.43] Local version/idstring: SSH-2.0-paramiko_3.1.0
2023-04-24 20:29:39,916 p=6700 u=root | paramiko [172.16.2.43] Remote version/idstring: SSH-2.0-OpenSSH_8.3 PKIX[12.5.1]
2023-04-24 20:29:39,916 p=6700 u=root | paramiko [172.16.2.43] Connected (version 2.0, client OpenSSH_8.3)
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] === Key exchange possibilities ===
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] kex algos: curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group16-sha512, diffie-hellman-group14-sha1, diffie-hellman-group14-sha256
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] server key: ssh-rsa, rsa-sha2-256
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] client encrypt: aes128-ctr, aes256-ctr, aes256-gcm@openssh.com, aes128-gcm@openssh.com, chacha20-poly1305@openssh.com
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] server encrypt: aes128-ctr, aes256-ctr, aes256-gcm@openssh.com, aes128-gcm@openssh.com, chacha20-poly1305@openssh.com
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] client mac: hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] server mac: hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha1-etm@openssh.com, hmac-sha2-256, hmac-sha2-512, hmac-sha1
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] client compress: none, zlib@openssh.com
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] server compress: none, zlib@openssh.com
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] client lang: <none>
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] server lang: <none>
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] kex follows: False
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] === Key exchange agreements ===
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] Kex: curve25519-sha256@libssh.org
2023-04-24 20:29:39,929 p=6700 u=root | paramiko [172.16.2.43] HostKey: ssh-rsa
2023-04-24 20:29:39,930 p=6700 u=root | paramiko [172.16.2.43] Cipher: aes128-ctr
2023-04-24 20:29:39,930 p=6700 u=root | paramiko [172.16.2.43] MAC: hmac-sha2-256
2023-04-24 20:29:39,930 p=6700 u=root | paramiko [172.16.2.43] Compression: none
2023-04-24 20:29:39,930 p=6700 u=root | paramiko [172.16.2.43] === End of kex handshake ===
2023-04-24 20:29:39,963 p=6700 u=root | paramiko [172.16.2.43] kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
2023-04-24 20:29:39,964 p=6700 u=root | paramiko [172.16.2.43] Switch to new keys ...
2023-04-24 20:29:39,965 p=6700 u=root | paramiko [172.16.2.43] Got EXT_INFO: {'publickey-algorithms@roumenpetrov.info': b'x509v3-sign-rsa,x509v3-sign-dss,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512', 'server-sig-algs': b'x509v3-sign-rsa,x509v3-sign-dss,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512'}
2023-04-24 20:29:40,066 p=6700 u=root | paramiko [172.16.2.43] userauth is OK
2023-04-24 20:29:40,520 p=6700 u=root | paramiko [172.16.2.43] Auth banner: b'User Access Verification\n'
2023-04-24 20:29:40,520 p=6700 u=root | paramiko [172.16.2.43] Authentication (password) successful!
2023-04-24 20:29:40,521 p=6700 u=root | paramiko [172.16.2.43] [chan 0] Max packet in: 32768 bytes
2023-04-24 20:29:40,588 p=6700 u=root | paramiko [172.16.2.43] Received global request "hostkeys-00@openssh.com"
2023-04-24 20:29:40,588 p=6700 u=root | paramiko [172.16.2.43] Rejecting "hostkeys-00@openssh.com" global request from server.
2023-04-24 20:29:40,630 p=6700 u=root | paramiko [172.16.2.43] [chan 0] Max packet out: 32768 bytes
2023-04-24 20:29:40,630 p=6700 u=root | paramiko [172.16.2.43] Secsh channel 0 opened.
2023-04-24 20:29:40,636 p=6700 u=root | paramiko [172.16.2.43] [chan 0] Sesch channel 0 request ok
2023-04-24 20:29:40,642 p=6700 u=root | paramiko [172.16.2.43] [chan 0] Sesch channel 0 request ok
2023-04-24 20:32:22,895 p=6493 u=root |  command timeout triggered, timeout value is 180 secs.
See the timeout setting options in the Network Debug and Troubleshooting Guide.

```EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-local-8154qrdgdy77/ansible-tmp-1682412624.6944456-145983820951559 `" && echo ansible-tmp-1682412624.6944456-145983820951559="` echo /root/.ansible/tmp/ansible-local-8154qrdgdy77/ansible-tmp-1682412624.6944456-145983820951559 `" ) && sleep 0'
Using module file /usr/local/lib/python3.6/site-packages/ansible/modules/network/nxos/nxos_config.py
PUT /root/.ansible/tmp/ansible-local-8154qrdgdy77/tmp066q1jq2 TO /root/.ansible/tmp/ansible-local-8154qrdgdy77/ansible-tmp-1682412624.6944456-145983820951559/AnsiballZ_nxos_config.py
EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-local-8154qrdgdy77/ansible-tmp-1682412624.6944456-145983820951559/ /root/.ansible/tmp/ansible-local-8154qrdgdy77/ansible-tmp-1682412624.6944456-145983820951559/AnsiballZ_nxos_config.py && sleep 0'
 EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-local-8154qrdgdy77/ansible-tmp-1682412624.6944456-145983820951559/AnsiballZ_nxos_config.py && sleep 0'
 EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-local-81597nbbrokz/ansible-tmp-1682412446.7001665-93340127680926/ > /dev/null 2>&1 && sleep 0'
FAILED - RETRYING: Removing the mgmt route maps for the tenant. (1 retries left).Result was: {
    "attempts": 3,
    "changed": false,
    "invocation": {
        "module_args": {
            "after": null,
            "auth_pass": null,
            "authorize": null,
            "backup": false,
            "before": null,
            "defaults": false,
            "diff_against": null,
            "diff_ignore_lines": null,
            "force": false,
            "host": null,
            "intended_config": null,
            "lines": [
                "no route-map RMAP-TNT905-MGMT-TO-PRIV-LEAK permit 30"
            ],
            "match": "line",
            "parents": null,
            "password": null,
            "port": null,
            "provider": null,
            "replace": "line",
            "replace_src": null,
            "running_config": null,
            "save": false,
            "save_when": "never",
            "src": null,
            "ssh_keyfile": null,
            "timeout": null,
            "transport": null,
            "use_ssl": null,
            "username": null,
            "validate_certs": null
        }
    },
    "msg": "command timeout triggered, timeout value is 180 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide.",`

Answer 5 · 2023-04-26T06:05:49.000Z

@arpitmidha Does running the playbook with ansible_network_cli_ssh_type: libssh give the same result?

Are these logs from a run that uses cisco.nxos v4.2.0 or v2.7.1? What is the version of ansible.netcommon being used?

The ansible-core v2.11 is at it's EOL. Could you please upgrade to ansible-core==2.14.x or 2.15.x (latest) and try this again?

Answer 6 · 2023-04-26T06:09:13.000Z

@Qalthos any ideas on what might be causing this issue?

Answer 7 · 2023-05-17T16:39:21.000Z

There is an issue when ansible_become is set to tru, ansible cannot program the nxos 10.3 devices when ansible_become is set to true, if ansible_become is set to false, it is able to perform the configuration.
The reason for this is, in Cisco nxos 10.x devices, "show privilege" command is not available, and when ansible_become is set to true in ansible inventory then the nxos module tries to run "show privilege" command

Answer 8 · 2023-05-19T05:37:02.000Z

This has been explained with more details in #304 (comment).

There are two solutions to this:

Turn off privilege escalation (ansible_become=False) for hosts that are running NX-OS >= 10.0.0
Or, set ansible_network_become_errors to determine how privilege escalation failures are handled. This option was added in ansible.netcommon v2.5.0, so if you're running an older version of netcommon than this, you would have to upgrade it.

Hope that resolves this issue for you.

Answer 9 · 2023-05-19T05:37:57.000Z

Please let us know if turning off privilege escalation or suppressing the errors fixes the original problem reported in this ticket. Thanks!