ktbyers/netmiko

Netmiko truncates the output of `show datapath session table` on Aruba 7210 AMC/WLCs.

somenath-sinha opened this issue · 10 comments

Netmiko truncates the output of show datapath session table on Aruba 7210 AMC/WLCs.

The full output is supposed to be around 12844 lines long, but netmiko.send_command() and netmiko.send_command_expect() both only partially get data till around the 84th line of the output, and then stop. What's even more perplexing is - when debugging is enabled, I can see that the rest of the data does show up in the logs, but not in the output from the commands.

Note: Please check https://guides.github.com/features/mastering-markdown/
to see how to properly format your request.

Setup

Netmiko version

(Paste verbatim output from pip freeze | grep netmiko between quotes below)

> pip freeze | grep netmiko
netmiko==4.3.0

Netmiko device_type (if relevant to the issue)

(Paste device_type between quotes below)

'device_type': 'aruba_os',

Steps to Reproduce the Issue

SSH into the device running Aruba OS (WLC/Mobility Controller) and run the show datapath sessions table command with netmiko. As is general behaviour, no paging is already in effect, clear from the fact that around the 84th line of output is captured, but it should be 12K+ lines of output.

Error Traceback

(Paste the complete traceback of the exception between quotes below)

n/a - incomplete output, but no explicit error. 

Relevant Python code

(Please try to essentialize your Python code to the minimum code needed to reproduce the issue)
(Paste the code between the quotes below)

datapath_sess_expected.txt
Main file:

controller = AMC_Connection_Handler().connect()

# Start getting the data
op = controller.send_command_expect("show datapath session table")
print(op)

AMC.py file:

class AMC_Connection_Handler(): 
    def __init__(self):
        self.controller = AMC_Connection_Handler.get_primary_controller()
    
    def connect(self):
        return ConnectHandler(**self.controller)

Problem example

Output Log

[last 3 lines - output ommitted above]
172.31.248.1      172.31.248.145  17   8494  8209   0/0     0    0   1   tunnel 2453 e    0          0          FYI             14
172.31.248.182    172.31.248.1    17   8209  8222   0/0     0    0   1   tunnel 1390 10   0
PS path> 

Debug Log (relevant section only) [lines: 684 - 783]

...
Source IP or MAC  Destination IP  Prot SPort DPort Cntr     Prio ToS Age Destination TAge Packets    Bytes      Flags           CPU ID  
----------------- --------------- ---- ----- ----- -------- ---- --- --- ----------- ---- ---------- ---------- --------------- ------- 
135.12.209.4      172.27.5.122    17   6881  52683  0/0     0    10  0   tunnel 2293 9    4          198        F               11       
172.31.248.1      172.31.251.225  17   17110 8209   0/0     0    0   1   tunnel 2733 10   0          0          FYI             14       
172.27.9.50       103.228.155.199 6    56397 443    0/0     0    0   0   tunnel 2536 1025 1665       149339     C               15       
172.31.250.175    172.31.248.1    17   8209  17110  0/0     0    0   1   tunnel 2344 c    0          0          FYCI            11       

172.27.7.214      104.18.29.147   6    60180 443    0/0     0    0   1   tunnel 1777 44   24         3988       Ci              13       
172.27.8.8        8.8.4.4         17   25001 53     0/0     0    0   0   tunnel 1650 a    1          72         FCI             14       
10.147.22.100     165.225.120.46  6    60169 443    0/0     0    0   1   tunnel 1891 3b6  144        27612      Ch              13       
172.27.7.132      8.8.4.4         17   47219 53     0/0     0    0   0   tunnel 1764 a    2          148        FCI             11       

DEBUG:netmiko:read_channel: 
172.31.251.76     172.31.248.1    17   8209  8222   0/0     0    0   0   tunnel 2496 b    0          0          FYCI            15       
172.31.248.1      172.31.249.206  17   8419  8209   0/0     0    0   1   tunnel 1419 c    0          0          FYI             14       
172.31.248.1      172.31.249.221  17   4500  4500   0/0     0    0   8   pc2         8d   0          0          FY              15       
172.31.248.1      172.31.251.52   17   17110 8209   0/0     0    0   0   tunnel 2543 8    0          0          FYI             13       

10.147.22.90      165.225.120.47  6    63645 443    0/0     0    0   0   tunnel 1600 a    17         6785       Ch              11       
172.31.249.10     10.148.124.254  17   8211  8224   0/0     0    0   0   local       4    0          0          FYI             11       
172.31.248.1      172.31.248.233  47   0     0      0/0     2    16  0   pc2         9aad 28880180   846478135  F               12       
52.113.194.132    172.27.7.214    6    443   60236  0/0     0    46  1   tunnel 1777 17   28         30695      i               12       

172.27.5.64       142.251.42.67   6    40392 443    0/0     0    0   1   tunnel 1814 17   11         1171       FCi             14       
172.64.154.86     172.27.8.229    6    443   62854  0/0     0    4   2   tunnel 2536 53   36         27650      i               12       
172.27.8.60       23.211.137.10   6    52777 80     0/0     0    0   0   tunnel 237  1    5          414        FNC             13       
172.31.248.1      172.31.249.125  17   8494  8209   0/0     0    0   1   tunnel 1069 11   0          0          FYI             15       

52.108.50.36      10.147.22.119   6    443   52330  0/0     0    24  1   tunnel 878  1d   15         10259      Fi              12       
172.31.248.1      172.31.251.198  17   4500  4500   0/0     0    0   2   pc2         2b   0          0          FY              12       
10.147.22.100     52.137.110.235  6    62753 443    0/0     0    0   5   tunnel 1891 55   12         2076       Cih             13       
172.31.248.1      172.31.251.213  17   8419  8209   0/0     0    0   1   tunnel 2805 14   0          0          FYI             15       

172.27.7.214      142.250.182.200 6    60199 443    0/0     0    0   0   tunnel 1777 30   9          2193       Ci              13       
10.148.124.254    172.31.251.247  17   8494  8211   1/0     0    0   0   local       1    1          129        FCI             11       
172.27.5.122      31.136.159.217  6    54535 27079  0/0     0    0   1   pc2         14   0          0          F               15       
172.27.5.122      195.68.137.34   6    56747 6881   0/0     0    0   0   tunnel 2293 b    5          363        FC              11       

DEBUG:netmiko:read_channel: 
DEBUG:netmiko:read_channel: 
DEBUG:netmiko:read_channel: 
172.27.8.229      162.247.243.39  6    62674 443    0/0     0    0   2   tunnel 2536 b4   37         2767       Ci              15       
172.31.248.1      172.31.251.102  17   8494  8209   0/0     0    0   1   tunnel 1876 11   0          0          FYI             15       
172.31.251.151    172.31.248.1    17   8209  8494   1/0     0    0   0   tunnel 146  1    0          0          FYCI            15       
172.31.248.1      172.31.251.19   17   4500  4500   0/0     0    0   1   pc2         16   0          0          FY              13       

172.31.250.117    10.148.124.254  17   8211  8419   0/0     0    0   0   local       1    0          0          FYI             11       
172.31.250.169    172.31.248.1    17   8209  8494   0/0     0    0   1   tunnel 3399 14   0          0          FYCI            11       
172.31.248.255    172.31.248.1    17   8209  8515   0/0     0    0   1   tunnel 2385 16   0          0          FYCI            15       
172.27.9.19       8.8.4.4         17   40342 53     0/0     0    0   0   tunnel 1249 1    2          148        FCI             13       

172.27.8.229      35.74.166.74    6    62913 443    0/0     0    0   2   tunnel 2536 25   7          1110       Ci              15       
10.148.124.254    172.31.251.85   17   8515  8211   1/0     0    0   0   local       4    1          242        FCI             11       
172.31.248.1      172.31.251.30   47   0     0      0/0     4    32  0   pc2         9acc 58454289   3503716006 F               13       
172.31.250.127    172.31.248.1    17   8209  17110  0/0     0    0   0   tunnel 770  8    0          0          FYCI            15       

172.31.248.1      172.31.251.196  17   8515  8209   0/0     0    0   0   tunnel 479  3    0          0          FYI             13       
157.240.192.4     172.27.13.111   6    443   37822  0/0     0    46  15  tunnel 1938 208  54         7030                       13       
f8:04:2e:00:d4:40                 806               0/0     0    0   0   tunnel 1021 273e 132454     271265792  F               15       
172.31.248.1      172.31.251.43   17   8209  8209   0/0     0    0   0   tunnel 777  6    0          0          FYI             11       

172.27.5.122      101.0.62.93     6    56624 40819  0/0     0    0   1   tunnel 2293 1f   5          260        YC              11       

DEBUG:netmiko:read_channel: 
DEBUG:netmiko:read_channel: 172.31.248.1      172.31.251.203  47   0     0      0/0     4    24  0   pc2         9acd 87976229   133332246  F               13       
10.148.124.254    172.31.250.206  17   8494  8211   0/0     0    0   0   local       8    1          128        FCI             11       
116.119.53.76     172.27.13.66    6    443   59720  0/0     0    46  0   tunnel 2423 5    1378       1932241    i               11       

172.31.248.1      172.31.250.255  17   4500  4500   0/0     0    0   1   pc2         1a   0          0          FY              11       
172.31.251.156    172.31.248.1    17   8209  8222   0/0     0    0   1   tunnel 1434 11   0          0          FYCI            13       
172.31.249.143    10.148.124.254  17   8211  8419   0/0     0    0   0   local       8    0          0          FYI             11       
172.31.248.1      172.31.250.95   17   8494  8209   0/0     0    0   1   tunnel 2662 1b   0          0          FYI             14       

172.31.251.47     172.31.248.1    17   8209  8494   0/0     0    0   1   tunnel 2260 19   0          0          FYCI            12       
172.27.7.214      142.250.183.14  17   51903 443    0/0     0    0   1   tunnel 1777 10   18         10406      FC              13       
86.227.41.227     172.27.5.122    6    34417 56688  0/0     0    0   1   tunnel 2293 19   5          428        F               15       
172.27.5.17       20.244.240.164  6    36690 443    0/0     0    0   2   tunnel 502  160  172        151506     C               13       

10.148.124.254    172.27.9.14     6    8080  43374  0/0     0    0   0   tunnel 2986 0    5          674        FS              11       
35.186.224.35     172.27.5.17     6    443   58422  0/0     0    46  1   tunnel 502  73   25         11230                      15       
172.31.249.140    172.31.248.1    17   8209  8421   1/0     0    0   0   tunnel 683  4    0          0          FYCI            15       
172.27.9.5        23.208.65.107   6    55397 443    0/0     0    0   3   tunnel 3156 3c   8          1008       Ci              11       

172.27.5.122      103.157.176.90  17   52683 12686  0/0     0    0   1   tunnel 2293 12   2          164        FC              11       
172.31.249.85     172.31.248.1    17   8209  17110  0/0     0    0   1   tunnel 2059 19   0          0          FYCI            11       
172.27.11.64      142.250.192.3   6    60699 443    0/0     0    0   1   tunnel 1858 67   16         2560       Ci              15       
172.31.248.1      172.31.249.229  17   8209  8209   0/0     0    0   2   tunnel 2086 2a   0          0          FYI             13       

10.147.24.224     165.225.120.44  6    65236 443    0/0     0    24  0   tunnel 2248 c66  220        16136      TC              15       
172.27.8.8        163.181.81.236  6    48952 443    0/0     0    0   1   tunnel 1650 d    11         1322       C               14       
172.31.251.241    172.31.248.1    17   8209  8444   1/0     0    0   0   tunnel 13   7    0          0          FYCI            15       
b8:27:eb:a4:a2:64                 806               0/0     0    0   1   tunnel 3265 f    1          2048       F               13       

172.31.251.41     172.31.248.1    17   8209  17110  1/0     0    0   0   tunnel 671  9    0          0          FYCI            13       
10.148.124.254    172.31.249.78   17   8515  8211   1/0     0    0   0   local       4    1          242        FCI             11       
172.31.248.1      172.31.249.5    47   0     0      0/0     4    32  0   pc2         9acd 44273561   140053077  F               15       
203.202.241.38    172.27.5.122    6    52376 56600  0/0     0    0   1   tunnel 2293 20   5          200        F               15       

10.147.24.9       40.100.72.2     6    55408 443    0/0     0    0   6   tunnel 1815 65   25         8836       Ci              14       
172.31.248.1      172.31.250.216  17   17110 8209   0/0     0    0   0   tunnel 1430 4    0          0          FYI             12       
172.31.248.1      172.31.248.145  17   8494  8209   0/0     0    0   1   tunnel 2453 e    0          0          FYI             14       
172.31.248.182    172.31.248.1    17   8209  8222   0/0     0    0   1   tunnel 1390 10   0          0          FYCI            13       

<--------------------------OUTPUT STOPS AROUND HERE (Line # 783), but the logs go on! 

172.31.251.109    172.31.248.1    17   8209  8515   1/0     0    0   0   tunnel 1233 1    0          0          FYCI            15       
172.31.250.121    172.31.248.1    17   8209  8494   0/0     0    0   1   tunnel 697  13   0          0          FYCI            14       
10.147.22.90      165.225.120.47  6    60464 443    0/0     0    0   0   tunnel 1600 1129 224        21638      Ch              11       

Complete Log Files and Output (expected vs received on Netmiko).

datapath_sess_truncated.txt
datapath_sess_expected.txt
test.log

@ktbyers / Kirk, Your tool has been invaluable to me but I just can't figure out why it almost randomly stops accepting data even though it's clearly being sent by the device, and is in the log!

Please check this out as soon as you can, and let me know?!

It doesn't look like it is truncated from your test.log file:

DEBUG:netmiko:read_channel: 
172.31.248.83     172.31.248.1    17   8209  8494   1/0     0    0   0   tunnel 3301 4    0          0          FYCI            13       
10.148.124.254    10.236.0.160    6    22    45900  0/0     0    0   1   pc2         15   4          233        F               11       
172.31.248.1      172.31.251.120  17   8222  8209   0/0     0    0   1   tunnel 2834 19   0          0          FYI             12       
172.31.248.1      172.31.250.213  17   17110 8209   0/0     0    0   0   tunnel 2384 4    0          0          FYI             11       

172.27.0.3        224.0.0.102     17   1985  1985   0/0     0    48  0   pc2         9222 32471      2597680    FC              14       
10.147.22.119     224.0.0.252     17   62513 5355   0/0     0    24  0   tunnel 878  f    2          96         FTC             13       
149.154.167.50    172.27.8.8      6    443   47060  0/0     0    0   0   tunnel 1650 11   3          169        F               15       
10.250.1.43       10.148.124.254  6    61377 21     1/15784 0    0   1   pc2         17   2          80         FCI             11       

10.147.22.119     23.215.205.230  6    52203 443    0/0     0    0   0   tunnel 878  4b   13         1550       FCi             13       
172.31.249.5      172.31.248.1    17   8209  8494   0/0     0    0   0   tunnel 2432 e    0          0          FYCI            14       
172.31.248.1      172.31.248.233  17   4500  4500   0/0     0    0   0   pc2         6    0          0          FY              12       
172.31.248.1      172.31.248.250  17   8419  8209   0/0     0    0   0   tunnel 1221 0    0          0          FYI             12       

172.27.9.153      172.217.194.188 6    52388 5228   0/0     0    0   2   tunnel 1816 1a41 162        8817       C               14       
172.31.249.182    172.31.248.1    17   8209  8222   0/0     0    0   1   tunnel 1974 1a   0          0          FYCI            15       
172.31.248.1      172.31.249.221  47   0     0      0/0     2    24  0   pc2         9aa4 31585372   1089898200 F               12       
10.148.124.254    172.31.249.150  17   8515  8211   1/0     0    0   0   local       9    1          242        FCI             11       

172.31.248.178    10.148.124.254  17   8211  8224   0/0     0    0   0   local       0    0          0          FYI             11       
172.27.5.122      46.232.210.90   17   52683 6881   0/0     0    0   0   tunnel 2293 d    2          96         FC              11       
(INMUM-02274-MB01SVR-WC02) *[mynode] #

The prompt comes back at the end there.

Also your session_log files don't look like Netmiko's standard session_log files?

Also what do you mean you don't see an error--does this mean you terminate the program yourself or are catching the error (or something else)?

@ktbyers - I agree, the test.log file has the complete output, but the part that send_command() or send_command_expect() shows as the output is truncated - which is where the confusion lies. For example, this is the output when I run the script now. As you'll see, the number of entries is much smaller, and the last line doesn't even complete before send_command_expect() presents the output.

cd 'c:\Users\scripts\Aruba_SessionTracer'; & 'c:\Users\scripts\Aruba_SessionTracer\env\Scripts\python.exe' 'c:\Users\.vscode\extensions\ms-python.debugpy-2024.6.0-win32-x64\bundled\libs\debugpy\adapter/../..\debugpy\launcher' '53671' '--' 'C:\Users\scripts\Aruba_SessionTracer\tracker.py' 
Connecting to: <device 1>
Connecting to: <device 2>
<device 2> is the primary controller - running commands on this. 

Datapath Session Table Entries
------------------------------

Flags: F - fast age, S - src NAT, N - dest NAT
       D - deny, R - redirect, Y - no syn
       H - high prio, P - set prio, T - set ToS
       C - client, M - mirror, V - VOIP
       Q - Real-Time Quality analysis
       u - Upstream Real-Time Quality analysis
       I - Deep inspect, U - Locally destined
       E - Media Deep Inspect, G - media signal
       r - Route Nexthop, h - High Value
       A - Application Firewall Inspect
       i - Session classified on first packet
       B - Permanent, O - Openflow
       L - Log, o - Openflow config revision mismatched

Source IP or MAC  Destination IP  Prot SPort DPort Cntr     Prio ToS Age Destination TAge Packets    Bytes      Flags           CPU ID
----------------- --------------- ---- ----- ----- -------- ---- --- --- ----------- ---- ---------- ---------- --------------- -------
172.31.251.145    172.31.248.1    17   8209  17110  0/0     0    0   1   tunnel 1307 10   0          0          FYCI            12
172.27.9.50       103.228.155.199 6    56397 443    0/0     0    0   1   tunnel 2536 3975 5809       458983     C               15
10.148.124.254    172.31.250.221  17   8222  8211   0/0     0    0   1   local       4    3          402        FCI             11
172.31.249.237    172.31.248.1    17   8209  17110  0/0     0    0   1   tunnel 491  f    0          0          FYCI            14

212.93.114.10     172.27.5.122    6    54039 59904  0/0     0    0   2   tunnel 2293 1a   1          44                         11
172.31.248.1      172.31.249.168  17   8494  8209   0/0     0    0   1   tunnel 3247 9    0          0          FYI             15
172.31.251.76     172.31.248.1    17   8209  8222   0/0     0    0   0   tunnel 327  2    0          0          FYCI            15
162.247.243.24    172.27.13.41    6    443   46634  0/0     0    0   5   tunnel 707  bf   13         2070                       11

172.31.251.35     10.148.124.254  17   8211  8419   0/0     0    0   2   local       19   0          0          FYI             11
172.31.248.1      172.31.248.233  47   0     0      0/0     2    16  0   pc2         c3fd 29084259   1088271364 F               11
172.31.250.193    172.31.248.1    17   8209  8494   0/0     0    0   1   tunnel 2977 9    0          0          FYCI            13
3.64.42.113       10.147.19.105   6    443   53560  0/0     0    16  4   tunnel 3058 830  96         24558      h               15

10.147.22.120     20.212.88.117   6    53997 443    0/0     0    0   12  tunnel 3166 58d  25         11721      Cih             14
52.108.50.37      10.147.22.99    6    443   51617  0/0     0    24  1   tunnel 1651 1482 415        44264      i               15
172.31.248.187    172.31.248.1    17   8209  17110  0/0     0    0   1   tunnel 143  6    0          0          FYCI            11
172.31.248.1      172.31.251.198  17   4500  4500   0/0     0    0   2   pc2         86   2          253        F               11

172.31.248.98     172.31.248.1    17   8209  8421   0/0     0    0   1   tunnel 2945 10   0          0          FYCI            13
142.250.67.150    172.27.13.248   17   443   60631  0/0     0    46  1   tunnel 3356 c    11         7099       F               15
10.148.124.254    172.31.251.247  17   8494  8211   0/0     0    0   1   local       12   1          129        FCI             11
106.70.168.210    172.27.5.122    6    6881  59787  0/0     0    0   0   pc2         1    1          40         FC              14

10.148.124.254    172.31.251.68   17   8419  8211   0/0     0    0   1   local       10   2          1815       FCI             11
172.31.248.1      172.31.251.102  17   8494  8209   0/0     0    0   0   tunnel 3005 1    0          0          FYI             15
52.59.167.127     10.147.16.213   6    443   53400  0/0     0    16  3   tunnel 848  44e  55         13918      h               12
172.31.250.169    172.31.248.1    17   8209  8494   1/0     0    0   0   tunnel 2360 3    0          0          FYCI            11

110.42.235.42     172.27.5.122    6    49155 59472  0/0     0    0   2   tunnel 2293 2772 171        6984                       12
198.244.190.234   172.27.5.122    6    6881  59827  0/0     0    11  3   tunnel 2293 2b   2          92                         13
10.148.124.254    172.31.249.129  17   8224  8211   0/0     0    0   1   local       d    2          1975       FCI             11
172.31.248.1      172.31.251.179  17   8494  8209   0/0     0    0   1   tunnel 10   4    0          0          FYI             12

172.31.248.1      172.31.251.30   47   0     0      0/0     4    32  0   pc2         c41c 58505658   3512790498 F               14
10.147.24.224     52.108.240.79   6    53273 443    0/0     0    24  0   tunnel 2248 176c 457        26854      TCi             15
205.185.114.24    172.27.5.122    6    6898  59934  0/0     0    10  1   tunnel 2293 14   4          172        F               12
172.31.248.1      172.31.249.110  17   8222  8209   0/0     0    0   0   tunnel 53   2    0          0          FYI             15

10.147.24.9       165.225.120.45  6    62913 443    0/0     0    24  1   tunnel 1815 da9  244        17651      TC              14
10.148.124.254    172.31.250.168  17   8419  8211   0/0     0    0   1   local       d    2          1107       FCI             11
f8:04:2e:00:d4:40                 806               0/0     0    0   0   tunnel 1021 508e 137599     281802752  F               15
172.31.248.1      172.31.251.43   17   8209  8209   0/0     0    0   8   tunnel 2589 83   0          0          FYI             11

142.250.70.118    172.27.5.37     6    443   47860  0/0     0    0   3   tunnel 1477 32   8          6282                       15
10.148.124.254    172.31.251.182  17   8421  8211   0/0     0    0   1   local       7    1          104        FCI             11
172.31.248.1      172.31.251.203  47   0     0      0/0     4    24  0   pc2         c41d 87986536   134239262  F               14
172.31.248.1      172.31.250.255  17   4500  4500   0/0     0    0   2   pc2         1f   1          29         F               11

172.31.248.1      172.31.250.95   17   8494  8209   0/0     0    0   1   tunnel 2662 9    0          0          FYI             14
172.31.251.47     172.31.248.1    17   8209  8494   0/0     0    0   1   tunnel 2260 8    0          0          FYCI            12
52.108.240.79     10.147.22.119   6    443   59197  0/0     0    24  1   tunnel 878  1f2e 634        62776      i               14
10.148.124.254    172.31.251.49   17   8222  8211   1/0     0    0   0   local       0    1          177        FCI             11

172.31.248.1      172.31.249.10   17   8515  8209   0/0     0    0   0   tunnel 1419 3    0          0          FYI             14
8.8.8.8           172.27.0.101    6    443   52754  0/0     0    46  1   tunnel 2957 26c  1222       224389     i               12
54.254.133.151    172.27.9.138    6    2957  35460  0/0     0    0   0   tunnel 1717 2201 3048       159506                     13
172.31.248.1      172.31.251.160  17   8222  8209   0/0     0    0   1   tunnel 1555 6    0          0          FYI             13

20.38.108.228     172.27.9.138    6    443   44194  0/0     0    0   1   tunnel 1717 9    306        24924      Fi              12
172.31.249.141    172.31.248.1    17   8209  8444   0/0     0    0   1   tunnel 2660 e    0          0          FYCI            11
172.31.251.41     172.31.248.1    17   8209  17110  0/0     0    0   1   tunnel 2508 b    0          0          FYCI            13
172.31.248.1      172.31.249.5    47   0     0      0/0     4    32  0   pc2         c41d 45590077   1832812700 F               14

10.148.124.254    172.31.251.228  17   8222  8211   0/0     0    0   1   local       4    1          177        FCI             11
172.27.5.122      146.70.204.168  6    60059 64386  0/0     0    0   0   tunnel 2293 2    2          104        YC              11
172.27.13.228     142.250.183.202 6    45436 443    0/0     0    0   0   tunnel 1477 0    9          1151       NCIi            13
45.57.41.1        172.27.5.17     6    443   56518  0/0     0    46  1   tunnel 502  29d  175        38880                      11

172.27.13.41      163.70.143.15   6    41482 443    0/0     0    0   12  tunnel 707  bf   15         4478       C               12
172.31.251.71     172.31.248.1    17   8209  8494   0/0     0    0   1   tunnel 2348 9    0          0          FYCI            12
172.31.250.121    172.31.248.1    17   8209  8494   1/0     0    0   0   tunnel 1614 2    0          0          FYCI            14
172.31.251.244    172.31.248.1    17   8209  8222   0/0     0    0   1   tunnel 1131 6    0          0          FYCI            11

172.31.248.1      172.31.249.48   17   8209  8209   0/0     0    0   2   tunnel 2757 14   0          0          FYI             13
172.31.249.206    172.31.248.1    17   8209  8209   0/0     0    0   0   tunnel 1360 8    7          1361       FCI             14
172.31.248.1      172.31.251.151  17   8209  8209   0/0     0    0   20  tunnel 2161 14a  0          0          FYI             15
172.27.0.101      116.119.74.209  6    57215 443    0/0     0    0   1   tunnel 2957 1da  149        23275      Ci              14

172.31.248.240    172.31.248.1    17   8209  8209   0/0     0    0   1   tunnel 2384 2a   22         13122      FCI             14
52.123.137.139    172.27.8.68     6    443   54204  0/0     0    0   1   tunnel 2004 6e92 724        72999      i               12
172.31.248.1      172.31.251.120  17   8515  8209   0/0     0    0   1   tunnel 1463 c    0          0          FYI             12
172.31.251.178    172.31.248.1    17   8209  8209   0/0     0    0   1   tunnel 2391 2d   19         6776       FCI             12

10.147.24.224     165.225.120.44  6    53319 443    0/0     0    24  1   tunnel 2248 174c
PS C:\Users\scripts\Aruba_SessionTracer> 

And finally the "session logs" are just the output from the device (expected) and the terminal when the program is run (truncated).

P.S. Really appreciate you taking the time to help solve this!

@ktbyers - also a weird thing I've noticed it - the data from the AMC kind of "stutters", and basically when the period of watch for the next chunk is a bit long is when the output just stops. Maybe that's what's causing the incomplete output?

I'm still not sure why send_command_expect() won't wait for the prompt to come back though - especially since I tried setting the read_timeout to 300 seconds.

Please note that this command takes a lil time to run, just like show tech-support from Cisco's OSs, and takes about 40s to complete.

I've since figured out:

  • This most likely isn't a timing issue: The send_command() fn waits till the complete output is acquired and the EOF (shown below) is received before printing the output to the screen, and
  • For some reason, even though the entire data is available in the logs, it just chooses the first 9K bytes or so during printing.

imcomplete_out

For some reason, even though the entire data is available in the logs, it just chooses the first 9K bytes 
or so during printing.

Meaning your print(op) isn't printing the entire output?

Yep - that seems to be the case @ktbyers . I even tried with Paramiko, but basically a similar / same output:

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname, username=config.get("connection", "uid"), password=config.get("connection", "upass"))
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(LOOKUP['datapath_table'])
ssh_stderr="".join(ssh_stderr.readlines())
ssh.close()
print(ssh_stderr)

Output:



Datapath Session Table Entries
------------------------------

Flags: F - fast age, S - src NAT, N - dest NAT
       D - deny, R - redirect, Y - no syn
       H - high prio, P - set prio, T - set ToS
       C - client, M - mirror, V - VOIP
       Q - Real-Time Quality analysis
       u - Upstream Real-Time Quality analysis
       I - Deep inspect, U - Locally destined
       E - Media Deep Inspect, G - media signal
       r - Route Nexthop, h - High Value
       A - Application Firewall Inspect
       i - Session classified on first packet
       B - Permanent, O - Openflow
       L - Log, o - Openflow config revision mismatched

Source IP or MAC  Destination IP  Prot SPort DPort Cntr     Prio ToS Age Destination TAge Packets    Bytes      Flags           CPU ID
----------------- --------------- ---- ----- ----- -------- ---- --- --- ----------- ---- ---------- ---------- --------------- -------
172.31.251.145    172.31.248.1    17   8209  17110  0/0     0    0   1   tunnel 950  f    0          0          FYCI            12
172.31.248.1      172.31.251.225  17   17110 8209   0/0     0    0   1   tunnel 1455 5    0          0          FYI             14
172.27.9.50       103.228.155.199 6    56397 443    0/0     0    0   1   tunnel 2536 5bd5 9258       715919     C               15
165.225.120.44    10.147.24.224   6    443   61252  0/0     0    16  1   tunnel 2248 15f  34         11049      h               12

172.27.1.46       8.8.4.4         17   47536 53     0/0     0    0   1   tunnel 2862 7    2          148        FCI             13
10.148.124.254    172.27.12.141   6    8080  56753  0/0     0    0   1   tunnel 2536 9    5          661        FSi             11
172.31.248.1      172.31.249.168  17   8494  8209   0/0     0    0   1   tunnel 2405 12   0          0          FYI             15
142.250.192.74    172.27.14.41    6    443   44420  0/0     0    32  11  tunnel 564  b5   26         21873      i               14

[OUTPUT OMITTED]

172.31.249.85     172.31.248.1    17   8209  17110  0/0     0    0   1   tunnel 2626 8    0          0          FYCI            11
172.31.248.178    172.31.248.1    17   8209  8421   0/0     0    0   1   tunnel 2679 8    0          0          FYCI            15
172.31.250.206    172.31.248.1    17   8209  8421   0/0     0    0   1   tunnel 2760 c    0          0          FYCI            15
172.31.248.1      172.31.249.229  17   8209  8209   0/0     0    0   2   tunnel 176  1e   0          0          FYI             13

172.31.251.41     172.31.248.1

Netmiko should read all of the data (and print should print out all of the data). Netmiko should generate an exception if it can't read all of the data in read_timeout seconds.

What does your send_command_expect() method look like (since you are wrapping this all in your class)?

Hey @ktbyers - you Sir, are a genius!

Netmiko is indeed getting all the data, but the print statement's output was being truncated.
I tried running the script directly in the terminal - and lo and behold --- I get the entire output!

VSCode just decided that it didn't want to show the entire output, and didn't think it'd be necessary to mention anything about shortening the output!

Thanks a lot mate, @ktbyers -- P.S. Long time user of netmiko, can't even begin to tell you how big a fan I am of your work!