Different buffers during basic image transfer example

Question

Different buffers during basic image transfer example

sergey-gra opened this issue 3 years ago · 9 comments

Hello. I'm trying to reproduce vnx-basic-image-transfer.ipynb Notebook example. I have three Alveo U50 boards on the same host and two of those are connected via QSFP28.
The first issue is that I got this warning /usr/local/lib/python3.6/site-packages/pynq/pl_server/xrt_device.py:89: UserWarning: xbutil failed to run - unable to determine XRT version 'xbutil failed to run - unable to determine XRT version') when running that part of code from JupiterLab

from vnx_utils import *
import pynq
%run dask_pynq.py

But after the second time running the same cell the warning isn't shown.
My XRT version: 2.12.427
Vitis version: 2021.1
Pynq version: 2.7.0

The second issue that I got an error from this command ol_w1.networklayer_0.get_arp_table()
The error is shown below

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-17-fc14260aa3d0> in <module>
      8 ol_w1.networklayer_0.arp_discovery()
      9 #4
---> 10 ol_w1.networklayer_0.get_arp_table()

AttributeError: 'NetworkLayer' object has no attribute 'get_arp_table'

The third one is that mm2s buffer is different than received s2mm buffer
Output:

[ 46 142  17 ...   2   7   3]
[ 46 142  17 ...   0   0   0]

Output from register_map after image transfer
ol_w0

RegisterMap {
  mac_address = Register(value=35888057852358624),
  ip_address = Register(value=3232235525),
  gateway = Register(value=3232235521),
  ip_mask = Register(value=4294967040),
  eth_in_cycles = Register(value=92629932406),
  eth_in_bytes = Register(value=7866168),
  eth_in_packets = Register(value=6161),
  pkth_in_cycles = Register(value=92629932407),
  pkth_in_bytes = Register(value=7790652),
  pkth_in_packets = Register(value=6161),
  arp_in_cycles = Register(value=91131066246),
  arp_in_bytes = Register(value=46020),
  arp_in_packets = Register(value=767),
  arp_out_cycles = Register(value=91408372667),
  arp_out_bytes = Register(value=45660),
  arp_out_packets = Register(value=761),
  icmp_in_cycles = Register(value=0),
  icmp_in_bytes = Register(value=0),
  icmp_in_packets = Register(value=0),
  icmp_out_cycles = Register(value=0),
  icmp_out_bytes = Register(value=0),
  icmp_out_packets = Register(value=0),
  ethhi_out_cycles = Register(value=0),
  ethhi_out_bytes = Register(value=0),
  ethhi_out_packets = Register(value=0),
  eth_out_cycles = Register(value=91408372667),
  eth_out_bytes = Register(value=45660),
  eth_out_packets = Register(value=761),
  udp_in_cycles = Register(value=83370835570),
  udp_in_bytes = Register(value=7744632),
  udp_in_packets = Register(value=5394),
  app_out_cycles = Register(value=8485351701),
  app_out_bytes = Register(value=1835008),
  app_out_packets = Register(value=1316),
  udp_out_cycles = Register(value=0),
  udp_out_bytes = Register(value=0),
  udp_out_packets = Register(value=0),
  app_in_cycles = Register(value=83370835565),
  app_in_bytes = Register(value=7593600),
  app_in_packets = Register(value=5394),
  debug_reset_counters = Register(value=0),
  arp_discovery = Register(value=0),
  arp_valid_offset = Register(value=0),
  arp_ip_addr_offset = Register(value=0),
  arp_mac_addr_offset = Register(value=0),
  udp_theirIP_offset = Register(value=0),
  udp_theirPort_offset = Register(value=0),
  udp_myPort_offset = Register(value=0),
  udp_valid_offset = Register(value=0),
  udp_number_sockets = Register(value=16)
}

ol_w1

RegisterMap {
  mac_address = Register(value=43839036682),
  ip_address = Register(value=3232235530),
  gateway = Register(value=3232235521),
  ip_mask = Register(value=4294967040),
  eth_in_cycles = Register(value=91408371989),
  eth_in_bytes = Register(value=45660),
  eth_in_packets = Register(value=761),
  pkth_in_cycles = Register(value=91408371989),
  pkth_in_bytes = Register(value=45660),
  pkth_in_packets = Register(value=761),
  arp_in_cycles = Register(value=91408371989),
  arp_in_bytes = Register(value=45660),
  arp_in_packets = Register(value=761),
  arp_out_cycles = Register(value=91131065572),
  arp_out_bytes = Register(value=46020),
  arp_out_packets = Register(value=767),
  icmp_in_cycles = Register(value=0),
  icmp_in_bytes = Register(value=0),
  icmp_in_packets = Register(value=0),
  icmp_out_cycles = Register(value=0),
  icmp_out_bytes = Register(value=0),
  icmp_out_packets = Register(value=0),
  ethhi_out_cycles = Register(value=83370834158),
  ethhi_out_bytes = Register(value=7820148),
  ethhi_out_packets = Register(value=5394),
  eth_out_cycles = Register(value=92629930872),
  eth_out_bytes = Register(value=7866168),
  eth_out_packets = Register(value=6161),
  udp_in_cycles = Register(value=0),
  udp_in_bytes = Register(value=0),
  udp_in_packets = Register(value=0),
  app_out_cycles = Register(value=83370833748),
  app_out_bytes = Register(value=7593600),
  app_out_packets = Register(value=5394),
  udp_out_cycles = Register(value=83370834136),
  udp_out_bytes = Register(value=7744632),
  udp_out_packets = Register(value=5394),
  app_in_cycles = Register(value=0),
  app_in_bytes = Register(value=0),
  app_in_packets = Register(value=0),
  debug_reset_counters = Register(value=0),
  arp_discovery = Register(value=0),
  arp_valid_offset = Register(value=0),
  arp_ip_addr_offset = Register(value=0),
  arp_mac_addr_offset = Register(value=0),
  udp_theirIP_offset = Register(value=0),
  udp_theirPort_offset = Register(value=0),
  udp_myPort_offset = Register(value=0),
  udp_valid_offset = Register(value=0),
  udp_number_sockets = Register(value=16)
}

Thanks is advance.

Answer 1 · 2022-05-31T08:03:20.000Z

Ho @sergey-gra,

UserWarning: xbutil failed to run - unable to determine XRT version 'xbutil failed to run - unable to determine XRT version'

Did you apply the patch discussed in the notes here for XRT 2.12 or newer? https://github.com/Xilinx/xup_vitis_network_example#tools

The second issue that I got an error from this command ol_w1.networklayer_0.get_arp_table()

I'll be fixing this soon.

The third one is that mm2s buffer is different than received s2mm buffer

What notebook are you using?

Answer 2 · 2022-05-31T09:06:49.000Z

Please, pull the latest changes from the repo to fix AttributeError: 'NetworkLayer' object has no attribute 'get_arp_table'

Answer 3 · 2022-06-01T05:40:18.000Z

Hi @mariodruiz. Thank you for your replies.

Did you apply the patch discussed in the notes here for XRT 2.12 or newer? https://github.com/Xilinx/xup_vitis_network_example#tools

Please, pull the latest changes from the repo to fix AttributeError: 'NetworkLayer' object has no attribute 'get_arp_table'

After applying that changes errors are fixed. Thank you.

What notebook are you using?

I'm using vnx-basic-image-transfer.ipynb. s2mm and mm2s buffers outputs are different, but images are the same after plotting.
Output of command msg = "Yes!" if np.array_equal(mm2s_buf, s2mm_buf) else "No!" is No.

Answer 4 · 2022-06-01T05:49:41.000Z

Query embedded debug probes output

rx app: total payload packets 0	bytes 0	 throughput 0.000 Gbps
tx app: total payload packets 0	bytes 0	 throughput 0.000 Gbps

Answer 5 · 2022-06-01T08:53:52.000Z

I'm using vnx-basic-image-transfer.ipynb. s2mm and mm2s buffers outputs are different, but images are the same after plotting.

That's very interesting. Does the cell with s2mm_wh.wait() finish?

Instead of using showDebugProbes(), can you try?

ol_w0.networklayer_0.get_debug_stats

The same for the other Alveo card.

A few more questions

What shell are you using?
What cable are you using to connect the Alveos? I suppose you are connecting them point to point.
Can you check if the cmac in the Rx Alveo reports any value non 0 with the string bad on it, e.g. ol_w0.cmac_0.get_stats()
Did you check using the other Alveo card? Or even changing which alveo sends and which receives?

My feeling is that the data is being transmitted, this is based on the stats you reported above. But, for some reason there are some bit errors. These bit errors won't be noticeable in the plotted image, but the numpy array comparison will detect them.

Answer 6 · 2022-06-01T10:56:45.000Z

Did you check using the other Alveo card? Or even changing which alveo sends and which receives?

I changed sequence of Alveo devices in the code(replaced sender and receiver) and it works fine! Thanks a lot @mariodruiz .

Answer 7 · 2022-06-01T11:03:47.000Z

That's great!

It would be good to understand why in the other scenario it is not working. The stats from the cmac are key for this.

Answer 8 · 2022-06-01T11:21:38.000Z

Below I provide outputs for case when buffers were different.

Output of ol_w0.networklayer_0.get_debug_stats for first device
{'tx_path': {'arp': {'packets': 1532, 'bytes': 91920, 'cycles': 4976427235855}, 'icmp': {'packets': 0, 'bytes': 0, 'cycles': 0}, 'ethernet_header_inserter': {'packets': 0, 'bytes': 0, 'cycles': 0}, 'ethernet': {'packets': 1532, 'bytes': 91920, 'cycles': 4976427235855}, 'app': {'packets': 0, 'bytes': 0, 'cycles': 0}, 'udp': {'packets': 0, 'bytes': 0, 'cycles': 0}}, 'rx_path': {'ethernet': {'packets': 12575, 'bytes': 15747516, 'cycles': 4976491476719}, 'packet_handler': {'packets': 12575, 'bytes': 15596484, 'cycles': 4976491476720}, 'arp': {'packets': 1787, 'bytes': 107220, 'cycles': 4976422514422}, 'icmp': {'packets': 0, 'bytes': 0, 'cycles': 0}, 'udp': {'packets': 10788, 'bytes': 15489264, 'cycles': 4971225531389}, 'app': {'packets': 10788, 'bytes': 15187200, 'cycles': 4971225531384}}}

Output of ol_w1.networklayer_0.get_debug_stats for second device

{'tx_path': {'arp': {'packets': 1787, 'bytes': 107220, 'cycles': 4976422543748}, 'icmp': {'packets': 0, 'bytes': 0, 'cycles': 0}, 'ethernet_header_inserter': {'packets': 10788, 'bytes': 15640296, 'cycles': 4971225559904}, 'ethernet': {'packets': 12575, 'bytes': 15747516, 'cycles': 4976491505184}, 'app': {'packets': 10788, 'bytes': 15187200, 'cycles': 4971225559495}, 'udp': {'packets': 10788, 'bytes': 15489264, 'cycles': 4971225559884}}, 'rx_path': {'ethernet': {'packets': 1532, 'bytes': 91920, 'cycles': 4976427265181}, 'packet_handler': {'packets': 1532, 'bytes': 91920, 'cycles': 4976427265181}, 'arp': {'packets': 1532, 'bytes': 91920, 'cycles': 4976427265181}, 'icmp': {'packets': 0, 'bytes': 0, 'cycles': 0}, 'udp': {'packets': 0, 'bytes': 0, 'cycles': 0}, 'app': {'packets': 0, 'bytes': 0, 'cycles': 0}}}

Output of ol_w0.cmac_0.get_stats() for first device

{'tx': {'packets': 255, 'good_packets': 255, 'bytes': 16320, 'good_bytes': 16320, 'packets_64B': 255, 'packets_65_127B': 0, 'packets_128_255B': 0, 'packets_256_511B': 0, 'packets_512_1023B': 0, 'packets_1024_1518B': 0, 'packets_1519_1522B': 0, 'packets_1523_1548B': 0, 'packets_1549_2047B': 0, 'packets_2048_4095B': 0, 'packets_4096_8191B': 0, 'packets_8192_9215B': 0, 'packets_large': 0, 'packets_small': 0, 'bad_fcs': 0, 'pause': 0, 'user_pause': 0}, 'rx': {'packets': 2053, 'good_packets': 2053, 'bytes': 2630228, 'good_bytes': 2630228, 'packets_64B': 255, 'packets_65_127B': 0, 'packets_128_255B': 0, 'packets_256_511B': 0, 'packets_512_1023B': 0, 'packets_1024_1518B': 1798, 'packets_1519_1522B': 0, 'packets_1523_1548B': 0, 'packets_1549_2047B': 0, 'packets_2048_4095B': 0, 'packets_4096_8191B': 0, 'packets_8192_9215B': 0, 'packets_large': 0, 'packets_small': 0, 'packets_undersize': 0, 'packets_fragmented': 0, 'packets_oversize': 0, 'packets_toolong': 0, 'packets_jabber': 0, 'bad_fcs': 0, 'packets_bad_fcs': 0, 'stomped_fcs': 0, 'pause': 0, 'user_pause': 0}, 'cycle_count': 18633460065}

Output of ol_w1.cmac_0.get_stats() for second device
{'tx': {'packets': 2053, 'good_packets': 2053, 'bytes': 2630228, 'good_bytes': 2630228, 'packets_64B': 255, 'packets_65_127B': 0, 'packets_128_255B': 0, 'packets_256_511B': 0, 'packets_512_1023B': 0, 'packets_1024_1518B': 1798, 'packets_1519_1522B': 0, 'packets_1523_1548B': 0, 'packets_1549_2047B': 0, 'packets_2048_4095B': 0, 'packets_4096_8191B': 0, 'packets_8192_9215B': 0, 'packets_large': 0, 'packets_small': 0, 'bad_fcs': 0, 'pause': 0, 'user_pause': 0}, 'rx': {'packets': 255, 'good_packets': 255, 'bytes': 16320, 'good_bytes': 16320, 'packets_64B': 255, 'packets_65_127B': 0, 'packets_128_255B': 0, 'packets_256_511B': 0, 'packets_512_1023B': 0, 'packets_1024_1518B': 0, 'packets_1519_1522B': 0, 'packets_1523_1548B': 0, 'packets_1549_2047B': 0, 'packets_2048_4095B': 0, 'packets_4096_8191B': 0, 'packets_8192_9215B': 0, 'packets_large': 0, 'packets_small': 0, 'packets_undersize': 0, 'packets_fragmented': 0, 'packets_oversize': 0, 'packets_toolong': 0, 'packets_jabber': 0, 'bad_fcs': 0, 'packets_bad_fcs': 0, 'stomped_fcs': 0, 'pause': 0, 'user_pause': 0}, 'cycle_count': 18632910452}

As we see there is not "bad" string parameter with non 0 value.

Answer 9 · 2022-06-01T11:49:36.000Z

It's intriguing because of the stats look good. If you find this again let me know.