addr_sign in udp_tx_mux stuck for some clk cycles - development for Intel Stratix 10 MX

[AleCamplani]

Hi everybody ,
I am working on making use of your IPbus code on the Intel Stratix 10 MX development kit.
This with the intention of using it for an ATLAS Phase II project.

I have been recently in contact with A. Thea, who gave me a few suggestions on how to proceed.
Taking into account that my Intel MAC IPcore

• uses 32 bit words
• uses Avalont ST protocol

I have written an interface between my MAC and the IPbus block (based on what the ATLAS LAr group have also been doing)

Also the same MAC IPcore

• runs at 156.25 MHz

and for this A. Thea suggested me to keep the 156.25 clock as mac_clk, divide it by 4 and feed the ipbus clk with this ~39MHz clk.

Given these info, I was able to mimic in simulation ARP, ping and read requests, getting back meaningful replies.

Unfortunately, going in hardware I found a discrepancy.
This is the link to my slides with my most recent results (from slide 6 on..), as well as my recent problem (slide 1 to 6), mentioned in the title on this ticket.

Please let me know if you can open the slides and if you can help/suggest me how to proceed.
Let me know if more information is needed.
I would also be available for a Skype chat.

Thanks in advance for your help.
Best regards,

Alessandra Camplani

[dpcsankey]

This is a bug in your mapping from Xilinx 8 bit AXI to Intel Avalon ST 32 bit. Addr_sig pauses because Mac_tx_ready has gone low - you say you are not ready for the packet so I stop sending it (so my address counter pauses until you are ready). Dave

[AleCamplani]

Hi David, thanks a lot for the fast reply!

I see your point, I didn't notice the ready going low (and I kept the valid high as it comes from IPbus).
I am trying now to adjust the valid as well as to understand better the specifications of my MAC (if there must be a certain latency between the ready going high again and the data coming in).

I'll let you know asap if this solves the problem. Thanks again.

Alessandra

[dpcsankey]

I’m expecting your flow control to behave as per figure 7.7 in https://www.xilinx.com/support/documentation/ip_documentation/tri_mode_eth_mac/v5_2/ug777_tri_mode_eth_mac.pdf I can cope with pauses anywhere in the data stream (and indeed ready held high), the diagram just shows the behaviour of the actual MAC with it sucking in the first two octets and then pausing. In your figure on slide 4 I see that there had already been a 1 tick pause during the transmission of the destination Ethernet address (that duplicate 0x02), but this is almost certainly the same ignored flow control. As a separate issue regarding MAC addresses CERN do have a pool of official ones rather than using our example ‘oddball’ ones, this would be extremely sensible going forward. D

[AleCamplani]

Hi David,
ok, I misunderstood. Now I clearly see the problem

Figure 11, page 39 at this link shows the same expectation for the Intel MAC IP.
I need a moment to think about how to fix it in the converter.

(The long 02 is due to my MAC address 0x020ddba11510.)

Thanks for the suggestion about the MAC addresses at CERN. I'll keep this in mind.
(At the moment I am testing from home...)

Thanks again for your help and best regard,
Alessandra

[dpcsankey]

OK. I don’t need any pauses but can cope with them, so this could ease the flow control for you. Might be easiest to suck whole frame from me into a FIFO and then send this? D

On 24 Nov 2020, at 11:12, AleCamplani ***@***.***> wrote: Hi David, ok, I misunderstood. Now I clearly see the problem Figure 11, page 39 at this link shows the same expectation for the Intel MAC IP. I need a moment to think about how to fix it in the converter. (The long 02 is due to my MAC address 0x020ddba11510.) Thanks for the suggestion about the MAC addresses at CERN. I'll keep this in mind. (At the moment I am testing from home...) Thanks again for your help and best regard, Alessandra — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

[AleCamplani]

Hi David, this is an interesting suggestion.

For me to understand.. Are you suggesting that I could:

1. set the mac_tx_ready signal (in input to the IPbus module) to '1'
2. write the IPbus output mac_tx_data just upon valid in the FIFO (..possibly already in 32 bit)
3. and then read the FIFO upon the "real" MAC ready signal?

Thanks and best regards,

Alessandra

[dpcsankey]

Exactly so. You just need to think about what to do about the ’next packet’ whilst you empty the FIFO (and there is a bit of pause required for the IFG and Ethernet preamble, hinted at in figure 10 in the Intel document). But this could just be suitably associating the mac_tx_ready with the FIFO fill level as I am happy to pause. D

On 24 Nov 2020, at 12:30, AleCamplani ***@***.***> wrote: Hi David, this is an interesting suggestion. For me to understand.. Are you suggesting that I could: • set the mac_tx_ready signal (in input to the IPbus module) to '1' • write the IPbus output mac_tx_data just upon valid in the FIFO (..possibly already in 32 bit) • and then read the FIFO upon the "real" MAC ready signal? Thanks and best regards, Alessandra — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

[AleCamplani]

Hi David, let me reply between the lines.
Thanks and best regards.

Alessandra

Exactly so. You just need to think about what to do about the ’next packet’ whilst you empty the FIFO (and there is a bit of pause required for the IFG and Ethernet preamble, hinted at in figure 10 in the Intel document).

Regarding this part, I understand. Two packets could actually come so close that the first one is not yet completely readout while the next one is in written. And then there is not gap.
But are the IFG and Ethernet preamble already created in the IPbus packet?
Also, is there a specific timing between two IPbus replies?

But this could just be suitably associating the mac_tx_ready with the FIFO fill level as I am happy to pause.

I have to admit that using the mac_tx_ready (coming from the MAC) to also write the data is not obvious to me.

Let me try to explain my doubt: for the fifo read I might need to use the Show-Ahead Mode to keep the data_out aligned with the ready coming from the MAC (as in Figure 11 of the MAC document).

But how can I keep that alignment if I use the same mac_tx_ready also for writing?
I thought I need at least 1clk cycle delay between the signal used to write and the one used to read.
Am I wrong? Maybe I am missing something.

D
On 24 Nov 2020, at 12:30, AleCamplani @.***> wrote: Hi David, this is an interesting suggestion. For me to understand.. Are you suggesting that I could: • set the mac_tx_ready signal (in input to the IPbus module) to '1' • write the IPbus output mac_tx_data just upon valid in the FIFO (..possibly already in 32 bit) • and then read the FIFO upon the "real" MAC ready signal? Thanks and best regards, Alessandra — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.
This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

[dpcsankey]

By mac_tx_ready I mean the Xilinx AXI signal (so maybe less confusing to call it axi_tx_ready). So say you have a FIFO that is AXI 8 bit + axi_tx_last coming in on the IPBus side and Avalon ST 32 bit + avalon_st_tx_startofpacket and avalon_st_tx_endofpacket going out on the other with a signal fifo_nearly_full back to the AXI side. On the AXI side you hold axi_tx_ready high whilst fifo_nearly_full is low and take data into the FIFO on axi_tx_valid (with appropriate handling when you take axi_tx_ready low to pause filling). On the Avalon ST side you decide when to start sending data - certainly with the Xilinx MAC you cannot pause so need have enough data in hand to be able to send continuously - but assert avalon_st_tx_startofpacket appropriately and avalon_st_tx_endofpacket at end of packet and read from the FIFO on avalon_st_tx_ready. After avalon_st_tx_endofpacket I would take avalon_st_tx_valid low for a few ticks before trying to send the next frame, expecting the Intel MAC to be like the Xilinx one and only assert avalon_st_tx_ready when it is ready (so it handles the IFG etc). On the IPBus side I think my state machine has two ticks between attempting to send consecutive packets, so the IFG is all handled by the MAC - the data I send corresponds to the top line in Figure 10. OK? D

…

On 24 Nov 2020, at 13:45, AleCamplani ***@***.***> wrote: Hi David, let me reply between the lines. Thanks and best regards. Alessandra Exactly so. You just need to think about what to do about the ’next packet’ whilst you empty the FIFO (and there is a bit of pause required for the IFG and Ethernet preamble, hinted at in figure 10 in the Intel document). Regarding this part, I understand. Two packets could actually come so close that the first one is not yet completely readout while the next one is in written. And then there is not gap. But are the IFG and Ethernet preamble already created in the IPbus packet? Also, is there a specific timing between two IPbus replies? But this could just be suitably associating the mac_tx_ready with the FIFO fill level as I am happy to pause. I have to admit that using the mac_tx_ready (coming from the MAC) to also write the data is not obvious to me. Let me try to explain my doubt: for the fifo read I might need to use the Show-Ahead Mode to keep the data_out aligned with the ready coming from the MAC (as in Figure 11 of the MAC document). But how can I keep that alignment if I use the same mac_tx_ready also for writing? I thought I need at least 1clk cycle delay between the signal used to write and the one used to read. Am I wrong? Maybe I am missing something. D On 24 Nov 2020, at 12:30, AleCamplani @.***> wrote: Hi David, this is an interesting suggestion. For me to understand.. Are you suggesting that I could: • set the mac_tx_ready signal (in input to the IPbus module) to '1' • write the IPbus output mac_tx_data just upon valid in the FIFO (..possibly already in 32 bit) • and then read the FIFO upon the "real" MAC ready signal? Thanks and best regards, Alessandra — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[AleCamplani]

Hi David, ok I think I understood.
Let me try tomorrow to implement it, and run some tests in sim and hw.
I'll let you know asap.

Thanks a lot again for all the help!
Alessandra

[AleCamplani]

Hi David,
the situation looks now a bit better (picture attached).

I don't have yet a ping reply on the PC, but I think I need to understand first why my start signal and my first two packages become 2clk cycles long.
-> Quick update: I probablt know why, it should be my show-ahead fifo. I'll try to fix that.

Thanks again,
Alessandra

[AleCamplani]

Dear David, the previously mentioned problem is solved.
The only thing that I now notice is that (as also from previous pictures) the IP address of the sender (the FPGA board, when replying to ARP) is set to 0.0.0.0. In my code that's actually not the default value.
While the MAC actually is propagated nicely.
Here in the screenshot I wrote down the packet that comes back from IPbus.

Let me now if you have suggestions (and if I should open another ticket).
Thanks in advance and best regards,

Alessandra

[dpcsankey]

That is not right, the IP address should be filled in there from My_IP_addr_sig signal in udp_if, which has come from the ip_addr port on ipbus_ctrl. The same signal is used in udp_packet_parser to recognise the ARP request, so this is very odd… D

On 1 Dec 2020, at 16:43, AleCamplani ***@***.***> wrote: Dear David, the previously mentioned problem is solved. The only thing that I now notice is that (as also from previous pictures) the IP address of the sender (the FPGA board, when replying to ARP) is set to 0.0.0.0. In my code that's actually not the default value. While the MAC actually is propagated nicely. Here in the screenshot I wrote down the packet that comes back from IPbus. Let me now if you have suggestions (and if I should open another ticket). Thanks in advance and best regards, Alessandra — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

[AleCamplani]

Hi David,

agreed, this is odd. In simulation after a reset this looked correct.
I'll have a look if in HW the reset is not propagated properly (and if the My_IP_addr_sig is actually picking up the correct value), adding these signals to SignalTap.

Thanks and best regards,
Alessandra

[AleCamplani]

Hi David,

this is now also solved. It was due to a problem in the packet conversion before IPbus.

I am attaching a couple of pictures.
I think now I just need to understand why, given a possible good package, the MAC tx gmii part (response going from MAC to PHY) does not look fine. The gmii tx error line is not visible in the second picture, but I remember it to signal an error at the end of the packet.

I have a avalon_st_txstatus_error that should give me soon some more hints.

But I thought that maybe the padding done when passing from 8 bit to a complete 32 bit word might not be as expected.
E.g. in the picture with the converted signals form 8 to 32 bit, the last words on the AXI side are C8 and 02.
The total number of words is not a multiple of 4, so I needed to pad my last 32 bit word with zeroes.. but maybe some more zeroes should be added (as we have been also mentioning also above).

Once I get the error type, I should be able to tell you more.
Thanks and best regards,

Alessandra

[AleCamplani]

Dear David,

after some debugging the situation looks much cleaner (first picture).
The converter from IPbus to MAC has been updated, also to have a valid that never goes down inside a package (to avoid Underflow error flag).

I still have an error flag in the package that I don't quite understand.
The data are now properly converted from the IPbus block towards the MAC, but and Undersized frame is flagged.
Looking at the avalon_st_txstatus_data (following table 44 at this link) the expected packet length is 3C (=60) while the payload length is expeted to be 2A (=42, that is the length of the IPbus packet itself).
Do you have any suggestion regarding this point?

Thanks and best regards,

Alessandra

[AleCamplani]

Ok, no worries. This was related to a wrong setting in my MAC.
It looks like I can ping now :)

Thanks for everything! Alessandra