IPBus split packets
Closed this issue · 15 comments
Hello,
In my project I am using IPBus on Spartan 6 (CERN's SPEC card) connected to 1Gbit Ethernet with SFP and during increasing data rate it seems that if almost every packet is split then incoming data gets mixed. Did any of you had similar problem?
The issue looks like IPBus master do not read correct amount of data - which is very important for me.
Hi,
I cannot recall having encountered any problems similar to the one that you described.
In order to provide help, I'd need more information about the unexpected bahaviour that you are seeing. For example, are you only reading a single block memory, reading several different registers as well as block memory, or writing to some registers/memories as well as reading from them? What observations suggest to you that the IPbus master did not read the correct amount of data?
Also, please could you let us know what version of the software and firmware you're using?
Cheers,
Tom
Also, in your application, what URI are you using for communicating with the hardware (should start with ipbusudp- or chtcp-)?
Hello again,
sorry for late response, I was trying to find any workaround but without any success.
I am reading single block memory (FIFO) and there are no other transactions during readout. First transaction checks register for how many bytes needs to be downloaded, second transaction reads this amount of data from a single register (FIFO) - here a split IPBus frame can happen.
The data read from the device is parallelised serial data coming from an external device. I know protocol and position of each byte. If data coming from external device is below one IPBus frame (one downloaded frame is 1120B) then there is no problem. On the other hand when I have enabled downloading multiple values of 1120B then my software checking each incoming frame gets desynchronised and I read rubbish - it looks like one part of frame was lost. It not happens right away - only when more than 50% of frames are split.
FIFO is secured that way it cannot be overfilled and if there is not enough empty space then whole frame will be dropped - so no problem here.
I am using older version of IPBus Firmware, from CERN SVN- but after comparing each file I have not found any critical change (only those with LEDs, etc.) - anyway I have upgraded each file and the problem has not been repaired.
As for software (which is written in C++), I am using direct uhal libraries with its libraries (uhal, connectionmanager etc.).
Thank you for your help.
Hi,
What URI(s) are you using for communicating with the hardware? If your HwInterface objects are all created from information specified in a connections file, then this would be the value of the id attributes for the connections that you're using from that file. If you're creating your HwInterface objects without a connections file - i.e. using the 3-parameter, static ConnectionManager::getDevice - then this would be the value of the 2nd parameter
Cheers,
Tom
I am using connection files, each time data is downloaded this line is executed:
ValVector<uint32_t>getBlock(std::string connectionFile, std::string dev, std::string block, int size){
ConnectionManager manager(connectionFile);
HwInterface hw = manager.getDevice (dev);
ValVector< uint32_t > mem = hw.getNode ( block ).readBlock( size );
hw.dispatch();
if (!mem.valid()){
std::cout << "Returned memory is invalid" << std::endl;
}
return mem;
}
Do your URIs in the connections file begin with ipbusudp-2.0 or chtcp-2.0?
ipbusudp-2.0
OK, thanks! This means that the packets are being sent directly from uHAL to the hardware. Could you try routing them via the ControlHub, by using a chtcp-2.0 URI as outlined at http://ipbus.web.cern.ch/ipbus/doc/user/html/software/uhalQuickTutorial.html#working-with-the-controlhub ?
I will try it tomorrow (I do not have hardware with me) and I will let you know - thank you for your response :)
I have tried routing those packets via ControlHub and during communication the problem still happens - but after longer period of time. Also incoming data rates gets lower (from 20Mbit/s with ipbusudp-2.0 to 4 Mbit/s with chtcp-2.0). I am running SLC 6 on VM - which is for sure responsible for lowering data rate, but I have never though that it would be so low.
Hi Folks.
If what's being observed is packet re-ordering on a direct UDP connection on the response side, then yes, we have seen this before. The problem was tracked down to a too-clever ethernet card and driver that ran across several cores of the host. The control hub can't help in all cases, if the packets are re-ordered between the control hub and receiving process.
However: this simply should not happen with a TCP connection between control hub and process, so there is still a mystery.
OP, do you have a simple test design that we could run on a board here and try to debug?
Dave
Hi @mkuklewski
Are you still encountering this problem? If so, as Dave suggested, do you have a simple test design that we could run on a board here and try to debug? If the problem has been resolved, could you let us know, so that we can close this ticket?
Thanks,
tom
Hello @tswilliams
The problem still exist, I just did not have time to test it more (I will try to do it during this and next week).
And honestly speaking I do not have any idea how I can narrow this problem. I am not sure if the problem is inside FPGA firmware (like not reading enough data), if there is something wrong with transaction on the bus or frames gets mixed on PC.
Best regards,
Michael Kuklewski
Hi,
I'm closing this ticket since there have not been any comments for several. If you are still suffering from this problem and/or have further questions, please feel free to reopen this ticket.
Cheers,
Tom