300 packets arriving during one tick causing unreal to freeze up?
Opened this issue · 6 comments
I'm receiving some 3d scanning data over UDP, packets of roughly 1000 characters containing vertices and meshes. The data arrives in kind of a bursty way something like 300 of these packets might arrive in the time between ticks. I'll have a couple ticks where nothing is received, then one where 300 packets are. Unreal is running on a very fast PC.
When the packet volume gets high enough, unreal completely freezes, no rendering and no message. If the traffic volume drops for a bit, it will recover without seeming to have lost any data. Processing 1,000,000 bytes of data all at once. Even if traffic stops completely, it can take several seconds to recover. Also even if unreal freezes for 10 seconds or more, the delta-T reported in the next on-event-tick will still be 1/30th of a second.
When the freeze happens, none of my CPU cores seem to have a full load on them.
The blueprint inside my actor is very simple, the receive data event is connected to a blueprint that parses out around 200 comma separated numbers in the packet and puts them into an array, the array is is already the right size.
Any idea what might be going on here and how to fix it?
My best guess is that 300 packets arriving in one tick creates 300 individual events that Unreal is trying to process all at once.
Something internally gets deadlocked and everything freezes till the inrush of packets stops.
Any help or ideas would be greatly appreciated.
By default this plugin copies and receives data on gamethread for simple use cases, but with your data volume, you probably want to change the UDPSettings to receive on background thread (set https://github.com/getnamo/UDP-Unreal/blob/master/Source/UDPWrapper/Public/UDPComponent.h#L64 to false). Then handle your processing (while recommended to be done in C++ with high volume, you can do this in blueprint) with being mindful of not creating or destroying uobjects on the background thread. Then when processed you copy or signal to the game thread your data is ready to read. See https://github.com/getnamo/SocketIOClient-Unreal#blueprint-multithreading on how to deal with multi-threading in blueprint (experimental, but functional)
Thanks for the reply! Just to clarify,
- This means anything attached to the "On Received Bytes" event in the blueprint will be running on the background thread?
- I can't create/delete an actor from the background thread?
- Can I still reference an existing actor and copy data into it's arrays from the background thread?
- Can I call the interface of an existing actor from the background thread?
I can see where using background thread would help but it also causes some issues if I can't create actors from it. One of the cases of this is it receives a message that creates an actor, then receives more messages that parse and copy data into that actor. Finally it calls an interface on the actor to trigger it to create the procedural geometry and become visible. Within the space of one tick I might receive several actor creation messages.
So if I can't create actors as a result of receiving a packet, then I have noplace to put the data that will show up in later messages.
Thinking about writing C++ now, though will this actually solve the issue? Wouldn't my code still need to run in a background thread where it couldn't create actors?
I appreciate your help.
You need to break up your logic.
- Prepare by creating the actor first on the game thread. (If you don't know about this ahead of time, do step 2 first, then do step 1 and 3 together)
- Then receive the data on the background thread (Yes on received bytes will be called on a background thread). Do any processing you need to do and prepare it into an easy to consume format (e.g. copy into the arrays)
- Then go back to the game thread to do the update (Interface call, if it is unsafe to do on bg thread, can test it, worst case it crashes and you know you can't do that action on a background thread)
It is fairly straightforward to chain this logic with the https://github.com/getnamo/SocketIOClient-Unreal#blueprint-multithreading utility functions that are available to the contained socketio plugin (also used for udp/string conversion)
Just keep in mind, this type of processing is normally done in C++ for performance reasons (but it is possible to do in Bp if you follow the steps above)
Also if you're using UDP, keep in mind it's unreliable and your packets may be received out of order, for your use case a TCP connection sounds more appropriate.
Thanks a lot.
It sounds like I should try to code something in C++. Do you know of any example code that receives UDP or TCP and does object spawning based on messages? I'm just not that familiar with Unreal's class library and internal "plumbing", a basic example would give me a starting point, then I can code the rest.
The app sending me all this data does full updates a lot, so I have structured the UDP messages so if one is dropped or arrives out of order it won't matter, the error will get corrected by the next update within a second or two. If I switched to TCP it might result in sending fewer packets and less data, but not a lot less.
So let me ask, if the receiving actor has a reference to an existing target actor, would it be safe to directly copy data from the background thread of the receiving actor into an existing target actor? An array resize might be involved.
My current setup is I have a I single receiver actor that receives the messages. Then there is a target actor definition that has arrays for vertex/face/normal data in in it and a blueprint for creating procedural geometry from it. The target actor auto-creates the procedural geometry during it's tick when it detects all the data it needs has been loaded into it's arrays.
The receiver actor has an (initially empty) index of target actors it has created, The packets are all of the form "this is for actor X, copy the following numbers into it's array starting with index Y" so when a packet arrives it gets the reference to the target from the index, uses that reference to access the target's array and copy the data directly into it. If the target actor isn't in the index it spawns one.
At the beginning I don't know how many target actors I'm going to need, but I could pre-create a pool of them. But can I copy data into their arrays from the background thread safely?
I could have the receiving actor background thread parse and save the data it receives in temporary structures. Then when it ticks in foreground I could create all the actors and copy over all the data. But this still seems like a lot of work in the foreground tick. During one tick I could get messages for as many as a dozen actors.
Seems like I might have similar issues implementing this in C++ unless the thread management is easier there.
I'm doing more studying, your pointers have been very helpful!