Memory Leak due to socketio
youliangtan opened this issue · 2 comments
Issue Description
This mem leak issue can be seen by launching the server and dashboard. To test this, the user will need to keep the dashboard open, and then observe ram usage api_server by using htop. We can observe that the ram usage will grow over time, till the server is beyond usable due to OOM. This increment gets more severe when we switch to the "task tab" on the dashboard, along with multiple tasks showing on the page.
It is also observed that the issue happens more frequently when we have more tasks being stored in the db. Switching the db between sqlite and postgres isn't helpful.
After multiple investigations, it is found that socketeio is the main culprit of this memory leak issue. This issue is identified when seeing multiple "subscribe" and "unsubscribe" events performed in every update. Constantly connections and disconnection is also expensive (ref). This gets worst when we have more tasks or robots showing up on the dashboard. To view this behavior on chrome, go to the "developer console", select "network", then select the ws connection to the api-server.
Temporary fix
As a stopgap, I tried only to "subscribe" once to fleet states during init. Also, as for testing, I also removed the socketio component for task states. The commit is shown here. Limiting the number of "subscribe" event greatly help to prevent the growth of the api-server mem usage. The dashboard will still work since we are also doing constant polling of http requests.
The proper way to solve this issue is to look into the implementation of FastIO. The server should still work even with high frequency of subscription events.
Thanks for the investigaton, I think I have a very likely cause for the leak now. The problem is probably not with socketio, rather it is with rxpy, particularly this line
leaks rxpy subscriptions.Can you also make a PR to fix the client side always re-subscribing? As mentioned in the socketio issue, this is a wrong usage, there is no need to constantly reset the subscriptions.