Help Wanted | Memory Usage buildup in Long Running Program
bcpeinhardt opened this issue · 5 comments
Hello Steve!
First of all I should say thank you for all the help you've already given and how responsive you've been, as it has been invaluable!
Thanks to that help, I've managed to construct a program that automates previously manual tasks that arrive in a queue. I have also made copy of that program without any of the previous commits that contained sensitive data and published it here:
https://github.com/bcpeinhardt/site_impact_order_processor_public_version
So there is finally a reference for my questions haha. Currently I'm facing the issue where when I run the program for extended periods of time, I start to build up memory, until I'm using up all my laptops 32GB and have to shutdown and restart. This is actually less of a hassle than it sounds like, but still I have to figure out how I can fix this.
As far as I can tell from my research online, the problem likely involves building up my cache with repeated page visits (it visits urls unique to each "order" to process them). In that case, I suppose the solution is to clear my cache and log back in every few minutes.
I'm happy to do this, but I think there is a high likelihood that the issue is actually stemming from my misuse of tokio and/or thirtyfour. The main areas where I think the trouble may be are main, processor, and listener. Obviously nobody should have to do code review for a stranger, but if something is glaringly wrong to an experienced eye it could save me a lot of heartache.
Tbh I haven't actually tested thirtyfour with a long-running program either so I'm keen to check this out. I should be able to get to this in the next day or so.
I'm assuming from what you said that the process that consumes all of the memory is yours and not the browser, but just wanted to confirm.
Just had a quick look at the listener. You've got a Vec called enforced_orders which you seem to push to but never clear. This is going to accumulate memory and never release it, which will result in what you're seeing. Not sure if that's solely responsible for chewing up all 32GB though. If it is, consider replacing it with some kind of MFU/MRU cache or similar so you can keep the cache at a fixed size.
I really appreciate you taking a look! I will try this, but I'm really hesitant to blame enforced_orders. I only run the program a day at a time at most, and it processes max 500 - 600 orders, so the enforced_orders vec will only ever hold that many strings at a time. On my task manager it's Firefox that registers as using the memory, but that's not to say my usage of the web driver isn't to blame. The website itself is ... hefty xD It makes use of several web workers and opening the dashboard on google chrome spawns 12 sub-processes (I don't know what the correct terminology is here but task manager gives me a little collapse arrow to display the individual processes). I really think it's just an impressively large website which I'm navigating to over and over, but I just wanted to make sure I didn't have any obvious concurrency mistakes.
I will convert enforced_orders to a fixed size cache (I think you're right that it presents a problem, especially if this memory gets handled and I do want to run it continuously) and begin looking at clearing the cache at regular intervals for each driver, probably based on a simple counter for how many times each browser has navigated to a new page.
Oh, so if it's Firefox consuming the memory then your code is most likely not to blame. It would do the same if you navigated to the page on Firefox and performed the same operations manually. Does Chrome do the same? The web app itself may have a memory leak (circular refs in JS etc) or something like that. One thing you could try is actually refreshing the page periodically (via WebDriver::refresh()), which should flush out any state that accumulated on the JS side and hopefully trigger GC.
Just wanted to give an update, the refresh worked like a charm, the OMS had a memory leak and refreshing every so often seems to trigger the GC. Thank you very much!