Running sitespeed test in pods, after few hours the pods go down
Opened this issue · 12 comments
Your question
Hi Team,
while running sitespeed tests in pods, I observed that memory usages increases continuously and the tests were scheduled to run every 15 minutes, after few hours the pods go down, we set up the --max-old-space-size to 8GB but still the pod goes down once it reached the threshold. please advise
Thanks,
Valiyullah
I assume you are running continually in the same pod; consider using a cronjob to run your tests so the control plane spins up a new pod for each schedule.
Hi @mdvaliyullah can you share more on how you run? I'm thinking maybe there could be some tweak in how we handle memory that could be done. Else as @madan said, starting a new container every time is good.
For long running containers (over days) I've think I have work todo to cleanup Chrome issues, I've seen tmp files etc that probably should be cleaned, but my use case has always been to spin up a new container.
Hi @soulgalore my use case is to run different urls list during each schedule, so i wanted to keep the container running for long period, is there way to clean up the unused memory after each run?
Ok, so the size of the container meets the limits is reached or what happens? As long as the NodeJs process is exited, the only thing that should be in the conainer are the result files like html/videos etc. Are they removed in your current setup?
Yes results folders are cleaned up before starting a new test, once the memory reaches the heap limits the pod get restarted
Ok, so do you have your own container or have you overridden the start script or how do you do it, can you share a little more?
I have own container with sitespeed image. No changes to startup script, just passing the arguments details like url list, iteration when starting the test to the script.
Ok. How does it work, how do you keep the container open, isn't closed after a test run?
Yes it is not closed after the test run, the sitespeed image is used as a base image in the container
But you exit node? So like for each test you run, you run the start script? Do you have anything setup to monitor the memory usage, is it constant increasing?
Yes, I checked there is no process running once the test completes. I run the start script for every test once the previously running test completes.
memory usage gradually increases for each test once it reaches the usage threshold the pods gets restarted.
Ok, can you get more granularity to see what uses the memory? Since you use your own container, do you have something else running in it?