Speedtest processes stall causing reporting to stop working
Closed this issue · 9 comments
Sometimes the speedtest binary stalls and eventually waitress refuses to create more threads.
This results in no more tests being run and no metrics being presented to prometheus
The logs include:
WARNING:waitress.queue:Task queue depth is 1
WARNING:waitress.queue:Task queue depth is 2
WARNING:waitress.queue:Task queue depth is 3
WARNING:waitress.queue:Task queue depth is 4
WARNING:waitress.queue:Task queue depth is 5
WARNING:waitress.queue:Task queue depth is 6
WARNING:waitress.queue:Task queue depth is 7
WARNING:waitress.queue:Task queue depth is 8
WARNING:waitress.queue:Task queue depth is 9
Container shows multiple speedtests running:
When stalled speedtest processes are killed things return to normal.
A possible solution would be to have a timeout for the subprocess execution that would kill the thread if the speedtest was taking longer than expected.
Hey,
This is a good idea. I will take a look on the PR.
Really thanks,
MiguelNDeCarvalho
@MiguelNdeCarvalho I have built an image based on my PR that I have been using for the past few days and it seems to be working fine https://hub.docker.com/layers/158995196/markfarrell/speedtest-exporter/3.3.0/images/sha256-8236e265f0bba1454da0c612c36ce6789ac9e984f0b4f7ca6e44405fc0140d5f?context=repo
Really thanks for your contribuition @markafarrell. I have made some comments from what we should change before merge it
@MiguelNdeCarvalho I can't seen any comments anywhere? Did you need me to change something?
Hey,
I have just requested some changes on the PR, can you see them?
Thanks,
MiguelNdeCarvalho
@MiguelNdeCarvalho Nope. I don't see anything in #84
Can you check now please? Thanks
I see them now. Thanks
Fixed in #86, really thanks for the help. Feel free to reopen if needed