healthcheck high load

Question

healthcheck high load

visibilityspots opened this issue 4 years ago · 11 comments

I had to force my nomad based piaware docker setup down to the image with label 3.8.0 due to the health check in version 3.8.1. The node it was running on got a really high load making it unstable.

I'll dig into it once I find some time to debug the issue but thought I already give a headsup.

Answer 1 · 2020-07-05T02:53:38.000Z

Hi Jan,

Sorry this is causing you problems.

You can just disable the health check, this might be a better option for you.

With docker run add --no-healthcheck.

With docker-compose add the following to the piaware service:

healthcheck:
      disable: true

Keep me posted!

Answer 2 · 2020-07-08T02:57:09.000Z

Hi @visibilityspots,

Any updates?

Answer 3 · 2020-07-23T18:41:43.000Z

Finally found some time to further investigate the issue. It's definitely the healthcheck which is causing the high load every 30s as you can see in the image:

The peaks are topping over 600Mhz during the healthcheck..

Unfortunately nomad doesn't support the disabling of the HEALTHCHECK from a docker image.. So or I have to stick with 3.8.0 or I have to rebuild a custom image based on yours without the healthcheck :(

Besides the fact nomad can't disable health checks I'm not sure if a healthcheck which causes such a higher load compared to the actual service is advisable?

Answer 4 · 2020-07-23T19:39:31.000Z

Yes, this is not ideal. I will rewrite the healthcheck.

In the meantime, I've created you mikenye/piaware:latest_no_healthcheck and mikenye/piaware:v3.8.1_no_healthcheck.

Feel free to use these. Once I've fixed the healthcheck I'll get you to test and revert to the normal stream if that's OK.

Thanks.

Answer 5 · 2020-07-23T23:12:28.000Z

Hi @visibilityspots, can you pull and try out mikenye/piaware:development? This contains a reworked healthcheck using bash instead of python.

If this solves your problem, I'll merge into master.

Thanks.

Answer 6 · 2020-07-24T12:08:38.000Z

Hi,

I tried the development tagged image but the issue still pops up now only every 300s :)

Answer 7 · 2020-07-27T10:34:54.000Z

Annoying. I hoped that moving the healthcheck from python to bash would alleviate the problem.

Looking at the healthcheck.sh script, I can't figure out what could be contributing to the high load. My best guess is running piaware-status.

Are you able to run piaware-status manually to see whether this causes a CPU spike?

Answer 8 · 2020-07-28T02:44:09.000Z

Would you like me to maintain an additional "latest_nohealthcheck" version of the image?

Answer 9 · 2020-08-03T07:41:53.000Z

if that's possible that would be great to keep the noheathcheck version alive. I double checked with the piaware-status but that particular command didn't really spiked the CPU...

Answer 10 · 2020-08-03T07:55:36.000Z

Hi @visibilityspots,

It would be my pleasure. I'll update my nightly build scripts to build the nohealthcheck version nightly.

Answer 11 · 2020-08-03T08:39:51.000Z

Hi @visibilityspots,

This has been completed. I changed the name of the tag to _nohealthcheck, so you should be able to pull latest_nohealthcheck, and this will be built nightly from master branch.

Cheers!