kata-containers/proxy

Limit the number of threads golang can utilise

grahamwhaley opened this issue · 6 comments

grahamwhaley commented

Description of problem

It has been seen that the proxy will, over time under certain circumstances, grow more and more processor threads (possibly via the go runtime garbage collector), which then predominantly sit idle. They do however consume host PIDs, which then limits the number of containers we can run on a given host (there is a finite, if somewhat large, limit to the number of active PIDs).

See:
kata-containers/runtime#807
for more details and diagnostics.

One fix is to set the GOMAXPROCS setting in the golang runtime to restrict/reduce the number of threads that will be consumed.

jodh-intel commented

I wonder if we want to add a CLI option to change this as we could then have an option in the runtimes configuration.toml file that will specify that option when the runtime launches the proxy? Power users could then tweak the config option as desired to override the default we bake into the proxy.

grahamwhaley commented

Maybe - but, hopefully if we set the max level correctly, we won't be restricting the performance of the shim or proxy. neither of them is that parallel or work processing heavy, so I'm not sure it is worth the effort.
My fix will look like:

if os.Getenv("GOMAXPROCS") == "" {
		runtime.GOMAXPROCS(6)
	}

so the value is over-rideable with the 'normal' golang env setting - of course, this does not mean there is a nice way to set that via the runtime - you could set it in the dockerd systemd env, but then it would effect all golang programs launched under docker.

I'll fire in the simple proxy and shim PRs (and they would probably still hold even if we do decide to add the config ability into the runtime), and we can then debate if we want to make the runtime able to pass the config request/override on. You can see what sort of runtime 'hack' in that Issue, or in my gist.

jodh-intel commented

That seems like a pragmatic solution and the admin can set the variable in /etc/systemd/system/docker.service.d/kata-containers.conf if needed. We could consider creating something like:

That is referenced by all the docker install docs and adds some verbage about how (and why) you can set GOMAXPROCS.

marcov commented

@grahamwhaley, I did some research and it seems that by default GOMAXPROCS matches the HW setup, in term of cores / CPU or whatever.

So shouldn't the cap be applied only to machines with many cores, checking e.g. runtime.NumCPU()?

grahamwhaley commented

Hi @marcov Indeed, the default is set to the number of cpus of the host.
We could thus only set our cap if the GOMAXPROCS has not already been over-ridden and the number of cores is larger than our max. Having a look at the go (1.10) source, I think if you set GOMAXPROCS > physcpus, it will still go ahead and create the GOMAXPROCS number of OS threads. So, yeah, maybe only setting the cap if necessary makes sense. I'll update my pending patches (I've not PR'd as I'm just running some footprint/speed tests to see if there is any noticeable difference).