[question] Is possible to use Docker Server with TT enabled in kernel?
ionutnechita opened this issue · 14 comments
Hi Hamad Al Marri,
Is it possible to have the TT in kernel and the docker server working properly?
Is there a problem with CGROUPs?
// level=warning msg="Unable to find cpu cgroup in mounts"
A good day.
Ionut.
@ionutnechita
Should work, but i dont think you wont profit much on it. Since the scheduler is mostly thought to provide a low latency. But you ca try it.
I tried today, but the docker containers don't start.
Maybe tomorrow I'll give you more details.
Do you have autogroup disabled ?
I use this options:
CONFIG_TT_SCHED=y
CONFIG_TT_ACCOUNTING_STATS=y
And AUTOGROUP is disabled.
Patch set:
tt-5.15.patch - 0.3.7
high-hz.patch
Sorry, i was wrong. It does not work, since autogroup is disabled at default.
could you please upload the .config
docker sometimes works without autogroup and sched_group
I'm sending you 2 .config, one without TT and one with TT.
-
Without TT:
config-5.15.9-lowlatency-non-tt.txt -
With TT:
config-5.15.9-lowlatency-with-tt.txt
My scenario:
- Docker Server is installed on my host.
- Use Kind tool for create new cluster. Docker Server is backend for new cluster.
./kind create cluster --config test.yml -v 1
Without TT, working correctly:
Creating cluster "kind" ...
DEBUG: docker/images.go:58] Image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6 present locally
✓ Ensuring node image (kindest/node:v1.21.1) 🖼
✓ Preparing nodes 📦 📦 📦 📦 📦 📦
✓ Configuring the external load balancer ⚖️
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
✓ Joining more control-plane nodes 🎮
✓ Joining worker nodes 🚜
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Have a nice day! 👋
./kind create cluster --config test.yml -v 1
With TT, not working correctly:
Creating cluster "kind" ...
DEBUG: docker/images.go:58] Image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6 present locally
✓ Ensuring node image (kindest/node:v1.21.1) 🖼
✓ Preparing nodes 📦 📦 📦 📦 📦 📦
✓ Configuring the external load balancer ⚖️
✗ Writing configuration 📜
ERROR: failed to create cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: failed to get file: command "docker exec --privileged kind-worker3 cat /kind/version" failed with error: exit status 1
Command Output: Error response from daemon: Container b411eec646c160287d2c770eefa0fffc3498d30ecb9debdd838b4966c309094f is not running
Stack Trace:
sigs.k8s.io/kind/pkg/errors.WithStack
sigs.k8s.io/kind/pkg/errors/errors.go:59
sigs.k8s.io/kind/pkg/exec.(*LocalCmd).Run
sigs.k8s.io/kind/pkg/exec/local.go:124
sigs.k8s.io/kind/pkg/cluster/internal/providers/docker.(*nodeCmd).Run
sigs.k8s.io/kind/pkg/cluster/internal/providers/docker/node.go:146
sigs.k8s.io/kind/pkg/exec.OutputLines
sigs.k8s.io/kind/pkg/exec/helpers.go:81
sigs.k8s.io/kind/pkg/cluster/nodeutils.KubeVersion
sigs.k8s.io/kind/pkg/cluster/nodeutils/util.go:35
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config.getKubeadmConfig
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config/config.go:208
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config.(*Action).Execute.func1.1
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config/config.go:90
sigs.k8s.io/kind/pkg/errors.UntilErrorConcurrent.func1
sigs.k8s.io/kind/pkg/errors/concurrent.go:30
runtime.goexit
runtime/asm_amd64.s:1371
A good day.
Ionut.
There was also some time ago a cfs release, maybe that is a try worth.
Also you could use hamad's old scheduler, the cacule scheduler.
I also run that on my buildserver, the RDB version at it runs great.
Here you find the patches for 5.15 and 5.16.
Could you please try with
UCLAMP_TASK enabled and also with HZ=300 or 1000
Hi @hamadmarri
I try with UCLAMP_TASK enabled and also with HZ=300.
But not working.
config-5.15.9-lowlatency-with-tt-uclamp.txt
And i notice this in journalctl:
journalctl -u docker
...
level=warning msg="Unable to find cpu cgroup in mounts"
I will test the next few days.
Thank you for help.
Happy New Year.
You might also try the BORE scheduler https://github.com/firelzrd/bore-scheduler
Hi @hamadmarri
I tried with BORE scheduler and working fine.
I think in the future I will use BORE or Cacule scheduler.
I don't think TT design supports that full docker support.
But this TT scheduler is very powerful.
Thanks for support.
A good day.