Missing equivalent for --cgroupns=[host|private]

Question

Missing equivalent for --cgroupns=[host|private]

thediveo opened this issue 4 years ago · 26 comments

On hosts with only a unified cgroup v2 controller hierarchy Docker now defaults (unless configured otherwise) to automatically creating private cgroup namespaces for created containers. The rationale here is to reduce leakage of potentially sensitive information about the cgroup hierarchy and configuration of the host, and thus other containers. Still, there are valid usecases to allow only specific containers to use the initial (host) cgroup namespace, especially for system diagnosis containers. For this, docker run introduced the CLI flag --cgroupns=[host|private] as of Docker-CE 20.0.0 and API 1.41.

Unfortunately, docker-compose doesn't offer any means to use the CLI flag functionality from docker compose files. This makes it impossible to use system diagnosis containers, such as cAdvisor, ... on cgroup v2 unified hierarchy-only systems anymore.

This situation would be solved by adding a cgroup field to service objects that would allow to specify the values of either "private" or "host", mirroring today's CLI flag functionality for handling the cgroup namespace creation for new containers.

services:
  foo:
    pid: host
    cgroup: host

huapox commented 2 years ago

+1

litnimax commented 2 years ago

+1

Answer 1 · 2022-01-14T20:16:18.000Z

I think this has become an important limitation for many users since Docker Desktop uses cgroupv2. Any plans of adding this?

Answer 2 · 2022-01-17T19:09:47.000Z

This require changes in the compose file format, please open a discussion on https://github.com/compose-spec/compose-spec

Answer 3 · 2022-01-17T22:02:09.000Z

already opened beginning of march 21: compose-spec/compose-spec#148

Answer 4 · 2022-01-18T07:10:19.000Z

@thediveo right. Would you like to create a PR to add this attribute to the compose spec?

Answer 5 · 2022-01-18T08:55:45.000Z

I would like to, but I'm currently completely tied up in (software) architectural work for my employer and as I'm yet lacking the required orientation in nerdctl's software architecture I wouldn't be of any help or use. What might be helpful in the future to onboard more hands to nerdctl might be some kind of sw orientation document to better understand where to add/plug-in various features and how nerdctl way of interacting with runc is (which is an area where I'm currently lost).

Answer 6 · 2022-01-27T19:36:06.000Z

+1 here. This would be huge. This is getting overwhelming as new docker desktop versions for windows no longer support cgroupv1, and mac is hanging by a thread with the settings.json deprecatedcgroupv1 change.

Staying on an older version of docker desktop for now...but who knows how long that'll last

I think this has become an important limitation for many users since Docker Desktop uses cgroupv2. Any plans of adding this?

Answer 7 · 2022-07-05T08:40:37.000Z

(bump) any E.T.A.?

Answer 8 · 2022-07-07T07:48:49.000Z

I was able to work around this limitation of compose by setting "default-cgroupns-mode": "host" in daemon.json.
Docker's documentation is misleading, since it claims that "host" is the default value, where in reality it defaults to "private".

So if you can change the daemon's defaults, this could be a valid workaround for the meantime.

Answer 9 · 2022-07-07T08:29:34.000Z

So if you can change the daemon's defaults, this could be a valid workaround for the meantime.

Unfortunately, that doesn't work for me/us. We have diagnosis containers that get deployed to production systems (not cloud production, but we're talking about real production systems for tangible goods) in order to check and diagnose workload-related issues that only tend to raise their ugly heads in production, but not in testing. And these are "sealed" Docker hosts where I simply cannot change the Docker daemon configuration. I fully agree with customers that this is an absolute "no-no-NO-NO-NOOOOO" 😀

Answer 10 · 2022-07-07T09:28:44.000Z

Then you could work around the issue by not using Docker Compose at all, but by passing --cgroupns=host to docker run.

Answer 11 · 2022-07-07T10:02:45.000Z

That's not an option either, as the sealed system enforces Docker compose for deployment. FYI, https://industrial-edge.io/

Answer 12 · 2022-11-17T19:14:41.000Z

Docker compose's lack of ability to configure containers is not "stale".

It's impact only grows.

Answer 13 · 2022-12-20T22:57:23.000Z

fixed by #10106

Answer 14 · 2023-01-05T10:53:45.000Z

I've seen #10106 was referenced in the changelog of Docker Compose 2.15.0, so I've done a quick check:

❯ /tmp/docker-compose version
Docker Compose version v2.15.0

❯ cat /tmp/docker-compose-test.yml
---
services:
  foo:
    image: ubuntu:jammy
    cgroup: host

❯ /tmp/docker-compose --file /tmp/docker-compose-test.yml ps 
services.foo Additional property cgroup is not allowed

I guess I'm missing something obvious here but I don't know what. Any ideas?

Answer 15 · 2023-01-05T11:05:38.000Z

indeed, something went wrong here. Investigating

Answer 16 · 2023-06-01T09:57:16.000Z

@ndeloof any news on this ?
I've just updated my docker desktop, coming with docker compose 2.18.1, and still:

services.foo Additional property cgroup is not allowed

Answer 17 · 2023-06-01T10:12:12.000Z

$ cat compose.yaml 
services:
   web:
      cgroup: host
      image: nginx
$ docker compose config
services:
  web:
    cgroup: host
    image: nginx
...

please check docker compose version

Answer 18 · 2023-06-01T10:19:07.000Z

@ndeloof thanks for your response :)

$ docker compose version
Docker Compose version v2.18.1

But i've just realized i use mutagen-compose as a wrapper to docker compose, which is stick on older versions (https://github.com/mutagen-io/mutagen-compose/blob/main/go.mod)

Sorry for the inconvenience, i will open an issue on their repo 😗

Answer 19 · 2023-11-02T12:57:41.000Z

Hey, I'm also experiencing an issue similiar to @carlosabalde and @nervo . I'm not sure if I haven't written group correctly as an element, but it seems good according to the docs.

# docker-compose.yaml
version: '3.8'
services:
   potato:
    build: .
    cgroup: host

I run the commands

$ docker compose version
Docker Compose version v2.23.0-desktop.1
$ docker-compose up --build
ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.potato: 'cgroup'

Does anyone know if I am doing something incorrectly?

Answer 20 · 2023-11-02T13:01:28.000Z

@EFinish your docker compose version is probably outdated. Check docker compose version

Answer 21 · 2023-11-02T13:14:40.000Z

@ndeloof I showed in my first post that I checked already. The version is 2.23.0. I was previously running 2.15 and I had the same issue nonetheless.

Answer 22 · 2023-11-02T13:48:32.000Z

@EFinish you mix both docker compose v1 ($ docker-compose) and v2 ($ docker compose) in your commands :)

Answer 23 · 2023-11-02T16:37:03.000Z

@EFinish You should indeed use docker compose to run commands, not docker-compose:

$ cat compose.yaml 
services:
   potato:
    build: .
    cgroup: host
 $ docker compose config
services:
  potato:
    build:
      context: /Users/nicolas/truc
      dockerfile: Dockerfile
    cgroup: host
...

Answer 24 · 2023-11-03T10:33:24.000Z

Yeah, that was it. Thanks for the help @ndeloof @nervo