docker/compose

[BUG] service can't be used with `extends` as it declare `depends_on`

r3nor opened this issue Β· 47 comments

r3nor commented

Description

When trying to bring up a compose project that makes use of extends feature, I am getting the following error:

service "nginx" can't be used with `extends` as it declare `depends_on`

Steps To Reproduce

  1. Have a docker-compose.yml project with extends, like this:
nginx:
    extends:
      file: ./nginx/compose.yml
      service: nginx
    container_name: "${SERVER_NAME:}-nginx"
    env_file: .env

and the nginx/compose.yml:

version: "3.9"

services:
  nginx:
    image: nginx:1
    restart: unless-stopped
    depends_on:
      - web_app
    volumes:
      - ./html:/var/www/html
  1. docker compose up

  2. See error: service "nginx" can't be used with extends as it declare depends_on

Compose Version

2.24.6

Docker Environment

N/A

Anything else?

No response

This seems to be a new restriction added in 2.24.6. We have an extends combined with depends_on and everything was working fine with 2.24.5 and previous versions. Not sure why this wasn't flagged as a breaking change?

r3nor commented

Yeah, same for me. Was working fine on 2.24.5 and broke on this new 2.24.6 release.

This makes reusing blocks much harder, as fragments cannot be used across files. As far as I know, using extends was the only way to have reusable depends_on blocks while allowing to separate services in multiple files and have a common file to extend from.

restriction already was implemented in 2.23.x, new compose parser used in 2.24.x prior to latest release was just missing this check.

@meriouma maybe you want to check out include: https://docs.docker.com/compose/compose-file/14-include/

That is not sufficient. It is common to do stuff like have an image like "background worker" and then have 3 instances of it - high-pri, medium-pri, and low-pri. You could then have the base depend on stuff like your DB and message bus and such, then just have 3 instances that extend the base. Now you have to repeat those dependencies over and over in the instantiations. This is definitely more degradation in compose.

We have used Compose for development purposes with a Kubernetes cluster for deployment. It deeply feels like Compose has been losing functionality while Kubernetes has been gaining it, and Kind is becoming more and more compelling to replace Compose from these sorts of things.

encountering this too. why is a breaking change being included in a patch release at all?

what's also frustrating here is that everything used to work as-intended before compose 2.24 and its use of a not-even-prod-ready compose-go project, but there wasn't really a clear alert that everything was going to break.

@davidlougheed compose-go/v2 is not "not-even-prod-ready". We released as RC as we anticipated the need for some additional refactoring, and don't want third party library to suffer from API changes.

THAT BEING SAID: as compose-go includes a consistency check, it would be able to detect reference to another service is orphaned after being used by extends, such a check docker compose v1 didn't implemented (so this rule in the compose file format, inherited by the compose specification). With the updated architecture of compose-go/v2 we can now relax this constraint, and offer more flexibility to users (which still should read the docs before they complain about a feature :trollface:). I'll write a proposal in the compose-spec repo for this purpose, so we can get this both fixed in the docs and in the code

This is the expected behavior, see https://github.com/compose-spec/compose-spec/blob/master/05-services.md#restrictions

@ndeloof So, what is the supported way of extending services that depend on other services?

@Leonid99 there's no way currently

It's possible to merge depends_on "later":

$ find *.yaml -print -exec cat {} \;
compose.yaml
include:
  - path:
      - defs.yaml
      - deps.yaml
defs.yaml
services:
  x:
    image: x

  y:
    image: y
#    depends_on:
#      - x

  z:
   extends: y
deps.yaml
services:
  y:
    depends_on:
      - x
$ docker compose config
name: deps
services:
  x:
    image: x
    networks:
      default: null
  "y":
    depends_on:
      x:
        condition: service_started
        required: true
    image: "y"
    networks:
      default: null
  z:
    image: "y"
    networks:
      default: null
networks:
  default:
    name: deps_default

... and if you want to "inherit" the depends_on, deps.yaml could be like this:

services:
  y: &idem
    depends_on:
      - x
  z: *idem

Got this error with ubuntu update

from docker-compose-plugin (2.24.6-1ubuntu.22.04jammy)
to (2.24.5-1ubuntu.22.04jammy)

It's quite straightforward to see why is was not supposed to work if you consider such a case:

With a compose.yaml like this:

services:
  service:
    extends:
      file: file.yaml
      service: service

that extends a service defined in a file.yaml like this:

services:
  hidden:
    image: hidden

  service:
    image: image
    depends_on:
      - hidden

sure, but the same applies to other resources, without such a constraint:

file.yaml

services:
  service:
    image: image
    volumes:
      - hidden:/work

volumes:
  hidden: ...

about that, extending services defined like this: https://docs.docker.com/compose/compose-file/05-services/#ipv4_address-ipv6_address doesn't work either, but it's not caught by checks if there's a homonym network around

that's my point: consistency check just validate there's a resource for any declared reference, whenever this might not have been declared in the original compose file. As we already do for networks/volumes, I don't see any reason we don't do the same for dependent services

hmm I removed my extends in favour of includes + merging and now I get bizarre network errors where services are not joining some (but not all) of the declared networks, leading to name resolution errors. maybe related, or maybe not. curious if anyone else encounters this.

Edit: ah no it's unrelated, I'm also encountering a different compose-go bug: #11533

This is the cleanest solution I found for this issue:

services:
  app-abstract:
    image: nginx
    profiles: [abstract]
  app-1:
    extends: app-abstract
    profiles: !reset []
    depends_on: ...
  app-2:
    extends: app-abstract
    profiles: !reset []
    depends_on: ...

But it's far from ideal having to add and reset profiles like this, not to mention VS Code does not like the !reset thing.

This is to say that there is no clean solution for this issue, one that would still allow to extend services within the same compose file without having to create extra compose files.

Perhaps, @ndeloof, you can consider allowing services to be marked as abstract: true. Such services would not be rendered by docker compose, they'd rather just be useful for other services extend from.

Does the change mean that affected depends_on never actually worked and can therefore be safely removed? What did compose previously do when it encountered such an depends_on?

Extending depends_on was working before this change.

rbax commented

I want to reuse another developers service without making them change their code. If I 'include' I am stuck with their implementation. why can't I just overwrite with my own 'depends_on' and move on?

BTW I don't think you were able to overwrite depends_on before. What you happen is that their depends_on would be merged with yours.

@ndeloof please take a look how many linked issues this change produced. I vote for reverting this behavior. Мy company also has tens of projects with currently broken depends_on

This issue was closed two weeks ago as not planned.

Shouldn't it be reopened, now that the relevant restriction in compose-spec has been lifted?

Thanks @ndeloof for fixing the spec.

The change broke my project as I have the base docker-compose.yaml which declares my API server (which depends_on a database image) built from the project source (ie: compiling code) and a .devcontainer/docker-compose-dev.yaml which extends the base service, but changes the build.target so that an image is used that has all the developer dependencies and mounts the source files.

This used to work so that the developer image/container would still depend on the DB which worked well for the development team.

Now that the spec has been updated, is there a projected timeline on updating the compose project to meet the revised spec?

will be part of tomorrow's release

Thanks for the fast update @ndeloof!

@r3nor Try now, latest published version fixed it for me (2.25.0).

$ docker compose version
Docker Compose version v2.25.0
$ docker --version
Docker version 26.0.0, build 2ae903e

Thanks @ndeloof !

Hello there, do we have an ETA for the version 2.25 to be released on Docker Desktop? My company heavily depends on this fix, for now I workaround the issue by download docker-compose instead and working with it, but the entire company is trained on the platform to use docker compose - would be nice to have the fix in place.

@ndeloof does this mean we should stop extending services as this feature will be deprecated in future?

@matthiasPOE no, we relaxed this restriction in the spec compose-spec/compose-spec#470
issue is closed by release v2.26.0

Hello there, do we have an ETA for the version 2.25 to be released on Docker Desktop? My company heavily depends on this fix, for now I workaround the issue by download docker-compose instead and working with it, but the entire company is trained on the platform to use docker compose - would be nice to have the fix in place.

I was looking for the directory where this plugin is located to replace it manually. After several unsuccessful attempts, I replaced docker-compose in all found directories and one of them worked. I don't know which one, but you can also try it yourself.

https://github.com/docker/compose#linux


Don't rely on the about page, it still shows the old version:

obraz

docker compose version
# Docker Compose version v2.26.0
/usr/lib/docker/cli-plugins/docker-compose

@gander are you referring to OSX Docker Desktop? That location does not exist in my version of Sonoma.

@Morriz

@gander are you referring to OSX Docker Desktop? That location does not exist in my version of Sonoma.

No, my Linux Mint location. Check other from https://github.com/docker/compose or use find or command to locate your docker-compose

Yeah, tnx. I was able to follow the original install instructions from Docker.

Any idea about when the fix will be released in OSX Docker Desktop?

should happen in a few days

In case someone else is facing issues because Docker Desktop is still running an old version of the docker-compose plugin, I've found this useful video explaining how to easily upgrade the docker-compose plugin used by Docker Desktop.

@beni0888 you didn't link the video πŸ™‚

You're right! Fixed!!