docker/compose

Port conflict with multiple "host:<port range>:port" services

andrei-m opened this issue · 30 comments

Description of the issue

If a compose file defines multiple services that share an overlapping port range, a port conflict occurs when docker-compose up is executed. This behavior started to happen for me when upgrading to Docker Desktop 2.2.0 (Stable) for Mac. The version I was previously running (I believe 2.1.5) was able to select distinct ports for the two services without conflicting. Here is a POC Docker Compose file:

version: '2.1'
services:
  postgres-foo:
    image: postgres
    ports:
      - "127.0.0.1:32768-61000:5432"

  postgres-bar:
    image: postgres
    ports:
      - "127.0.0.1:32768-61000:5432"

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.25.2, build 698e2846
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

services:
  postgres-bar:
    image: postgres
    ports:
    - 127.0.0.1:32768-61000:5432/tcp
  postgres-foo:
    image: postgres
    ports:
    - 127.0.0.1:32768-61000:5432/tcp
version: '2.1'

Steps to reproduce the issue

  1. docker-compose up

Observed result

Services fail to start and a port conflict error occurs

Expected result

Services start up. An available port within the overlapping range is selected for each service.

Stacktrace / full error message

docker-compose up
Starting docker-bug-poc_postgres-foo_1 ...
Starting docker-bug-poc_postgres-bar_1 ... error
Starting docker-bug-poc_postgres-foo_1 ... error
ERROR: for docker-bug-poc_postgres-bar_1  Cannot start service postgres-bar: Ports are not available: listen tcp 127.0.0.1:32768: bind: address already in use

ERROR: for docker-bug-poc_postgres-foo_1  Cannot start service postgres-foo: Ports are not available: listen tcp 127.0.0.1:37587: socket: too many open files

ERROR: for postgres-bar  Cannot start service postgres-bar: Ports are not available: listen tcp 127.0.0.1:32768: bind: address already in use

ERROR: for postgres-foo  Cannot start service postgres-foo: Ports are not available: listen tcp 127.0.0.1:37587: socket: too many open files
ERROR: Encountered errors while bringing up the project.

Additional information

I uploaded diagnostic information with the ID 37C41A27-71E7-4431-AFBE-5DA0EEC74A3C/20200126225028

I am experiencing a similar issue that may come from the same cause.
The context is the same: upgrading to Docker Desktop 2.2.0 on MacOS breaks the port publishing when trying to run an NFS server container.
To reproduce:

docker run -d --privileged --restart=always -v /exported_folder:/exported_folder -e NFS_EXPORT_DIR_1=/exported_folder -e NFS_EXPORT_DOMAIN_1=\* -e NFS_EXPORT_OPTIONS_1=rw,no_root_squash -p 111:111 -p 111:111/udp -p 2049:2049 -p 2049:2049/udp -p 32765:32765 -p 32765:32765/udp -p 32766:32766 -p 32766:32766/udp -p 32767:32767 -p 32767:32767/udp fuzzle/docker-nfs-server:latest

The error shown:

docker: Error response from daemon: driver failed programming external connectivity on endpoint <CONTAINER_NAME> (ad6b1dccbfe77e3708e687b5eed9311cf1a6f828e36c739f18f5daa5803461b7): Error starting userland proxy: listen tcp 0.0.0.0:111: bind: address already in use

I have already taken down every component related to NFS on the Mac itself, and nothing is occupying that port.
The output of sudo lsof -iTCP -sTCP:LISTEN -n -P:

COMMAND    PID    USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
launchd      1    root    8u  IPv4 0xd9679bbeeddece79      0t0  TCP *:22 (LISTEN)
launchd      1    root    9u  IPv6 0xd9679bbeedde7269      0t0  TCP *:22 (LISTEN)
launchd      1    root   11u  IPv4 0xd9679bbeeddece79      0t0  TCP *:22 (LISTEN)
launchd      1    root   14u  IPv6 0xd9679bbeedde7269      0t0  TCP *:22 (LISTEN)
dnscrypt-  955  nobody    8u  IPv4 0xd9679bbef0fb0e79      0t0  TCP 127.0.0.1:53 (LISTEN)
com.docke 1190 <USERNAME>    8u  IPv4 0xd9679bbef5ad2801      0t0  TCP 127.0.0.1:49273 (LISTEN)

Just downgrading to Docker Desktop 2.1.x solves everything.
Does anybody have any idea for a workaround?

Wondering how this would've worked before, as it wouldn't be possible to have two processes listening on the same port 🤔

@thaJeztah should I file a separate issue, as it is not directly related to Docker compose?

Wondering how this would've worked before, as it wouldn't be possible to have two processes listening on the same port 🤔

Also when you add scale and run the command multiple times, it will create more containers than the "scale" value. For example when I ran it with --scale=3 for the first time it creates one container successfully and two containers fail with port conflicts. Second time 2 containers success 1 fails, 3rd time 2 success 1 fails. when I execute docker ps it shows 5 containers.
Note: I also have port range in my compose file. Before the 2.2.0 update It worked perfectly and created 3 containers and assigned ports within the given range.

Any news on this? After upgrading docker on mac to > 2.0 my compose script using port ranges on scalable service also fails. It works perfectly on docker < 2.0.

@MartinLyne Did you find any solutions for this?

Same problem on MacOS with Docker Desktop v3.1.0 (51484) with the following docker-compose.yml:

version: '3.5'

services:
  zookeeper:
    image: strimzi/kafka:0.19.0-kafka-2.5.0
    command:
      - sh
      - -c
      - bin/zookeeper-server-start.sh config/zookeeper.properties
    ports:
      - "2181-2182:2181"
    environment:
      LOG_DIR: /tmp/logs
  
  kafka:
    image: strimzi/kafka:0.19.0-kafka-2.5.0
    command:
      - sh
      - -c
      - bin/kafka-server-start.sh config/server.properties 
        --override listeners=$${KAFKA_LISTENERS} 
        --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} 
        --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT} 
        --override num.partitions=$${KAFKA_NUM_PARTITIONS}
    depends_on:
      - zookeeper
    ports:
      - "9092-9094:9092"
    environment:
      LOG_DIR: /tmp/logs
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_NUM_PARTITIONS: 3

First call fails, second call works:

    # first call fails
    docker-compose up -d --scale kafka=3 --scale zookeeper=2 kafka
    Creating network "kafka_default" with the default driver
    WARNING: The "zookeeper" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
    Creating kafka_zookeeper_1 ... error
    Creating kafka_zookeeper_2 ... done

    ERROR: for kafka_zookeeper_1  Cannot start service zookeeper: Ports are not available: listen tcp 0.0.0.0:2181: bind: address already in use

    ERROR: for zookeeper  Cannot start service zookeeper: Ports are not available: listen tcp 0.0.0.0:2181: bind: address already in use
    ERROR: Encountered errors while bringing up the project.

    # second call works
    docker-compose up -d --scale kafka=3 --scale zookeeper=2 kafka
    WARNING: The "zookeeper" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
    Starting kafka_zookeeper_1 ... done
    WARNING: The "kafka" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
    Creating kafka_kafka_1     ... done
    Creating kafka_kafka_2     ... done
    Creating kafka_kafka_3     ... done

So I tried: docker-compose up -d kafka && docker-compose up -d --scale kafka=3 --scale zookeeper=2 kafka and this works as well. It seems there is a problem with initial start & scaling and the same time.

The error occurs whenever a services which exposes a host port range is scaled, e. g.

version : "3"

services:
  hello:
    image: tutum/hello-world
    ports:
      - "8080-8082:80"

Running 3 instances of this services with

docker-compose up --scale hello=3

results in a port conflict. Running this command three times resolves the problem.

I use docker version 19.03.12 and docker-compose 1.29.0 on Windows 10 (WSL 2).

mac89 commented

Still occurs on docker version v20.10.7, docker-compose 1.29.2 on Windows 10 (WSL 2).

Same issue with latest versions in Win 10 (WSL 2)

Similar issue on Docker v20.10.7 and docker-compose v2.0.0 on Windows 10 (WSL 2).

Compose File

version: "3.8"

services:
  app:
    image: mishroleapp
    environment:
      MONGO_URL: "mongodb://mongodb:27017/test"
    depends_on:
      - mongodb
    ports:
      - "3000-3001:3000"

  mongodb:
    image: mongo

docker-compose up

[+] Running 2/4
 - Network docker_default      Created                                                                                                                            0.2s
 - Container docker_mongodb_1  Started                                                                                                                            5.4s
 - Container docker_app_2      Starting                                                                                                                           4.9s
 - Container docker_app_1      Starting                                                                                                                           4.9s
Error response from daemon: Ports are not available: listen tcp 0.0.0.0:3000: bind: Solo se permite un uso de cada dirección de socket (protocolo/dirección de red/puerto)

docker-compose ps

NAME                COMMAND                  SERVICE             STATUS              PORTS
docker_app_1        "docker-entrypoint.s…"   app                 running             0.0.0.0:3001->3000/tcp, :::3001->3000/tcp, 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp
docker_app_2        "docker-entrypoint.s…"   app                 created
docker_mongodb_1    "docker-entrypoint.s…"   mongodb             running             27017/tcp

Updates? 😪

dr3x commented

Same issue pulling down the most recent Confluent platform binaries:
https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html
curl --silent --output docker-compose.yml
https://raw.githubusercontent.com/confluentinc/cp-all-in-one/7.0.0-post/cp-all-in-one/docker-compose.yml

docker-compose up -d

ERROR: for zookeeper Cannot start service zookeeper: Ports are not available: listen tcp 0.0.0.0:2181: bind: An attempt was made to access a socket in a way forbidden by its access permissions.

Issue happens with both Docker Desktop 3.0.0 / 4.2.0

Update
After upgrading and rebooting machine, was able to launch the instance. So may be a timing issue with initial startup.

Here's a snippet of the yaml with the definitions:

`

version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:7.0.0
hostname: zookeeper
container_name: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000

broker:
image: confluentinc/cp-server:7.0.0
hostname: broker
container_name: broker
depends_on:
- zookeeper
ports:
- "9092:9092"
- "9101:9101"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
`

doesnt work on:

having the same issue on Docker Desktop 4.3.2 (72729) win10 2004
both with compose v2 and when disabling v2
came from this issue trying to help this guy with this issue
https://stackoverflow.com/questions/70516739/how-to-configure-port-mapping-for-replicated-containers-in-docker-compose

works on:

rocky linux 8.5
docker:

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:22 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:44 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker-compose:

docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

compose file:

version : "3"
services:
  web:
    image: "nginx:latest"
    ports:
      - "8000-8020:80"

docker command:

docker-compose up --scale web=5

Still not working:

uname -a
# => Darwin my-username 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:45:05 PDT 2022; root:xnu-8020.101.4~15/RELEASE_X86_64 x86_64

docker --version
# => Docker version 20.10.13, build a224086

docker-compose --version
# => docker-compose version 1.29.2, build 5becea4c

docker-compose.yml

version: '3.7'
services:
  solr:
    image: solr:latest
    ports:
      - 8983-8993:8983

scaling fails:

docker-compose up --scale solr=5
Creating network "solr_plugins_default" with the default driver
WARNING: The "solr" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating solr_plugins_solr_1 ... done
Creating solr_plugins_solr_2 ... error
Creating solr_plugins_solr_3 ... done
Creating solr_plugins_solr_4 ... done
Creating solr_plugins_solr_5 ... done

ERROR: for solr_plugins_solr_2  Cannot start service solr: Ports are not available: listen tcp 0.0.0.0:8984: bind: address already in use

ERROR: for solr  Cannot start service solr: Ports are not available: listen tcp 0.0.0.0:8984: bind: address already in use
ERROR: Encountered errors while bringing up the project.

I'm also experiencing this same issue for my microservices project. I can't scale my services due to port conflicts. I have the range 40000 to 41000, and I just have to get lucky to get the three instances of the service running.

Windows 10 Version 21H2 (Build 19044.1766)
Docker version 20.10.16, build aa7e414
docker-compose version 1.29.2, build 5becea4c

Still having the same issue:

docker-compose.yaml

version: "3.7"
services:
  jobs:
    image: coralproject/talk:7
    restart: always
    ports:
      - "5000-5010:5000"
    environment:
      - MONGODB_URI=${MONGODB_URI:-mongodb://host.docker.internal:27017/coral}
      - REDIS_URI=${REDIS_URI:-redis://host.docker.internal:6379}
      - SIGNING_SECRET=${SIGNING_SECRET}

running docker-compose up -d --scale jobs=5 results in Error response from daemon: driver failed programming external connectivity on endpoint job-pods-mise-en-place-jobs-3 (1dc2c980ae5558438c074a08fcb109762573957eca2f9930f45eb1c67b5a9b0c): Bind for 0.0.0.0:5004 failed: port is already allocated

But running docker-compose ps -a doesn't show any containers using that port (and nothing else on my machine is). This also happens after changing the port range.

Still having the same issue:

version: '3.8'

networks:
  app-tier:
    driver: bridge

services:
  redis:
    image: docker.io/bitnami/redis:7.0.5
    environment:
      - REDIS_REPLICATION_MODE=master
      - REDIS_PASSWORD=bitnami
    networks:
      - app-tier
    ports:
      - '6379'
  redis-slave:
    image: docker.io/bitnami/redis:7.0.5
    environment:
      - REDIS_REPLICATION_MODE=slave
      - REDIS_MASTER_HOST=redis
      - REDIS_MASTER_PASSWORD=bitnami
      - REDIS_PASSWORD=bitnami
    ports:
      - '6379'
    depends_on:
      - redis
    networks:
      - app-tier
  redis-sentinel:
    image: docker.io/bitnami/redis-sentinel:7.0.5
    environment:
      - REDIS_MASTER_PASSWORD=bitnami
    depends_on:
      - redis
      - redis-slave
    ports:
      - '26379-26381:26379'
    networks:
      - app-tier
docker compose up --scale redis-sentinel=3

image

Same issue here on both macOS and linux.

Has anyone from Docker org actually commented on this? possibly providing some work around?
Using scale or deploy/replicas produces the same thing

  memcached:
    image: "memcached-1.6"
    deploy:
      mode: replicated
      replicas: 3
    ports:
      - "10400-10409:11211-11220"
# uname -a
Linux ip-10-200-77-70.ec2.internal 5.4.94 #1 SMP Mon Jun 27 21:13:52 UTC 2022 x86_64 Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz GenuineIntel GNU/Linux

 # docker-compose -v
docker-compose version 1.27.3, build 4092ae5d
# docker -v
Docker version 19.03.15, build 99e3ed8

# docker ps |grep memcache
901aaf04ec43        memcached-1.6                                     "docker-entrypoint.s…"   22 seconds ago      Up 20 seconds           0.0.0.0:10400->11211/tcp, 0.0.0.0:10401->11212/tcp, 0.0.0.0:10402->11213/tcp, 0.0.0.0:10403->11214/tcp, 0.0.0.0:10404->11215/tcp, 0.0.0.0:10405->11216/tcp, 0.0.0.0:10406->11217/tcp, 0.0.0.0:10407->11218/tcp, 0.0.0.0:10408->11219/tcp, 0.0.0.0:10409->11220/tcp   app_memcached_2

and the same on mac:

❯ uname -a
Darwin C02GN1CC1PG2 21.6.0 Darwin Kernel Version 21.6.0: Thu Sep 29 20:12:57 PDT 2022; root:xnu-8020.240.7~1/RELEASE_X86_64 x86_64

❯ docker -v
Docker version 20.10.14, build a224086

❯ docker-compose -v
docker-compose version 1.29.2, build 5becea4c

@thaJeztah this seems to be the same race condition issue we discussed on #10067
Shall we apply a comparable workaround to start containers sequentially to workaround race condition in libnetwork?

As suggest by @ndeloof, introducing delays may mitigate the problem.
I had the same issue resolved by update_config:

deploy:
      mode: replicated
      replicas: 3
      update_config:
        delay: 1s

Configure delay seconds based on your container startup.

@robertoporceddu that's weird you get this fixed this way, as Docker Compose doesn't use update_config to create containers

Still same problem here and update_config didn't help 😭

order-service:
    build:
      context: ./
      dockerfile: ./Dockerfile.dev
    deploy:
      replicas: 2
    ports:
      - 8000-8010:8000

docker-compose up -d

Error response from daemon: Ports are not available: exposing port TCP 0.0.0.0:8000 -> 0.0.0.0:0: listen tcp 0.0.0.0:8000: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

docker --version

Docker version 20.10.21, build baeda1f

docker-compose --version

Docker Compose version v2.12.2

wsl --version

WSL version: 1.2.5.0
Kernel version: 5.15.90.1

If I create replicas of a container, and use a port range, I still get that same error message:

Compose file excerpt:

    ports:
      - target: 3000
        published: "3000-3010"
        mode: host
    deploy:
      mode: replicated
      replicas: 3
      restart_policy:
        condition: any
        delay: 5s
        window: 120s
      update_config:
        delay: 5s
        failure_action: rollback
        max_failure_ratio: .5
        monitor: 5s
        parallelism: 1

Error message:

Error response from daemon: Ports are not available: exposing port TCP 0.0.0.0:3001 -> 0.0.0.0:0: listen tcp 0.0.0.0:3001: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

update_config delay made no difference.

I would expect this to be a pretty normal use case, is there no way to run replicas on one host without port conflicts?

If I create replicas of a container, and use a port range, I still get that same error message:

Compose file excerpt:

    ports:
      - target: 3000
        published: "3000-3010"
        mode: host
    deploy:
      mode: replicated
      replicas: 3
      restart_policy:
        condition: any
        delay: 5s
        window: 120s
      update_config:
        delay: 5s
        failure_action: rollback
        max_failure_ratio: .5
        monitor: 5s
        parallelism: 1

Error message:

Error response from daemon: Ports are not available: exposing port TCP 0.0.0.0:3001 -> 0.0.0.0:0: listen tcp 0.0.0.0:3001: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

update_config delay made no difference.

I would expect this to be a pretty normal use case, is there no way to run replicas on one host without port conflicts?

I found a setting that works, at least for the abovementioned problem (replicas causing port conflict). Just leave the published ports unassigned (host-side ports), which forces Docker itself to randomly assign them. In other words:

Works:

    ports:
      - target: 3000

or:

    ports: "3000-3010"

Does Not Work:

    ports:
      - target: 3000
        published: "3000-3010"

or:

    ports: "3000-3010:3000-3010"

or:

    ports: "3000:3000-3010"
hwgn commented

I found a setting that works, at least for the abovementioned problem (replicas causing port conflict). Just leave the published ports unassigned (host-side ports), which forces Docker itself to randomly assign them. In other words:
[...]

    ports:
      - target: 3000

This actually worked for me, thank you! All the previous approaches did sadly not. There was one fix I needed to do, which was to specify the protocol, otherwise the communication via node failed:

ports:
  - target: 3000
    protocol: tcp

Leaving protocol undefined shouldn't cause any issues (as this, according to documentation, simply allows any protocol), but specifying it was the only way to make it work for me.

jbvsmo commented

Hi any news on this issue? There are no good workarounds for us

Hi. Same problem, anything new about this?

8000-8010:80 syntax already tells engine to select a random available port within the 8000-8010 range
since #10067 (v2.14.1) we start containers sequentially to workaround race condition in engine selecting such a port.
Can you please confirm this issue persists with a recent release ?

I am using version 2.23.0 and the problem persists