(HTTP code 409) container stopped/paused - container ... is not running
mejenborg opened this issue · 2 comments
mejenborg commented
Expected Behaviour
That a database container was successfully launched
Actual Behaviour
It fails launching
Testcontainer Logs
testcontainers [DEBUG] Checking container runtime strategy "TestcontainersHostStrategy"... +0ms
testcontainers [DEBUG] Container runtime strategy "TestcontainersHostStrategy" is not applicable +1ms
testcontainers [DEBUG] Checking container runtime strategy "ConfigurationStrategy"... +0ms
testcontainers [DEBUG] Container runtime strategy "ConfigurationStrategy" is not applicable +0ms
testcontainers [DEBUG] Checking container runtime strategy "UnixSocketStrategy"... +0ms
testcontainers [TRACE] Fetching Docker info... +0ms
testcontainers [TRACE] Fetching remote container runtime socket path... +17ms
testcontainers [TRACE] Resolving host... +0ms
testcontainers [TRACE] Fetching Compose info... +0ms
testcontainers [TRACE] Looking up host IPs... +810ms
testcontainers [TRACE] Initialising clients... +18ms
testcontainers [TRACE] Container runtime info:
testcontainers {
testcontainers "node": {
testcontainers "version": "v22.11.0",
testcontainers "architecture": "arm64",
testcontainers "platform": "darwin"
testcontainers },
testcontainers "containerRuntime": {
testcontainers "host": "localhost",
testcontainers "hostIps": [
testcontainers {
testcontainers "address": "::1",
testcontainers "family": 6
testcontainers },
testcontainers {
testcontainers "address": "127.0.0.1",
testcontainers "family": 4
testcontainers }
testcontainers ],
testcontainers "remoteSocketPath": "/var/run/docker.sock",
testcontainers "indexServerAddress": "https://index.docker.io/v1/",
testcontainers "serverVersion": "27.3.1",
testcontainers "operatingSystem": "Docker Desktop",
testcontainers "operatingSystemType": "linux",
testcontainers "architecture": "aarch64",
testcontainers "cpus": 10,
testcontainers "memory": 8217968640,
testcontainers "runtimes": [
testcontainers "io.containerd.runc.v2",
testcontainers "runc"
testcontainers ],
testcontainers "labels": [
testcontainers "com.docker.desktop.address=unix:///Users/dkKasSoe/Library/Containers/com.docker.docker/Data/docker-cli.sock"
testcontainers ]
testcontainers },
testcontainers "compose": {
testcontainers "version": "2.30.3-desktop.1",
testcontainers "compatability": "v2"
testcontainers }
testcontainers } +1ms
testcontainers [DEBUG] Container runtime strategy "UnixSocketStrategy" works +1ms
testcontainers [DEBUG] Checking if image exists "mysql:8.0.31"... +1ms
testcontainers [DEBUG] Checked if image exists "mysql:8.0.31" +9ms
testcontainers [DEBUG] Image "mysql:8.0.31" already exists +1ms
testcontainers [DEBUG] Acquiring lock file "/var/folders/dt/y51xshd153zc7rm15rdhyqdc0000gp/T/testcontainers-node.lock"... +40ms
testcontainers [DEBUG] Acquired lock file "/var/folders/dt/y51xshd153zc7rm15rdhyqdc0000gp/T/testcontainers-node.lock" +4ms
testcontainers [DEBUG] Listing containers... +0ms
testcontainers [DEBUG] Listed containers +8ms
testcontainers [DEBUG] Creating new Reaper for session "7f5af60dfd7c" with socket path "/var/run/docker.sock"... +1ms
testcontainers [DEBUG] Checking if image exists "testcontainers/ryuk:0.5.1"... +0ms
testcontainers [DEBUG] Checked if image exists "testcontainers/ryuk:0.5.1" +7ms
testcontainers [DEBUG] Image "testcontainers/ryuk:0.5.1" already exists +0ms
testcontainers [DEBUG] Creating container for image "testcontainers/ryuk:0.5.1"... +0ms
testcontainers [DEBUG] [d25b93c75e8c] Created container for image "testcontainers/ryuk:0.5.1" +31ms
testcontainers [INFO] [d25b93c75e8c] Starting container for image "testcontainers/ryuk:0.5.1"... +0ms
testcontainers [DEBUG] [d25b93c75e8c] Starting container... +0ms
testcontainers [DEBUG] [d25b93c75e8c] Started container +92ms
testcontainers [INFO] [d25b93c75e8c] Started container for image "testcontainers/ryuk:0.5.1" +1ms
testcontainers [DEBUG] [d25b93c75e8c] Inspecting container... +0ms
testcontainers [DEBUG] [d25b93c75e8c] Inspected container +3ms
testcontainers [DEBUG] [d25b93c75e8c] Fetching container logs... +0ms
testcontainers [DEBUG] [d25b93c75e8c] Demuxing stream... +3ms
testcontainers [DEBUG] [d25b93c75e8c] Demuxed stream +0ms
testcontainers [DEBUG] [d25b93c75e8c] Fetched container logs +0ms
testcontainers [DEBUG] [d25b93c75e8c] Waiting for container to be ready... +0ms
testcontainers [DEBUG] [d25b93c75e8c] Waiting for log message "/.+ Started!/"... +0ms
testcontainers [DEBUG] [d25b93c75e8c] Fetching container logs... +0ms
testcontainers:containers [d25b93c75e8c] 2024/12/06 23:08:33 Pinging Docker... +0ms
testcontainers:containers [d25b93c75e8c] 2024/12/06 23:08:33 Docker daemon is available! +0ms
testcontainers:containers [d25b93c75e8c] 2024/12/06 23:08:33 Starting on port 8080... +0ms
testcontainers:containers [d25b93c75e8c] 2024/12/06 23:08:33 Started! +0ms
testcontainers [DEBUG] [d25b93c75e8c] Demuxing stream... +3ms
testcontainers [DEBUG] [d25b93c75e8c] Demuxed stream +0ms
testcontainers [DEBUG] [d25b93c75e8c] Fetched container logs +1ms
testcontainers [DEBUG] [d25b93c75e8c] Log wait strategy complete +2ms
testcontainers [INFO] [d25b93c75e8c] Container is ready +1ms
testcontainers [DEBUG] [d25b93c75e8c] Connecting to Reaper (attempt 1) on "localhost:55012"... +0ms
testcontainers [DEBUG] [d25b93c75e8c] Connected to Reaper +2ms
testcontainers [DEBUG] Releasing lock file "/var/folders/dt/y51xshd153zc7rm15rdhyqdc0000gp/T/testcontainers-node.lock"... +0ms
testcontainers [DEBUG] Released lock file "/var/folders/dt/y51xshd153zc7rm15rdhyqdc0000gp/T/testcontainers-node.lock" +1ms
testcontainers [DEBUG] Creating container for image "mysql:8.0.31"... +0ms
testcontainers:containers [d25b93c75e8c] 2024/12/06 23:08:33 New client connected: 172.17.0.1:57630 +12ms
testcontainers:containers [d25b93c75e8c] 2024/12/06 23:08:33 Adding {"label":{"org.testcontainers.session-id=7f5af60dfd7c":true}} +2ms
testcontainers [DEBUG] [39f675b64955] Created container for image "mysql:8.0.31" +23ms
testcontainers [INFO] [39f675b64955] Starting container for image "mysql:8.0.31"... +0ms
testcontainers [DEBUG] [39f675b64955] Starting container... +1ms
testcontainers [DEBUG] [39f675b64955] Started container +99ms
testcontainers [INFO] [39f675b64955] Started container for image "mysql:8.0.31" +0ms
testcontainers [DEBUG] [39f675b64955] Inspecting container... +0ms
testcontainers [DEBUG] [39f675b64955] Inspected container +2ms
testcontainers [DEBUG] [39f675b64955] Fetching container logs... +0ms
testcontainers [DEBUG] [39f675b64955] Demuxing stream... +5ms
testcontainers [DEBUG] [39f675b64955] Demuxed stream +0ms
testcontainers [DEBUG] [39f675b64955] Fetched container logs +0ms
testcontainers [DEBUG] [39f675b64955] Waiting for container to be ready... +0ms
testcontainers [DEBUG] [39f675b64955] Waiting for host port 55013... +1ms
testcontainers [DEBUG] [39f675b64955] Waiting for internal port 3306... +0ms
testcontainers:containers [39f675b64955] 2024-12-06 23:08:33+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.31-1.el8 started. +127ms
testcontainers [DEBUG] [39f675b64955] Host port 55013 ready +3ms
testcontainers [DEBUG] [39f675b64955] Host port wait strategy complete +0ms
testcontainers [TRACE] [39f675b64955] Port check result exit code 127: /bin/sh: nc: command not found +41ms
testcontainers [TRACE] [39f675b64955] Port check result exit code 1: /bin/bash: connect: Connection refused
testcontainers /bin/bash: /dev/tcp/localhost/3306: Connection refused +0ms
testcontainers:containers [39f675b64955] 2024-12-06 23:08:33+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' +72ms
testcontainers:containers [39f675b64955] 2024-12-06 23:08:33+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.31-1.el8 started. +4ms
testcontainers:containers [39f675b64955] 2024-12-06 23:08:33+00:00 [Note] [Entrypoint]: Initializing database files +69ms
testcontainers:containers [39f675b64955] 2024-12-06T23:08:33.806686Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead. +4ms
testcontainers:containers [39f675b64955] 2024-12-06T23:08:33.806729Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.31) initializing of server in progress as process 100 +0ms
testcontainers:containers [39f675b64955] mysqld: Error writing file '/var/lib/mysql/auto.cnf' (OS errno 28 - No space left on device) +1ms
testcontainers:containers [39f675b64955] 2024-12-06T23:08:33.809446Z 0 [ERROR] [MY-010076] [Server] Initialization of the server's UUID failed because it could not be read from the auto.cnf file. If this is a new server, the initialization failed because it was not possible to generate a new UUID. +0ms
testcontainers:containers [39f675b64955] 2024-12-06T23:08:33.809453Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it. +1ms
testcontainers:containers [39f675b64955] 2024-12-06T23:08:33.809456Z 0 [ERROR] [MY-010119] [Server] Aborting +0ms
testcontainers:containers [39f675b64955] 2024-12-06T23:08:33.809556Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.31) MySQL Community Server - GPL. +0ms
testcontainers [ERROR] [39f675b64955] The HostPortWaitStrategy will not work on a distroless image, use an alternate wait strategy +118ms
testcontainers [ERROR] [39f675b64955] Failed to exec container with command "/bin/bash -c </dev/tcp/localhost/3306": Error: (HTTP code 409) container stopped/paused - container 39f675b64955b7b5176795361fe5bf42a9a2cffe4a362d1f99ad78d49b02e78c is not running : +106ms
testcontainers [ERROR] [39f675b64955] Container failed to be ready: Error: (HTTP code 409) container stopped/paused - container 39f675b64955b7b5176795361fe5bf42a9a2cffe4a362d1f99ad78d49b02e78c is not running +0ms
testcontainers [DEBUG] [39f675b64955] Stopping container... +1ms
testcontainers [ERROR] [39f675b64955] Failed to exec container with command "/bin/sh -c cat /proc/net/tcp* | awk '{print $2}' | grep -i :0cea": Error: (HTTP code 409) container stopped/paused - container 39f675b64955b7b5176795361fe5bf42a9a2cffe4a362d1f99ad78d49b02e78c is not running : +1ms
testcontainers [ERROR] [39f675b64955] Failed to exec container with command "/bin/sh -c nc -vz -w 1 localhost 3306": Error: (HTTP code 409) container stopped/paused - container 39f675b64955b7b5176795361fe5bf42a9a2cffe4a362d1f99ad78d49b02e78c is not running : +2ms
testcontainers [DEBUG] [39f675b64955] Container already stopped +0ms
testcontainers [DEBUG] [39f675b64955] Removing container... +1ms
testcontainers [DEBUG] [39f675b64955] Removed container +5msSteps to Reproduce
I've reproduced the error in this repository https://github.com/mejenborg/testcontainers-node-fails
It's a clean NodeJS project, with typeorm, @nestjs/typeorm, mysql2, and @testcontainers/mysql packages installed.
When executing $ npm run test, all tests fails with (HTTP code 409) container stopped/paused - container 39f675b64955b7b5176795361fe5bf42a9a2cffe4a362d1f99ad78d49b02e78c is not running
Environment Information
- Operating System: Mac OS Sequoia 15.1.1
- Docker Version: 27.3.1
- Node version: 22.11.0
- NestJS version: 10.4.13
- Jest version: 29.7.0
- Testcontainers version: 10.16.0
gustavotrapp commented
Also have this problem, with a similar setup. Any updates
mejenborg commented
I was so determined that these two lines were the root cause:
testcontainers [TRACE] [39f675b64955] Port check result exit code 127: /bin/sh: nc: command not found +41ms
testcontainers [TRACE] [39f675b64955] Port check result exit code 1: /bin/bash: connect: Connection refused
But I noticed testcontainers:containers [39f675b64955] mysqld: Error writing file '/var/lib/mysql/auto.cnf' (OS errno 28 - No space left on device) +1ms, which didn't really make sense to me, as I have over 600GB available on my disk. So I removed all unused docker images, which amounted to ~60gb, using the $ docker system prune command - now it works!
I wasn't aware that Docker had a soft limit; a "Virtual disk limit" of 64 GB, which can be found under Settings -> Resources -> Virtual disk limit.
So now I know!