error on launching docker-compose by piping to sh ( echo 'docker-compose ... ' | sh )
keyolk opened this issue Β· 38 comments
I've a application which has multi container. to easy installation, I decide to use package image which contains docker-comopse.yaml file.
what I expected is like below
$ docker run my_application install | sh
this will pull all related image
$ docker run my_application up | sh
this will do "docker-compose up" and some intial job for application using "docker-compose exec"
But I got below error
$ echo "docker-compose exec cassandra cqlsh -e 'desc keyspaces'" | sh
Traceback (most recent call last):
File "<string>", line 3, in <module>
File "compose/cli/main.py", line 57, in main
File "compose/cli/main.py", line 108, in perform_command
File "compose/cli/main.py", line 353, in exec_command
File ".tox/py27/lib/python2.7/site-packages/dockerpty/pty.py", line 338, in start
File ".tox/py27/lib/python2.7/site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1
and I confiremd that below two commands works well
$ echo "docker exec my_application_cassandra_1 cqlsh -e 'desc keyspaces'" | sh
system_traces system
$ sh -c "docker-compose exec cassandra cqlsh -e 'desc keyspaces'"
system_traces system
Does someone have any idea why the error comes?
below is my version info and logs from docker-compose
$ docker version
Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Sat Mar 12 19:18:57 2016
OS/Arch: linux/amd64
Server:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Sat Mar 12 19:18:57 2016
OS/Arch: linux/amd64
$ docker-compose version
docker-compose version 1.7.0rc1, build 1ad8866
docker-py version: 1.8.0-rc2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
$ sh -c "docker-compose --verbose exec cassandra cqlsh -e 'desc keyspaces'"
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.auth.auth.load_config: Found 'auths' section
docker.auth.auth.parse_auth: Found entry (registry=u'dtr.test.org', username=u'keyolk')
docker.auth.auth.parse_auth: Found entry (registry=u'test.org', username=u'$oauthtoken')
compose.cli.command.get_client: docker-compose version 1.7.0rc1, build 1ad8866
docker-py version: 1.8.0-rc2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
compose.cli.command.get_client: Docker base_url: http+docker://localunixsocket
compose.cli.command.get_client: Docker version: KernelVersion=4.3.2-0-vanilla, Os=linux, BuildTime=2016-03-12T19:18:57.542338402+01:00, ApiVersion=1.22, Version=1.10.3, GitCommit=20f81dd, Arch=amd64, GoVersion=go1.5.3
compose.cli.verbose_proxy.proxy_callable: docker containers <- (filters={u'label': [u'com.docker.compose.project=insator', u'com.docker.compose.service=cassandra', u'com.docker.compose.oneoff=False', u'com.docker.compose.container-number=1']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker exec_create <- (u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253', ['cqlsh', '-e', 'desc keyspaces'], tty=True, stdin=True, privileged=False, user=None)
compose.cli.verbose_proxy.proxy_callable: docker exec_create -> {u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'}
compose.cli.verbose_proxy.proxy_callable: docker exec_start <- ({u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'}, tty=True, socket=True)
compose.cli.verbose_proxy.proxy_callable: docker exec_start -> <socket object, fd=7, family=1, type=1, protocol=0>
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect <- ({u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'})
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect -> {u'CanRemove': False,
u'ContainerID': u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253',
u'DetachKeys': u'',
u'ExitCode': None,
u'ID': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725',
u'OpenStderr': True,
u'OpenStdin': True,
u'OpenStdout': True,
u'ProcessConfig': {u'arguments': [u'-e', u'desc keyspaces'],
u'entrypoint': u'cqlsh',
...
compose.cli.verbose_proxy.proxy_callable: docker exec_resize <- ({u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'}, width=190, height=42)
compose.cli.verbose_proxy.proxy_callable: docker exec_resize -> None
system_traces system
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect <- ({u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'})
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect -> {u'CanRemove': False,
u'ContainerID': u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253',
u'DetachKeys': u'',
u'ExitCode': 0,
u'ID': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725',
u'OpenStderr': True,
u'OpenStdin': True,
u'OpenStdout': True,
u'ProcessConfig': {u'arguments': [u'-e', u'desc keyspaces'],
u'entrypoint': u'cqlsh',
...
echo "docker-compose --verbose exec cassandra cqlsh -e 'desc keyspaces'" | sh [2/22580]
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.auth.auth.load_config: Found 'auths' section
docker.auth.auth.parse_auth: Found entry (registry=u'dtr.test.org', username=u'keyolk')
docker.auth.auth.parse_auth: Found entry (registry=u'test.org', username=u'$oauthtoken')
compose.cli.command.get_client: docker-compose version 1.7.0rc1, build 1ad8866
docker-py version: 1.8.0-rc2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
compose.cli.command.get_client: Docker base_url: http+docker://localunixsocket
compose.cli.command.get_client: Docker version: KernelVersion=4.3.2-0-vanilla, Os=linux, BuildTime=2016-03-12T19:18:57.542338402+01:00, ApiVersion=1.22, Version=1.10.3, GitCommit=20f81dd, A$ch=amd64, GoVersion=go1.5.3
compose.cli.verbose_proxy.proxy_callable: docker containers <- (filters={u'label': [u'com.docker.compose.project=insator', u'com.docker.compose.service=cassandra', u'com.docker.compose.oneo$f=False', u'com.docker.compose.container-number=1']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker exec_create <- (u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253', ['cqlsh', '-e', 'desc keyspaces'], tty=True, stdin=True$ privileged=False, user=None)
compose.cli.verbose_proxy.proxy_callable: docker exec_create -> {u'Id': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01'}
compose.cli.verbose_proxy.proxy_callable: docker exec_start <- ({u'Id': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01'}, tty=True, socket=True)
compose.cli.verbose_proxy.proxy_callable: docker exec_start -> <socket object, fd=7, family=1, type=1, protocol=0>
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect <- ({u'Id': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01'})
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect -> {u'CanRemove': False,
u'ContainerID': u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253',
u'DetachKeys': u'',
u'ExitCode': None,
u'ID': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01',
u'OpenStderr': True,
u'OpenStdin': True,
u'OpenStdout': True,
u'ProcessConfig': {u'arguments': [u'-e', u'desc keyspaces'],
u'entrypoint': u'cqlsh',
...
compose.cli.verbose_proxy.proxy_callable: docker exec_resize <- ({u'Id': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01'}, width=190, height=42)
compose.cli.verbose_proxy.proxy_callable: docker exec_resize -> None
Traceback (most recent call last):
File "<string>", line 3, in <module>
File "compose/cli/main.py", line 57, in main
File "compose/cli/main.py", line 108, in perform_command
File "compose/cli/main.py", line 353, in exec_command
File ".tox/py27/lib/python2.7/site-packages/dockerpty/pty.py", line 338, in start
File ".tox/py27/lib/python2.7/site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1
I'm having the same problem, when piping docker-compose
output via gzip to a file. When I run the command myself in Bash, it works just fine. However, as a cron job, I'm getting the same error as you β and the same stack trace file & line numbers & function names, as you, and return code -1.
Here's the command:
/usr/local/bin/docker-compose exec postgres pg_dumpall --username=postgres \
| gzip > $postgres_backup_path
(I have that line in a backup script that I run sometimes manually (works fine) and sometimes as a cron job (won't work)).
Now I've worked around my problem by calling Docker directly like so:
/usr/bin/docker exec edm_postgres_1 pg_dumpall --username=postgres \
| gzip > $postgres_backup_path
β works fine from both Cron and manually in Bash.
I do have the same issue when using docker-compose exec
in a cronjob:
something like
cd ~/piwik && docker-compose exec piwik sudo -u www-data /usr/bin/php5 /app/console core:archive --url=https://piwik-host/ > ~/piwik-archive.log
does not work - neither as a one-liner in the crontab, nor as a shell script called from crontab
I could create a workaround with bash script like:
#!/bin/bash
piwikid=$(docker ps -f "ancestor=piwik" -q)
/usr/bin/docker exec ${piwikid} /usr/bin/php5 /app/console core:archive --url=https://piwik-host/ > ~/piwik-archive.log
I could get it to work by adding the -T
parameter to not create a Pseudo-TTY.
docker-compose exec -T container_name ./build.sh
for example.
The -T option worked for me, but the script doesn't return to command prompt even after it finishes the last command.
I'm having the same issue when running:
docker-compose exec mycontainer cmd_that_takes_stdin < my/file/with/content
Everything seems to work after all (the command executes as expected), it's just that nasty crash that docker-compose gives me afterwards:
$ docker-compose exec mycontainer cmd_that_takes_stdin < my/file/with/content
Traceback (most recent call last):
File "<string>", line 3, in <module>
File "compose/cli/main.py", line 61, in main
File "compose/cli/main.py", line 113, in perform_command
File "compose/cli/main.py", line 441, in exec_command
File "site-packages/dockerpty/pty.py", line 338, in start
File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1
$ echo $?
255
When I use the -T
flag as recommended though, my command doesn't work and I even get stuck on the docker-compose command (doesn't return to my shell but just hangs instead).
Traceback (most recent call last):
File "", line 3, in
File "compose/cli/main.py", line 62, in main
File "compose/cli/main.py", line 114, in perform_command
File "compose/cli/main.py", line 442, in exec_command
File "site-packages/dockerpty/pty.py", line 334, in start
File "site-packages/dockerpty/pty.py", line 370, in _hijack_tty
File "site-packages/dockerpty/io.py", line 164, in do_write
OSError: [Errno 32] Broken pipe
docker-compose returned -1
I have the same problem, but just by launching this from jenkins (it fails just there):
docker-compose exec drupal bash -c "bin/behat --profile=ci"
, now changed to docker exec ${COMPOSE_PROJECT_NAME}_drupal_1 bash -c "bin/behat --profile=ci"
.
Same here. docker-compose exec
fails when running as part of a Jenkins job (works locally), but docker exec
works. Thanks @paolomainardi !
Have you tried adding the -T flag to docker-compose exec?
Tried it now, adding -T
to docker compose exec solve this for me as well.
-T
isn't working for me. My only workaround is to use docker exec
directly but means I have to parse the container ID from docker ps
which isn't ideal.
To test I have a basic python script that just prints stdin
$ cat myScript.py
#!/usr/bin/python
import sys
print "started"
for line in sys.stdin:
print line
$ echo test | docker-compose exec -T myContainer /usr/src/myScript.py
.. nothing happens, just waits ..
$ echo test | docker-compose exec myContainer /usr/src/myScript.py
Traceback (most recent call last):
File "<string>", line 3, in <module>
File "compose/cli/main.py", line 65, in main
File "compose/cli/main.py", line 117, in perform_command
File "compose/cli/main.py", line 462, in exec_command
File "site-packages/dockerpty/pty.py", line 338, in start
File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1
$ echo test | docker exec -i d6380314bb89 /usr/src/myScript.py
started
test
I'd say that the problem is that -T
in docker-compose exec
is mostly equivalent to not using any of -t
or -i
flags in docker run
, so with -T
there is not tty but also not attached stdin, and it seems that there is no way of attaching stdin without tty in docker-compose.
docker-compose exec
should probably have different flags for tty and for attaching stdin as docker run
has.
env: ansible playbook + ubuntu(vagrant)
error:
Traceback (most recent call last):
File "/usr/local/bin/docker-compose", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 65, in main
command()
File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 117, in perform_command
handler(command, command_options)
File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 462, in exec_command
pty.start()
File "/usr/local/lib/python2.7/dist-packages/dockerpty/pty.py", line 338, in start
io.set_blocking(pump, flag)
File "/usr/local/lib/python2.7/dist-packages/dockerpty/io.py", line 32, in set_blocking
old_flag = fcntl.fcntl(fd, fcntl.F_GETFL)
ValueError: file descriptor cannot be a negative integer (-1)
add -T
fix the issue.
for all those curious about getting around this while still getting information from docker-compose...
docker exec -i $(docker-compose ps -q cassandra) < someexample.cql
The -i keeps stdin open while not allotting a tty and the ps -q
on the docker-compose provides us with the container ID we want.
I assume this will provide multiple IDs if you have many running, you'll have to do some munging to use this method.
It would appear docker-compose isn't implicitly using a -i
when passing the exec along.
Just want to leave this minimal test-case for docker-compose:
echo 123 | docker-compose exec container_name tee
It triggers error:
Traceback (most recent call last):
File "bin/docker-compose", line 3, in <module>
File "compose/cli/main.py", line 88, in main
File "compose/cli/main.py", line 140, in perform_command
File "compose/cli/main.py", line 486, in exec_command
File "site-packages/dockerpty/pty.py", line 338, in start
File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
Failed to execute script docker-compose
I can reproduce this bug just when docker-compose gets called from jenkins or gitlab-ci
Same here in Jenkins.
reproduced by gitlab-ci
reproduced in gitlab-ci for me too
same problem on jenkins, solved by using -T:
docker-compose exec -T container_name cmd
I do have the same problem when using docker-compose exec in a cronjob
Traceback (most recent call last):
File "bin/docker-compose", line 3, in <module>
File "compose/cli/main.py", line 67, in main
File "compose/cli/main.py", line 117, in perform_command
File "compose/cli/main.py", line 453, in exec_command
File "site-packages/dockerpty/pty.py", line 338, in start
File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
Failed to execute script docker-compose
Command exited with non-zero status 255
solved by using -T
/usr/local/bin/docker-compose -f /path/to/docker-compose.yml exec -T php ./yii command
where php
is service name
I have no input
and
I have output from command
In our case (we are running these scripts by Makefile) docker-compose
command returns code 2
, solved by wrapping this running in $(shell docker-compose <whatever you want>)
@YurkinPark may be absolute path needed?
I have reproduced the bug on GitLab CI. Note, that adding -T
flag solved it.
Here is the full .gitlab-ci.yml
:
image: docker:17.04.0-ce
variables:
# See https://docs.gitlab.com/ce/ci/docker/using_docker_build.html#using-the-overlayfs-driver
DOCKER_DRIVER: overlay
services:
# See https://docs.gitlab.com/ce/ci/docker/using_docker_build.html#use-docker-in-docker-executor
- docker:dind
before_script:
# Install docker-compose
- apk add --no-cache py-pip
- pip install docker-compose
- docker --version
- docker-compose --version
# Verify that Docker CE is installed correctly
- docker run hello-world
# Prepare environment
- cp .env.example .env
- docker-compose up -d
- docker-compose ps
# If you're perplexed as to why we took a detour using bash -c
# see this issue https://github.com/docker/compose/issues/3379
- docker-compose exec php bash -c 'yarn install --non-interactive --no-progress --no-emoji; (exit $?)'
- docker-compose exec php bash -c 'yarn run production; (exit $?)'
jest:
script: docker-compose exec php bash -c 'yarn run test; (exit $?)'
Output after the encounter with the first docker-compose exec
:
Traceback (most recent call last):
File "/usr/bin/docker-compose", line 11, in <module>
sys.exit(main())
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 67, in main
command()
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 117, in perform_command
handler(command, command_options)
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 453, in exec_command
pty.start()
File "/usr/lib/python2.7/site-packages/dockerpty/pty.py", line 338, in start
io.set_blocking(pump, flag)
File "/usr/lib/python2.7/site-packages/dockerpty/io.py", line 32, in set_blocking
old_flag = fcntl.fcntl(fd, fcntl.F_GETFL)
ValueError: file descriptor cannot be a negative integer (-1)
Adding -T flag is solving problem, but command output become corrupted, it doesn't show all text as the normal command execution suppose to. I'm using docker-compose commands inside make file on gitlab ci.
So basically, you need -i
for docker exec
when you are piping in data to the command.
And you should not use -t
for docker exec
when you are piping data out from the command.
And docker-compose exec
only have -T
, which means neither -i
or -t
. But you cannot represent -i
without -t
in docker-compose
.
That's the issue, right?
Usecase | TTY | Stdin | Option |
---|---|---|---|
< |
Does not matter | Yes | N/A (This issue!) |
> |
No | Does not matter | -T |
Neither | Does not matter | Does not matter | Does not matter |
Both | No | Yes | N/A (This issue!) |
We're executing a test suite within jenkins that uses pytest-sugar
for color-coded test failures and this issue(?) seems to be preventing this from working. Without using -T
to disable pseudo-tty (and therefore strip out all color), docker-compose exec
fails with ValueError: file descriptor cannot be a negative integer (-1)
I'm running into this exact issue as well. Would love to be able to pipe data into containers via docker-compose.
For people who are stumbling into this while trying to run docker-compose exec
on Jenkins pipeline like myself:
The -T
flag mentioned bypasses the error, but it does not actually execute the command for me. Jenkins just thinks the command executed successfully and moves on.
I had to fall back to docker exec
, like so:
environment {
PROJECT_NAME = "${env.BRANCH_NAME}.${env.BUILD_ID}"
CONTAINER_PREFIX = "${PROJECT_NAME.replace('-', '').replace('.', '').toLowerCase()}"
}
stages {
stage('run') {
steps {
sh 'docker-compose -p $PROJECT_NAME up -d --build'
sh "docker exec ${CONTAINER_PREFIX}_app_1 bash -c \"hello world\""
}
}
}
The $CONTAINER_PREFIX
is a workaround to get the docker compose prefix. It's not pretty, and a little too verbose, but it's the only way I could get it to work for me.
What is docker-compose -q app
?
I think he means you can use docker-compose ps -q {app}
to get the container ID. For example,
docker exec $(docker-compose ps -q {app}) bash -c ...
docker-compose exec container_name cat logfile > logfile
also fails for me, but with -T
docker-compose exec -t container_name cat logfile > logfile
it just produce empty file, so no any output! Even there no error, it is unusable.
So, as I understand, docker exec
is the solution for now?
echo 123 | docker exec -i $(docker-compose ps -q service) cat -
for now...
I would add a bit of changes for someone who have a container-name set .Based on @michaelarnauts
This worked for me, docker-compose exec -T app ./build.sh
where app is the service defined in the docker-compose
now I try in cron (again)
crontab -e
*/10 * * * * time -p /usr/local/bin/docker-compose -f /path/to/docker-compose.yml exec php ./yii command
now I have same problem in cron (again)
[17362] Failed to execute script docker-compose
Traceback (most recent call last):
File "site-packages/dockerpty/pty.py", line 334, in start
File "site-packages/dockerpty/pty.py", line 367, in _hijack_tty
File "site-packages/dockerpty/io.py", line 59, in select
File "site-packages/dockerpty/io.py", line 351, in fileno
File "site-packages/dockerpty/io.py", line 103, in fileno
File "socket.py", line 635, in fileno
ValueError: I/O operation on closed file.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "bin/docker-compose", line 6, in <module>
File "compose/cli/main.py", line 71, in main
File "compose/cli/main.py", line 127, in perform_command
File "compose/cli/main.py", line 519, in exec_command
File "site-packages/dockerpty/pty.py", line 338, in start
File "site-packages/dockerpty/io.py", line 32, in set_blocking
File "site-packages/dockerpty/io.py", line 351, in fileno
File "site-packages/dockerpty/io.py", line 103, in fileno
File "socket.py", line 635, in fileno
ValueError: I/O operation on closed file.
Command exited with non-zero status 255
real 2.09
user 0.98
after add into crontab -e
COMPOSE_INTERACTIVE_NO_CLI=1
# Edit this file to introduce tasks to be run by cron.
https://stackoverflow.com/a/10657111/7670492
I try again with -T
/usr/local/bin/docker-compose -f /path/to/docker-compose.yml exec -T php ./yii command
success