datastack-net/dockerized

--shell tests failing for specific commands

boukeversteegh opened this issue · 2 comments

boukeversteegh commented

Most mysterious bug ever.

Expected:

  • dockerized --shell <command> -c env should print the current environment

Actual:

  • In some specific conditions, there is no output from the container at all, and dockerized just exits.

    Run bin/dockerized -v --shell perl -c env | tee ~/shell.log
Dockerized root: /home/runner/work/dockerized/dockerized
Loading: '/home/runner/work/dockerized/dockerized/.env'
Compose files: /home/runner/work/dockerized/dockerized/docker-compose.yml
Setting up shell in container for perl...
Entrypoint: [/bin/bash]
Command:    [-c env]
Preparing compose environment...
Network dockerized_default  Creating
Network dockerized_default  Created
Volume "dockerized_node_modules"  Creating
Volume "dockerized_node_modules"  Created
Volume "dockerized_python_modules"  Creating
Volume "dockerized_python_modules"  Created
Volume "dockerized_pip_cache"  Creating
Volume "dockerized_pip_cache"  Created
Volume "dockerized_go_cache"  Creating
Volume "dockerized_go_cache"  Created
Volume "dockerized_home"  Creating
Volume "dockerized_home"  Created
Running one-off container...
Container exited with code 0.

Observations:

  • These shell tests worked perfectly in march, and suddenly broke.
  • Many official docker images were updated under the same tags. Alpine and busybox were changed.
  • Affected commands:
    • go, python, perl, rustc
  • Still works perfectly locally on windows
  • It only breaks the first time. Running the command twice makes it work somehow.
  • Running docker-compose before dockerized also causes the --shell test to work.
  • For some commands, the issue is with bash. Switching to sh for --shell fixes the bug.
  • sh is not a universal fix. rustc uses sh and still doesn't work.
  • Was able to reproduce once on local Ubuntu VM. Cannot reproduce after. Even docker system prune -af won't help to reproduce it, running without a tty. (true | dockerized --shell perl -c env | cat)
  • docker-compose run --rm <command> always seems to work.

Hypothese (and motivation):

  • Something related to setting up the networks / compose project
    • There could be small implementation differences between docker-compose run and dockerized
    • Since docker-compose seems to work without failure, there must be something different causing the bug.
  • Something TTY related
    • Github Actions have no TTY whereas local running from a shell will provide a TTY, explaining reproduction difficulty
    • ❌ Then why does it work the second time?
    • Running echo helloworld the first time didn't fix perl, but disabling TTY did. So TTY is involved.
  • Related to updates in Bash and Alpine in official docker images
    • This did work a month ago and suddenly broke. Re-ran an old successful action and it didn't work anymore.
    • Specifying the exact alpine version in the docker image tag has fixed the issue for some commands. Unfortunately, the old digests are not published by dockerhub, so it's not possible to restore all commands. Also, this is not a good long term solution.
    • ❌ Then why does it work the second time?
  • Something specifically related to the env command

Excluded causes:

  • Related to pulling the images the first time.
    • ❌ Added dockerized --pull. Running this before the shell test doesn't make a difference.
  • Related to a race condition in creating the network.
    • ❌ The issue occurs consistently with a specific subset of commands and is not unpredictable
boukeversteegh commented

Another weird observation:

  • When running the --shell option 2x, all commands succeed (except zip, for different reasons).
  • When running first --shell -c 'echo hello world' before the env test, all commands succeed, except 'perl'.

https://github.com/datastack-net/dockerized/runs/6093814155

boukeversteegh commented

Found a solution, although the observed behavior is still a big mystery.

  • Only run container with TTY when dockerized is run from an interactive terminal.

Remaining questions:

  • Why does the forceful assignment of a TTY to a non-tty session:
    • Fail on some commands, but not others?
    • Didn't fail before, but failed now (after image updates)?
    • Fail the first time within a shell session, but not the second time?
    • Fail the first time within a shell session, and also the second time, if the first session only ran echo?
    • Fail the first time, but not after running docker-compose run first?