spurin/diveintoansible-lab

WSL2 docker for windows / docker-compose up issues

Closed this issue · 23 comments

oscarmparedes commented

Hi!

this is my .env file:

# sshd ports
UBUNTUC_PORT_SSHD=2221
UBUNTU1_PORT_SSHD=2222
UBUNTU2_PORT_SSHD=2223
UBUNTU3_PORT_SSHD=2224
CENTOS1_PORT_SSHD=2225
CENTOS2_PORT_SSHD=2226
CENTOS3_PORT_SSHD=2227

# ttyd (web terminal) ports
UBUNTUC_PORT_TTYD=7681
UBUNTU1_PORT_TTYD=7682
UBUNTU2_PORT_TTYD=7683
UBUNTU3_PORT_TTYD=7684
CENTOS1_PORT_TTYD=7685
CENTOS2_PORT_TTYD=7686
CENTOS3_PORT_TTYD=7687

# Shared config volume
CONFIG=/home/{{user}}/diveintoansible-lab

# Shared home directories
ANSIBLE_HOME=/home/{{user}}/.ansible`

This is the output of docker-compose up command:

`docker-compose up
Creating network "diveinto.io" with the default driver
Creating ubuntu-c ... done
Creating centos3  ... done
Creating ubuntu1  ... done
Creating ubuntu2  ... done
Creating centos2  ... done
Creating docker   ... done
Creating centos1  ... done
Creating ubuntu3  ... done
Creating portal   ... done
Attaching to centos3, ubuntu-c, centos2, ubuntu1, ubuntu2, docker, ubuntu3, centos1, portal
ubuntu-c exited with code 255
ubuntu1 exited with code 255
ubuntu2 exited with code 255
ubuntu3 exited with code 255
portal      | 2020/12/18 17:51:29 [emerg] 1#1: host not found in upstream "ubuntu1" in /etc/nginx/conf.d/default.conf:46
portal      | nginx: [emerg] host not found in upstream "ubuntu1" in /etc/nginx/conf.d/default.conf:46
portal exited with code 1

Given that the autocreate dirs feature for bind volumes is deprecated, I created dirs manually just in case, with these commands:
docker/compose#2781

`mkdir -p ~/.ansible/ubuntu[1,2,3]
mkdir -p ~/.ansible/centos[1,2,3]
mkdir -p ~/.ansible/ubuntu-c
mkdir -p ~/.ansible/shared

ls -la ~/.ansible/
total 40
drwxr-xr-x  4 root  root  4096 Dec 18 18:16 centos1
drwxr-xr-x  4 root  root  4096 Dec 18 18:16 centos2
drwxr-xr-x  4 root  root  4096 Dec 18 18:16 centos3
drwxr-xr-x  2 root  root  4096 Dec 18 18:16 shared
drwxr-xr-x  4 root  root  4096 Dec 18 18:16 ubuntu-c
drwxr-xr-x  4 root  root  4096 Dec 18 18:16 ubuntu1
drwxr-xr-x  4 root  root  4096 Dec 18 18:16 ubuntu2
drwxr-xr-x  4 root  root  4096 Dec 18 18:16 ubuntu3
`

Client: Docker Engine - Community
 Cloud integration: 1.0.2
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:02:36 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:07:04 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

lsb_release -d
Description:    Ubuntu 20.04.1 LTS

docker-compose --version
docker-compose version 1.27.4, build 40524192

Still I couldn't make it work. Any suggestions?, centos images run but ubuntu ones don't. I also checked the layers but I really couldn't find what is happening.
https://hub.docker.com/layers/spurin/diveintoansible/ubuntu/images/sha256-3ee94e9bffdb108bc1c34b78558a415a0690779082d8edb15b59b952f0691b76?context=explore

Thanks in advance, and hope this helps others.

spurin commented
  • 👍1
spurin commented
  • 👍1
oscarmparedes commented

Hi Spurin!,

Thanks for your quick reply. Ok, for some reason I copied over the .env file without the config part. Now I didn't create any dirs manually, you are totally right it creates the dirs automatically, but not sure why docker-compose creates the dirs as root.

.env

# sshd ports
UBUNTUC_PORT_SSHD=2221
UBUNTU1_PORT_SSHD=2222
UBUNTU2_PORT_SSHD=2223
UBUNTU3_PORT_SSHD=2224
CENTOS1_PORT_SSHD=2225
CENTOS2_PORT_SSHD=2226
CENTOS3_PORT_SSHD=2227

# ttyd (web terminal) ports
UBUNTUC_PORT_TTYD=7681
UBUNTU1_PORT_TTYD=7682
UBUNTU2_PORT_TTYD=7683
UBUNTU3_PORT_TTYD=7684
CENTOS1_PORT_TTYD=7685
CENTOS2_PORT_TTYD=7686
CENTOS3_PORT_TTYD=7687

# Shared config volume
CONFIG=/home/oscar/diveintoansible-lab/config

# Shared home directories
ANSIBLE_HOME=/home/oscar/diveintoansible-lab/ansible_home

ls -la
total 40
drwxr-xr-x 10 root  root  4096 Dec 18 19:43 .
drwxrwxrwx  5 oscar oscar 4096 Dec 18 19:43 ..
drwxr-xr-x  4 root  root  4096 Dec 18 19:43 centos1
drwxr-xr-x  4 root  root  4096 Dec 18 19:43 centos2
drwxr-xr-x  4 root  root  4096 Dec 18 19:43 centos3
drwxr-xr-x  2 root  root  4096 Dec 18 19:43 shared
drwxr-xr-x  4 root  root  4096 Dec 18 19:43 ubuntu-c
drwxr-xr-x  4 root  root  4096 Dec 18 19:43 ubuntu1
drwxr-xr-x  4 root  root  4096 Dec 18 19:43 ubuntu2
drwxr-xr-x  4 root  root  4096 Dec 18 19:43 ubuntu3

and still same error, but this is the verbose output:

Starting portal   ... done
compose.parallel.feed_queue: Pending: set()
compose.parallel.parallel_execute_iter: Finished processing: <Service: portal>
compose.parallel.feed_queue: Pending: set()
Attaching to ubuntu-c, ubuntu1, centos1, centos3, centos2, docker, ubuntu2, ubuntu3, portal
compose.cli.verbose_proxy.proxy_callable: docker wait <- ('c3fb9366a0029847b4f3092970caa08eda6fc6243f0460bd259702e8e33d900d')
compose.cli.verbose_proxy.proxy_callable: docker wait <- ('e4b4c2d76513107dce494a93f38d6fc7a48aef80c268ac2142158d40f81ee01a')
compose.cli.verbose_proxy.proxy_callable: docker wait <- ('7425ec0e49b54a346d085286d1f91491bcca9ccc1ff48e0bba8032c3ea42504f')
compose.cli.verbose_proxy.proxy_callable: docker wait <- ('4a5d830c214417942e9cbc5ea408d5b7ae3bff5e103d40d0526dca41faf62ff6')
compose.cli.verbose_proxy.proxy_callable: docker events <- (filters={'label': ['com.docker.compose.project=diveintoansible-lab', 'com.docker.compose.oneoff=False']}, decode=True)
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.40/containers/c3fb9366a0029847b4f3092970caa08eda6fc6243f0460bd259702e8e33d900d/wait HTTP/1.1" 200 None
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.40/containers/e4b4c2d76513107dce494a93f38d6fc7a48aef80c268ac2142158d40f81ee01a/wait HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker wait -> {'Error': None, 'StatusCode': 255}
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.40/containers/4a5d830c214417942e9cbc5ea408d5b7ae3bff5e103d40d0526dca41faf62ff6/wait HTTP/1.1" 200 None
ubuntu-c exited with code 255
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.40/events?filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Ddiveintoansible-lab%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 None
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.40/containers/7425ec0e49b54a346d085286d1f91491bcca9ccc1ff48e0bba8032c3ea42504f/wait HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker wait -> {'Error': None, 'StatusCode': 255}
compose.cli.verbose_proxy.proxy_callable: docker events -> <docker.types.daemon.CancellableStream object at 0x7f6b6b8b9c90>
ubuntu1 exited with code 255
compose.cli.verbose_proxy.proxy_callable: docker wait -> {'Error': None, 'StatusCode': 255}
portal      | 2020/12/18 18:48:44 [emerg] 1#1: host not found in upstream "ubuntu1" in /etc/nginx/conf.d/default.conf:46
compose.cli.verbose_proxy.proxy_callable: docker wait -> {'Error': None, 'StatusCode': 255}
portal      | nginx: [emerg] host not found in upstream "ubuntu1" in /etc/nginx/conf.d/default.conf:46
ubuntu3 exited with code 255
ubuntu2 exited with code 255
compose.cli.verbose_proxy.proxy_callable: docker wait <- ('84f75b87f92561b65ebaa7f6a7506b93f6ba628eca00dcfcf383a4de3d4770fb')
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('84f75b87f92561b65ebaa7f6a7506b93f6ba628eca00dcfcf383a4de3d4770fb')
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.40/containers/84f75b87f92561b65ebaa7f6a7506b93f6ba628eca00dcfcf383a4de3d4770fb/wait HTTP/1.1" 200 None
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.40/containers/84f75b87f92561b65ebaa7f6a7506b93f6ba628eca00dcfcf383a4de3d4770fb/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker wait -> {'Error': None, 'StatusCode': 1}
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['nginx', '-g', 'daemon off;'],
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['nginx', '-g', 'daemon off;'],
            'Domainname': '',
            'Entrypoint': ['/docker-entrypoint.sh'],
            'Env': ['NGINX_ENTRYPOINT_QUIET_LOGS=1',
                    'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
...
portal exited with code 1
spurin commented

Hi Oscar,

I think you can ignore the permissions on the directory, Docker is running as root and docker-compose is passing off to Docker, hence the permissions. We'll get to the bottom of this, even if we need to break it down into stages.

Just seeing how we can break this up for further testing, will update shortly.

oscarmparedes commented

Thank you very much. I could make this work perfect in a vagrant box, so I am concerned it maybe has to do with something in docker vs using WSL2 engine.

spurin commented

It's a good opportunity to find out whats wrong so we'll try and troubleshoot this for now. Okay, some stages -

Let's see if you can run the ansible container as it is, quite simply, you should get a login and then just be able to exit -

docker run --rm -p 12345:7681 --privileged -it spurin/diveintoansible:ansible bash
oscarmparedes commented

This works.

docker run --rm -p 12345:7681 --privileged -it spurin/diveintoansible:ansible bash
root@49c0e35574e7:/# exit
exit
spurin commented

Next, let's test the directory mounts -

docker run -v /home/oscar/diveintoansible-lab/config:/config -v /home/oscar/diveintoansible-lab/ansible_home --rm -p 12345:7681 --privileged -it spurin/diveintoansible:ansible ls /config /ansible_home
oscarmparedes commented 
 docker run -v /home/oscar/diveintoansible-lab/config:/config -v /home/oscar/diveintoansible-lab/ansible_home --rm -p 12345:7681 --privileged -it spurin/diveintoansible:ansible ls /config /ansible_home
ls: cannot access '/ansible_home': No such file or directory
/config:
guest_name  guest_passwd  guest_shell  root_passwd
spurin commented

Okay so thats interesting. By the way, are you running this as a standalone Linux Ubuntu system or is it running in another virtualisation layer?

Let's see the permissions -

ls -ald /home/oscar/diveintoansible-lab/ansible_home
oscarmparedes commented

Ok this is docker desktop for windows with WSL2 engine enabled. I am using a WLS2 ubuntu 20.04 image.

 ls -ald /home/oscar/diveintoansible-lab/ansible_home
drwxr-xr-x 10 root root 4096 Dec 18 19:43 /home/oscar/diveintoansible-lab/ansible_home

It was docker-compose that created this dir btw. It does work perfectly fine in an Ubuntu 20.04 Vagrant box.

spurin commented

Understood. You could also directly run the lab from Windows and I don't think you'll have these problems.

Let's try this -

id
mkdir -p /home/oscar/diveintoansible-lab/ansible_home2
docker run -v /home/oscar/diveintoansible-lab/config:/config -v /home/oscar/diveintoansible-lab/ansible_home2 --rm -p 12345:7681 --privileged -it spurin/diveintoansible:ansible ls /config /ansible_home
oscarmparedes commented

Sure, I am actually testing this for a friend that started your course. I am using my work laptop (windows only), and I need WSL2 to have linux for work. I think the WSL2 is doing something strange with permissions.

id
uid=1000(oscar) gid=1000(oscar) groups=1000(oscar),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),117(netdev),1001(docker)
:~/diveintoansible-lab$ mkdir -p /home/oscar/diveintoansible-lab/ansible_home2
:~/diveintoansible-lab$ docker run -v /home/oscar/diveintoansible-lab/config:/config -v /home/oscar/diveintoansible-lab/ansible_home2 --rm -p 12345:7681 --privileged -it spurin/diveintoansible:ansible ls /config /ansible_home
ls: cannot access '/ansible_home': No such file or directory
/config:
guest_name  guest_passwd  guest_shell  root_passwd
spurin commented

Try this -

docker run -v /home/oscar/diveintoansible-lab/config:/config -v /home/oscar/diveintoansible-lab/ansible_home:/ansible_home --rm -p 12345:7681 --privileged -it spurin/diveintoansible:ansible ls /config /ansible_home

oscarmparedes commented 
docker run -v /home/oscar/diveintoansible-lab/config:/config -v /home/oscar/diveintoansible-lab/ansible_home:/ansible_home --rm -p 12345:7681 --privileged -it spurin/diveintoansible:ansible ls /config /ansible_home
/ansible_home:
centos1  centos2  centos3  shared  ubuntu-c  ubuntu1  ubuntu2  ubuntu3

/config:
guest_name  guest_passwd  guest_shell  root_passwd
spurin commented

Okay great, earlier on it was missing the :/ansible_lab bit, can you try this for me next please -

docker run --name diveintodebug -v /home/oscar/diveintoansible-lab/config:/config -v /home/oscar/diveintoansible-lab/ansible_home/ubuntu-c/ansible:/home/ansible -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -p 12345:7681 --privileged -it spurin/diveintoansible:ansible

You should, if all goes well be able to login to http://localhost:12345

Can you try this out and share the output, if it's not working. Note, to stop this, you'll need a new terminal and will need to do -

docker stop diveintodebug
oscarmparedes commented 
docker run --name diveintodebug -v /home/oscar/diveintoansible-lab/config:/config -v /home/oscar/diveintoansible-lab/ansible_home/ubuntu-c/ansible:/home/ansible -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -p 12345:7681 --privileged -it spurin/diveintoansible:ansible
Failed to mount cgroup at /sys/fs/cgroup/systemd: No such file or directory
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
spurin commented
oscarmparedes commented

This is great and I hope it is useful for someone else I think I found a fix. So for the record:

Looks like it works by using this. So if anyone is using Docker for windows with WSL2 engine, maybe this is the fix for your lab to work in this kind of environment. microsoft/WSL#4189

spurin commented
  • 👍1
oscarmparedes commented

More precisely, if using WSL2, and Docker for windows with WSL2 engine support enabled, try running like this:

sudo mkdir -p /sys/fs/cgroup/systemd
:~/diveintoansible-lab$ sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd
:~/diveintoansible-lab$ docker-compose up
Creating network "diveinto.io" with the default driver
Creating ubuntu1  ... done
Creating ubuntu2  ... done
Creating ubuntu-c ... done
Creating centos2  ... done
Creating centos1  ... done
Creating centos3  ... done
Creating ubuntu3  ... done
Creating docker   ... done
Creating portal   ... done
Attaching to ubuntu2, ubuntu1, ubuntu-c, centos2, centos1, ubuntu3, docker, centos3, portal

As per answering your questions, the tests we were doing before were done from WSL2 Ubuntu 20.04 box, not Vagrant. Vagrant worked fine from the beginning is just this setup. It's just a workaround, not a proper fix, I will keep investigating.

Very cool lab by the way. Congrats!

  • 👍2
spurin commented
oscarmparedes commented

Welcome! Happy to help!