Issue deploying with docker-compose
djuillard opened this issue · 4 comments
Hi, I've read all the documentation and this project seems really nice; I tried to set it up mon my computer but I got some errors during the docker build (about 5 min after it started building). But actually, I'm not sure I am doing it the right way :
- I am using a windows 11 computer with visual studio + docker desktop and WSL
- I first make the git clone on my computer
- I then copy and set up the .env file in the ./docker/ folder (because that's where I found the docker-compose.yaml file so I thought it was the right place to put it).
- and then, still in the docker folder, I run the cli : docker-compose up --build -> it pulls a lot of file, but crashes after 5 min (strange message about VM crashing)
I do not have the opportunity to run the "make" command lines in the backend (and to be honnest, I was wondering where I should run those CLI anyway : in the docker Inguess ?)
Any help would be appreciated.
Thanks
hi @djuillard ,
Thank you for picking up the project; I don't have the Window environment, so I haven't tested it yet. I assume the WSL is a Linux-like environment, so I assume it should be similar to any Linux (correct me if I'm wrong).
From your above statement, the .env needs to put into build/ but it's not the reason for the docker compose build, since docker compose build does not require it.
I think to narrow down the crash reason, we need to further understand the detailed VM crashing reason, maybe due to OOM or disk full or others. Maybe you want to another try and catch the reasons to see.
Last, for the make command, sorry for the confusion. To run it, we basically stay at the root of the auto-news folder. I will enhance the wiki to point this out 👍
thank you for your reply. Indeed the VM crash may be linked to something else. I freed up some space, and did not have the same issue.... but another one :
PS C:\Users\damien\Documents\3. DEV\auto-news\auto-news\docker> docker-compose up --build
time="2024-04-30T12:33:10+02:00" level=warning msg="The "MYSQL_PROJ_DIR" variable is not set. Defaulting to a blank string."
time="2024-04-30T12:33:10+02:00" level=warning msg="The "POSTGRES_PROJ_DIR" variable is not set. Defaulting to a blank string."
time="2024-04-30T12:33:10+02:00" level=warning msg="The "REDIS_PROJ_DIR" variable is not set. Defaulting to a blank string."
time="2024-04-30T12:33:10+02:00" level=warning msg="The "AIRFLOW_UID" variable is not set. Defaulting to a blank string."
time="2024-04-30T12:33:10+02:00" level=warning msg="The "AIRFLOW_UID" variable is not set. Defaulting to a blank string."
[+] Building 512.4s (17/17) docker:default
[+] Building 512.5s (17/17) docker:default
=> [airflow-init internal] load build definition from Dockerfile 0.0s
[+] Building 515.5s (42/48) docker:default
=> [airflow-init internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.75kB 0.0s
=> [airflow-init-user internal] load metadata for docker.io/apache/airflow:2.7.3-python3.11 2.7s
=> [airflow-init auth] apache/airflow:pull token for registry-1.docker.io 0.0s
=> [airflow-init internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [airflow-init-user 1/11] FROM docker.io/apache/airflow:2.7.3-python3.11@sha256:a8c67ae095c03868e56c2d95c9a57a1b9a47e87247d923e329cb7c15e6d47ee5 0.0s
=> [airflow-init internal] load build context 0.0s
=> => transferring context: 38B 0.0s
=> CACHED [airflow-init-user 2/11] WORKDIR /opt/airflow 0.0s
=> CACHED [airflow-init-user 3/11] RUN set -ex; deps=' make g++ vim net-tools curl wget lsof less iputils-ping telnet procps git libgomp1 jq ffmpeg '; apt-get update && apt-get install -y --no-install-recomm 0.0s
=> CACHED [airflow-init-user 4/11] RUN set -ex; git config --global --add safe.directory /opt/airflow/run/auto-news && git config --global pull.ff only 0.0s
=> CACHED [airflow-init-user 5/11] RUN set -ex; echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf 0.0s
=> CACHED [airflow-init-user 6/11] RUN set -ex; mkdir -p ~/.ssh && echo "StrictHostKeyChecking=no" > ~/.ssh/config 0.0s
=> CACHED [airflow-init-user 7/11] RUN set -ex; ln -snf /opt/airflow ~/airflow 0.0s
=> CACHED [airflow-init-user 8/11] RUN set -ex; git config --global --add safe.directory /opt/airflow/run/auto-news && git config --global pull.ff only 0.0s
=> CACHED [airflow-init-user 9/11] RUN set -ex; ln -snf /opt/airflow ~/airflow 0.0s
=> CACHED [airflow-init 10/11] ADD requirements.txt . 0.0s
=> [airflow-init 11/11] RUN set -ex; pip install --no-cache-dir -r requirements.txt && rm -rf ~/.cache/pip 465.2s
=> [airflow-init] exporting to image 45.7s
=> => exporting layers 45.6s
=> => writing image sha256:09618fd1c31621df93d956d9bbb6aec427092c0f5469a63e8d5e7869b3e1251d 0.0s
=> => naming to docker.io/library/docker-airflow-init 0.0s
=> [airflow-scheduler internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 1.75kB 0.0s
=> [airflow-worker internal] load build definition from Dockerfile 0.2s
=> => transferring dockerfile: 1.75kB 0.0s
=> [airflow-webserver internal] load build definition from Dockerfile 0.2s
=> => transferring dockerfile: 1.75kB 0.0s
=> [airflow-triggerer internal] load build definition from Dockerfile 0.2s
=> => transferring dockerfile: 1.75kB 0.0s
=> [airflow-scheduler auth] apache/airflow:pull token for registry-1.docker.io 0.0s
=> [airflow-worker internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [airflow-triggerer internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [airflow-webserver internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [airflow-scheduler internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [airflow-webserver internal] load build context 0.1s
=> => transferring context: 38B 0.0s
=> [airflow-worker internal] load build context 0.1s
=> => transferring context: 719B 0.0s
=> [airflow-scheduler internal] load build context 0.1s
=> => transferring context: 719B 0.0s
=> CACHED [airflow-webserver 10/11] ADD requirements.txt . 0.0s
=> CACHED [airflow-webserver 11/11] RUN set -ex; pip install --no-cache-dir -r requirements.txt && rm -rf ~/.cache/pip 0.0s
=> [airflow-triggerer internal] load build context 0.1s
=> => transferring context: 719B 0.0s
=> [airflow-webserver] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:741feb25559205c565e915cacfada8a67768cbce5db915984b120d11acba69f5 0.0s
=> => naming to docker.io/library/docker-airflow-webserver 0.0s
=> [airflow-worker] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:570b30efe7338b39410146191a0b66c6856ec7b733cc4c2be51dd742c262683c 0.0s
=> => naming to docker.io/library/docker-airflow-worker 0.0s
=> [airflow-triggerer] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:5551fba784629af820b763b91a161efc10c59c3a237a3d74fdba3f63b35e5a06 0.0s
=> => naming to docker.io/library/docker-airflow-triggerer 0.0s
=> [airflow-scheduler] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:bb28df26a9ac680806d4049b2b5523e3f38c85fcedf4f3cdb6218d4291821128 0.0s
=> => naming to docker.io/library/docker-airflow-scheduler 0.0s
=> [airflow-init-user internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.75kB 0.0s
=> [airflow-init-user internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [airflow-init-user internal] load build context 0.0s
=> => transferring context: 38B 0.0s
=> CACHED [airflow-init-user 10/11] ADD requirements.txt . 0.0s
=> CACHED [airflow-init-user 11/11] RUN set -ex; pip install --no-cache-dir -r requirements.txt && rm -rf ~/.cache/pip 0.0s
=> [airflow-init-user] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:23b0dbf882a09b719a1a868c04ddf49c20a600b459a5ab58507c5b6b1e38558b 0.0s
=> => naming to docker.io/library/docker-airflow-init-user 0.0s
Error response from daemon: network bot_network not found
PS C:\Users\damien\Documents\3. DEV\auto-news\auto-news\docker>
I see, thank you for the detailed logs, which give us the direct clue, the reason is missing the preparation step before building the image, from the wiki, the command to build the backend is:
make deps && make build && make deploy && make initThe first make command will finish the preparation step, aka create necessary folders and create "docker network", it's recommended to run above make command for the building (install it before actual building), in case not, we can manually run the commands line by line here: https://github.com/finaldie/auto-news/blob/main/Makefile#L49-L71
And, last, similar for make deploy and make init, require some manual fix if not using make. Hope that helps a little.
hi @djuillard ,
Can I assume that the issue has been solved? close it for now, and reopen it if need further discussion, thanks a lot.