Docker Install
emhl opened this issue · 35 comments
since Ubuntu 22.04 ships php 8.1 i wasn't able to try out this application yet.
In order to make deployments easier and less dependant on the php version of the host system and run it in a containerized environment, it would be awesome if a docker image would be provided for this project
the work by the symfony-docker project could probably be used as a great starting point to create a docker image
https://github.com/dunglas/symfony-docker/blob/main/docs/existing-project.md
Hi @emhl,
I just added Docker support following this template. Can you tell me if it works for you?
Thanks!
since no image has been published on docker hub with the name app-php the image: line needed to be removed.
seperation. the use of the compose.override.yml makes this setup a bit confusing.
btw. if the containers are all in the same docker network (they are by default if defined in the same compose file) the database and mail ports don't need to be exposed publicly anymore.
and so caddy doesn't try to obtain a tls cert for localhost SERVER_NAME could be set to http://localhost instead of localhost (in order to fix a caddy error i needed to add the libnss3-tools package as well)
with those changes the webserver seems to run, but i get an error from symfony now:
An exception has been thrown during the rendering of a template ("Asset manifest file "/app/public/build/manifest.json" does not exist. Did you forget to build the assets with npm or yarn?")
the build for the frontend seems to be missing from the dockerfile and the app wasn´t copied to the /app directory. i tried to modify the Dockerfile a bit in order to get it running, but could not get it into a working state
Thanks for your feedback and testing @emhl
The app-php image is built locally before being used. The database is not exposed publicly :
Port 5432 is internal to the container group and is not exposed.
Indeed, the frontend was not built. I just fixed that. From my first tests, it seems to work.
Commands to launch the production environment:
- Build the Docker images:
docker compose -f compose.yaml -f compose.prod.yaml build --pull --no-cache
- Start the project in prod:
docker compose -f compose.yaml -f compose.prod.yaml up
We are reaching the goal of dockerization!
Thanks for your work. this is getting ready quite fast
The app-php image is built locally before being used. The database is not exposed publicly :
Port 5432 is internal to the container group and is not exposed.
if you don't specify the compose file it uses the compose.yaml and compose.override.yaml file as configuration, because of the content of the compose.override.yaml file the database port was exposed outside of the docker network for me
Indeed, the frontend was not built. I just fixed that. From my first tests, it seems to work.
the frontend seems to work now, but the static text doesn't seem to load.
and when trying to create an account with this setup, the backend application throws an error about the MAILER_DSN environment variable not being set. i guess the app is trying to send a verification email but the service isn't confugured.
php-1 | "message":"Matched route \"user_register\".","context":{"route":"user_register","route_parameters":{"_route":"user_register","_api_resource_class":"App\\Entity\\User","_api_operation_name":"register","_controller":"App\\Controller\\RegistrationController::register"},"request_uri":"http://localhist:80/api/register","method":"POST"},"level":200,"level_name":"INFO","channel":"request","datetime":"2024-08-11T19:16:18.751651+00:00","extra":{}}
php-1 | {"message":"Checking for authenticator support.","context":{"firewall_name":"api","authenticators":1},"level":100,"level_name":"DEBUG","channel":"security","datetime":"2024-08-11T19:16:18.751787+00:00","extra":{}}
php-1 | {"message":"Checking support on authenticator.","context":{"firewall_name":"api","authenticator":"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\JWTAuthenticator"},"level":100,"level_name":"DEBUG","channel":"security","datetime":"2024-08-11T19:16:18.751792+00:00","extra":{}}
php-1 | {"message":"Authenticator does not support the request.","context":{"firewall_name":"api","authenticator":"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\JWTAuthenticator"},"level":100,"level_name":"DEBUG","channel":"security","datetime":"2024-08-11T19:16:18.751800+00:00","extra":{}}
php-1 | {"message":"Uncaught PHP Exception InvalidArgumentException: \"The controller for URI \"/api/register\" is not callable: Environment variable not found: \"MAILER_DSN\".\" at ControllerResolver.php line 97","context":{"exception":{"class":"InvalidArgumentException","message":"The controller for URI \"/api/register\" is not callable: Environment variable not found: \"MAILER_DSN\".","code":0,"file":"/app/vendor/symfony/http-kernel/Controller/ControllerResolver.php:97","previous":{"class":"Symfony\\Component\\DependencyInjection\\Exception\\EnvNotFoundException","message":"Environment variable not found: \"MAILER_DSN\".","code":0,"file":"/app/vendor/symfony/dependency-injection/EnvVarProcessor.php:221"}}},"level":500,"level_name":"CRITICAL","channel":"request","datetime":"2024-08-11T19:16:18.774164+00:00","extra":{}}
i haven't found any documentation on how to create accounts through the cli or from environment variables yet, or if there are default credentials
edit: the static text not loading seems to be because the markdown files and images in the public folder aren't present in this repository by default
public/content/home.md
public/content/privacy.md
public/content/tos.md
public/content/faq.md
public/images/icons-512.png
public/images/banner.png
public/favicon.ico
I made the text files fully customizable. See here for the list.
The text files are Markdown files. These include the homepage, the TOS and the privacy policy (different content for each instance).
There are no default credentials. You have to create an account by making sure that the flag REGISTRATION_ENABLED=true
is present.
Indeed, the installation is not yet properly documented. I will be careful to document the installation well.
In the meantime, I invite you to add the flag MAILER_DSN=null://null
to avoid sending emails. Afterwards, you can validate the user in db and your configuration will be operational.
Thank you for your time
thank's for all the hints. btw the error about the mail address being unvalidated is currently silent in the frontend. and it would be nice if the email confirmation link would get printed to the log when setting up a new account, so one doesn't need to manually jump into the database or set up email delivery
{"message":"Uncaught PHP Exception Symfony\\Component\\HttpKernel\\Exception\\AccessDeniedHttpException: \"You have not yet validated your email address.\" at JWTAuthenticator.php line 35","context":{"exception":{"class":"Symfony\\Component\\HttpKernel\\Exception\\AccessDeniedHttpException","message":"You have not yet validated your email address.","code":0,"file":"/app/src/Security/JWTAuthenticator.php:35"}},"level":400,"level_name":"ERROR","channel":"request","datetime":"2024-08-11T20:19:09.371511+00:00","extra":{}}
The text files are Markdown files. These include the homepage, the TOS and the privacy policy (different content for each instance).
it is awesome that they are customizable, but a bit confusing that they are empty by default.
Good idea for the logs!
I will add the link in the logs and close this issue as soon as it is fixed.
it is awesome that they are customizable, but a bit confusing that they are empty by default.
I'm not sure if proposing a template for legal files is a good idea, I'll think about it. For the homepage it could be done I guess.
I just implemented the validation link logging :
When a new user registers, the validation link is logged. This allows to easily retrieve this link without touching the database.
I will add documentation to explain the installation process via Docker.
Thanks for your feedback, I think this issue can be closed.
i noticed docker support in a recent release - thanks! are there any plans for prebuilt images for those looking to avoid the complexity of building their own?
Hi @halphalp,
Indeed, the project is now deployable with Docker.
An image is published on the Docker Hub for each new release as well as on GitHub Packages.
You probably need to adapt the compose.yaml file to use this remote image. Feel free to leave a message if you have any difficulties with the deployment ;)
Hi @halphalp,
Indeed, the project is now deployable with Docker. An image is published on the Docker Hub for each new release as well as on GitHub Packages.
You probably need to adapt the compose.yaml file to use this remote image. Feel free to leave a message if you have any difficulties with the deployment ;)
this doesn't seem to work - i'm trying to use existing installations of caddy and postgres along with this project. any thoughts?
domainwatchdog:
image: maelgangloff/domain-watchdog
restart: unless-stopped
environment:
SERVER_NAME: 0.0.0.0
DATABASE_URL: ${DATABASE_URL}
REGISTRATION_ENABLED: 'true'
depends_on:
- postgresql
networks:
- caddy
- postgresql
Definitely, this containerization will not have been so easy ( maybe some kind of black magic 🪄 )
I just deployed a new version that allows to display the logs when the application is deployed within a Docker container.
@halphalp :
I just tested and it seems to work. I advise you to pull the new version from the Docker Hub first.
docker pull maelgangloff/domain-watchdog:latest
Here is a part of my docker-compose.yaml file:
...
domainwatchdog:
image: maelgangloff/domain-watchdog:latest
restart: unless-stopped
environment:
SERVER_NAME: localhost
DATABASE_URL: postgresql://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-!ChangeMe!}@database:5432/${POSTGRES_DB:-app}?serverVersion=${POSTGRES_VERSION:-15}&charset=${POSTGRES_CHARSET:-utf8}
SYMFONY_VERSION: ${SYMFONY_VERSION:-}
STABILITY: ${STABILITY:-stable}
APP_ENV: prod
APP_SECRET: mysecret
MAILER_DSN: null://null
MAILER_SENDER_NAME: "Domain Watchdog"
MAILER_SENDER_EMAIL: notifications@example.com
REGISTRATION_ENABLED: true
LIMITED_FEATURES: false
LIMIT_MAX_WATCHLIST: 0
LIMIT_MAX_WATCHLIST_DOMAINS: 0
volumes:
- caddy_data:/data
- caddy_config:/config
ports:
- target: 80
published: ${HTTP_PORT:-80}
protocol: tcp
...
Be careful not to set SERVER_NAME=0.0.0.0
. This is not a value expected by Caddy. In your case, you probably wanted to configure like this:
SERVER_NAME: '*'
Here is the documentation of the Caddyfile: https://caddyserver.com/docs/caddyfile
I hope you can launch the image this time (or at least get some logs).
I'll leave this issue open in the meantime. Thanks!
Definitely, this containerization will not have been so easy ( maybe some kind of black magic 🪄 ) I just deployed a new version that allows to display the logs when the application is deployed within a Docker container.
@halphalp : I just tested and it seems to work. I advise you to pull the new version from the Docker Hub first.
docker pull maelgangloff/domain-watchdog:latestHere is a part of my docker-compose.yaml file:
... domainwatchdog: image: maelgangloff/domain-watchdog:latest restart: unless-stopped environment: SERVER_NAME: localhost DATABASE_URL: postgresql://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-!ChangeMe!}@database:5432/${POSTGRES_DB:-app}?serverVersion=${POSTGRES_VERSION:-15}&charset=${POSTGRES_CHARSET:-utf8} SYMFONY_VERSION: ${SYMFONY_VERSION:-} STABILITY: ${STABILITY:-stable} APP_ENV: prod APP_SECRET: mysecret MAILER_DSN: null://null MAILER_SENDER_NAME: "Domain Watchdog" MAILER_SENDER_EMAIL: notifications@example.com REGISTRATION_ENABLED: true LIMITED_FEATURES: false LIMIT_MAX_WATCHLIST: 0 LIMIT_MAX_WATCHLIST_DOMAINS: 0 volumes: - caddy_data:/data - caddy_config:/config ports: - target: 80 published: ${HTTP_PORT:-80} protocol: tcp ...Be careful not to set
SERVER_NAME=0.0.0.0
. This is not a value expected by Caddy. In your case, you probably wanted to configure like this:SERVER_NAME: '*'Here is the documentation of the Caddyfile: https://caddyserver.com/docs/caddyfile I hope you can launch the image this time (or at least get some logs).
I'll leave this issue open in the meantime. Thanks!
i just realized the container is also deploying caddy. is there a way to deploy just domain-watchdog and proxy to it using an instance of caddy i use for other services?
i just realized the container is also deploying caddy. is there a way to deploy just domain-watchdog and proxy to it using an instance of caddy i use for other services?
No, I don't think that's possible. You definitely need a web server for the Docker container to be accessible from the outside. You can't just deploy a php application, you need a web server (Caddy, Nginx, Apache, ...)
I suggest you use a different port and use your Caddy server to proxy the requests. Did you successfully deploy this project?
i just realized the container is also deploying caddy. is there a way to deploy just domain-watchdog and proxy to it using an instance of caddy i use for other services?
No, I don't think that's possible. You definitely need a web server for the Docker container to be accessible from the outside. You can't just deploy a php application, you need a web server (Caddy, Nginx, Apache, ...)
I suggest you use a different port and use your Caddy server to proxy the requests. Did you successfully deploy this project?
this is what i'm strugglign with. the domain-watchdog container seems to deploy correctly - i just can't find a suitable caddy directive to serve it with in my existing caddy installation.
i've tried:
reverse_proxy domain-watchdog:80
reverse_proxy domain-watchdog:443
domain-watchdog is in my 'caddy' docker network, so i'm not sure where the disconnect is.
@halphalp In your docker-compose file, did you name the service domain-watchdog
or domainwatchdog
?
(you used both of these service names in this issue)
Be sure to have the same hostname in both your Caddyfile and your docker-compose file, otherwise Docker won't understand which container you're referring to ;)
I advise you not to use HTTPS inside your containers, this will avoid self-signed certificate issues and reduce latency a bit.
@halphalp In your docker-compose file, did you name the service
domain-watchdog
ordomainwatchdog
?(you used both of these service names in this issue)
Be sure to have the same hostname in both your Caddyfile and your docker-compose file, otherwise Docker won't understand which container you're referring to ;)
I advise you not to use HTTPS inside your containers, this will avoid self-signed certificate issues and reduce latency a bit.
good callout - i changed it between comments above, but have named it properly in my compose file and caddyfile but still no luck. here's my latest compose file:
domain-watchdog:
image: maelgangloff/domain-watchdog:latest
restart: unless-stopped
environment:
SERVER_NAME: '*'
DATABASE_URL: ${DATABASE_URL}
APP_ENV: prod
APP_SECRET: ${DOMAIN_WATCHDOG_SECRET}
REGISTRATION_ENABLED: 'true'
LIMITED_FEATURES: 'false'
LIMIT_MAX_WATCHLIST: 0
LIMIT_MAX_WATCHLIST_DOMAINS: 0
depends_on:
- postgres
networks:
- caddy
- postgres
@halphalp Don't forget to add MAILER_DSN=null://null
to prevent mails from being sent if you don't have an SMTP server configured.
Can you try setting SERVER_NAME=http://*
?
To prevent Caddy from redirecting to the SSL port that is not connected to your proxy
@halphalp Don't forget to add
MAILER_DSN=null://null
to prevent mails from being sent if you don't have an SMTP server configured.Can you try setting
SERVER_NAME=http://*
? To prevent Caddy from redirecting to the SSL port that is not connected to your proxy
so close! making those two changes now loads a blank white page instead of an error.
@halphalp if caddy is defined in s different stack the hostname for the dw container is probably domain-watcher-domain-watcher-1 but you could set it to domain-watcher with the container_name option
Setting SERVER_NAME=http://* leads to a blank page with a red ghost favicon for me as well. Caddy probably can't handle this configuration.
I use traefik as my reverse proxy. In order to get it working I set SERVER_NAME=http://domain-watcher.my-domain.net so caddy doesn't try to get a certificate since the Webserver isn't publicly accessible anyways.
And used Host(domain-watcher.my-domain.net
) in traefik, in addition to the normal config pointing to the DW docker container
After setting up the docker container and proxying through nginx using the following compose:
database:
image: postgres
restart: unless-stopped
environment:
POSTGRES_PASSWORD: BestPassword
POSTGRES_USER: app
POSTGRES_DB: app
volumes:
- /home/ubuntu/docker/domain-watchdog/postgresql-data:/var/lib/postgresql/data
domainwatchdog:
image: maelgangloff/domain-watchdog:latest
restart: unless-stopped
environment:
SERVER_NAME: http://domain
DATABASE_URL: postgresql://app:BestPassword@database:5432/app?serverVersion=${POSTGRES_VERSION:-15}&charset=${POSTGRES_CHARSET:-utf8}
SYMFONY_VERSION: ${SYMFONY_VERSION:-}
STABILITY: ${STABILITY:-stable}
APP_ENV: prod
APP_SECRET: "base64:AAAAAAAAAAAAAAAAAA+YIhppD4LeURsj7x1C46UaB9o="
MAILER_DSN: null://null
MAILER_SENDER_NAME: "Domain Watchdog"
MAILER_SENDER_EMAIL: notifications@example.com
#LIMITED_FEATURES: "false"
#LIMIT_MAX_WATCHLIST: "0"
#LIMIT_MAX_WATCHLIST_DOMAINS: "0"
volumes:
- dwd_caddy_data:/data
- dwd_caddy_config:/config
ports:
- "8974:80"
volumes:
dwd_caddy_data:
dwd_caddy_config:
I get the following errors and thus the website is constantly loading.
GET http://domain/content/home.md -> Returns 404 -> No route found
GET http://domain/api/me -> Returns 401 -> JWT Token not found
@skyracer2012 These errors are normal for an instance that has just been deployed.
Regarding the first request mentioned, you need to add the following files:
public/content/home.md
public/content/privacy.md
public/content/tos.md
public/content/faq.md
public/images/icons-512.png
public/images/banner.png
public/favicon.ico
These files correspond to files that correspond to your own instance (legal documents in particular).
Regarding the second request: The frontend is trying to find out if you are connected or not. This response is completely normal if you are not connected.
I advise you to add the missing files by mounting a volume on /app/public/content
.
Once you have registered, look at the logs to retrieve the link to validate the account (this link would normally have been sent by email if an SMTP server is configured).
The error you describe is not blocking the use of your DW instance locally.
okay, definitely making progress. the page loads, but i can't do anything and get a bunch of /api and /#/login errors in the console when trying to create an account or log in.
my current docker compose file:
domain-watchdog:
image: maelgangloff/domain-watchdog:latest
restart: unless-stopped
environment:
SERVER_NAME: http://domainwatchdog.mydomain.com
DATABASE_URL: ${DOMAINWATCHDOG_DB_URL}
APP_ENV: prod
APP_SECRET: ${DOMAINWATCHDOG_SECRET}
MAILER_DSN: null://null
REGISTRATION_ENABLED: 'true'
LIMITED_FEATURES: 'false'
LIMIT_MAX_WATCHLIST: 0
LIMIT_MAX_WATCHLIST_DOMAINS: 0
volumes:
- ./domain-watchdog:/app/public/contents
depends_on:
- postgres
networks:
- caddy
- postgres
my caddy file is simply:
reverse_proxy domain-watchdog:80
and here are the console errors i'm seeing:
@halphalp These errors are normal for an instance that has just been deployed.
These are the same HTTP errors encountered by @skyracer2012.
I refer you to my last message. I draw your attention to the fact that these errors are not blocking for personal use.
You must retrieve the validation link from the logs to be able to validate the account. This link is usually sent by email if you have configured an SMTP server.
Is your instance functional?
So I have been able to set it up with the config I mentioned above. I just added the REGISTRATION_ENABLED parameter. After registering I did not see the confirmation url in the docker logs so I verified myself in the database. After Login I get a 200 response from the https://domain/api/login
endpoint. The reply is however {"token":""}
When trying to find a Domain I do also get {"code":401,"message":"JWT Token not found"}
. Does the docker image not generate a JWT when logging? I have tried to set APP_SECRET to a random string aswell as base64:xxxxx
.
Also the comment you made above. It seems like the path is /app/public/content
instead of /app/public/contents
@skyracer2012 Yes, typo, it is public/content
and not public/contents
.
Thanks for reporting this error. I think I found the problem, the JWT key pair is not generated on first launch. I am deploying a fix and publishing a release afterwards.
Thanks for the feedback!
Release v0.1.5 has just been published 🎉
When starting the container, if no JWT key pair is detected, it will be automatically generated.
Otherwise, the key pair is not overwritten.
docker pull maelgangloff/domain-watchdog:latest
I hope this will fix the problem ;)
EDIT: I just noticed that the list of RDAP servers to contact is empty when deploying with Docker. I'll see to make sure that this list is instantiated on the first launch (otherwise you would have to wait 24 hours...)
Adding this to your docker compose should instantiate the list of RDAP servers on first startup and process Watchlists:
...
php-worker:
image: maelgangloff/domain-watchdog:latest
restart: always
command: php /app/bin/console messenger:consume --all --time-limit=3600 -vvv
environment:
DATABASE_URL: ${DATABASE_URL}
...
i was experiencing the same issues as @skyracer2012 before v0.1.5 (could register, log in, but the JW token error prevented me from doing anything) and have now updated my compose file to include the new command and pull v0.1.6 and can no longer access the web interface (the container is marked as unhealthy). here are the last ~25 lines of the container logs:
domain-watchdog | {"message":"App\\Message\\UpdateRdapServers was handled successfully (acknowledging to transport).","context":{"class":"App\\Message\\UpdateRdapServers","message_id":1},"level":200,"level_name":"INFO","channel":"messenger","datetime":"2024-08-15T19:54:32.173448+00:00","extra":{}}
domain-watchdog | 19:54:32 INFO [messenger] App\Message\UpdateRdapServers was handled successfully (acknowledging to transport).
domain-watchdog | [
domain-watchdog | "class" => "App\Message\UpdateRdapServers",
domain-watchdog | "message_id" => 1
domain-watchdog | ]
domain-watchdog | 19:54:32 INFO [cache] Lock acquired, now computing item "scheduler_checkpoint_notif_watchlist"
domain-watchdog | [
domain-watchdog | "key" => "scheduler_checkpoint_notif_watchlist"
domain-watchdog | ]
domain-watchdog | 19:54:32 INFO [cache] Lock acquired, now computing item "scheduler_checkpoint_notif_watchlist"
domain-watchdog | [
domain-watchdog | "key" => "scheduler_checkpoint_notif_watchlist"
domain-watchdog | ]
domain-watchdog | 19:54:32 INFO [cache] Lock acquired, now computing item "scheduler_checkpoint_rdap_udpate"
domain-watchdog | [
domain-watchdog | "key" => "scheduler_checkpoint_rdap_udpate"
domain-watchdog | ]
domain-watchdog | 19:54:32 INFO [cache] Lock acquired, now computing item "scheduler_checkpoint_rdap_udpate"
domain-watchdog | [
domain-watchdog | "key" => "scheduler_checkpoint_rdap_udpate"
domain-watchdog | ]
thanks for all of the work and quick responses on this, by the way! anxious to start using this.
It's hard to debug remotely and identify the problem you're having. I've created a Gist file with a working configuration. Here is the link https://gist.github.com/maelgangloff/ca1dfa18becefccb1ac1ff82c4a73ef8
(I tested it on two different machines)
Be careful to delete browser cookies as JWT keys have changed.
Looking forward to hearing from you with good news 🙌
It's hard to debug remotely and identify the problem you're having. I've created a Gist file with a working configuration. Here is the link https://gist.github.com/maelgangloff/ca1dfa18becefccb1ac1ff82c4a73ef8
(I tested it on two different machines)
Be careful to delete browser cookies as JWT keys have changed.
Looking forward to hearing from you with good news 🙌
do i need the new php-worker every time, or just run it once the very first time i deploy?
do i need the new php-worker every time, or just run it once the very first time i deploy?
The worker must be active continuously. It handles asynchronous tasks such as:
- Updating the list of RDAP servers to contact and the TLDs
- Processing of Watchlists: updating the domain names present in the Watchlist, tracking events and automatic purchase
- Sending event notification emails
Have you successfully deployed this project?
It works really nice!
The only thing is that I wasnt able to use the email server I have running on the same machine but that might be one me.
One Problem I did encounter is the php-worker due to using the command:
argument is not being marked as "started" but as "starting". This caused my monitoring to go all crazy every hour due to the container being marked as unhealthy.
Can we just add the php-worker to the original docker container using supervisor. This would make the docker-compose simpler. There is documentation on this in the docker docs here.
Hi @skyracer2012
The Docker image worked, that's great! I'm reassured.
I updated the Docker Compose to disable the healthcheck. I think it's better to do that than to include the worker in the web app container.
php-worker
restarts every hour but is no longer marked as unhealthy.
Hopefully this will fix this issue ;)