maelgangloff/domain-watchdog

Docker Install

emhl opened this issue · 35 comments

emhl commented

since Ubuntu 22.04 ships php 8.1 i wasn't able to try out this application yet.
In order to make deployments easier and less dependant on the php version of the host system and run it in a containerized environment, it would be awesome if a docker image would be provided for this project

the work by the symfony-docker project could probably be used as a great starting point to create a docker image
https://github.com/dunglas/symfony-docker/blob/main/docs/existing-project.md

Hi @emhl,

I just added Docker support following this template. Can you tell me if it works for you?

Thanks!

emhl commented

since no image has been published on docker hub with the name app-php the image: line needed to be removed.
seperation. the use of the compose.override.yml makes this setup a bit confusing.

btw. if the containers are all in the same docker network (they are by default if defined in the same compose file) the database and mail ports don't need to be exposed publicly anymore.

and so caddy doesn't try to obtain a tls cert for localhost SERVER_NAME could be set to http://localhost instead of localhost (in order to fix a caddy error i needed to add the libnss3-tools package as well)
with those changes the webserver seems to run, but i get an error from symfony now:

An exception has been thrown during the rendering of a template ("Asset manifest file "/app/public/build/manifest.json" does not exist. Did you forget to build the assets with npm or yarn?")

the build for the frontend seems to be missing from the dockerfile and the app wasn´t copied to the /app directory. i tried to modify the Dockerfile a bit in order to get it running, but could not get it into a working state

Thanks for your feedback and testing @emhl

The app-php image is built locally before being used. The database is not exposed publicly :

screenshot

Port 5432 is internal to the container group and is not exposed.

Indeed, the frontend was not built. I just fixed that. From my first tests, it seems to work.

Commands to launch the production environment:

  1. Build the Docker images:
docker compose -f compose.yaml -f compose.prod.yaml build --pull --no-cache
  1. Start the project in prod:
docker compose -f compose.yaml -f compose.prod.yaml up

We are reaching the goal of dockerization!

emhl commented

Thanks for your work. this is getting ready quite fast

The app-php image is built locally before being used. The database is not exposed publicly :
Port 5432 is internal to the container group and is not exposed.

if you don't specify the compose file it uses the compose.yaml and compose.override.yaml file as configuration, because of the content of the compose.override.yaml file the database port was exposed outside of the docker network for me

Indeed, the frontend was not built. I just fixed that. From my first tests, it seems to work.

the frontend seems to work now, but the static text doesn't seem to load.

image

and when trying to create an account with this setup, the backend application throws an error about the MAILER_DSN environment variable not being set. i guess the app is trying to send a verification email but the service isn't confugured.

php-1       | "message":"Matched route \"user_register\".","context":{"route":"user_register","route_parameters":{"_route":"user_register","_api_resource_class":"App\\Entity\\User","_api_operation_name":"register","_controller":"App\\Controller\\RegistrationController::register"},"request_uri":"http://localhist:80/api/register","method":"POST"},"level":200,"level_name":"INFO","channel":"request","datetime":"2024-08-11T19:16:18.751651+00:00","extra":{}}
php-1       | {"message":"Checking for authenticator support.","context":{"firewall_name":"api","authenticators":1},"level":100,"level_name":"DEBUG","channel":"security","datetime":"2024-08-11T19:16:18.751787+00:00","extra":{}}
php-1       | {"message":"Checking support on authenticator.","context":{"firewall_name":"api","authenticator":"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\JWTAuthenticator"},"level":100,"level_name":"DEBUG","channel":"security","datetime":"2024-08-11T19:16:18.751792+00:00","extra":{}}
php-1       | {"message":"Authenticator does not support the request.","context":{"firewall_name":"api","authenticator":"Lexik\\Bundle\\JWTAuthenticationBundle\\Security\\Authenticator\\JWTAuthenticator"},"level":100,"level_name":"DEBUG","channel":"security","datetime":"2024-08-11T19:16:18.751800+00:00","extra":{}}
php-1       | {"message":"Uncaught PHP Exception InvalidArgumentException: \"The controller for URI \"/api/register\" is not callable: Environment variable not found: \"MAILER_DSN\".\" at ControllerResolver.php line 97","context":{"exception":{"class":"InvalidArgumentException","message":"The controller for URI \"/api/register\" is not callable: Environment variable not found: \"MAILER_DSN\".","code":0,"file":"/app/vendor/symfony/http-kernel/Controller/ControllerResolver.php:97","previous":{"class":"Symfony\\Component\\DependencyInjection\\Exception\\EnvNotFoundException","message":"Environment variable not found: \"MAILER_DSN\".","code":0,"file":"/app/vendor/symfony/dependency-injection/EnvVarProcessor.php:221"}}},"level":500,"level_name":"CRITICAL","channel":"request","datetime":"2024-08-11T19:16:18.774164+00:00","extra":{}}

i haven't found any documentation on how to create accounts through the cli or from environment variables yet, or if there are default credentials

edit: the static text not loading seems to be because the markdown files and images in the public folder aren't present in this repository by default

public/content/home.md
public/content/privacy.md
public/content/tos.md
public/content/faq.md
public/images/icons-512.png
public/images/banner.png
public/favicon.ico

I made the text files fully customizable. See here for the list.

The text files are Markdown files. These include the homepage, the TOS and the privacy policy (different content for each instance).

There are no default credentials. You have to create an account by making sure that the flag REGISTRATION_ENABLED=true is present.

Indeed, the installation is not yet properly documented. I will be careful to document the installation well.
In the meantime, I invite you to add the flag MAILER_DSN=null://null to avoid sending emails. Afterwards, you can validate the user in db and your configuration will be operational.

Thank you for your time

emhl commented

thank's for all the hints. btw the error about the mail address being unvalidated is currently silent in the frontend. and it would be nice if the email confirmation link would get printed to the log when setting up a new account, so one doesn't need to manually jump into the database or set up email delivery

{"message":"Uncaught PHP Exception Symfony\\Component\\HttpKernel\\Exception\\AccessDeniedHttpException: \"You have not yet validated your email address.\" at JWTAuthenticator.php line 35","context":{"exception":{"class":"Symfony\\Component\\HttpKernel\\Exception\\AccessDeniedHttpException","message":"You have not yet validated your email address.","code":0,"file":"/app/src/Security/JWTAuthenticator.php:35"}},"level":400,"level_name":"ERROR","channel":"request","datetime":"2024-08-11T20:19:09.371511+00:00","extra":{}}

The text files are Markdown files. These include the homepage, the TOS and the privacy policy (different content for each instance).

it is awesome that they are customizable, but a bit confusing that they are empty by default.

Good idea for the logs!
I will add the link in the logs and close this issue as soon as it is fixed.

it is awesome that they are customizable, but a bit confusing that they are empty by default.

I'm not sure if proposing a template for legal files is a good idea, I'll think about it. For the homepage it could be done I guess.

I just implemented the validation link logging :
When a new user registers, the validation link is logged. This allows to easily retrieve this link without touching the database.

I will add documentation to explain the installation process via Docker.

screenshot

Thanks for your feedback, I think this issue can be closed.

i noticed docker support in a recent release - thanks! are there any plans for prebuilt images for those looking to avoid the complexity of building their own?

Hi @halphalp,

Indeed, the project is now deployable with Docker.
An image is published on the Docker Hub for each new release as well as on GitHub Packages.

You probably need to adapt the compose.yaml file to use this remote image. Feel free to leave a message if you have any difficulties with the deployment ;)

Hi @halphalp,

Indeed, the project is now deployable with Docker. An image is published on the Docker Hub for each new release as well as on GitHub Packages.

You probably need to adapt the compose.yaml file to use this remote image. Feel free to leave a message if you have any difficulties with the deployment ;)

this doesn't seem to work - i'm trying to use existing installations of caddy and postgres along with this project. any thoughts?

  domainwatchdog:
    image: maelgangloff/domain-watchdog
    restart: unless-stopped
    environment:
      SERVER_NAME: 0.0.0.0
      DATABASE_URL: ${DATABASE_URL}
      REGISTRATION_ENABLED: 'true'
    depends_on:
      - postgresql
    networks:
      - caddy
      - postgresql

Definitely, this containerization will not have been so easy ( maybe some kind of black magic 🪄 )
I just deployed a new version that allows to display the logs when the application is deployed within a Docker container.

@halphalp :
I just tested and it seems to work. I advise you to pull the new version from the Docker Hub first.

docker pull maelgangloff/domain-watchdog:latest

Here is a part of my docker-compose.yaml file:

...
  domainwatchdog:
    image: maelgangloff/domain-watchdog:latest
    restart: unless-stopped
    environment:
      SERVER_NAME: localhost
      DATABASE_URL: postgresql://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-!ChangeMe!}@database:5432/${POSTGRES_DB:-app}?serverVersion=${POSTGRES_VERSION:-15}&charset=${POSTGRES_CHARSET:-utf8}
      SYMFONY_VERSION: ${SYMFONY_VERSION:-}
      STABILITY: ${STABILITY:-stable}
      APP_ENV: prod
      APP_SECRET: mysecret
      MAILER_DSN: null://null
      MAILER_SENDER_NAME: "Domain Watchdog"
      MAILER_SENDER_EMAIL: notifications@example.com
      REGISTRATION_ENABLED: true
      LIMITED_FEATURES: false
      LIMIT_MAX_WATCHLIST: 0
      LIMIT_MAX_WATCHLIST_DOMAINS: 0
    volumes:
      - caddy_data:/data
      - caddy_config:/config
    ports:
      - target: 80
        published: ${HTTP_PORT:-80}
        protocol: tcp
 ...

Be careful not to set SERVER_NAME=0.0.0.0. This is not a value expected by Caddy. In your case, you probably wanted to configure like this:

SERVER_NAME: '*'

Here is the documentation of the Caddyfile: https://caddyserver.com/docs/caddyfile
I hope you can launch the image this time (or at least get some logs).

I'll leave this issue open in the meantime. Thanks!

Definitely, this containerization will not have been so easy ( maybe some kind of black magic 🪄 ) I just deployed a new version that allows to display the logs when the application is deployed within a Docker container.

@halphalp : I just tested and it seems to work. I advise you to pull the new version from the Docker Hub first.

docker pull maelgangloff/domain-watchdog:latest

Here is a part of my docker-compose.yaml file:

...
  domainwatchdog:
    image: maelgangloff/domain-watchdog:latest
    restart: unless-stopped
    environment:
      SERVER_NAME: localhost
      DATABASE_URL: postgresql://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-!ChangeMe!}@database:5432/${POSTGRES_DB:-app}?serverVersion=${POSTGRES_VERSION:-15}&charset=${POSTGRES_CHARSET:-utf8}
      SYMFONY_VERSION: ${SYMFONY_VERSION:-}
      STABILITY: ${STABILITY:-stable}
      APP_ENV: prod
      APP_SECRET: mysecret
      MAILER_DSN: null://null
      MAILER_SENDER_NAME: "Domain Watchdog"
      MAILER_SENDER_EMAIL: notifications@example.com
      REGISTRATION_ENABLED: true
      LIMITED_FEATURES: false
      LIMIT_MAX_WATCHLIST: 0
      LIMIT_MAX_WATCHLIST_DOMAINS: 0
    volumes:
      - caddy_data:/data
      - caddy_config:/config
    ports:
      - target: 80
        published: ${HTTP_PORT:-80}
        protocol: tcp
 ...

Be careful not to set SERVER_NAME=0.0.0.0. This is not a value expected by Caddy. In your case, you probably wanted to configure like this:

SERVER_NAME: '*'

Here is the documentation of the Caddyfile: https://caddyserver.com/docs/caddyfile I hope you can launch the image this time (or at least get some logs).

I'll leave this issue open in the meantime. Thanks!

i just realized the container is also deploying caddy. is there a way to deploy just domain-watchdog and proxy to it using an instance of caddy i use for other services?

i just realized the container is also deploying caddy. is there a way to deploy just domain-watchdog and proxy to it using an instance of caddy i use for other services?

No, I don't think that's possible. You definitely need a web server for the Docker container to be accessible from the outside. You can't just deploy a php application, you need a web server (Caddy, Nginx, Apache, ...)

I suggest you use a different port and use your Caddy server to proxy the requests. Did you successfully deploy this project?

i just realized the container is also deploying caddy. is there a way to deploy just domain-watchdog and proxy to it using an instance of caddy i use for other services?

No, I don't think that's possible. You definitely need a web server for the Docker container to be accessible from the outside. You can't just deploy a php application, you need a web server (Caddy, Nginx, Apache, ...)

I suggest you use a different port and use your Caddy server to proxy the requests. Did you successfully deploy this project?

this is what i'm strugglign with. the domain-watchdog container seems to deploy correctly - i just can't find a suitable caddy directive to serve it with in my existing caddy installation.

i've tried:

reverse_proxy domain-watchdog:80
reverse_proxy domain-watchdog:443

domain-watchdog is in my 'caddy' docker network, so i'm not sure where the disconnect is.

@halphalp In your docker-compose file, did you name the service domain-watchdog or domainwatchdog?

(you used both of these service names in this issue)

Be sure to have the same hostname in both your Caddyfile and your docker-compose file, otherwise Docker won't understand which container you're referring to ;)

I advise you not to use HTTPS inside your containers, this will avoid self-signed certificate issues and reduce latency a bit.

@halphalp In your docker-compose file, did you name the service domain-watchdog or domainwatchdog?

(you used both of these service names in this issue)

Be sure to have the same hostname in both your Caddyfile and your docker-compose file, otherwise Docker won't understand which container you're referring to ;)

I advise you not to use HTTPS inside your containers, this will avoid self-signed certificate issues and reduce latency a bit.

good callout - i changed it between comments above, but have named it properly in my compose file and caddyfile but still no luck. here's my latest compose file:

  domain-watchdog:
    image: maelgangloff/domain-watchdog:latest
    restart: unless-stopped
    environment:
      SERVER_NAME: '*'
      DATABASE_URL: ${DATABASE_URL}
      APP_ENV: prod
      APP_SECRET: ${DOMAIN_WATCHDOG_SECRET}
      REGISTRATION_ENABLED: 'true'
      LIMITED_FEATURES: 'false'
      LIMIT_MAX_WATCHLIST: 0
      LIMIT_MAX_WATCHLIST_DOMAINS: 0
    depends_on:
      - postgres
    networks:
      - caddy
      - postgres

@halphalp Don't forget to add MAILER_DSN=null://null to prevent mails from being sent if you don't have an SMTP server configured.

Can you try setting SERVER_NAME=http://* ?
To prevent Caddy from redirecting to the SSL port that is not connected to your proxy

@halphalp Don't forget to add MAILER_DSN=null://null to prevent mails from being sent if you don't have an SMTP server configured.

Can you try setting SERVER_NAME=http://* ? To prevent Caddy from redirecting to the SSL port that is not connected to your proxy

so close! making those two changes now loads a blank white page instead of an error.

emhl commented

@halphalp if caddy is defined in s different stack the hostname for the dw container is probably domain-watcher-domain-watcher-1 but you could set it to domain-watcher with the container_name option

emhl commented

Setting SERVER_NAME=http://* leads to a blank page with a red ghost favicon for me as well. Caddy probably can't handle this configuration.

I use traefik as my reverse proxy. In order to get it working I set SERVER_NAME=http://domain-watcher.my-domain.net so caddy doesn't try to get a certificate since the Webserver isn't publicly accessible anyways.

And used Host(domain-watcher.my-domain.net) in traefik, in addition to the normal config pointing to the DW docker container

After setting up the docker container and proxying through nginx using the following compose:

    database:
      image: postgres
      restart: unless-stopped
      environment:
        POSTGRES_PASSWORD: BestPassword
        POSTGRES_USER: app
        POSTGRES_DB: app
      volumes:
          - /home/ubuntu/docker/domain-watchdog/postgresql-data:/var/lib/postgresql/data
    domainwatchdog:
        image: maelgangloff/domain-watchdog:latest
        restart: unless-stopped
        environment:
            SERVER_NAME: http://domain
            DATABASE_URL: postgresql://app:BestPassword@database:5432/app?serverVersion=${POSTGRES_VERSION:-15}&charset=${POSTGRES_CHARSET:-utf8}
            SYMFONY_VERSION: ${SYMFONY_VERSION:-}
            STABILITY: ${STABILITY:-stable}
            APP_ENV: prod
            APP_SECRET: "base64:AAAAAAAAAAAAAAAAAA+YIhppD4LeURsj7x1C46UaB9o="
            MAILER_DSN: null://null
            MAILER_SENDER_NAME: "Domain Watchdog"
            MAILER_SENDER_EMAIL: notifications@example.com
            #LIMITED_FEATURES: "false"
            #LIMIT_MAX_WATCHLIST: "0"
            #LIMIT_MAX_WATCHLIST_DOMAINS: "0"
        volumes:
            - dwd_caddy_data:/data
            - dwd_caddy_config:/config
        ports:
            - "8974:80"
volumes:
  dwd_caddy_data:
  dwd_caddy_config:

I get the following errors and thus the website is constantly loading.

GET http://domain/content/home.md -> Returns 404 -> No route found
GET http://domain/api/me -> Returns 401 -> JWT Token not found

@skyracer2012 These errors are normal for an instance that has just been deployed.

Regarding the first request mentioned, you need to add the following files:

public/content/home.md
public/content/privacy.md
public/content/tos.md
public/content/faq.md
public/images/icons-512.png
public/images/banner.png
public/favicon.ico

These files correspond to files that correspond to your own instance (legal documents in particular).

Regarding the second request: The frontend is trying to find out if you are connected or not. This response is completely normal if you are not connected.

I advise you to add the missing files by mounting a volume on /app/public/content.
Once you have registered, look at the logs to retrieve the link to validate the account (this link would normally have been sent by email if an SMTP server is configured).

The error you describe is not blocking the use of your DW instance locally.

okay, definitely making progress. the page loads, but i can't do anything and get a bunch of /api and /#/login errors in the console when trying to create an account or log in.

my current docker compose file:

  domain-watchdog:
    image: maelgangloff/domain-watchdog:latest
    restart: unless-stopped
    environment:
      SERVER_NAME: http://domainwatchdog.mydomain.com
      DATABASE_URL: ${DOMAINWATCHDOG_DB_URL}
      APP_ENV: prod
      APP_SECRET: ${DOMAINWATCHDOG_SECRET}
      MAILER_DSN: null://null
      REGISTRATION_ENABLED: 'true'
      LIMITED_FEATURES: 'false'
      LIMIT_MAX_WATCHLIST: 0
      LIMIT_MAX_WATCHLIST_DOMAINS: 0
    volumes:
      - ./domain-watchdog:/app/public/contents
    depends_on:
      - postgres
    networks:
      - caddy
      - postgres

my caddy file is simply:

reverse_proxy domain-watchdog:80

and here are the console errors i'm seeing:

domain-watchdog-console-errors

@halphalp These errors are normal for an instance that has just been deployed.

These are the same HTTP errors encountered by @skyracer2012.

I refer you to my last message. I draw your attention to the fact that these errors are not blocking for personal use.

You must retrieve the validation link from the logs to be able to validate the account. This link is usually sent by email if you have configured an SMTP server.

Is your instance functional?

So I have been able to set it up with the config I mentioned above. I just added the REGISTRATION_ENABLED parameter. After registering I did not see the confirmation url in the docker logs so I verified myself in the database. After Login I get a 200 response from the https://domain/api/login endpoint. The reply is however {"token":""} When trying to find a Domain I do also get {"code":401,"message":"JWT Token not found"}. Does the docker image not generate a JWT when logging? I have tried to set APP_SECRET to a random string aswell as base64:xxxxx.

Also the comment you made above. It seems like the path is /app/public/content instead of /app/public/contents

@skyracer2012 Yes, typo, it is public/content and not public/contents.

Thanks for reporting this error. I think I found the problem, the JWT key pair is not generated on first launch. I am deploying a fix and publishing a release afterwards.

Thanks for the feedback!

Release v0.1.5 has just been published 🎉
When starting the container, if no JWT key pair is detected, it will be automatically generated.
Otherwise, the key pair is not overwritten.

docker pull maelgangloff/domain-watchdog:latest

I hope this will fix the problem ;)

EDIT: I just noticed that the list of RDAP servers to contact is empty when deploying with Docker. I'll see to make sure that this list is instantiated on the first launch (otherwise you would have to wait 24 hours...)

Adding this to your docker compose should instantiate the list of RDAP servers on first startup and process Watchlists:

...
  php-worker:
    image: maelgangloff/domain-watchdog:latest
    restart: always
    command: php /app/bin/console messenger:consume --all --time-limit=3600 -vvv
    environment:
      DATABASE_URL: ${DATABASE_URL}
...

i was experiencing the same issues as @skyracer2012 before v0.1.5 (could register, log in, but the JW token error prevented me from doing anything) and have now updated my compose file to include the new command and pull v0.1.6 and can no longer access the web interface (the container is marked as unhealthy). here are the last ~25 lines of the container logs:

domain-watchdog  | {"message":"App\\Message\\UpdateRdapServers was handled successfully (acknowledging to transport).","context":{"class":"App\\Message\\UpdateRdapServers","message_id":1},"level":200,"level_name":"INFO","channel":"messenger","datetime":"2024-08-15T19:54:32.173448+00:00","extra":{}}
domain-watchdog  | 19:54:32 INFO      [messenger] App\Message\UpdateRdapServers was handled successfully (acknowledging to transport).
domain-watchdog  | [
domain-watchdog  |   "class" => "App\Message\UpdateRdapServers",
domain-watchdog  |   "message_id" => 1
domain-watchdog  | ]
domain-watchdog  | 19:54:32 INFO      [cache] Lock acquired, now computing item "scheduler_checkpoint_notif_watchlist"
domain-watchdog  | [
domain-watchdog  |   "key" => "scheduler_checkpoint_notif_watchlist"
domain-watchdog  | ]
domain-watchdog  | 19:54:32 INFO      [cache] Lock acquired, now computing item "scheduler_checkpoint_notif_watchlist"
domain-watchdog  | [
domain-watchdog  |   "key" => "scheduler_checkpoint_notif_watchlist"
domain-watchdog  | ]
domain-watchdog  | 19:54:32 INFO      [cache] Lock acquired, now computing item "scheduler_checkpoint_rdap_udpate"
domain-watchdog  | [
domain-watchdog  |   "key" => "scheduler_checkpoint_rdap_udpate"
domain-watchdog  | ]
domain-watchdog  | 19:54:32 INFO      [cache] Lock acquired, now computing item "scheduler_checkpoint_rdap_udpate"
domain-watchdog  | [
domain-watchdog  |   "key" => "scheduler_checkpoint_rdap_udpate"
domain-watchdog  | ]

thanks for all of the work and quick responses on this, by the way! anxious to start using this.

It's hard to debug remotely and identify the problem you're having. I've created a Gist file with a working configuration. Here is the link https://gist.github.com/maelgangloff/ca1dfa18becefccb1ac1ff82c4a73ef8

(I tested it on two different machines)

Be careful to delete browser cookies as JWT keys have changed.

Looking forward to hearing from you with good news 🙌

It's hard to debug remotely and identify the problem you're having. I've created a Gist file with a working configuration. Here is the link https://gist.github.com/maelgangloff/ca1dfa18becefccb1ac1ff82c4a73ef8

(I tested it on two different machines)

Be careful to delete browser cookies as JWT keys have changed.

Looking forward to hearing from you with good news 🙌

do i need the new php-worker every time, or just run it once the very first time i deploy?

do i need the new php-worker every time, or just run it once the very first time i deploy?

The worker must be active continuously. It handles asynchronous tasks such as:

  • Updating the list of RDAP servers to contact and the TLDs
  • Processing of Watchlists: updating the domain names present in the Watchlist, tracking events and automatic purchase
  • Sending event notification emails

Have you successfully deployed this project?

It works really nice!
The only thing is that I wasnt able to use the email server I have running on the same machine but that might be one me.

One Problem I did encounter is the php-worker due to using the command: argument is not being marked as "started" but as "starting". This caused my monitoring to go all crazy every hour due to the container being marked as unhealthy.
Can we just add the php-worker to the original docker container using supervisor. This would make the docker-compose simpler. There is documentation on this in the docker docs here.

Hi @skyracer2012
The Docker image worked, that's great! I'm reassured.

I updated the Docker Compose to disable the healthcheck. I think it's better to do that than to include the worker in the web app container.
php-worker restarts every hour but is no longer marked as unhealthy.

Hopefully this will fix this issue ;)