Errors with the script not finding the TMP_DIR directory, its solution and other bug-fixes and improvements, plus the 502 Bad Gateway fix when using Mailu with Nginx Proxy Manager (NPM)

Question

Errors with the script not finding the TMP_DIR directory, its solution and other bug-fixes and improvements, plus the 502 Bad Gateway fix when using Mailu with Nginx Proxy Manager (NPM)

apainter2 opened this issue 4 years ago · 0 comments

Hi there,

So here are the changes to the following files, in relation to my issues thread over on the Mailu repo: Mailu/Mailu#1807

As it's easier to start with the service file, I run without the need for a WireGuard VPN, but I am interested in that, as I use a messier method to restrict access to the mail server ssh interface, so would like to discuss that one.

My sync-npm-mailu-ssl.service file:

[Unit]
Description=Nginx Proxy Manager SSL Cert Rsync

[Service]
Restart=always
WorkingDirectory=/root/ssl-sync/
ExecStart=/bin/bash /root/ssl-sync/sync-ssl.sh

[Install]
WantedBy=multi-user.target

I used a systemctl link option rather than copy the .service file directly to the /etc/systemd/system/ directory.

So it should look similar to the following:
lrwxrwxrwx 1 root root 41 May 21 16:35 sync-npm-mailu-ssl.service -> /root/ssl-sync/sync-npm-mailu-ssl.service

Now, my amendments to the .bash file, which for myself I renamed to .sh just out of habit.

My version of your sync script, which I named sync-ssl.sh:

#!/bin/bash
#This script syncs the ssl keys down to the mailu certs folder start this script as systemctl process
#use included systemctl file
# apt install inotify-tools
NPM_CERT_PATH="/home/user/docker_files/npm/letsencrypt/live/npm-3/"
MAILU_CERT_PATH="/home/user/docker_files/mailu/certs/"
SCRIPT_DIR="/root/ssl-sync"
TMP_DIR="$SCRIPT_DIR/tmp-sync"
CERT="fullchain.pem"
KEY="privkey.pem"
MAILU_FRONTEND="mailu_front_1"

function copySSLCerts() {
    rsync -avL $NPM_CERT_PATH $TMP_DIR #resolve symlinks
    cd $TMP_DIR
    mv $CERT $MAILU_CERT_PATH
    mv $KEY $MAILU_CERT_PATH
    rm -rf $TMP_DIR/*
    docker exec $MAILU_FRONTEND nginx -s reload #reload ngix service within mailu_front container
}

while inotifywait -r $NPM_CERT_PATH; do
  if [ ! -d "$TMP_DIR" ]; then
    mkdir -p $TMP_DIR
    copySSLCerts
  else
    copySSLCerts
  fi
done

As you can see, I've made quite a few alterations, which I'll happily breakdown for you:

I only like to type paths and the like once. So I declare the repetitive paths as variables, and sometimes, even a variable inside a variable.

So, the first is the Nginx Proxy Manager (NPM) location for the LetsEncrypt live certificate-store; it looks like NPM bastardises things here and actually uses SymLinks, which is not how the LetsEncrypt CertBot would prefer them to be, in fact, CertBot actually states not to define the certs as SymLinks within the README file located within the NPM-* directory, but that's an NPM author issue.

The second is for the Mailu certificate location, again Mailu prefers you to use /mailu off the root, but I'm not a fan of that so in a user-home directory it goes.

It's important to note, that you do not need to use the user-home directory, and if you used Docker volumes, you can point directly to the flat-directory structure of those too, but it's more fiddly and prone to messing up unless you know what you are doing.

The third variable is the bash script location, and I've left that as your default.

Forth, sets the temporary directory that holds the certificates and keys during transfer, I've used a directory off the parent ssl-sync directory, so to ensure that nothing is hard-coded I've incorporated the SCRIPT-DIR variable into it.

CERT variable declares the filename for the full-chain certificate that LetsEncrypt generates, I've left the filename as default, and I've declared a variable within the mailu.env file which I'll reference below to ensure that Mailu uses it.

The same with the KEY variable.

Now I had issues with the Mailu Front container, not seeing the new or updated certificates the solution for me was to reload the Nginx instance it uses as its reverse proxy, and as not everyone will name their Mailu Front container the same, I've declared the name of it a variable so we can reference it later in the script.

Now here is the big change from your current script iteration. I had issues with the way it handled the deletion of the TMP_DIR, which caused the entire import process to fail the second time it would run (i.e. on day 180 / second certificate renewal - as LE certs are valid for 90 days). What your script would do is upon moving the full-chain and the private key files to their new location, your script would delete the entire TMP_DIR, now this is fine, ordinarily, as the next time the script would run it would notice this and re-create the TMP_DIR directory, but that is not how inotifywait in this instance works. In this instance, the script never really exits but is stuck in a virtual loop and is paused and only issues a mkdir during the first run through. Nothing I could do would make it recreate the TMP_DIR a second time, so the script would fail and simply error out, and then pause for the next source file change (certificate update), yet never move the certificates over.

My fix for this was to simply check to see if the TMP_DIR existed, if it did not, it would create it and then proceed as normal. If it existed, it simply ignored the mkdir command and proceeded as normal.

You also have it so that it deletes the entire TMP_DIR directory, I've modified it so it only deletes the left-over files and leaves the directory itself in-situ, meaning in future runs, the script will function if left running via the systemd service manager. My amendment is to check for the existence of the directory and create it if it is missing, much like a 'first-time setup wizard' sanity check. This sanity check works each time the script is started, using systemctl (via a systemctl restart command) or a server reboot etc.

Now for the above to work in a way that minimises potential errors and to prevent code duplication, it makes sense to declare the following as a function:

the rsync command to move the LE certs from the NPM npm-* location to the TMP_DIR so we can work with them, and remove the symlinks that NPM creates. - I prefer the use of rsync here over cp, due to how it handles errors and more importantly due to the NPM author using symlinks in place of actual files for the LetsEncrypt certificate store, how it does not copy the symlink but the actual file.
Move into the TMP_DIR directory.
Move the full-chain certificate and private key from the TMP_DIR into the Mailu certificate directory.
A bit of clean up, so we delete the remaining leftover files from the rsync, these are the solo certificate, the chain (aka the intermediate/subCA) and rootCA certificate combined into a single 'chain' file), and the README file from the TMP_DIR directory, whilst keeping the TMP_DIR in-situ.
Reload the configuration of the Nginx instance within the defined Mailu Front container, this ensures that the new/updated certificates are reloaded.

Here is the function:

function copySSLCerts() {
    rsync -avL $NPM_CERT_PATH $TMP_DIR #resolve symlinks
    cd $TMP_DIR
    mv $CERT $MAILU_CERT_PATH
    mv $KEY $MAILU_CERT_PATH
    rm -rf $TMP_DIR/*
    docker exec $MAILU_FRONTEND nginx -s reload #reload ngix service within mailu_front container
}

Now I noticed that your using inotifywait, fantastic! I modified it to use an if/else statement to incorporate everything I've mentioned so far, so that will create the TMP_DIR if it does not exist and then calls the copySSLCert function then exits, or if the TMP_DIR directory does exist, it simply calls the function. This removes any duplication of code that would ordinarily be needed, thus removes the change of any code discrepancies between the TMP_DIR existing or not.

So, here is your while statement, with my if/else statement incorporated into it:

while inotifywait -r $NPM_CERT_PATH; do
  if [ ! -d "$TMP_DIR" ]; then
    mkdir -p $TMP_DIR
    copySSLCerts
  else
    copySSLCerts
  fi
done

For reference the contents of my /root/ssl-sync directory consist of:

-rw-r--r-- 1 root root  198 May 21 18:29 sync-npm-mailu-ssl.service
-rwxr-xr-x 1 root root  858 May 21 18:20 sync-ssl.sh
drwxr-xr-x 2 root root 4096 May 21 18:30 tmp-sync

Now, onto my changes to the mailu.env environment file, find the # Choose how secure connections will behave (value: letsencrypt, cert, notls, mail, mail-letsencrypt) section. I changed this area to the following:

TLS_FLAVOR=cert
# Custom certificame filenames
TLS_CERT_FILENAME=fullchain.pem
TLS_KEYPAIR_FILENAME=privkey.pem

Much like your own, but I specify the TLS_KEYPAIR_FILENAME as well.

Now as this started out as a Nginx Proxy Manager issue over on the Mailu GitHub repo, with NPM not able to talk to the Mailu services and serving a 502 - Bad Gateway. So, here is my solution to that with input from yourself.

Ensure that the SUBNET variable within the mailu.env file uses a 172.xxx.0.0/24 subnet, my example:

# Subnet of the docker network. This should not conflict with any networks to which your system is connected. (Internal and external!)
SUBNET=172.213.0.0/24

Modify the docker-compose.yml of Mailu, in relation to defining the listening IP address for the docker ports specifically for the 'Mailu Front' container HTTP and HTTPS ports, which pins all this together and removes the 502 Bad Gateway issue from NPM:

 front:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-1.7}
    restart: always
    env_file: mailu.env
    logging:
      driver: json-file
    ports:
      - "0.0.0.0:9080:80"
      - "0.0.0.0:9443:443"
      - "xxx.xxx.xxx.xxx:25:25"
      - "xxx.xxx.xxx.xxx:465:465"
      - "xxx.xxx.xxx.xxx:587:587"
      - "xxx.xxx.xxx.xxx:110:110"
      - "xxx.xxx.xxx.xxx:995:995"
      - "xxx.xxx.xxx.xxx:143:143"
      - "xxx.xxx.xxx.xxx:993:993"
    volumes:
      - "/home/user/docker_files/mailu/certs:/certs"
      - "/home/userdocker_files/mailu/overrides/nginx:/overrides"

And finally within NPN, create the proxy host to use proxy host address IP address to the IP address of the server that is running the docker instance for Mailu, as well as set the port to match the above (in this instance 9443). Specifying the container name or using 127.0.01/localhost as the address will not work. As per your original screenshot, shown here:
https://user-images.githubusercontent.com/6371945/118332237-0ef26880-b50a-11eb-80bd-1b2e57b32fa2.png

Now hopefully this entire post makes sense, happy to discuss and improve upon it further. But, one final thing!

Thank you for creating the original script and idea, it really helped me out of a bind when it came to ensuring that the Mailu certificates are kept up to date as the HTTPS port is fine (NPM can take care of that little issue) but the STARTTLS and SSL elements of POSTFIX and SMTP would not function; and your little script fixes that like a champ!