TwiN/gatus

Bind mounted config.yaml does not automatically reload

roycyt opened this issue · 10 comments

roycyt commented

Step to reproduce:

$ cat config/config.yaml
services:
  - name: example
    url: "https://example.org/"
    interval: 30s
    conditions:
      - "[STATUS] == 200"

$ docker run \
    --rm \
    -p 8080:8080 \
    -v "$(pwd)"/config/config.yaml:/config/config.yaml \
    twinproduction/gatus

Change the name field in the config/config.yaml.

Observation:

In the HasLoadedConfigurationFileBeenModified function, the result of the comparison of config.lastFileModTime.Unix() != fileInfo.ModTime().Unix() is always false.

TwiN commented

I've tried it with the exact steps you've outlined, and I wasn't able to reproduce the issue.

Note that the file changes aren't picked up instantly - there's a job for changes every 30 seconds.

roycyt commented

Thanks for the confirmation. Will double check and update later.

roycyt commented

I can reproduce at my side but I am not sure what is going wrong.
Since I can not provide more information.
I will re-open it when new information comes to light.

  • 👍1
1player commented

Not sure why was this closed, I'm having the same issue on a freshly-installed gatus instance.

FWIW the mount point is a glusterfs mount, but file modification time is updating every time I change the config file, so I have no idea why gatus isn't picking up the changes.

In fact, after multiple tests, only one time it has printed [main][listenToConfigurationFileChanges] Configuration file has been modified, reloaded itself and never again.

1player commented

I found a way to reproduce. tl;dr: this software doesn't play well with vim.

Steps to reproduce:

  1. Start gatus
  2. touch config.yaml, notice the service has picked up the changes
  3. nano config.yaml, notice the service has picked up the changes
  4. vim config.yaml, service doesn't notice the changes (this is a stock Ubuntu vim installation, important)
  5. Service stops picking up any change even with touch

Here's the reason: https://unix.stackexchange.com/a/188954

nano and touch update the file in place, which trigger gatus' configuration update mechanism, while vim by default writes on a temporary file and then overwrites the original file with it, and this breaks gatus. The biggest problem is once gatus reload system has been broken by vim, it will stop noticing any configuration change until it's restarted.

  • 👍1
TwiN commented

@1player Thanks for the heads up.

To answer your question, the person who created the issue is the one who closed it -- but it was somewhat reasonable at the time given that nobody else had reported this issue before.

In any case, I tried to reproduce it with my main machine, which runs Windows, to no avail:

  • Running Gatus locally, the changes were picked up
  • Running Gatus on Docker, the changes were picked up when modified from PS/CMD
  • Running Gatus on Docker, the changes were picked up when modified from bash (WSL)

But since this is the second person that brings it up, I gave it a shot my laptop which runs Manjaro, and:

  • Running Gatus locally, the changes were picked up
  • Running Gatus on Docker, the changes were not picked up

This is a pretty weird issue...

1player commented

I found the issue: https://stackoverflow.com/a/53548078

My Docker instance of gatus used a bind mount for the config.yaml file, as suggested by the README, as such: -v ./config/config.yaml:/config/config.yaml.

As that Stack Overflow link above suggests, bind mounts are based on inodes and when the file is deleted and recreated then the bind-mount is broken, so when vim replaces the original file with the updated one, the inode changes, and the bind mount breaks.

If you mount the whole parent directory instead of just the config file (-v ./config:/config), the app will reload as expected even when using vim.

EDIT: see also moby/moby#6011

  • 👍1
TwiN commented

@1player Great catch!

Would you be interested in updating the following examples?:

Basically, it's just adding a config folder, moving the config.yaml file in it and updating the docker-compose.yml files to bind the folder instead of the file.

(P.S. If you don't want to, no worries - I'll update them. I'm just offering it since you're the one who found the root cause)

1player commented

The first one to do it wins! :-)

A bit busy atm, if I find a moment to do it, I will push a PR, but if you're in front of the code, go ahead and thanks for taking care of updating the examples. Also perhaps we should update the README, and adding a note why binding the file itself is not the best.

TwiN commented

I added a small note in the README.md for now (see #166), but I'll let you take care of the examples if you don't mind 😄