fusionauth-app won't run due to filesystem permissions issue

[smcoll]

Having a permissions issue when running a fusionauth-app container. Same results for 1.15.2 all the way down to 1.13. For example:

$ docker run -it fusionauth/fusionauth-app:1.15.2
mkdir: cannot create directory '/usr/local/fusionauth/fusionauth-app/apache-tomcat/../../logs': Permission denied
Using CATALINA_BASE:   /usr/local/fusionauth/fusionauth-app/apache-tomcat
Using CATALINA_HOME:   /usr/local/fusionauth/fusionauth-app/apache-tomcat
Using CATALINA_TMPDIR: /usr/local/fusionauth/fusionauth-app/apache-tomcat/temp
Using JRE_HOME:        /usr/local/fusionauth/fusionauth-app/apache-tomcat/../../java/current
Using CLASSPATH:       /usr/local/fusionauth/fusionauth-app/apache-tomcat/bin/bootstrap.jar:/usr/local/fusionauth/fusionauth-app/apache-tomcat/bin/tomcat-juli.jar
Feb 20, 2020 5:28:25 PM org.apache.catalina.startup.Catalina load
WARNING: Unable to load server configuration from [/usr/local/fusionauth/fusionauth-app/apache-tomcat/conf/server.xml]
Feb 20, 2020 5:28:25 PM org.apache.catalina.startup.Catalina start
SEVERE: Cannot start server. Server instance is not configured.

Seems it's trying to create /usr/local/fusionauth/logs but unable due to permissions.

Using a dockerfile based on https://github.com/FusionAuth/fusionauth-containers/blob/1.15.2/docker/fusionauth/fusionauth-app/Dockerfile but using bash as the entrypoint, i'm seeing these permissions:

$ cd /usr/local/fusionauth/fusionauth-app/
$ ls -al
total 24
drwxr-sr-x 6 root fusionauth 4096 Feb 20 20:30 .
drwxr-sr-x 1 root fusionauth 4096 Feb 20 20:30 ..
drwxrwsr-x 8 root fusionauth 4096 Feb 19 11:58 3rd-party-licenses
drwxrwsr-x 6 root fusionauth 4096 Feb 19 11:58 apache-tomcat
drwxrwsr-x 2 root fusionauth 4096 Feb 19 11:58 template
drwxrwsr-x 7 root fusionauth 4096 Feb 19 11:58 web
$ cd ..
$ mkdir logs
mkdir: cannot create directory 'logs': Permission denied

Given that the Docker user is fusionauth, and root owns /usr/local/fusionauth, it doesn't work. Additionally, this user can't do anything in the apache-tomcat/conf directory:

$ cd /usr/local/fusionauth/fusionauth-app/apache-tomcat
$ ls -al
total 140
drwxrwsr-x 6 root fusionauth  4096 Feb 19 11:58 .
drwxr-sr-x 6 root fusionauth  4096 Feb 20 20:30 ..
-rw-r----- 1 root fusionauth 19534 Feb 19 11:58 BUILDING.txt
-rw-r----- 1 root fusionauth  5407 Feb 19 11:58 CONTRIBUTING.md
-rw-r----- 1 root fusionauth 57011 Feb 19 11:58 LICENSE
-rw-r----- 1 root fusionauth  1726 Feb 19 11:58 NOTICE
-rw-r----- 1 root fusionauth  3255 Feb 19 11:58 README.md
-rw-r----- 1 root fusionauth  7139 Feb 19 11:58 RELEASE-NOTES
-rw-r----- 1 root fusionauth 16262 Feb 19 11:58 RUNNING.txt
drwxr-s--- 2 root fusionauth  4096 Feb 19 11:58 bin
drwx--S--- 2 root fusionauth  4096 Feb 19 11:58 conf
drwxr-s--- 2 root fusionauth  4096 Feb 19 11:58 lib
drwxr-s--- 2 root fusionauth  4096 Feb 19 11:58 temp
$ cd conf/
bash: cd: conf/: Permission denied

At first i thought this was the same as #31, which was closed yesterday, but i don't think that's the case.

[robotdan]

@smcoll can you confirm that you have the latest images? 1.15.0 and 1.15.1 were re-released after we corrected the permission issue.

@tyduptyler13 can you take a look to make sure this is no longer an issue?

[smcoll]

Yes, i noticed in the conversation at #31 that the images were updated, so i deleted my local images and pulled again. Notably, though, this problem isn't only for 1.15.0 and 1.15.1 - i also see this issue with 1.15.2, 1.14.x, and 1.13.x (that's as far as i tested).

[tyduptyler13]

I can't seem to reproduce this problem. But based on your logs it does seem that your images have a permissions issue. The fact that it spans to 1.14.x and 1.13.x tells me that something odd is going on because the bug we introduced in 1.15.x (and fixed in #31) wouldn't affect those versions.

The logs issue (the reason we changed the docker file in 1.15.x) was fixed in 1.15.x. Is that what you mean when you say there is a permission issue in 1.13.x and 1.14.x?

docker run -itu root fusionauth/fusionauth-app:1.15.2 bash
root@36124d699a80:/# cd /usr/local/fusionauth/fusionauth-app/apache-tomcat/
root@36124d699a80:/usr/local/fusionauth/fusionauth-app/apache-tomcat# ls -lh
total 132K
-rw-r----- 1 fusionauth fusionauth  20K Feb 19 11:58 BUILDING.txt
-rw-r----- 1 fusionauth fusionauth 5.3K Feb 19 11:58 CONTRIBUTING.md
-rw-r----- 1 fusionauth fusionauth  56K Feb 19 11:58 LICENSE
-rw-r----- 1 fusionauth fusionauth 1.7K Feb 19 11:58 NOTICE
-rw-r----- 1 fusionauth fusionauth 3.2K Feb 19 11:58 README.md
-rw-r----- 1 fusionauth fusionauth 7.0K Feb 19 11:58 RELEASE-NOTES
-rw-r----- 1 fusionauth fusionauth  16K Feb 19 11:58 RUNNING.txt
drwxr-s--- 2 fusionauth fusionauth 4.0K Feb 19 11:58 bin
drwx--S--- 2 fusionauth fusionauth 4.0K Feb 19 11:58 conf
drwxr-s--- 2 fusionauth fusionauth 4.0K Feb 19 11:58 lib
drwxr-s--- 2 fusionauth fusionauth 4.0K Feb 19 11:58 temp

[smcoll]

Yes, i'm saying i have this problem with 1.13.x, 1.14.x, and 1.15.x. So it's a problem with my local Docker setup?

$ docker run -itu root fusionauth/fusionauth-app:1.15.2 bash
Unable to find image 'fusionauth/fusionauth-app:1.15.2' locally
1.15.2: Pulling from fusionauth/fusionauth-app
619014d83c02: Already exists 
7850519a51c4: Pull complete 
55d94306ce26: Pull complete 
301f440c018b: Pull complete 
Digest: sha256:c58f9b4895c0eaf851abd80adcc83012a1a0ecf615deba6a31f0a0dc35e69295
Status: Downloaded newer image for fusionauth/fusionauth-app:1.15.2
root@319576e05062:/# cd /usr/local/fusionauth/fusionauth-app/apache-tomcat/
root@319576e05062:/usr/local/fusionauth/fusionauth-app/apache-tomcat# ls -lh
total 132K
-rw-r----- 1 root fusionauth  20K Feb 19 11:58 BUILDING.txt
-rw-r----- 1 root fusionauth 5.3K Feb 19 11:58 CONTRIBUTING.md
-rw-r----- 1 root fusionauth  56K Feb 19 11:58 LICENSE
-rw-r----- 1 root fusionauth 1.7K Feb 19 11:58 NOTICE
-rw-r----- 1 root fusionauth 3.2K Feb 19 11:58 README.md
-rw-r----- 1 root fusionauth 7.0K Feb 19 11:58 RELEASE-NOTES
-rw-r----- 1 root fusionauth  16K Feb 19 11:58 RUNNING.txt
drwxr-s--- 2 root fusionauth 4.0K Feb 19 11:58 bin
drwx--S--- 2 root fusionauth 4.0K Feb 19 11:58 conf
drwxr-s--- 2 root fusionauth 4.0K Feb 19 11:58 lib
drwxr-s--- 2 root fusionauth 4.0K Feb 19 11:58 temp

[robotdan]

I am not a Docker expert so I could be mistaken, but it looks like you trying to run the image under the root user?

If you call docker run -it fusionauth/fusionauth-app:1.15.2 for example, it will default to the user specified in the Dockerfile which is fusionauth.

When you call docker run -itu root fusionauth/fusionauth-app:1.15.2 - you're starting the container as root - which is not a good idea, and possibly the cause of the permission issue.

@tyduptyler13 any thoughts on this?

[smcoll]

i think @tyduptyler13 was invoking the container as root to be able to see permissions on files and directories unavailable to the fusionauth user. That i'm seeing different file ownership altogether is the problem- it's probably something with my local setup.

[robotdan]

Ah, ok, sorry I missed that part. Thanks for the clarification.

[smcoll]

Yeah, the situation is that i have Docker user namespacing configured to map the container's root user to my host user. So when it runs inside the container as user 1000, there are permissions issues.

[smcoll]

For posterity, i have user namespace remapping configured on my linux host, to map processes to user uid 1000+. Since the search and fusionauth images have a user at uid 1000, this mapped to user 2000 (1000+1000) on my machine, which couldn't write to the mounted volumes. i disabled user namespace remapping for each service in the Dockerfile with userns_mode: "host" and it's working fine. i think this is contingent of my host user being uid 1000 in the first place, and note that the postgres user is uid 999.

[robotdan]

Thanks for the update @smcoll is there perhaps a better way for us to be handling the assigned UIDs int he Dockerfiles?

We've put together what we thought to be a common pattern, but we are still learning a bit about the Docker world.

[smcoll]

From what i've learned, there doesn't seem to be a standard solution for Linux hosts, for what is essentially a Unix issue, not a Docker problem per se. There's a long-standing Docker ticket, but not a clear path to resolution.

Windows and Mac hosts don't have these permissions issues because the uid/gid namespaces are being emulated, rather than leveraged, when managing volumes. i think it's up to the user to understand what uid will be running the containers and what uid will be used within each container (and so owning files created in the mounted volumes) , although a bit of documentation to that effect might eliminate some digging.

Here's a pretty good overview of the situation, and here is a list of some common approaches a developer might take to work around it.