Insufficient permissions on /data volume
Closed this issue · 24 comments
I am starting a project from scratch (make clean, make prepare, make build, make start) with normal user inside docker group.
Following error occurs in the OCR-D Manager when running the ocr script in Kitodo.Production.
today at 12:48:25Sep 1 10:48:24 ocrd-manager for_production.sh: insufficient permissions on /data volume
The data permissions and user group assignments are following in "kitodo" directory. For the OCR-D Manager only the "metadata" directory is mounted:
dr-xrwxrwx 2 root root 4096 Sep 1 10:29 config
drwxrwxrwx 2 root root 4096 Sep 1 10:29 debug
drwxrwxrwx 2 root root 4096 Sep 1 10:29 diagrams
dr-xrwxrwx 2 root root 4096 Sep 1 10:29 import
drwxrwxrwx 2 root root 4096 Sep 1 12:36 logs
dr-xrwxrwx 2 root root 4096 Sep 1 10:29 messages
drwxr-xr-x 3 weigelt domänen-benutzer 4096 Apr 5 10:14 metadata
drwxrwxrwx 2 root root 4096 Sep 1 10:29 modules
drwxr-xr-x 2 weigelt domänen-benutzer 4096 Jun 30 14:01 ocr_workflows
dr-xrwxrwx 7 root root 4096 Sep 1 10:29 plugins
drwxr-xr-x 2 weigelt domänen-benutzer 4096 Jun 17 12:16 rulesets
drwxr-xr-x 2 weigelt domänen-benutzer 4096 Jun 15 11:22 scripts
drwxrwxrwx 2 root root 4096 Sep 1 10:29 swap
drwxrwxrwx 2 root root 4096 Sep 1 10:29 temp
drwxrwxrwx 2 root root 4096 Sep 1 10:29 users
drwxr-xr-x 2 weigelt domänen-benutzer 4096 Jul 1 15:37 xslt
I think unzipping the resource data.zip explains the rights of files and directories that are not root. So i can change them to root but i wonder why the files and directories run on the root user. Cause with our UID and GID implementations the files and directories should have the current user and group as owne.
In my case current user id is "946828167" and group id "946800513" and for root user id is "0" group id "0".
When i go into the OCR-D Manager container UID and GID have the default value "1001" of .env file. So it seems env variables are not set correctly and probably root is then the fallback.
When i print the environment variables of Makefile following output is generated:
CONTROLLER_ENV_UID is 946828167
CONTROLLER_ENV_GID is 946800513
MANAGER_ENV_UID is 946828167
MANAGER_ENV_GID is 946800513
MODE is managed
COMPOSE_FILE is docker-compose.yml:docker-compose.kitodo-app.yml:docker-compose.managed.
docker inspect kitodo_production_ocrd-ocrd-manager-1 shows that env variables are not replaced with environment variables defined in Makefile.
"Config": {
"Hostname": "ocrd-manager",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": true,
"AttachStderr": true,
"ExposedPorts": {
"22": {},
"22/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"GID=1001",
"UMASK=0002",
"CONTROLLER=ocrd-controller:22",
"ACTIVEMQ=kitodo-mq:61616",
"UID=1001",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"DEBIAN_FRONTEND=noninteractive",
"PYTHONIOENCODING=utf8",
"LC_ALL=C.UTF-8",
"LANG=C.UTF-8",
"HOME=/",
"ACTIVEMQ_CLIENT_LOG4J2=/opt/kitodo-activemq-client/log4j2.properties",
"ACTIVEMQ_CLIENT=/opt/kitodo-activemq-client/kitodo-activemq-client-0.2.jar",
"PREFIX=/usr",
"VIRTUAL_ENV=/usr"
],
@bertsky Here is the permission problem what i mentioned in our dev meeting last week. Do you have any idea?
I think unzipping the resource data.zip explains the rights of files and directories that are not root.
Yes.
So i can change them to root but i wonder why the files and directories run on the root user. Cause with our UID and GID implementations the files and directories should have the current user and group as owne.
Not necessarily: we do pass the current $(id -u) and $(id -g) values as $UID and $GID into Manager and Controller containers, but not into Kitodo.
I don't know how specifically Kitodo then proceeds to create all the files under APP_DATA from _resources/data, but likely it just runs as root and therefore does not care about the ownership of the latter.
docker inspect kitodo_production_ocrd-ocrd-manager-1shows that env variables are not replaced with environment variables defined in Makefile.
Yes, that's odd. It works in my (also non-default) case:
"UID=1003",
"GID=100",
"UMASK=0002",
"CONTROLLER=sdvocr.slub-dresden.de:8022",
Perhaps the .EXPORT_ALL_VARIABLES failed. Which version of make are you running?
(Mine is 4.2.1)
Another possible explanation: you still have some prior definitions of these variables in your shell. Since in the Makefile we only define these as opt-in…
…any shell env value would override them.
I think unzipping the resource data.zip explains the rights of files and directories that are not root.
Yes.
So i can change them to root but i wonder why the files and directories run on the root user. Cause with our UID and GID implementations the files and directories should have the current user and group as owne.
Not necessarily: we do pass the current
$(id -u)and$(id -g)values as$UIDand$GIDinto Manager and Controller containers, but not into Kitodo.
Oh yes that is the reason! If I use only Kitodo.Production without the OCR-D implementations i run also in the permission problems so therefore I will adapt the permissions analogous the OCR-D services to fix it for all usages.
I don't know how specifically Kitodo then proceeds to create all the files under
APP_DATAfrom_resources/data, but likely it just runs as root and therefore does not care about the ownership of the latter.
docker inspect kitodo_production_ocrd-ocrd-manager-1shows that env variables are not replaced with environment variables defined in Makefile.Yes, that's odd. It works in my (also non-default) case:
"UID=1003", "GID=100", "UMASK=0002", "CONTROLLER=sdvocr.slub-dresden.de:8022",Perhaps the
.EXPORT_ALL_VARIABLESfailed. Which version ofmakeare you running?(Mine is 4.2.1)
I use 4.2.1 too.
…any shell env value would override them.
For example, this could have happened if you ever sourced your .env file.
Another possible explanation: you still have some prior definitions of these variables in your shell. Since in the Makefile we only define these as opt-in…
…any shell env value would override them.
Yes. In my these variables are not defined in shell before using make.
…any shell env value would override them.
For example, this could have happened if you ever sourced your
.envfile.
I don't include using source and i used new terminal too.
I don't include using source and i used new terminal too.
Could be in .bashrc though. So there is no $MANAGER_ENV_UID?
Ah, you answered that one above already.
I think .EXPORT_ALL_VARIABLES fails. So that will be the reason.
Cause in my local (atm) make "info" target, env values are defined by make correctly.
CONTROLLER_ENV_UID is 946828167
CONTROLLER_ENV_GID is 946800513
MANAGER_ENV_UID is 946828167
MANAGER_ENV_GID is 946800513
I think
.EXPORT_ALL_VARIABLESfails. So that will be the reason.
It's hard to think how it actually could, though.
What does make help show for these values?
It could still be some .bashrc thing. Your terminal's shell might be differently initialized than the bash which make is invoking...
I think
.EXPORT_ALL_VARIABLESfails. So that will be the reason.It's hard to think how it actually could, though.
What does
make helpshow for these values?It could still be some .bashrc thing. Your terminal's shell might be differently initialized than the bash which make is invoking...
Variables:
- CONTROLLER_ENV_UID user id to use on the OCR-D Controller (default: 946828167)
- CONTROLLER_ENV_GID group id to use on the OCR-D Controller (default: 946800513)
- MANAGER_ENV_UID user id to use on the OCR-D Manager (default: 946828167)
- MANAGER_ENV_GID group id to use on the OCR-D Manager (default: 946800513)
- MODE if 'managed', also starts/stops OCR-D Controller here (default: managed)
- in addition, all variables defined in .env can be overridden via shell or make
Same. I overlooked that make help prints these variables too so the make info is not necessary anymore. I have not made any adjustments to the bash.rc. Beside that when i printed all my env over printenv an these envs are not defined.
Do you happen to have a $BASH_ENV, or some Manager related defs in ~/.profile or ~/.bash_profile?
Also, for diagnostics, could you add the following recipe lines to help (after the EOF line, but before endef:
echo UID=$$MANAGER_ENV_UID GID=$$MANAGER_ENV_GIDDo you happen to have a
$BASH_ENV, or some Manager related defs in~/.profileor~/.bash_profile?
No ~/.bash_profile does not exist and ~/.profile has not such variable.
Also, for diagnostics, could you add the following recipe lines to
help(after theEOFline, but beforeendef:echo UID=$$MANAGER_ENV_UID GID=$$MANAGER_ENV_GID
Change results in following output:
...
Variables:
- CONTROLLER_ENV_UID user id to use on the OCR-D Controller (default: 946828167)
- CONTROLLER_ENV_GID group id to use on the OCR-D Controller (default: 946800513)
- MANAGER_ENV_UID user id to use on the OCR-D Manager (default: 946828167)
- MANAGER_ENV_GID group id to use on the OCR-D Manager (default: 946800513)
- MODE if 'managed', also starts/stops OCR-D Controller here (default: managed)
- in addition, all variables defined in .env can be overridden via shell or make
UID=946828167 GID=946800513
Ok, so the problem is not on make's side, the variables do get exported.
You are absolutely sure the Container was started afresh (not from a stopped earlier instance)?
(Try:
docker stop kitodo_production_ocrd_ocrd-manager_1
docker container rm kitodo_production_ocrd_ocrd-manager_1
make start
docker inspect kitodo_production_ocrd_ocrd-manager_1 | grep -e UID -e GID)
Yes i'm absolutely sure that the Container is fresh. I checkout fpr test purposes your PR and run a new setup an for with explicite docker-compose build --no-cache.
In addition I have also executed the script. I only had to replace the container name because it is different for me.
After running application i got following result:
weigelt@LDV163:~/Work/OCR-D/kitodo_production_ocrd$ docker stop kitodo_production_ocrd-ocrd-manager-1
kitodo_production_ocrd-ocrd-manager-1
weigelt@LDV163:~/Work/OCR-D/kitodo_production_ocrd$ docker container rm kitodo_production_ocrd-ocrd-manager-1
kitodo_production_ocrd-ocrd-manager-1
weigelt@LDV163:~/Work/OCR-D/kitodo_production_ocrd$ make start
docker-compose up -d
[+] Running 8/8
⠿ Container kitodo_production_ocrd-dozzle-1 Running 0.0s
⠿ Container kitodo_production_ocrd-kitodo-mq-1 Running 0.0s
⠿ Container kitodo_production_ocrd-ocrd-controller-1 Running 0.0s
⠿ Container kitodo_production_ocrd-ocrd-monitor-1 Running 0.0s
⠿ Container kitodo_production_ocrd-kitodo-es-1 Running 0.0s
⠿ Container kitodo_production_ocrd-ocrd-manager-1 Started 0.4s
⠿ Container kitodo_production_ocrd-kitodo-db-1 Running 0.0s
⠿ Container kitodo_production_ocrd-kitodo-app-1 Running 0.0s
weigelt@LDV163:~/Work/OCR-D/kitodo_production_ocrd$ docker inspect kitodo_production_ocrd-ocrd-manager-1 | grep -e UID -e GID
"GID=1001",
"UID=1001",
weigelt@LDV163:~/Work/OCR-D/kitodo_production_ocrd$ make help
Targets:
- prepare: create directories, SSH key files and examples
- build: `docker-compose build` all images
- start: `docker-compose up` all containers (in detached mode)
- down: `docker-compose down` all containers (i.e. stop and remove)
- stop: `docker-compose stop` all containers (i.e. only stop)
- config: dump all the composed files
- status: list running containers
Variables:
- CONTROLLER_ENV_UID user id to use on the OCR-D Controller (default: 946828167)
- CONTROLLER_ENV_GID group id to use on the OCR-D Controller (default: 946800513)
- MANAGER_ENV_UID user id to use on the OCR-D Manager (default: 946828167)
- MANAGER_ENV_GID group id to use on the OCR-D Manager (default: 946800513)
- MODE if 'managed', also starts/stops OCR-D Controller here (default: managed)
- in addition, all variables defined in .env can be overridden via shell or make
UID=946828167 GID=946800513
Mysterious!
What versions do you use? (I have docker-compose 1.29 und Docker 20.10. But the environment key has been there for a long time, and so has the general variable interpolation mechanism. Thus I doubt versions are the problem.)
Yes it is! My versions are Docker Compose Version 2.9.0 and Docker Version 20.10.12.
So i will add the user to the kitodo-production-docker repo and check if same problem occurs there too.
Ooof. I thought the latest is 1.29 (since this is the most recent version on PyPI, and the most recent docker-compose for Debian is also 1.29 for unstable and even 1.21 for stable/buster).
I can get docker-compose-plugin though for my distro (via Docker apt repo), which brings docker compose (mind the gap) 2.6. The two in parallel don't even conflict. Very strange.
Could you elaborate where you got your compose from (and which you are referring to, docker-compose --version or docker compose version)?
I installed both from the https://download.docker.com/linux/ubuntu source in my linux mint environment.
Okay, regardless how and why on your system the official Docker repo offers a more up to date version of docker-compose than mine: Let's analyse what specifically broke in v2 with our environment variable passing.
I think this warning (fixed here) might be the clue. Could you please try updating to either 2.10 or downgrading to 2.8?
Upgraded docker-compose to 2.10.x and now it works for me again.
docker inspect kitodo_production_ocrd_ocrd-manager_1 | grep -e UID -e GID
"UID=946828167",
"GID=946800513",
docker inspect kitodo_production_ocrd_ocrd-controller_1 | grep -e UID -e GID
"UID=946828167",
"GID=946800513",
I can get
docker-compose-pluginthough for my distro (via Docker apt repo), which bringsdocker compose(mind the gap) 2.6. The two in parallel don't even conflict. Very strange.
Yes docker compose and docker-compose seemed to be different. I uninstalled docker-compose and added a script docker-compose under /usr/bin witch include a compatibility call docker compose --compatibility "$@"
Seems to work with new Docker Compose version so i close this issue.