Stopping the container removes existing content in modified folders
ohuseyinoglu opened this issue · 5 comments
At first, I thought this had to do with the particular plugin I was working with, but I simplified the repro case much more, and it happens by just starting, and stopping the container!?! So, it probably is about docker, or the way this image has been created (or, in the way these interact with my specific environment)
Repro case:
docker pull redmine:latest
# We don't care about DB setup (sqlite is fine), or port mapping
docker run -d --name rdmndbg redmine
# Check the differences made on file system (just by booting up redmine): note the lines in [1]
docker diff rdmndbg
# Stop the container
docker stop rdmndbg
# Get the diff again: Note the lines with "D" (deleted?) in [2]
docker diff rdmndbg
With the plugin install, since the bundle/gem folders are also touched, many more files get deleted, and the whole container falls apart!
I'm guessing such a major issue had to be specific to my environment somehow. If so, I'd appreciate a confirmation, or pointers in debugging the culprit. Thanks!
Environment:
- Oracle Linux 7.4: Kernel: 4.1.12-94.3.9.el7uek.x86_64
- docker-engine: 19.03.11.ol Release: 13.el7
- redmine:latest (i.e. 5.0.1 => "Created": "2022-05-29T04:39:32.038408404Z")
[1]
(Since the "public" folder exists in the base image, the "C"reation (?) here should not be necessary, but I'm guessing this is a typical way it's reported when create directory OS calls are made (for "plugin_assets") in the container)
C /usr/src/redmine/public
C /usr/src/redmine/public/plugin_assets
C /usr/src/redmine/public/plugin_assets/empty
[2]
D /usr/src/redmine/public/stylesheets
D /usr/src/redmine/public/404.html
D /usr/src/redmine/public/htaccess.fcgi.example
D /usr/src/redmine/public/images
D /usr/src/redmine/public/help
D /usr/src/redmine/public/500.html
D /usr/src/redmine/public/favicon.ico
D /usr/src/redmine/public/javascripts
D /usr/src/redmine/public/dispatch.fcgi.example
D /usr/src/redmine/public/themes
Additional Info: The same problem (repro case) occurs with 5.0.1-alpine3.15 and 4.2.6-bullseye images as well.
When I try it I don't get any of the D line additions in the second diff
$ docker run -d --name rdmndbg redmine
e71db795973f57358999fdb093410b11b6ba6b681a52d0ba76e2c98bdc939a33
$ docker diff rdmndbg > diff1
$ docker stop rdmndbg
rdmndbg
$ docker diff rdmndbg > diff2diff -y --suppress-common-lines diff1 diff2
> C /usr/src/redmine/log
> C /usr/src/redmine/db
> A /usr/src/redmine/db/schema.rb
> C /usr/src/redmine/public
> C /usr/src/redmine/public/plugin_assets
> C /usr/src/redmine/public/plugin_assets/empty
> C /usr/src/redmine/sqlite
> A /usr/src/redmine/sqlite/redmine.db
> C /usr/src/redmine/tmp
> C /usr/src/redmine/tmp/sockets
> C /usr/src/redmine/tmp/test
> C /usr/src/redmine/tmp/test/empty
> C /usr/src/redmine/tmp/thumbnails
> C /usr/src/redmine/tmp/thumbnails/empty
> C /usr/src/redmine/tmp/cache
> C /usr/src/redmine/tmp/imports
> C /usr/src/redmine/tmp/pdf
> C /usr/src/redmine/tmp/pids
> C /usr/src/redmine/tmp/sessions
> C /usr/src/redmine/config/additional_environment.rb.example
> C /usr/src/redmine/config/application.rb
> C /usr/src/redmine/config/boot.rb
C /usr/src/redmine/config/database.yml.example <
> C /usr/src/redmine/config/initializers/30-redmine.rb
> C /usr/src/redmine/config/initializers/backtrace_silencers.rb
> C /usr/src/redmine/config/initializers/inflections.rb
C /usr/src/redmine/config/initializers/30-redmine.rb | C /usr/src/redmine/config/settings.yml
C /usr/src/redmine/config/initializers/backtrace_silencers.rb | C /usr/src/redmine/config/additional_environment.rb
C /usr/src/redmine/config/initializers/inflections.rb | A /usr/src/redmine/config/database.yml
> C /usr/src/redmine/config/database.yml.example
> C /usr/src/redmine/config/environment.rb
> C /usr/src/redmine/config/environments
> C /usr/src/redmine/config/environments/development.rb
> C /usr/src/redmine/config/environments/production.rb
> C /usr/src/redmine/config/environments/test.rb
> C /usr/src/redmine/config/environments/test_pgsql.rb
> C /usr/src/redmine/config/environments/test_sqlite3.rb
C /usr/src/redmine/config/locales/eu.yml | C /usr/src/redmine/config/locales/el.yml
C /usr/src/redmine/config/locales/fi.yml | C /usr/src/redmine/config/locales/en-GB.yml
C /usr/src/redmine/config/locales/gl.yml | C /usr/src/redmine/config/locales/es.yml
C /usr/src/redmine/config/locales/he.yml | C /usr/src/redmine/config/locales/ro.yml
C /usr/src/redmine/config/locales/id.yml <
C /usr/src/redmine/config/locales/no.yml <
C /usr/src/redmine/config/locales/bs.yml <
C /usr/src/redmine/config/locales/zh.yml <
> C /usr/src/redmine/config/locales/uk.yml
C /usr/src/redmine/config/locales/ja.yml | C /usr/src/redmine/config/locales/es-PA.yml
C /usr/src/redmine/config/locales/pt.yml | C /usr/src/redmine/config/locales/et.yml
> C /usr/src/redmine/config/locales/eu.yml
> C /usr/src/redmine/config/locales/fi.yml
> C /usr/src/redmine/config/locales/no.yml
> C /usr/src/redmine/config/locales/pl.yml
> C /usr/src/redmine/config/locales/he.yml
> C /usr/src/redmine/config/locales/ko.yml
> C /usr/src/redmine/config/locales/sk.yml
> C /usr/src/redmine/config/locales/sr-YU.yml
> C /usr/src/redmine/config/locales/zh.yml
> C /usr/src/redmine/config/locales/bs.yml
> C /usr/src/redmine/config/locales/ca.yml
C /usr/src/redmine/config/locales/tr.yml | C /usr/src/redmine/config/locales/gl.yml
C /usr/src/redmine/config/locales/uk.yml | C /usr/src/redmine/config/locales/it.yml
C /usr/src/redmine/config/locales/ko.yml | C /usr/src/redmine/config/locales/pt.yml
C /usr/src/redmine/config/locales/el.yml | C /usr/src/redmine/config/locales/zh-TW.yml
C /usr/src/redmine/config/locales/es-PA.yml <
C /usr/src/redmine/config/locales/et.yml <
C /usr/src/redmine/config/locales/fa.yml <
C /usr/src/redmine/config/locales/fr.yml <
C /usr/src/redmine/config/locales/mk.yml <
C /usr/src/redmine/config/locales/nl.yml <
C /usr/src/redmine/config/locales/es.yml <
C /usr/src/redmine/config/locales/hu.yml <
C /usr/src/redmine/config/locales/pt-BR.yml <
C /usr/src/redmine/config/locales/ro.yml <
C /usr/src/redmine/config/locales/sr.yml <
C /usr/src/redmine/config/locales/en-GB.yml | C /usr/src/redmine/config/locales/mk.yml
C /usr/src/redmine/config/locales/it.yml | C /usr/src/redmine/config/locales/pt-BR.yml
C /usr/src/redmine/config/locales/lt.yml <
C /usr/src/redmine/config/locales/ca.yml <
C /usr/src/redmine/config/locales/sk.yml <
C /usr/src/redmine/config/locales/sr-YU.yml | C /usr/src/redmine/config/locales/sr.yml
C /usr/src/redmine/config/locales/zh-TW.yml | C /usr/src/redmine/config/locales/tr.yml
C /usr/src/redmine/config/locales/bg.yml | C /usr/src/redmine/config/locales/ar.yml
C /usr/src/redmine/config/locales/hr.yml | C /usr/src/redmine/config/locales/fr.yml
> C /usr/src/redmine/config/locales/hu.yml
> C /usr/src/redmine/config/locales/id.yml
> C /usr/src/redmine/config/locales/lt.yml
> C /usr/src/redmine/config/locales/bg.yml
> C /usr/src/redmine/config/locales/fa.yml
> C /usr/src/redmine/config/locales/hr.yml
> C /usr/src/redmine/config/locales/ja.yml
C /usr/src/redmine/config/locales/pl.yml | C /usr/src/redmine/config/locales/nl.yml
C /usr/src/redmine/config/locales/ar.yml <
C /usr/src/redmine/config/additional_environment.rb.example <
C /usr/src/redmine/config/application.rb <
C /usr/src/redmine/config/boot.rb <
A /usr/src/redmine/config/database.yml <
C /usr/src/redmine/config/environment.rb <
C /usr/src/redmine/config/environments <
C /usr/src/redmine/config/environments/development.rb <
C /usr/src/redmine/config/environments/production.rb <
C /usr/src/redmine/config/environments/test.rb <
C /usr/src/redmine/config/environments/test_pgsql.rb <
C /usr/src/redmine/config/environments/test_sqlite3.rb <
C /usr/src/redmine/config/settings.yml <
C /usr/src/redmine/config/additional_environment.rb <
C /usr/src/redmine/public <
C /usr/src/redmine/public/plugin_assets <
C /usr/src/redmine/public/plugin_assets/empty <
C /usr/src/redmine/log <
C /usr/src/redmine/tmp <
C /usr/src/redmine/tmp/sockets <
C /usr/src/redmine/tmp/test <
C /usr/src/redmine/tmp/test/empty <
C /usr/src/redmine/tmp/thumbnails <
C /usr/src/redmine/tmp/thumbnails/empty <
C /usr/src/redmine/tmp/cache <
C /usr/src/redmine/tmp/imports <
C /usr/src/redmine/tmp/pdf <
C /usr/src/redmine/tmp/pids <
C /usr/src/redmine/tmp/sessions <
C /usr/src/redmine/sqlite <
A /usr/src/redmine/sqlite/redmine.db <
A /usr/src/redmine/sqlite/redmine.db-journal <My host's OS is Debian Buster, it could be related to the host's filesystem or maybe Docker's storage driver?
$ docker info | grep -i storage
Storage Driver: overlay2I sorted the two docker diff files and the only difference is the pid file (it is unfortunate that docker diff doesn't have a consistent output order):
$ diff -u diff1 diff2
--- diff1 2022-06-10 10:48:33.512605652 -0700
+++ diff2 2022-06-10 10:48:45.956660768 -0700
@@ -2,7 +2,6 @@
A /usr/src/redmine/config/initializers/secret_token.rb
A /usr/src/redmine/db/schema.rb
A /usr/src/redmine/sqlite/redmine.db
-A /usr/src/redmine/tmp/pids/server.pid
C /usr
C /usr/src
C /usr/src/redmine
Thank you both for following up. I was also somewhat sure this had to do with my environment, since, (even though a big portion of containers are used in a throw-away fashion) a non-restartable redmine image would definitely be a major issue for all! :)
So, I tried a couple more things to narrow down the root cause:
- Other, older images:
- 4.2.6-passenger: Reproducible, issue exists
- 3.4.10: NOT reproducible! The public folder is kept as-is after the restart, no deletion happened.
- (didn't have time to disect/b-search further for the "first" image with the issue - let me know please though, if there were major changes b/w certain tags, and I'll try to make time)
- Docker Upgrade: 20.10.17: Reproducible, issue exists with the
latestimage (which was recently refreshed apparenty, and not quite the same with the one I've tried (even though it's still based on the same 5.0.1 redmine version)).
@wglambert , after learning a bit more about the way images, and docker storage driver works, I was also suspicious of the way "overlays" works (since it looks like the container layer changes were overriding too much of the image layer data). I also checked the info output, Didn't see anything striking right away (was not sure about Native Overlay Diff but the relevant pages I found were relating it to directory renames):
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: false
userxattr: false
Sorry, for further help debugging what you're seeing, I'd suggest a dedicated support forum, such as the Docker Community Forums, the Docker Community Slack, or Stack Overflow. 🙇