docker/for-win

Inotify on shared drives does not work

DeltaWhy opened this issue ยท 58 comments

DeltaWhy commented

I'm using docker-compose to run a Node.js application with nodemon. The application source is on the Windows host and mounted on the container so that I can make changes without rebuilding. However, nodemon does not pick up file changes. If I exec a shell inside the container and touch one of the files from there, nodemon picks it up and restarts the app, but any changes from Windows don't seem to trigger the inotify events.

friism commented

@DeltaWhy thanks, this is a known limitation that we're trying to figure out how to address.

nodemon has a fallback polling mode that I recommend you try: https://github.com/remy/nodemon#application-isnt-restarting

@londoncalling I checked the FAQ, and it looks like we don't have a mention that inotify doesn't work on Windows - it might be good to add.

  • ๐Ÿ‘9
  • ๐Ÿ‘€1
londoncalling commented

@friism adding it now to the FAQs

rn commented

@DeltaWhy We are using SMB for the file sharing and the Linux SMB implementation does not support change notifications. We have update the FAQ accordingly andI'm closing this issues.
Thanks for reporting

garveen commented

https://wiki.samba.org/index.php/Samba_3.0_Features_added/changed

Samba 3.0.25:

New file change notify subsystem which is able to make use of inotify on Linux.

rn commented

@garveen unfortunately, the linux kernel implementation of SMB (in ./fs/cifs) does not support inotify. There are various comments in the code that the older dnotify once was supported (and it is marked as experimental) but none of the code calls into the fsnotify API.

UPDATE: I think the above refers to the Linux SMB server being able to generate SMB change notification from inotify events. The opposite direction is what is needed by Docker for Windows: SMB change notifications generated by the server on the host need to be translated into the events on Linux

garveen commented

@rneugeba maybe other than smb:

If we have a tool that listen on native Windows' file system APIs, communicate with MobyLinux and generate Linux events, it would be great.

Does this idea practicable?

DeltaWhy commented

I assume Hyper-V doesn't have anything akin to VirtualBox shared folders? Inotify works great with Docker Toolbox as well as Vagrant.

rn commented

@DeltaWhy I haven't seen anything in the VBox source suggesting they support inotify on the shared folders, but I may have looked at the wrong place, but there seems plenty of evidence on the web that it doesn't, like for example: https://www.virtualbox.org/ticket/10660?cversion=0&cnum_hist=1

kidlj commented

@rneugeba @friism
Vagrant can make bi-directional file changes synchronization between Windows host and Linux guest machines. And Vagrant takes several ways to achieve this:

SMB: https://www.vagrantup.com/docs/synced-folders/smb.html
RSync: https://www.vagrantup.com/docs/synced-folders/rsync.html
Virtualbox shared folders: https://www.vagrantup.com/docs/synced-folders/virtualbox.html

Hope you can get some inspirations from there! Thank you D4W developers.

friism commented

@kidlj yes, moving files and file changes back and forth works fine (as it does with Docker for Windows), but the inotify change notifications are not, and not with any of those solutions that you linked to either, at least not that I know of. Do you have an example of inotify working with either SMB, RSync or vbox shared folders?

kidlj commented

@friism Thank you for the explanation. I thought inotify was the same as file changes synchronization, but now I get it.

I just found a vagrant solution for the same inotify issue, could it help?
https://github.com/adrienkohlbecker/vagrant-fsnotify

friism commented

@kidlj I don't think it works for Windows, but might be an interesting model. Maybe @rneugeba has thoughts.

DeltaWhy commented

I guess I was mistaken. It looks like mocha --watch works on both (I guess it's using polling) and nodemon doesn't by default, which is maybe why I got confused.

Rsync with rsync-auto should trigger inotify events since it's touching files on the Linux native FS, right? But that probably won't work for sharing a whole drive ๐Ÿ˜‰

Vagrant-fsnotify looks very interesting but I wonder if that approach would also have performance issues when sharing a whole drive. You might need a Linux kernel module to tell what's being watched on the CIFS mount so the host side can watch those same things rather than the whole drive.

blaggacao commented

This hurts too much for beeing closed.

  • ๐Ÿ˜„2
frankgreco commented

Why is this issue closed? I want to use Docker Native for local development across my entire organization but can't util inotify works for windows as well. I don't want to use polling because I want seamless integration between windows/mac code bases and don't want to change my webpack config files for example to use polling if you have a windows machine.

Is there someplace for this feature/enhancement can be tracked so I know that status of it.

  • ๐Ÿ‘41
  • โค๏ธ19
blaggacao commented

I guess strategic bets are to wait for windows linux subsystem. But some cracky C# person wouldn't be that limited as most of us to fix this in upstream samba quickly.

frankgreco commented

It would be nice if there was a Docker Native backlog that I could refer to so that I know when Docker is planning on address this (they say they will eventually as they want Docker for Mac/Windows to be the same). That way, I'd have an idea on when I can plan on having this implemented in my organization.

friism commented

[..] C# person wouldn't be that limited as most of us to fix this in upstream samba quickly

@blaggacao Samba is implemented in C: https://github.com/samba-team/samba

asolopovas commented

It looks like issue is so difficult to implement that developers don't even want to talk about it, that's why it gets closed without even discussing it.

  • ๐Ÿ˜•2
rn commented

@asolopovas

  • To implement this correctly with SMB/CIFS requires significant changes to the generic Linux Virtual Filesystem layer, not just improvements to the Linux Kernel SMB implementation.
  • Another option is to go the route we have taken with Docker for Mac, which uses a user-space file system (FUSE) in Linux and then passes this to a daemon on the host which implements the requires semantics. This requires mapping of Linux filesystem semantics (including users, permissions, symlinks, inotify etc) to NTFS filesystem semantics. The filesystems are very different and it's not possible in all cases to map the semantics. WSL helps a little bit, but it too has mis-matched semantics.
  • The other "solutions" mentioned in this thread all seem like hacks which come with fairly big caveats, so they may work in one case but maybe problematic in others.
  • ๐Ÿ‘1
  • โค๏ธ2
frankgreco commented

@rneugeba Is Docker planning on implementing this? Here is says, "Currently, inotify does not work on Docker for Windows. This will become evident, for example, when an application needs to read/write to a container across a mounted drive. This is a known issue that the team is working on."

Is there an estimated time for this functionality being available? Or, is there somewhere on the Docker Github where future/current work like this is officially proposed/tracked?

  • ๐Ÿ‘9
friism commented

@frankgreco I've sent a PR to update the docs. As @rneugeba pointed out, we don't have a lot of good options at the moment. We're aware that this is an annoying limitation, and we're continuously evaluating progress in WSL and Samba to see what improvements we might incorporate.

@frankgreco @asolopovas what programming framework are you using? Does it not have a polling mode for looking for filesystem changes, similar to nodemon?

  • ๐Ÿ‘1
  • ๐Ÿ˜•1
asolopovas commented

@friism I am using browsersync haven't found any polling modes in it but was searching for. I know that webpack has it but I use combination of browsersync with webpack and hot module reload and it does not seem to work if I enable poling just in webpack. I simply installed all necessary tools on my host so that I can operate looks like we just have to be a bit more patient.

trodrigues commented

I know +1s in issue comments don't help but I'd just like to add my voice here and say this is an issue that I also care about very much and would love to see it addressed in the future. In fact, I found this issue here today because I saw this was an issue a few months ago and decided to come back and look for what the current status of this situation was.

  • ๐Ÿ‘27
trodrigues commented

I was reading the updates about the new stuff in the Windows Subsystem for Linux on the Creators Update, and saw there's some bits there about inotify. Given that @friism mentioned WSL before, I wonder if this might have any impact on this issue?

https://blogs.msdn.microsoft.com/commandline/2017/04/11/windows-10-creators-update-whats-new-in-bashwsl-windows-console/

  • ๐Ÿ‘2
friism commented

@trodrigues I don't think so, I'm afraid. That code is not open source and the WSL Linux/Windows architecture is very different from Docker for Windows.

therealmikz commented

For anyone still struggling with this issue, there's a tool that works around this issue: http://blog.subjectify.us/miscellaneous/2017/04/24/docker-for-windows-watch-bindings.html

You just need to have python installed.

  • ๐Ÿ‘9
  • ๐ŸŽ‰1
  • โค๏ธ2
Corsseir commented

Any progress on this issue? This is really frustrating and the poll is not an option for me as it consumes much more cpu.

ninrod commented

any progress on this?

friism commented

As we make the transition to using Linux Containers on Windows, this is an enhancement that can be provided by the Windows platform: https://blog.docker.com/2017/09/preview-linux-containers-on-windows/

  • ๐Ÿ‘4
lightrow commented

why is this issue closed? It still doesn't work, even with LCOW on latest 18.02.0 version, and im still forced to use dirty workarounds

  • ๐Ÿ‘20
wgwz commented

Just for anyone else who ever runs into this, I followed the hint in this comment:

I don't want to use polling

And I started using polling :D
It works for me so far.

Running this in the container with volume mounted src, detects changes and reloads OK:

webpack --config webpack.debug.js --watch --watch-poll 1

Win10, source code volume mounted into the container like so:

  watch:
    build:
      context: .
      dockerfile: Dockerfile.local
    volumes:
     - .:/usr/src/app
    command: ['bash']
  • ๐Ÿ˜„1
  • ๐Ÿ˜•5
wintercounter commented

This really helped me too, worked like a charm: #56 (comment)
Guys, please add such support into Docker until there's a final solution, shouldn't be that hard, blog also tells what they're exactly doing.

nemzes commented

Another workaround that seems to do the job (from WSL too): https://github.com/FrodeHus/docker-windows-volume-watcher/releases

  • ๐Ÿ‘1
iki commented

The @merofeev docker-windows-volume-watcher in python mentioned by @therealmikz works perfectly. If you prefer go, there's also @FrodeHus docker-windows-volume-watcher in go mentioned by @nemzes.

In our case, I didn't use the common watchmedo command from @gorakhargosh python watchdog package, because it was too fast and restarted the service in docker several times on each change. Instead, I picked inotifywait from @rvoicilas inotifytools, and put sleep there. The Attrib events need to be monitored in this case, because the docker-windows-volume-watcher in python rewrites file permissions to trigger inotify. Also, you need to apt-get install inotify-tools in the Dockerfile, and it's helpful to know the trick to use inotifywait exclude regexp to actually only include particular extension.

This is how we use it in Makefile.

  • make run runs the service (inside docker container)
  • make dev runs the service and reloads on code change (inside docker container)
  • make start starts the docker container, and if run on Windows it also waits 10s and runs the volume watcher on background
PROJECT = project-name
CODE_DIR ?= src

run:
	# Starting local service...
	cd $(CODE_DIR) && python service.py

run-detached:
	make run &

dev:
	# Starting local service in development mode (will reload on code change)...
	@while true; do \
	  make run-detached; \
	  inotifywait -e create -e modify -e delete -e move -e attrib -r $(CODE_DIR) --exclude '[^.]..$$|[^p].$$|[^y]$$'; \
	  pkill -exf 'python service.py'; \
	  sleep 1; \
	done

watcher:
	@if [ "$(OS)" = 'Windows_NT' ]; then if where docker-volume-watcher >/dev/null 2>&1; then \
          sleep 10; \
	  echo "# Starting Docker volume watcher for Windows to enable reloading on code change..."; \
	  echo "# See https://github.com/docker/for-win/issues/56"; \
	  docker-volume-watcher '$(PROJECT)_*' 2>&1 | grep -v 'ERROR:root:Exec run returned non-zero exit code: 1'; \
	else \
	  echo "# Install Docker volume watcher for Windows to enable reloading on code change..."; \
	  echo "# > pip install docker-windows-volume-watcher"; \
	fi; fi

start:
	make watcher &
	# Starting containers...
	docker-compose -p $(PROJECT) up
jasonwilliams commented

Does anyone know why this is closed? from what i understand the Docker Guest still isn't notified when i change a file on a windows host. @iki @friism ?

We use Webpack.
Polling is not an option here as it causes our CPU to go mental and on MacOS host its not needed.

  • ๐Ÿ‘11
TheKnightCoder commented

I think running the docker engine on WSL could be a potential solution for this problem. WSL supports Inotify.
Have a look at this thread microsoft/WSL#2291

  • ๐Ÿ‘Ž1
marxangels commented

docker-machine ssh default touch /path/to/file/under/shared/folder

zippoxer commented

If anyone's interested, I've forked docker-windows-volume-watcher and made many improvements: https://github.com/zippoxer/docker-windows-volume-watcher

I made it almost 2x faster, added -ignore, added -delay (default 100ms) to accommodate the way editors save files, and fixed some bugs.

I hope this helps someone ;-)

(Still disappointed to see the problem hasn't been fixed yet for over two years now..)

  • ๐Ÿ‘9
peteromano commented

@zippoxer Exactly. i havent gotten "dwvm" to work yet, but yes, i was gonna comment the same thing - why is this still an issue??

Ill try your fork btw instead. (i didnt spend much time on trying dwvm yet, but so far, im getting "no container found" errors.) its probably just something on my end tho

Lukortech commented

Hello everyone!
I am writing here since there was no better place yet to be found.
I am using Win10.pro with Docker, but it seems that there are no changes seen by webpack too, and I don't understand why won't my front-end env. refresh after changes to the .ts files of any sort.

I've looked for any kind of help and it al lbrought me here. Waiting for an update. Hopefully windows users won't be second-class citiziens soon.

wintercounter commented
mastacheata commented

Does anyone yet have any results with Docker Desktop Community 2.1.6.0/2.1.6.1 which should've fixed this problem?
I tried, but without any success.
Quite the opposite, actually. It seems that the container is now unaware of inotify events inside it's own filesystem as well and no longer just forgets to broadcast the events to the containers. (volume watcher tools as well as touching/chmoding a file from docker exec seem to have no effect on either python or node's file-watchers. :(

djs55 commented

@mastacheata thanks for your port. Could you try the following on the latest edge release:

  • create a directory for testing
  • share the directory with docker run -it -v <path>:/foo djs55/inotify-event --dir /foo
  • create a subdirectory on the host inside the testing directory

If the inotify event injection is working then you should see events inside the docker run like this:

watch=1 cookie=0 events=CREATE "foo"

Here's a screen capture of it working locally for me: https://youtu.be/M6lpN65yF38

scottkurz commented

@djs55 thx for the instructions... I tried your inotify-event image with the edge Docker Desktop v2.1.6.1 and saw nothing happened when creating a subdir from the host with PS, Win Explorer, etc. Would be happy to try something else.

mastacheata commented

Same here (Docker Desktop 2.1.6.1 Build 40900, Docker Engine 19.03.5):

ฮป  docker run -it -v C:\Users\benedikt\foo:/foo djs55/inotify-event --dir /foo
Unable to find image 'djs55/inotify-event:latest' locally
latest: Pulling from djs55/inotify-event
7622164d525b: Pull complete
Digest: sha256:ce5540b724b37808d8933f80cf393e07bcb0a9e865891aa4e75786d8b316f67c
Status: Downloaded newer image for djs55/inotify-event:latest

And that's it. No more output from the container after that when creating directories inside the mounted directory.

https://youtu.be/PI-u1YO3pfY

francescobianco commented

@DeltaWhy @friism @scottkurz My solution is create a clone of INOTIFYWAIT full bash based on polling using a diff screenshot of filesystem, please take a look and support it work perfect on WINDOWS systems
https://github.com/javanile/inotifywait-polling

mastacheata commented

@DeltaWhy @friism @scottkurz My solution is create a clone of INOTIFYWAIT full bash based on polling using a diff screenshot of filesystem, please take a look and support it work perfect on WINDOWS systems
https://github.com/javanile/inotifywait-polling

Well, there are already 4-5 different versions of that. They all have downsides like not being able to transfer permission changes back and forth.

Docker Desktop on WSL2 officially supports inotify events on mounted volumes. It's just not working for me or @scottkurz โ˜น๏ธ

  • ๐Ÿ˜„1
scottkurz commented

To clarify I'm not using WSL2 (or WSL).

mastacheata commented

To clarify I'm not using WSL2 (or WSL).

Well, than nothing is supposed to change. The inotify fix was only announced for docker running in WSL2 mode.

francescobianco commented

@mastacheata please could you link example of this

Well, there are already 4-5 different versions of that.

djs55 commented

The latest edge release of Docker Desktop (2.1.7.0) should have better support for inotify on shared drives (in regular Linux mode, not WSL2) -- It's worth trying if you have the time. There is a known bug when used with docker-compose, see #5318 (comment) for a workaround.

  • ๐Ÿ‘1
mastacheata commented

@mastacheata please could you link example of this

Well, there are already 4-5 different versions of that.

And of course all the forks of each.
They basically watch Windows for changes and then change the permissions using docker exec of those files that changed in order to trigger a proper inotify event without changing the dates of the file (previous tooling used to do this by issueing a touch command inside the container, which would cause unwanted side effects)

kamko commented

The support for inotify that @djs55 mentioned in edge release is now in latest (2.2.0.0) stable release

Docker Desktop now supports inotify events on shared filesystems.
https://docs.docker.com/docker-for-windows/release-notes/#docker-desktop-community-2200

  • ๐ŸŽ‰5
djs55 commented

I hope the inotify support works for you!

If not, please repro the problem (e.g. change a file on the host), capture a diagnostics and make a fresh ticket.

madacol commented

How lucky of me!!!
I encounter this bug today and hours later its fixed with the update.

I had the same problem as OP.
A NodeJs app with nodemon, using docker-compose

docker-robott commented

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked