Bind mounts are really slow
Miksu82 opened this issue · 39 comments
- I have tried with the latest version of my channel (Stable or Edge)
- I have uploaded Diagnostics
- Diagnostics ID: CE10E9E0-4D58-4D8C-8533-82134EF088B2/20190520070056
Disk IO from/to container to/from host is really slow. I am running some IO heavy unit/integration/end-to-end test workloads that take ages to complete. My co-workers using Linux don't have that problem
I know this has been discussed in several issues already (such as #2707, #1592) and buried inside them may have suggestions for configurations that should improve performance or comments that docker-sync should be used etc.
But it would be nice to have the following from the Docker team:
- Documentation what causes this issue.
- Documentation and how-to's to improve the performance (configurations?, VirtualBox instead of HyperKit?)
- Is there work on-going to improve situation. If there is, is there a pull-request/ticket/branch or something else where this work can be followed and collaborated.
Expected behavior
File reads/writes between container and host are ~same as inside container.
$ docker run --rm -it -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real 0m 0.37s
user 0m 0.03s
sys 0m 0.31s
Actual behavior
$ docker run --rm -it -v "$(PWD):/pwd:delegated" -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real 0m 31.82s
user 0m 0.34s
sys 0m 3.04s
$ docker run --rm -it -v "$(PWD):/pwd:cached" -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real 0m 34.16s
user 0m 0.21s
sys 0m 3.78s
Information
- macOS Version: 10.14.4
- Docker for Mac: 2.0.0.3
- Docker Engine: 18.09.2
- Docker Machine: 0.16.1
Steps to reproduce the behavior
$ docker run --rm -it -v "$(PWD):/pwd" -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
I'm also experiencing similar issue with the 2.0.0.3 version on Docker for mac, same configuration as user ahead. My web projects based on Drupal are now so slow I can not even load a page.
➜ docker run --rm -it -v "$(PWD):/pwd" -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real 0m 27.87s
user 0m 0.14s
sys 0m 3.53s
I also experience this with 2.0.0.3. (macOS version 10.14.3)
$ docker run --rm -it -v "$(PWD):/pwd" -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real 1m 16.87s
user 0m 0.47s
sys 0m 4.96s
Restarting the Docker daemon does help somewhat, albeit temporarily.
$ docker run --rm -it -v "$(PWD):/pwd" -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real 0m 40.75s
user 0m 0.26s
sys 0m 4.67s
Diagnostics: 7536A828-0956-4629-9EB0-47BB753EA663/20190604034728
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.
Prevent issues from auto-closing with an /lifecycle frozen comment.
If this issue is safe to close now please do so.
Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale
Not fixed
/remove-lifecycle stale
/lifecycle frozen
Just my experience:
I am currently using Docker Desktop 2.1.0.3 (macOS Mojave & Catalina) for develop in nodejs with nodemon.
Everytime nodemon restart the nodejs process due to a change, the entire startup process gets slower and slower till it becomes unusable, not only the actual container but also the other containers running on my Mac.
The only "workaround" is to restart Docker Desktop entirely.
Also my colleagues experience the same behaviour, they have grunt that reload the project for every change.
It is still happening.
Are there NOT enough Mac Docker user which would make this as a priority to fix ASAP?
Docker is a great tool if you are on Linux, not so much on Windows or Mac, but it could be if annoying performance errors/bugs like this would be fixed quick, as otherwise it is way more easier to use Docker on Linux natively which is more productive.
I have tried to use the docker-sync project, but I think it is not yet workable it was working for 1 week, than all of a sudden not and I did not even updated my Mac nor changed anything in the docker-compose file.
Why do we need an project like docker-sync when this should be a priority to fix quickly to natively work properly?
It seems because some project manager/leader in Google/Docker just seems not care enough to push the issue through to be fixed FAST!!!!
I hope it will be fixed, otherwise it is just a crap thing on the Mac which is used for marketing or false sense of quality.
+1 UP.
+1
8sec on a RaspberryPi
vs
50sec on a Macbook Pro
Dump your macbooks and start carrying a RPi.
How do you think, downgrading to boot2docker (CLI for virtualbox with guest linux system) solve this problem?
Any news?
8sec on a RaspberryPi
vs
50sec on a Macbook ProDump your macbooks and start carrying a RPi. – @sul4bh
Now this is where one can see how much interest there is from the docker Team!!!!! Sad but true! Action talks, bullsh*t walks!!
30USD RPi vs even my 3000USD MBP touchbar mac with fast 3000MB/s storage speed!!!
Go RPi!!!!
It is terrible to have such slow performance on bind mount.
I've read some articles about NFS and helper tools.
My working solution
- macOS Catalina 10.15.4
- Docker desktop 2.2.0.5
sudo nano /etc/exports
# Add following line
/System/Volumes/Data -alldirs -mapall=501:20 localhostWhere 501 is User ID id -u and 20 is Group ID id -g.
sudo nano /etc/nfs.conf
# Add following line
nfs.server.mount.require_resv_port = 0Restart NFS daemon:
sudo nfsd restartTo mount ${PWD}/html folder to /var/www/html inside container:
version: '2.4'
services:
httpd:
volumes:
- html:/var/www/html
volumes:
html:
driver: local
driver_opts:
type: nfs
o: addr=host.docker.internal,rw,nolock,hard,nointr
device: ":${PWD}/html"Limitation
All files and folders under mount are writable but chmod/chown commands doesn't work.
It seems to be fixed in the recent version of Docker Desktop Edge 2.3.1.0.
https://docs.docker.com/docker-for-mac/edge-release-notes/#docker-desktop-community-2310
Here is the video presentation from the official Docker youtube channel.
https://www.youtube.com/watch?v=gyddZyc8r48
@GarryOne just read your comment and I feel the need to tell you this information made my day. I watched the video, switched to edge, started using the sync feature and WOW. Thank you!
Tried Docker Desktop Edge 2.3.4.0 without any performance improvement over the stable one.
@AidasK did you set up file syncing for the directories in question? It's under preferences.
@krancour I have deleted my volume and changed :cached to :delegated and my problem was gone. It's now as fast as it should be
I guess it is back to square one with this issue. Quote from Edge 2.3.5.0 release notes
The Mutagen file sync feature that we have been experimenting with in recent Edge releases has been removed. Thanks to everyone who has provided feedback on this feature. We are reconsidering how to integrate it based on the feedback we have received so far.
Looking for an update or possible work around here:
We have developers on Windows + WSL2, and developers on Linux that can run a docker-compose and build our product over the bind-mounted volumes at near native speed. The IO is great.
For the Mac developers, it is abysmally slow to use the bind mount. It's taking 25secs 6secs on a Mac bind mount to write a 1Gig file: (edit: performed testing on a more equivalent machine, still 6x slower)
docker run --rm -it -v $PWD:/test ubuntu bash -c "dd if=/dev/zero of=/test/io.dmp bs=1G count=1 oflag=dsync && rm -f /test/io.dmp"
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.06602 s, 177 MB/s
On Windows? About 1 second!
For high IO workloads, this is the difference between a build taking 15mins vs. 1hr +
Is there any workaround for this?
Been going on for years now. Mutagen gave hope, then that was pulled (don't understand why). Today:
- Docker for Mac 3.1.0
- macOS 11.1
Bind mount (cached or delegated makes no difference):
$ docker run --rm -it -v "$(PWD):/pwd" -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 100000+0 records in
100000+0 records out
real 2m 1.91s
user 0m 0.35s
sys 0m 5.69s
Native:
$ docker run --rm -it -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real 0m 0.40s
user 0m 0.02s
sys 0m 0.36s
~300 times slower.
On a Macbook Air M1 processor with Docker Tech Preview still the same:
- Docker for Mac 3.1.0 Tech Preview
- MacOS 11.0.1
Bind mount:
docker run --rm -it -v "$(PWD):/pwd:delegated" -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real 0m 24.54s
user 0m 0.15s
sys 0m 2.12s
Native:
$ docker run --rm -it -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real 0m 0.16s
user 0m 0.01s
sys 0m 0.14s
Hello, I wonder if anyone can point me to current workarounds that you use to deal with this problem? Do you use VSCode feature to mount inside container and use git tools inside containers when develop? Do you ssh into container and sync it with your favorite editor? Do you use linux vm? Do you use mutagen still? Any ideas are very much appreciated. If you think there is a better place to ask this question, please let me know. Thank you
Thank you @adrubesh , since you use VM, you avoid this performance problem. I wonder where you run your development tools - IDE, network monitoring tools, browser? Do they run on your Mac still and you somehow connect to Linux VM via ssh or you do all development inside VM and these tools are installed in Linux?
I hope these questions don't annoy people from this thread too much. I feel it is a relevant conversation, but maybe not exactly relevant to the open issue.
Thank you @adrubesh , since you use VM, you avoid this performance problem. I wonder where you run your development tools - IDE, network monitoring tools, browser? Do they run on your Mac still and you somehow connect to Linux VM via ssh or you do all development inside VM and these tools are installed in Linux?
I hope these questions don't annoy people from this thread too much. I feel it is a relevant conversation, but maybe not exactly relevant to the open issue.
All tools are installed in the VM.
I use docker-sync. It creates a folder-sync between your fs-folder and a docker-volume. docker-volumes have great performance. Sure you use up double the space, but I still prefere this approach instead of having a VM.
I'm currently working on a huge (ugly) Django/Python Project with 90+ custom packages which sourcecode has to be mounted because it's part of the the project. Just running ./manage.py w/o any command takes nearly a minute with bind mounts. Completely unusable. With docker-sync it takes about 2 seconds. I really hope bind mounts can reach this lvl of performance someday.. but we are waiting since the first release and the improvements so far had been very little..
I'm on 3.3.0 Apple Silicon Preview.
Why is this such a low priority... are we ever going to see improvement here? It's been years 😞
docker-sync requires ruby, I'd prefer our devs not having to deal with ruby. I simply do not understand why this cannot be addressed on Mac - as the last comment states, its been years. The performance is unacceptable. M1 will make virtually no difference.
@justindoody, @kevinquillen, and so on, please do not pollute this thread. Thanks. The docker team already has explained why, and already knows how much a pity it is. They do not need fathering anymore.
@keywinf well they need to figure out something if they expect people to start paying for it now
Any updates?
Are there any plans to address this issue?
As slow as it is now it is virtually unusable ..
My build process is very slow in a docker container, but it is 5-7 times faster on a native platform.. gRPC fuse is very slow!
I have the same issue: Macos 12.0.1, docke for macos is very slow. I'm trying this on a basic springbot starter application
Have you tried the the VirtioFS experimental build? It does have a notable bug (opcode 40 doesn't work), but it should help tremendously in any situation where you don't run into that specific bug. Others have seen really good improvements.
Has anyone solved this problem, it's really bad to mount with docker on mac.
@hongliang5316 you should have a look at docker/roadmap#7
@mrvisser
I tried it, but it seems to be getting more stuck.