Support for deleted memory mapped files
jhead opened this issue · 2 comments
I noticed that one of the limitations due to unprivileged checkpointing is:
Memory mapped files that have been deleted are not supported.
Unfortunately, this appears to be a non-starter for any JVM application using Netty. It seems to immediately copy the .so into /tmp and map it into memory, then deletes it.
Error (criu/proc_parse.c:348): Can't open mapped [/tmp/libnetty_transport_native_epoll_x86_643796318373806122906.so (deleted)]: No such file or directory
Error (criu/proc_parse.c:672): Can't open 1000's mapfile link 7f363e210000: No such file or directory
Error (criu/cr-dump.c:1261): Collect mappings (pid: 1000) failed with -1
Error (criu/cr-dump.c:1779): Dumping FAILED.
tar: Removing leading `/' from member names
[ff.checkpoint] (0.329s) `criu dump --tree 1000 --leave-stopped --empty-ns net --tcp-established --skip-in-flight --tcp-close --ext-unix-sk --images-dir /var/fastfreeze/run --cpu-cap --shell-job --file-locks --file-validation filesize --stream` failed with exit_code=1
Any ideas how I might be able to work around this? Thanks!
I found a Netty-specific JVM flag io.netty.native.deleteLibAfterLoading=false to help work around this issue. With that in place, I was able to successfully checkpoint and restore my application!
I am curious though why this limitation exists. I did some debugging in criu with gdb, comparing the execution between unprivileged and privileged containers, and noticed that it hits an EPERM when accessing the /proc/1000/map_files/<file> corresponding to the above mmap'd library. I read through proc(5) and ptrace(2) and I don't fully understand why criu wouldn't have permission to access that file.
I did see that there's a fallback user space attempt to stat the file, which of course fails if the file has already been deleted. It seems like the best approach would be fixing those missing permissions to access map_files, though I'm at a loss for how to debug the access denial.
I am curious though why this limitation exists.
Nicolas discussed this limitation at LPC: https://youtu.be/fSyr_IXM21Y?t=2928