Singularity containers do not run when working directory is a sshfs mount
Closed this issue · 10 comments
When I try to run a singularity container from a command line, e.g. typing fsleyes in the terminal after loading the fsl module, and working directory is a sshfs mount (either the base of the mount, or a subdirectory), the container does not run, and I get this error:
ERROR : Failed to open current working directory: Permission denied
Strangely, the same directory is accessible from within the Singularity container, as long as the container was executed from a different working directory. When it comes to launching from the menu of VNM -- when launching the GUI app (fsleyesGUI 6.0.4) all works fine, but when launching the Singularity container interactively (e.g., fsl 6.0.4), I get the same error. I guess all depends on the working directory where the container is launched from in each scenario.
Can you reproduce it?
FYI, I do the sshfs the simplest way possible:
sshfs USER@TARGET_HOST:TARGET_PATH SOURCE_PATH
Should I mount differently?
Dear @civier
you need to add your ssh-mounted directory to your SINGULARITY_BINDPATH environment module variable and then it should work.
Cheers
Steffen
It IS added to SINGULARITY_BINDPATH
As I said, when I start the container from a different working directory, the ssh-mounted directory IS accessible from the container!
I attach a screenshot of the problem. /fred is the directory which is ssh-mounted on our HPC:
When trying to run the containers when the working directory is /, which has identical permissions to /fred, it works just fine.
neuro@929c10268e06:/$ ls -ld /
drwxr-xr-x 1 root root 4096 Mar 3 03:43 /
That's the output when running Singularity with the --debug flag:
neuro@929c10268e06:/fred$ export PWD=pwd -P
neuro@929c10268e06:/fred$ singularity --debug exec --pwd
DEBUG [U=1000,P=28855] persistentPreRun() Singularity version: 3.7.0
DEBUG [U=1000,P=28855] persistentPreRun() Parsing configuration file /usr/local/singularity/etc/singularity/singularity.conf
DEBUG [U=1000,P=28855] handleConfDir() /home/neuro/.singularity already exists. Not creating.
DEBUG [U=1000,P=28855] setValue() Updated flag 'bind' value to: [/afm01,/afm02,/90days,/30days,/QRISdata,/RDS,/data,/short,/proc_temp,/TMPDIR,/nvme,/local,/gpfs1,/working,/winmounts,/state,/autofs,/cluster,/local_mount,/scratch,/clusterdata,/nvmescratch,/vnm,/fred,/dagg,/home]
DEBUG [U=1000,P=28855] execStarter() Saving umask 0022 for propagation into container
DEBUG [U=1000,P=28855] execStarter() Checking for encrypted system partition
DEBUG [U=1000,P=28855] Init() Image format detection
DEBUG [U=1000,P=28855] Init() Check for sandbox image format
DEBUG [U=1000,P=28855] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1000,P=28855] Init() Check for sif image format
DEBUG [U=1000,P=28855] Init() sif format initializer returned: SIF magic not found
DEBUG [U=1000,P=28855] Init() Check for squashfs image format
DEBUG [U=1000,P=28855] Init() squashfs image format detected
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding SHELL environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding SUPERVISOR_GROUP_NAME environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding COLORTERM environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding SUPERVISOR_SERVER_URL environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding HOSTNAME environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding ModuleTable002 environment variable
VERBOSE [U=1000,P=28855] SetContainerEnv() Not forwarding SINGULARITY_VERSION environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding LMOD_DIR environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding PWD environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding MODULESHOME environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding ModuleTable_Sz environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding VTE_VERSION environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding LMOD_VERSION environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding LMOD_DEFAULT_MODULEPATH environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding GO_VERSION environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding LMOD_PKG environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding TERM environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding USER environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding LOADEDMODULES environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding DISPLAY environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding SHLVL environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding ModuleTable001 environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding LC_CTYPE environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding SUPERVISOR_PROCESS_NAME environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding MODULEPATH environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding LMFILES environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding LMOD_CMD environment variable
VERBOSE [U=1000,P=28855] SetContainerEnv() Not forwarding SINGULARITY_BINDPATH environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding DEBIAN_FRONTEND environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding OLDPWD environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding SUPERVISOR_ENABLED environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding BASH_FUNC_ml%% environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding BASH_FUNC_module%% environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding _ environment variable
DEBUG [U=1000,P=28855] SetContainerEnv() Forwarding USER_PATH environment variable
VERBOSE [U=1000,P=28855] SetContainerEnv() Setting HOME=/home/neuro
VERBOSE [U=1000,P=28855] SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG [U=1000,P=28855] init() Use starter binary /usr/local/singularity/libexec/singularity/bin/starter-suid
VERBOSE [U=0,P=28855] print() Set messagelevel to: 5
VERBOSE [U=0,P=28855] init() Starter initialization
DEBUG [U=0,P=28855] load_overlay_module() Trying to load overlay kernel module
DEBUG [U=0,P=28855] load_overlay_module() Overlay seems supported by the kernel
VERBOSE [U=0,P=28855] is_suid() Check if we are running as setuid
VERBOSE [U=0,P=28855] priv_drop() Drop root privileges
DEBUG [U=1000,P=28855] read_engine_config() Read engine configuration
DEBUG [U=1000,P=28855] init() Wait completion of stage1
VERBOSE [U=1000,P=28863] priv_drop() Drop root privileges permanently
DEBUG [U=1000,P=28863] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1000,P=28863] init() Spawn stage 1
DEBUG [U=1000,P=28863] startup() singularity runtime engine selected
VERBOSE [U=1000,P=28863] startup() Execute stage 1
DEBUG [U=1000,P=28863] StageOne() Entering stage 1
DEBUG [U=1000,P=28863] prepareAutofs() No autofs mount point found
DEBUG [U=1000,P=28863] Init() Image format detection
DEBUG [U=1000,P=28863] Init() Check for sandbox image format
DEBUG [U=1000,P=28863] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1000,P=28863] Init() Check for sif image format
DEBUG [U=1000,P=28863] Init() sif format initializer returned: SIF magic not found
DEBUG [U=1000,P=28863] Init() Check for squashfs image format
DEBUG [U=1000,P=28863] Init() squashfs image format detected
DEBUG [U=1000,P=28863] setSessionLayer() Overlay seems supported and allowed by kernel
DEBUG [U=1000,P=28863] setSessionLayer() Attempting to use overlayfs (enable overlay = try)
VERBOSE [U=1000,P=28855] wait_child() stage 1 exited with status 0
DEBUG [U=1000,P=28855] cleanup_fd() Close file descriptor 4
DEBUG [U=1000,P=28855] cleanup_fd() Close file descriptor 5
DEBUG [U=1000,P=28855] cleanup_fd() Close file descriptor 6
DEBUG [U=1000,P=28855] init() Set child signal mask
DEBUG [U=1000,P=28855] init() Create socketpair for master communication channel
DEBUG [U=1000,P=28855] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=1000,P=28855] priv_escalate() Get root privileges
VERBOSE [U=0,P=28855] priv_escalate() Change filesystem uid to 1000
VERBOSE [U=0,P=28855] init() Spawn master process
ERROR [U=0,P=28855] init() Failed to open current working directory: Permission denied
DEBUG [U=0,P=28869] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=28869] create_namespace() Create mount namespace
That might answer this, though a bit different:
apptainer/singularity#2638
I'll check later on
Interesting, but I don't know how to solve this problem.
The problem is that Singularity actually runs with suid, and before dropping the admin privileges, it tries to access the working directory as root. By default, sshfs mounts only allow the user that created them to assess the mount, and that is why Singularity gives the error:
ERROR : Failed to open current working directory: Permission denied
The solution is to allow the root user to access the sshfs mount. It is done in the following way:
- add the line "user_allow_other" to /etc/fuse.conf within the VNM
- add the option "-o allow_root" to the sshfs command
@stebo85 @aswinnarayanan Could you please add point 1 to the VNM container permanently? You may also want to have it as part of NeuroDesk installation script (in case user have admin rights)
Dear @civier
It would be good if you can try to add this yourself. It should only be one line in here:
https://github.com/NeuroDesk/neurodesk/blob/master/vnm/Dockerfile
great work @aswinnarayanan :) @civier can you test?
@civier should be fixed in vnmd/vnm:20210331.
Would you be able to test and update the instructions
add the line "user_allow_other" to /etc/fuse.conf within the VNM
add the option "-o allow_root" to the sshfs command
Resolved