Singularity support

Question

Singularity support

Closed this issue 5 years ago · 8 comments

Support for building singularity images for running on HPC clusters was requested (at the old repository) so I'm opening a new issue for it here.

This repository appears to provide the tools to convert the Docker image: https://github.com/singularityhub/docker2singularity

Perhaps the conversation process could be automated with CI services?

Answer 1 · 2019-08-23T03:30:16.000Z

Thanks @bdevans.

Apologies for the length of this.

I am able to build chaste and run apps (the chaste cardiac app in particular in my case) with singularity so long as I don't request more than a single HPC node.

For example, this works fine:

singularity exec chaste.sif mpirun -np 20 \
      /home/chaste/lib/apps/Chaste examples/Propagation1d/ChasteParameters.xml

within a SLURM job scheduling script, e.g.

sbatch --ntasks-per-node=20 --nodes=1 --mem=128gb --time=00:10:00 \
       examples/VCCRI/params/run.sh

(obviously 20 cores for this example is drastic overkill, but it's just an illustration).

Using more than 1 node, e.g. sbatch --ntasks-per-node=20 --nodes=2 ... and mpirun -np 40 ..., e.g.

singularity exec chaste.sif mpirun -np 40 /home/chaste/lib/apps/Chaste \
        examples/params/chaste-1042718.xml

causes mpirun to try to invoke srun which does not exist within the container, yielding this error:

The SLURM process starter for OpenMPI was unable to locate a
usable "srun" command in its path. Please check your path
and try again.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
An internal error has occurred in ORTE:

[[53876,0],0] FORCE-TERMINATE AT (null):1 - error plm_slurm_module.c(462)

This is something that should be reported to the developers.
--------------------------------------------------------------------------

So, I tried starting singularity itself with srun like this (where srun here is invoked from within the host, not in the container):

srun -n 40 singularity exec chaste.sif /home/chaste/lib/apps/Chaste \
        examples/params/chaste-1042718.xml

This fails with:

The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:

  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix.

  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.

Please configure as appropriate and try again.

This also fails:

srun -n 40 singularity exec chaste.sif mpirun -np 40 /home/chaste/lib/apps/Chaste \
        examples/params/chaste-1042718.xml

but with the first error shown above:

The SLURM process starter for OpenMPI was unable to locate a
usable "srun" command in its path. Please check your path
and try again.
...

So, from all of this, at the very least it seems that either:

srun must be present within the container, or
the container's OpenMPI must be built with SLURM support.

I have plenty of Docker experience but have just started using singularity. Happy to collaborate on solving this.

Answer 2 · 2019-09-13T16:02:23.000Z

Hi @davidbenncsiro,

Thanks for your offer of help. and sorry for the delay in responding - unfortunately I have not used and do not have access to a singularity cluster but hopefully together we can figure this out!

From your summary, it sounds like adding srun to the image might fix the problem. Could you check if it does by running sudo apt-get update && sudo apt-get install <PackageName> within a container to install whichever package it is that gives you srun (password: chaste) then run your simulations with multiple HPC nodes? If that works then I'm happy to build the necessary extra packages into the image. If OpenMPI also needs to be built with SLURM support (or if it would be useful anyway) I'm happy to look at what needs to be done to achieve that next.

Answer 3 · 2019-09-15T02:02:28.000Z

Thanks @bdevans. This morning I installed SLURM and have it working within a chaste docker container. I suspect there is more to do with respect to config yet, since it will need to know that there exists a cluster of nodes beyond the container. In any case, what I've done so far is this:

sudo apt update
sudo apt install slurm-wlm
sudo apt install systemctld
sudo apt install munge
sudo apt install vim
vi /etc/slurm-llnl/slurm.conf
sudo /etc/init.d/slurmd start
sudo /etc/init.d/slurmctld start   
sudo /etc/init.d/munge start

At this point I can ask for the chaste user's SLURM jobs and start a job:

squeue -u chaste
sbatch --job-name="test1" --nodes=1 --ntasks-per-node=4 --time=00:10:00 foo.sh

where foo.sh is:

#!/bin/bash
sleep 3
echo "done"

squeeze shows that the job is running and yields a slurm-<job-id>.out file with stout/err.

The /etc/slurm-llnl/slurm.conf file above has this:

# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ControlMachine=bd09adf8deef
#ControlAddr=
#
#MailProg=/bin/mail
MpiDefault=none
#MpiParams=ports=#-#
ProctrackType=proctrack/pgid
ReturnToService=1
SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
#SlurmctldPort=6817
SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
#SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
SlurmUser=slurm
#SlurmdUser=root
StateSaveLocation=/var/lib/slurm-llnl/slurmctld
SwitchType=switch/none
TaskPlugin=task/none
#
#
# TIMERS
#KillWait=30
#MinJobAge=300
#SlurmctldTimeout=120
#SlurmdTimeout=300
#
#
# SCHEDULING
FastSchedule=1
SchedulerType=sched/builtin
#SchedulerPort=7321
SelectType=select/linear
#
#
# LOGGING AND ACCOUNTING
AccountingStorageType=accounting_storage/none
ClusterName=bd09adf8deef
#JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
#SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
#SlurmdDebug=3
SlurmdLogFile=/var/log/slurm-llnl/slurmd.log
#
#
# COMPUTE NODES
NodeName=bd09adf8deef CPUs=4 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2 State=UNKNOWN
PartitionName=long Nodes=bd09adf8deef Default=YES MaxTime=INFINITE State=UP

See https://ubuntuforums.org/showthread.php?t=2404746

The only change needed from this is the hostname, in this case "bd09adf8deef".

To get this into a singularity context, what I need to do is to have a docker image I can access via singularity pull ... (from docker hub is how I'm doing it now) since creating a singularity image file from a local file on HPC requires sudo access. Either that or I have to install singularity on a non-HPC machine to create the image there first.

So, should I fork your chaste-docker and/or push a docker image to dockerhub to experiment with this until it works in singularity and then we can discuss further or would you prefer to handle this a different way?

It may be that it's necessary to compile OpenMPI with SLURM support, as you say, and as per some of the output I quoted in the earlier reply. One step at a time I guess. :)

Thanks!

Answer 4 · 2019-09-27T13:47:05.000Z

Hi @davidbenncsiro,

Thanks for the update. So if I were to add the packages slurm-wlm systemctld munge to the image it looks like that would get you most of the way there. The sticking point would be /etc/slurm-llnl/slurm.conf. This would have to be tailored to each setup so can't go in the image. Can singularity be configured to look elsewhere for this file?

The simplest way to experiment might be to create your own Dockerfile beginning with:
FROM chaste/base:disco
This will have the basic dependencies but no Chaste code or a VOLUME specified which may help simplify the development process. See how that goes...
Best,

Ben

Answer 5 · 2019-10-09T21:58:22.000Z

Sorry for the delay @bdevans

Two questions:

Is it better to start with chaste/base:disco or a full Dockerfile? Do you mind which? In experiments so far I started with the full dockerhub image and added to the container incrementally. Was going to just do the same in the Dockerfile thereafter.
Should I create a fork or just create the Dockerfile and we can correspond re: that?

Thanks.

David

Answer 6 · 2019-10-10T13:56:03.000Z

Hi @davidbenncsiro,
No worries!

Is it better to start with chaste/base:disco or a full Dockerfile? Do you mind which? In experiments so far I started with the full dockerhub image and added to the container incrementally. Was going to just do the same in the Dockerfile thereafter.

In practice, it shouldn't make much difference given your previous approach (as you shouldn't need to remove anything from the image), so you could save some build time and start with chaste/base:disco. However, this relates to your second question...

Should I create a fork or just create the Dockerfile and we can correspond re: that?

If you start with the base image, there is nothing to fork as it will be a separate version. I guess it depends on whether this is going to be generally useful and so should be included in the main image(s) (so best to fork https://github.com/Chaste/chaste-docker) or if it's more niche and should just be in a special version for HPC (in which case start with chaste/base:disco)... 🤔 @Chaste/developers - any thoughts?
Best,

Ben

Answer 7 · 2020-02-11T15:30:40.000Z

Hi @davidbenncsiro,
I'm closing this due to lack of recent activity but feel free to reopen it if have time to take another look.
Best,
Ben

Answer 8 · 2020-02-12T23:11:37.000Z

No worries @bdevans I have been a bit time poor in recent months and the importance of the use of singularity for me has decreased since then.