Test this environment on:
You can import any python package installed but you do not have the permission to install any new packages or modify any of the installed packages.
The focus of this setup is to demonstrate how to configure a conda environment for high performance computing on a multi-user system. This means one conda environment is installed which is used from multiple users and can be updated regularly by the administrative user. The challange in this approach is that we do not want to allow non-administrative users to modify the environment and at the same time as python is a just-in-time compiled language users typically need write access to execute python code. Therefore the official anaconda documentation recommends to give all users in the same group read, write and execute access: https://docs.anaconda.com/anaconda/install/multi-user/#multi-user-anaconda-installation-on-linux
But in this case users might be able to install new packages via pip or directly via conda or modify the python code of existing packages. Both is not recommended for a high performance computing environment which is focused on stability. Instead this tutorial proposes to provide the users with only read and execute access and only the administrative user with write access.
Just follow the single user setup and install conda in an directory which is accessible for all users, as explained in: https://docs.anaconda.com/anaconda/install/multi-user/#multi-user-anaconda-installation-on-linux
In the example above conda is installed to:
/opt/anaconda3
After the successful setup copy the conda activate commands to each users shell profile. For an installation in /opt/anaconda3 the activate commands might look like this:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/opt/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/opt/anaconda3/etc/profile.d/conda.sh" ]; then
. "/opt/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/opt/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
After the initialization of the user account (logout and login again) each user should be able to access the conda environment. To confirm this check the location of the following executables:
which conda
which python
Both should point to the newly installed conda environment. In addition the environment variable ${CONDA_PREFIX} should be set for all users.
Restrict the permissions of users in the group to read and execute access while limiting the write access to the atministrative account who is executing these commands:
chmod -R 750 ${CONDA_PREFIX}
Before modifying the environment it is recommended to add the conda-forge community channel and update all packages:
conda config --add channels conda-forge
conda update --all
To prevent conda from switching from anaconda to conda-forge, add the following line in .condarc.
channel_priority: strict
channels:
- conda-forge
- defaults
ssl_verify: true
Rather than having a ~/.condarc in the users home directory it is recommended to place it in the root of the conda environment ${CONDA_PREFIX}/.condarc. If conda already created a ~/.condarc in the users home directory, this should be removed to confirm that only the configuration in the root of the conda envrionment ${CONDA_PREFIX}/.condarc is used. After this step it is important to update conda again to superseed packages from the default conda channel with the ones from the conda-forge community channel, in case they were not updated previously:
conda update --all
You can confirm that the majority of the packages are now installed from conda-forge using:
conda list
To prevent conda from switching from MKL to openblas, add the following line in ${CONDA_PREFIX}/conda-meta/pinned:
libblas[build=*mkl]
Again after this step it is recommended to update the environment:
conda update --all
In case you prefer openblas you can fix openblas the same way. As a general rule of thumb mkl typically provides better performance for Linux systems running on intel hardware.
As the conda directory is set to read and execute only, all python files are compiled using:
find ${CONDA_PREFIX} -name "*.py" ! -path "${CONDA_PREFIX}pkgs/*" -exec ${CONDA_PREFIX}bin/python -m py_compile {} +
This step is essential for the non-administrative users to have access to the conda environment and it has to be executed after every update.
Some packages like nglview are distributed as two parts a conda package and a jupyterlab extension. It is therefore important to keep both versions synchronized. To check the status of the jupyter lab plugins use:
jupyter labextension list
When a jupyterlab extension was updated then it might be necessary to rebuild jupyterlab. You should see a corresponding warning message when you call the command above. To rebuild jupyterlab use:
jupyter lab build
When regularly updating the environment it makes sense to clean up afterwards. Conda offers a built-in clean up utility:
conda clean --all
In addition jupyterlab requires a clean up to remove the intermediate parts of the build process:
npm cache clean --force
jlpm cache clean
jupyter lab clean