Each student has been assigned to the HPC services. Make sure you are connected to the Imperial network if you are working from home, using a VPN.
- connect to your container:
ssh <user_name>@login.cx1.hpc.ic.ac.uk
- you should now be in:
rds/general/user/<user_name>/home
The containers provided by HPC already have all the modules you need to do your project. If you need a specific software, first check if it is available using module avail
, and discuss it with the HPC services directly.
- If you prefer to use python 3:
module load anaconda3/personal
anaconda-setup
- Launch Python:
python
A virtual environnment is like a small container where you can install all the packages you need for a project. You can create as many virtual environments as you like with conda
conda create -n <name environment> python=x.x
source activate <name environment>
: to activate your environment.source deactivate
: to deactivate your environment.
Install your packages (ex: pytorch):
source activate <name_environment>
conda install pytorch torchvision -c pytorch
conda install matplotlib
...
You can either use git or bitbucket.
For example, if you don't have (yet) any code:
git clone https://github.com/a-pouplin/use_HPC.git
In order to submit your job, you will need to submit a PBS file using a bashscript to HPC. Your job will be in a queue (which can last either a few minutes or a few hours). You can create your bashscript using vim: vim <pbs_name>.sh
You will need to write the following specifications in your <pbs_name>.sh
file:
#PBS -lwalltime=00:30:00
#PBS -lselect=1:ncpus=1:mem=8gb:ngpus=1
module load anaconda3/4.3.1
source activate <name_environment>
python $HOME/<path_to_code>/<code>.py
qsub <pbs_name>
: submit your job in the queue
qstat -s
: check your job statusqstat -q
: check the queue information (which GPU/ CPU are available)qdel <job_number>
: delete the current job
You will end up with two output:
<pbs_name>.sh.o<job_number>
: output file, containing all your print in your python job<pbs_name>.sh.e<job_number>
: error file, containing an error, if your job failed
Dowload your file with scp (you can also use FileZilla):
scp -r <user_name>@login.cx1.hpc.ic.ac.uk/<path_to_your_experiments> <path_on_your_computer>
You can ask us, or go directly to the weekly HPC drop-in clinic: every Tuesday from 14:00-16:00 in room 402, 4th Floor Sherfield Building. You can also have a look at their Getting started guide