This repository contains the Python files for the IAML labs. It also contains instructions for how to get started. We recommend that you use Notable to do the labs.
Gettting Started: Using Notable
Go to Notable. It might ask your UUN and Password, if you are not logged into Euclid. After login, the following screen will be available with few options to start the server.
Choose Standard Notebook (Python3) and click Start.
The jupyter notebook will open in the same window, as shown below. On the Notable home folder, click on the +GitRepo button.
In the Notebook server, a small pop-up window appears, as shown below:
Go to IAML github repo: iaml-labs and copy the HTTPS link by clicking on the green colored Code button.
Enter the copied Github link in the pop-up window on the Notebook server (keep branch as master). Now click Clone to copy the labs into Notable.
Notable will take few seconds to copy and load the IAML Labs.
Once the Notable is successfull, it will show the repository contents.
Go to Labs to find the available jupyter notebook and continue working on them.
The following instructions tell you how to setup Python and how to configure it for the IAML labs. If you successfully got Notable working you can skip this step. In this section we provide alternative ways in which you can run the labs - either on DICE or your own computer. The main steps are (i) installing Python (using conda), (ii) configuring the correct libraries required for IAML, and (iii) downloading the labs.
These instructions are primarily written for DICE. DICE refers to desktops and servers that run Unix and are managed by computing staff in the School of Informatics. It includes computers that are both physically in the labs and ones that you can access remotely.
In the instructions below, any text styled like this should be executed in
the terminal. You should enter these commands by hand,
one-by-one. This is to help detect any issues.
Please read and heed any warnings and especially errors you may encounter. We
are on standby in the labs to help if required.
For this course, you can either use (i) a DICE computer via remote access or (ii) your own personal computer running Linux, Windows, or MacOS. We have verified that these instructions work under DICE and while they should work on your machine too, we can not guarantee this for everyone. If you are using your own computer, you can skip to step 2.
If you choose to work on DICE, there are two main ways of using it remotely -- either via RDP (remote desktop) or via SSH. The first one is recommended, as it is the easiest. If you are running Linux, run the following command from your own computer:
xfreerdp +glyph-cache /relax-order-checks /u:s1234567 /v:s1234567.remote.inf.ed.ac.uk
Replace s1234567 with your username. This will open a remote DICE session and ask you to login. Then you can proceed with the setup instructions below. Guides for remote access using other operating systems can be found here: http://computing.help.inf.ed.ac.uk/remote-desktop.
If you have used SSH before/know what you are doing, you can also use the university SSH gateways. The guide is here: http://computing.help.inf.ed.ac.uk/external-login. Note, that you will need to setup port forwarding to be able to access Jupyter notebooks.
If you're on a DICE machine, in the top left click Applications -> Utilities -> Terminal. Alternatively, you may find the terminal under Applications -> System Tools -> MATE Terminal, depending on the system used.
Note that your space on DICE is allocated dynamically. If you are having problems it may be because you were using new space faster than it could be allocated to you!
All DICE users registered for IAML will automatically be allocated 20GB extra space over their default space values. Please register for the course ASAP to get this space.
- Check how much space you have on DICE. You will need at least 4.5GB.
freespace- If you don't have enough space, follow the instructions on this page.
- Check you don't already have conda installed!
which conda- if you already have it installed, skip ahead to Create an Environment
- It doesn't matter if you have miniconda3, or anaconda3 installed (it does not even matter if it is version 2).
- If you don't have conda, download the latest version of miniconda3
cd ~/Downloads(you can make a Downloads folder if you don't have one)- Download the installer (we prefer to use miniconda since it carries less baggage), depending on your system (you can check links here):
- Linux:
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh - Mac:
wget https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.shorcurl -LOk https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh - Or just simply download from the site
- Linux:
- Install miniconda3 with default settings
bash Miniconda3-latest-Linux-x86_64.sh- Follow the prompt - type
yesand hitenterto accept all default settings when asked
- Close Terminal and reopen
- Try executing
conda -h. If it works, you can delete the installerrm ~/Downloads/Miniconda3-latest-Linux-x86_64.sh
If you are installing conda on your own machine, you will find some instructions for Windows users at the end of this README.
- Update conda:
conda update conda - Create the environment for the course. Call it py3iaml and install python 3 (hence the name):
conda create -n py3iaml python=3.7
You can find more information in the Miscellaneous section below for how to work with conda environments. Briefly, source activate py3iaml will activate the new environment and conda deactivate will exit it.
Before installing required modules, we need to obtain the repository since it includes the specifications of the packages to use (as well as all lab material). Within your terminal:
- Navigate back to your home directory:
cd - Now you have two options:
- If and only if you are familiar and confident with using Git/GitHub, you can initialize a git directory, add the above repo as remote and pull everything into your local directory, something like:
- `git clone https://github.com/uoe-iaml/iaml-labs iaml-labs
- OTHERWISE, we recommend that you directly download a .zip file from https://github.com/uoe-iaml/iaml-labs which will contain everything you need and save it in the folder you have just created (by clicking Code -> Download ZIP from in your browser). Or you can do this from the terminal by typing (this also makes the directory structure equivalent to that obtained by git clone):
wget https://github.com/uoe-iaml/iaml-labs/archive/master.zipunzip master.zipmv iaml-labs-master iaml-labsrm master.zip
- If and only if you are familiar and confident with using Git/GitHub, you can initialize a git directory, add the above repo as remote and pull everything into your local directory, something like:
- Navigate to the new directory
cd iaml-labs
Supporting and teaching git is not in scope for this course so please only use it if you are happy to google your own solutions! That being said, git is a better alternative if you are familiar with it. We reccommend that you work with your own branch/fork as the git repository is read-only.
- Activate the environment:
source activate py3iaml - {May take 5 minutes} Install all required packages. We have done a iaml.req file for you to use:
conda install --file iaml.req. You can download this as part of the repository (see below). It is important to use this requirements file as this contains the specific version numbers so that the course is consistent regardless of when you start - Get some space back:
conda clean -a
Before starting any IAML work in a new terminal you must always activate the iaml conda environment using source activate py3iaml. If the environment is not activated, you will be using your base python with its own set of packages. If you are ever in any doubt of which python version is being used, execute which python and make sure that it points to where your environments are installed.
Once you have downloaded the material, you are now ready to start working with Jupyter notebooks. First you need to activate the software environment and then start a Jupyter Notebook session from within the folder where the material is stored. You will have to follow this procedure for all labs and assignments.
- Activate the conda environment:
source activate py3iaml - Enter the directory where you downloaded the course material:
cd iaml-labs/iaml-master - Start a jupyter notebook
jupyter notebook
- This should automatically open your browser
- Click on
00 - Introduction.ipynbto open it (it exists under the Labs directory)
- Click on
Now you are ready to start working on the labs!
- Conda getting started - 30 minute practical well worth going through https://conda.io/docs/user-guide/getting-started.html
- System Environment variables - https://en.wikipedia.org/wiki/Environment_variable
- Linux execution order - https://www.cyberciti.biz/tips/how-linux-or-unix-understand-which-program-to-run-part-i.html
Firstly, please note that your space on DICE is allocated dynamically. If you are having problems it may be because you were using new space faster than it could be allocated to you!
- Check how much space you have on DICE. You will need at least 4.5GB.
freespace- If you don't have enough space, follow the instructions on this page
- Try installing packages individually and executing
conda clean --allafter each installation
If you install incorrect packages, or a package breaks for some reason, you can
just delete your environment and start again. Execute conda remove --name py3iaml --all then install the package as described above.
This is fairly extreme but as a final resort can be done quickly and easily. Please note that you will lose all your environments if you do this, so check this will not affect you before proceeding...follow instructions here
Check that you downloaded the zip correctly! An error like:
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
means that the file you've downloaded is likely incomplete. Try downloading from the GitHub repo directly by clicking the green button and downloading the zip.
You do not have wget installed! Either install it, download from
the GitHub repo directly by clicking the green button and download the zip, or try using another program like curl e.g. curl -LOk https://github.com/michael-camilleri/IAML2018/archive/master.zip
DICE issue: DICE has a different set of bash startup mechanism, and you may need to edit some different files yourself. Do the below with ~/.benv instead. See here for more info.
Unix solution: First try closing your terminal and reopening. If that doesn't fix, it's likely that, in the conda installation, you didn't allow conda to add the it's bin directory to your $PATH. Check your home directory for ~/.brc or ~/.bashrc. You should have a line in one of those files that looks like this (the XX's represent your student number):
export PATH="$PATH:/afs/inf.ed.ac.uk/user/sXX/sXXXXXXX/miniconda3/bin"
If it does not exist, simply add it. Note: it does not normally matter if the PATH is prepended or appended, but I prefer to append. You will then need to activate this change. You can either exit the terminal and reopen a new one, or simply source it:
source ~/.brc
You're on windows aren't you! Please see the note at the top of the file (replace source with conda)
You're on windows aren't you! Please see the note at the top of the file (which = where on windows).
You're on windows aren't you! Please see the note at the top of the file (echo $PATH == echo %PATH% on windows).
We have found that people also taking MLP and/or ANLP (other courses that use conda) have installed multiple versions of conda. To check whether you've done this, simply list your home directory:
ls ~
If you see multiple folders called anaconda or miniconda, e.g. anaconda3 and miniconda2, you have installed multiple versions of conda! Another way to check is to print your PATH or view your .brc / .benv:
echo $PATH
cat ~/.brc
cat ~/.benv # if you're on DICE
This will show multiple conda directories.
You only need to use one installation of conda, and it doens't matter whether you use version 2 or 3 (there is no difference that will affect this course).
Simply recreate your environment(s) in one of the conda installations, and delete the other.
- https://conda.io/docs/user-guide/tasks/manage-environments.html#sharing-an-environment
- https://conda.io/docs/user-guide/install/linux.html#uninstalling-anaconda-or-miniconda
An environment is a collection of packages of specific versions. You can have multiple environments and switch between them for different projects. Conda is a tool for managing both environments and the packages within each environment. Here is a quick introduction:
- Show a list of your environments:
conda env list - Print
$PATH, one of your system's environment variables, in the terminal:echo $PATH$PATHis the list of directories your terminal can search to find anything you execute:
- Print a list of python installations on your
$PATH(the top one is the one that will get executed if you typepythonin the terminal):which python -a - Activate the new environment:
source activate py3iaml - Show list of python installations on your system now:
which python -a - Show your system
$PATHagain:echo $PATH - Deactivate the new environment:
source deactivate - Observer how your $PATH has changed again:
echo $PATH - Make an empty environment:
conda create --name empty - You can clone environments; this is useful for backing up:
conda create --name empty_bkp --clone empty - Make another python 3 environment with numpy already installed:
conda create --name py3 python=3 numpy conda env list- Activate py3:
source activate py3 - Show the installed packages:
conda list - Switch environments:
source deactivate; source activate empty conda listto show packages (note that python and, crucially, pip are not installed)- Q: What python would get used now?
which pythonA: the conda root environment installation of python i.e. not this environment's python. - Install numpy:
conda install numpy - Q: What python would get used now?
which pythonA: You may have clocked that conda installed a dependency of numpy (a python package)...python! - Let's delete these test environments:
source deactivateconda env listconda remove --name empty --allconda remove --name empty_bkp --allconda remove --name py3 --allconda env list
We recommend two options for working remotely (we prefer that you use option (1): i.e. your own machine)
- Use your own machine! Conda installation should work fine on your own computer. You must still have a DICE account: when submitting assignments, you will need to copy work up to DICE and submit from there. Instructions will be given for this in each assignment
- Use virtual dice - a virtual machine emulated on your own computer connected to the dice network. Please read here for installation instructions and more: http://computing.help.inf.ed.ac.uk/vdice
- After conda installation, all instructions are much the same
- Please follow conda installation instructions on their website here
- To activate the
py3iamlenvironment, note that you don't typesource activate py3iamlbut instead useconda activate py3iaml - You can ignore "What is an environment?" (though you can google windows equivalents of all the Unix commands given)