Using Python on ARC

The ARC team maintain centrally Python Anaconda installations for Anaconda 2 and Anaconda 3. These installations typically do not have any extra packages installed, they exist such that users of the ARC service can use them to install and maintaiin their own virtual environments.

Creating your own virtual environment

First you should decide which version of Python you wish to use, 2 or 3. There are Anaconda modules available for both versions, the current Anaconda versions can be found by typing:

module spider anaconda

To load the version of Anaconda you want, in this example version 2019.03, use one of the following commands:

Python 2
module load Anaconda2
Python 3
module load Anaconda3/2020.11 
or  one the versions shown by module spider. 

Once the module is loaded you can use the conda commands to create a virtual environment in your $DATA area. For example to create an environment named "myenv" in $DATA we can use the following commands:

export CONPREFIX=$DATA/myenv
Python 2
conda create --prefix $CONPREFIX --copy python=2.7
Python 3
conda create --prefix $CONPREFIX --copy python=3.8

Note: Please ignore any messages from conda which instruct you to run "conda init" - this command will add lines to your ~/.bashrc file which may cause undesirable behaviour in batch files.

You can now use (activate) the environment by running the following command:

source activate $CONPREFIX

You can then use the conda install or pip commands to install packages. We recommend the use of conda install where possible to maintain package version consistency in the virtual environment. For example

conda install numpy

or

pip install numpy

Using Anaconda from within a submission script

In order to use your installed virtual environment from a batch script, you will need to load the appropriate Anconda module, and activate your environment. Using values from the above example (and assuming Python version 3, Anaconda 2020/11):

# After SBATCH section

module load Anaconda3//2020.11

source activate $DATA/myenv

# Your Python commands here...

 

Important Anaconda Information

 

When using Anaconda on the ARC systems, please take note of the following:

 

1. Do not load Anaconda virtual environments automatically on log in from your .bashrc or .bash_profile scripts. These will cause issues to SLURM submitted jobs.

2. Ensure you have deactivated the virtual environment BEFORE submitting a SLURM job using sbatch, otherwise you will have issues with packages from your virtual environment not being found.

3. You should load all you require from the submission script - as in the submission script example above.

 

 

Common Conda Install Recipes

Bioconda

Use the instructions above to create a basic Python Anaconda 2 or 3 virtual environment, then use the following commands to ensure the bioconda repostories are enabled:

conda config --add channels defaults

conda config --add channels bioconda

conda config --add channels conda-forge

 

Bioconda packages may then be installed by using the conda install command, for example to install bwa:

conda install bwa

Tensorflow

See detailled instructions here: Apllication-Tensorflow

Jupyter 

In the following examples subsititue the string ARCUSERID for your ARC username (e.g. ouit0554)

1)  Connection from a computer on the University network or VPN:On your local desktop/laptop

ssh  -L  8081:127.0.0.1:8081  ARCUSERID@arc-login.arc.ox.ac.uk

this will open a connection to one of arc-c login nodes, with the right port open for 

Jupyter.  

On the terminal opened by the above process type: 

module load Anaconda3/2020.11
jupyter notebook --no-browser --port=8081

On the arc-login terminal you will see a number of http addresses , open one of them bu clicking on the link or use a browser on yoru local machien and cut and paste the addess.  for example:

http://127.0.0.1:8081/?token=685129fef76b99296498c700d6bc0f1144601d9a243 f01a6

 

2) From outisde campus via oscgate 

On the local desktop/laptop type:

ssh -L 8081:127.0.0.1:8081 ARCUSERID@arc-login01.arc.ox.ac.uk -J \
oscgate.arc.ox.ac.uk 

On the terminal opened by the above process type: 

module load Anaconda3/2020.11
jupyter notebook --no-browser --port=8081

On the arc-login terminal you will see a number of http addresses , open one of them by clicking on the link or use a browser on your local machine and cut and paste the addess.  For example:

On the terminal opened by the above process type :

http://127.0.0.1:8081/?token=685129fef76b99296498c700d6bc0f1144601d9a243f01a6

where the token is unique to each session and presented to you by jupyter . On the arc-login terminal you will see a number of http address , open one of them by clicking on the link or use a browser on yoru local machine and cut and paste the addess.  for example:

On the local desktop/laptop type:

ssh -L 8081:127.0.0.1:8081 ARCUSERID@arc-login01.arc.ox.ac.uk -J \
oscgate.arc.ox.ac.uk ARCUSERID@arc-login01.arc.ox.ac.uk

 

3) From University network or VPN via MobaXterm

 

Use MobaXterm to establish a standard SSH connection to arc-login01.arc.ox.ac.uk, then run the following commands:

module load Anaconda3/2020.11
jupyter notebook --no-browser --port=8081

On the arc-login terminal you will see a number of http addresses ensure the port 8081 is shown after the localhost: name. If not, Jupyter may be using a different port due to 8081 being in use. If the port is different make a note of it for the next section.

http://localhost:8081/?token=c8c9dcf43ee58a15a9d72515a17a3965e4f582fa8b61e291

 

In MobaXterm click on "Tunneling" on the top Icon bar. Select "New SSH Tunnel" and select "Local port forwarding", complete the boxes as below:

 

Forwarded Port:8081 (or as given by Jupyter above)

Remote Server: 127.0.0.1

Remote Port: 8081 (or as given by Jupyter above)

SSH server: arc-login01.arc.ox.ac.uk

SSH login: Your ARC login

SSH port: 22

Click "Save" then click "Start". 

 

You will be asked for your ARC password to complete the connection, once this is done you will be able to paste the http URL given by Jupyter into your local broswer.