Registering to the CISM/CECI

The first step to access Manneback is to register a CECI login by following the instructions here. You should also create a CISM account at the same time, even though it's not needed for access to Manneback, it can be useful for direct access to the jupyter notebook functionality (more on that soon).

Getting on Manneback

The first (and often most difficult) step in getting your program to run is to log in to Manneback through ssh. Starting information is available on the CECI site.

You should then choose if you are on Linux/Mac or Windows:

• Linux instructions
• Windows instructions

Don't hesitate to use the config wizard in order to create a correct configuration to help you in connecting to all servers in the cluster.

Once you followed the instructions above, you should be able to simply use ssh manneback to log in.

Using SLURM

Most of these instructions are based on this quick start guide.

First steps

The first step is often to check the availability of the resources. The main nodes we will use are mb-cas101 and mb-cas102 on the gpu partition.

The first command to use once you're on manneback is the following:

sinfo --format "%P %.5a %.10l %.6D %.6t %G %N"

This will show the different partitions, their (GPU) resources (column called GRES), and the nodes attached to it. Once you've found the resources that interest you, you can find further information with one line per node :

sinfo -N --format "%N %.6D %P %6t %G" | grep "gpu"

Without the grep, this command would give you a full list of all the (hundreds of) nodes, but thanks to grep you should only see lines that can interest you in some way. Let's now say that we are interested in a V100 GPU. Through our command, we see that both mb-cas101 and mb-cas102 has those GPUs. This will be helpful in creating our SLURM script.

Let's now look at the utilization of those nodes, we don't want to wait too long before having our program run !

squeue --nodes mb-cas101,mb-cas102

Look's like mistasse is on mb-cas101, but mb-cas102 is not used by anyone. In order to know what resources a job uses, you can use the following additional flag to squeue :

squeue --nodes mb-cas101,mb-cas102 -O "jobid,tres-per-node"

Now we see that one job is using one gpu (the other job finished between the two squeue commands).

As mb-cas102 is free, let's run an interactive session on it, using srun:

srun -w mb-cas102 --gres="gpu:1" --partition=gpu --time=00:10:00 --cpus-per-task=8 --pty bash -l

This command will demand one gpu --gres="gpu:1" and 8 CPUs --cpus-per-task=8 for 10 minutes --time=00:10:00.

If the resources are available, your shell should transparently be on the node, to see if it worked, you could try running

nvidia-smi

and check if you see the V100 GPU. Once you're on the node, you can run the code you've intended to run.

To exit, either wait for 10 minutes, or press Ctrl-D (maybe Cmd-D on Mac).

Important parameters to srun and inside sbatch scripts

There are a lot more parameters that could be useful, that you can find in the man pages for srun and sbatch (with man srun or available online at https://slurm.schedmd.com/srun.html and https://slurm.schedmd.com/sbatch.html)

Using with Jupyter Notebooks

Note: This section works for any use-case where you need access to a local website/api

This section is adapted from the CISM documentation.

The first step is to install sshuttle. (Sshuttle is only available on linux and mac, but you could use a linux VM as explained here or maybe Windows subsystem for linux).

Once it is installed, create a tunnel with it in a separate terminal window :

$ sshuttle -r manneback 10.0.0.0/8

You need sudo, if you don't ask your admin to run sshuttle --sudoers --sudoers-user ALL once.

Once the tunnel is available, you can launch an interactive session with jupyter notebook :

[user@manneback ~] $ srun --pty bash -c 'ml Python ; jupyter notebook --ip $(hostname -i)'

The --ip part is important : by default jupyter (and most server software) listens to localhost, but we want it to listen to its external ip address (in the 10.0.0.0/8 range).

Once the job starts, you should see something like

[vjoosdtb@mb-cas101.cism.ucl.ac.be] ~ ❯ jupyter notebook --ip $(hostname -i)                          
[I 10:34:03.178 NotebookApp] JupyterLab extension loaded from /home/ucl/elen/vjoosdtb/venv/lib/python3.6/site-packages/jupyterlab
[I 10:34:03.178 NotebookApp] JupyterLab application directory is /home/ucl/elen/vjoosdtb/venv/share/jupyter/lab
[I 10:34:03.181 NotebookApp] Serving notebooks from local directory: /auto/home/users/v/j/vjoosdtb
[I 10:34:03.181 NotebookApp] The Jupyter Notebook is running at:
[I 10:34:03.181 NotebookApp] http://10.3.220.101:8888/?token=9080eeb8cb7f81b635971a38eb8201cef20b0b1d79858975
[I 10:34:03.181 NotebookApp]  or http://127.0.0.1:8888/?token=9080eeb8cb7f81b635971a38eb8201cef20b0b1d79858975
[I 10:34:03.181 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 10:34:03.186 NotebookApp] No web browser found: could not locate runnable browser.
[C 10:34:03.186 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///auto/home/users/v/j/vjoosdtb/.local/share/jupyter/runtime/nbserver-2175-open.html
    Or copy and paste one of these URLs:
        http://10.3.220.101:8888/?token=9080eeb8cb7f81b635971a38eb8201cef20b0b1d79858975
     or http://127.0.0.1:8888/?token=9080eeb8cb7f81b635971a38eb8201cef20b0b1d79858975

Then copy (or click) the address starting with 10. in your browser, and off you go !

When you're done, don't forget to kill both the server and the sshuttle tunnel.

SLURM script

You often don't want to run code interactively, and that's good because SLURM was not made for interactive work, but to run batch jobs. In order to run a batch job, you will use the sbatch command, but first you need a slurm script.

As always, additional information is available here.

A SLURM script is a kind of shell script with comments that tell SLURM what kind of resources you need. Because it can be any kind of script, you could add SLURM comments to your python script, but don't ! (Except maybe for very simple cases).

A SLURM script (named submit.sh) looks like this :

#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=res.txt
#
#SBATCH --ntasks=1
#SBATCH --time=5-00:00:00
#
#SBATCH --cpus-per-task=8
#SBATCH --gres="gpu:TeslaV100:1"
#SBATCH --partition=gpu

srun nvidia-smi

This is almost completely equivalent to the srun command from earlier, with only a few differences. Firstly, you run it using sbatch submit.sh (or whatever you named the file). Secondly, we changed some parameters, mainly the time (5 days instead of 30 minutes) and a more precise definition for the GPU (gpu:TeslaV100:1). We don't mention any particular nodes, in order to be put directly on the first available GPU.

If your application is not CPU bound, it might make sense to leave a CPU available to srun directly on a node to check progress, but the easiest way is to output to standard output, which will be redirected to a file (res.txt). If you notice that you don't get every line directly in the file, it might because the file is buffering in memory before transferring to file (over the network), if you don't like that, you might want to flush memory more often.

Additional sbatch/srun that might be useful are the following :

#SBATCH --mail-user=your.mail@uclouvain.be
#SBATCH --mail-type=ALL

This will send you a mail when your script starts, and when it has ended. It will also give you more information, for example if it ended because of an error or lack of time.

While outside of the slurm script an srun command will run your command on another node, an srun command inside a slurm script will simply constitute a job step.

Module system

Please read the ceci documentation before continuing.

Most importantly for our use-cases is probably having access to python and CUDA, which you can do using

module load Python
module load CUDA

Because CUDA is probably not available on the main node, you will have to call this after being on a node with a gpu, either in the sbatch script or after having run srun.

If you need a specific version of Python (for example 3.6) you can load it using :

module load CUDA/3.6

If it says it is not available, you can try to find the problem with

module spider Python/3.6

It might for example say that you need a different release :

You will need to load all module(s) on any one of the lines below before the "Python/3.6.6-foss-2018b" module is available to load.

      releases/2018b

But be careful that this might make other software unavailable.

Questions

If you have any question, don't hesitate to send a mail to victor.joos@uclouvain.be or to create an issue in this project.