Helsinki cluster access

This is a short tutorial to help setup a computing account at the University of Helsinki’s computing cluster. The guide is only relevant for UH students and staff.

As a very first step, you need to ask your group leader to add you to the cluster user group. After that is done, you can follow these steps to log in and install runko.

Preliminary access

First, test that you can login to turso cluster with

ssh -YA username@turso.cs.helsinki.fi

where you need to replace username with your uni account name. The connection only works within the university’s eduroam internet network, i.e., you have to be physically at the campus. If you are greeted with the turso terminal, you can continue to the next step.

If you want to connect from outside the university you need to first jump through via e.g., melkinpaasi.cs.helsinki.fi. Tips for how to automate this are given below.

SSH connection

SSH keys for easier login

A regular ssh connection requires you to type the (university) password on every login. You can make your life a bit easier by adding your SSH public key to the accepted connections on turso.

The following steps need to be done only once per machine that you will use to login to the terminal.

First, we need to generate a public SSH key (if you dont already have one). On your own machine’s home directory (i.e., ~/) execute

mkdir .ssh 
ssh-keygen -t rsa

and press enter for the default suggested directory and for passphrase (i.e. it leave empty).

The command generates

• .ssh/id_rsa private ssh key
• .ssh/id_rsa.pub public ssh key (for sharing)

In order to whitelist your computer you need to add the id_rsa.pub key to the host machine’s SSH config. In practice, print out the public key on your own local machine and copy the content to clipboard with

cat .ssh/id_rsa.pub

Then, connect via ssh to turso (command above) and paste the content of the id_rsa.pub (on your own machine) to ~/.ssh/authorized_keys (on the host machine). Then, repeat the same process on melkinpaasi by issuing a ssh connection to username@melkinpaasi.cs.helsinki.fi and paste the same key to to there as well.

SSH shortcut to your .ssh/config

One final touch can be done by configuring your own SSH connections to include turso as a known host. The following steps need to be done only once per machine that you will use to login to the terminal.

Add to your own machine’s ~/.ssh/config

Host turso
    HostName turso.cs.helsinki.fi
    User your_username
    IdentityFile ~/.ssh/id_rsa
    ProxyJump your_username@melkinpaasi.cs.helsinki.fi

and replace your_username with the university account name (note that it appears in 2 places here). Note that the whitespace on the command is made via tabs (not spaces).

After this, you should be able to connect to turso from anywhere with

ssh turso

Runko installation

Modules

Next, we will automate the loading of the necessary HPC modules on turso. SSH to turso and move to the vakka work disk space, and clone runko there for later access:

ssh turso
cd /wrk-vakka/users/$USER
git clone --recursive https://github.com/natj/runko.git

Then, move back to the turso home directory, create a modules/runko folder and copy the module file template there with

cd ~
mkdir modules
cd modules
mkdir runko
cp /wrk-vakka/users/$USER/runko/archs/modules/5.0.0.lua ~/modules/runko/5.0.0.lua 

Next, we need to introduce our own module directory to lmod. Its best to automate this by adding to your ~/.bash_profile on turso a line (and create the file if does not exist)

module use --append /home/your_username/modules

where your_username needs to be replaced with the real one. Now, when you login to turso, the available module list will be automatically updated.

Virtual Python environment

Finally, we need to setup the virtual python environment. To do this, first load the (incomplete) modules we just created with

module use --append /home/$USER/modules

and create a python virtual environment on turso home directory with

mkdir venvs
cd ~/venvs
virtualenv runko

Then activate the environment with

source ~/venvs/runko/bin/activate

after which you should see the terminal status bar change to

(runko) username@turso2:~$ 

or similar.

Then, we can install the python requirements (stored and reloaded automatically when we activate the venv) with

pip install mpi4py h5py scipy matplotlib numpy

The computing environment is now ready for compilation.

Installation

Runko installation is now easy. We login to turso, load the runko module,

ssh turso
module load runko

and can compile runko in /wrk-vakka (which was the location where we cloned the code in the SSH setup stage) with

cd /wrk-vakka/users/$USER/runko
mkdir build
cd build
cmake ..
make -j8

After which you should see the compilation take place and the tests being run.

Runko and SLURM usage

Submitting an example job

The code can be ran by e.g., submitting an example SLURM job in the shock project directory

cd $RUNKODIR
cd projects/pic-shocks
cd jobs

and submitting the example job

sbatch 1ds3.turso

with the content of 1ds3.turso being something like

#!/bin/bash
#SBATCH -J 1ds3              # user-given SLURM job name
#SBATCH -M ukko              # machine name [ukko, kale, carrington, hile]
#SBATCH -p short             # partition; use "sinfo -M all" for options
#SBATCH --output=%J.out      # output file name
#SBATCH --error=%J.err       # output error file name
#SBATCH -t 0-05:00:00        # maximum job duration
#SBATCH --nodes=1            # number of computing nodes
#SBATCH --ntasks-per-node=16 # MPI tasks launched per node
#SBATCH --constraint=amd     # target specific nodes; [amd, intel]
#SBATCH --exclusive          # reserve the full node for the job

# INFO: ukko AMD nodes have 128 EPYC Rome cores

# modules
module use /home/jnattila/modules
module load runko

# HPC configurations
export OMP_NUM_THREADS=1
export PYTHONDONTWRITEBYTECODE=true
export HDF5_USE_FILE_LOCKING=FALSE

# go to working directory
cd $RUNKODIR/projects/pic-shocks/

mpirun -n 16 python pic.py --conf 2dsig3.ini

This uses ukko (-M ukko) to run a job in the short queue (-p short) on one node (--nodes=1) with 16 cores (--ntasks-per-node=16).

Basic SLURM commands

You can check the status of the SLURM queue with

squeue

and status of your own jobs with

sacct

Sometimes you might also need information about the available partitions which can be accessed with

sinfo -M all