Argo is ICTP HPC cluster, comprising of 153 hosts/nodes, with total count of 2588 CPUs, nearly 10 TB of memory, 40Gbps+ Infiniband interconnects, 1Gbps network and several houndreds of TB of dedicated NFS storage.
The available worker/compute nodes are organised in queues(partitions).
There are three more special cluster nodes: a master node that controls job execution and login nodes argo-login1 and argo-login2, where users login, submit jobs, and compile code.
Jobs can be submitted from argo (argo-login2), argo-login1 and argo-login2.
ssh argo.ictp.it
or
ssh argo-login1.ictp.it
Queue infomation can be listed with the sinfo command:
$ sinfo -s
PARTITION AVAIL TIMELIMIT NODES(A/I/O/T) NODELIST
cmsp up 1-00:00:00 35/5/0/40 node[01-16,161-184]
esp up 1-00:00:00 24/11/1/36 node[61-96]
esp1 up 1-00:00:00 16/11/1/28 node[101-128]
long* up 1-00:00:00 19/18/1/38 node[21-32,131-156]
gpu up 1-00:00:00 0/2/0/2 gpu[01-02]
serial up 7-00:00:00 0/2/0/2 serial[01-02]
testing up 6:00:00 0/2/0/2 testing[01-02]
westmere up 6:00:00 0/0/1/1 westmere01
nehalem up 6:00:00 0/2/0/2 nehalem[01-02]
esp_guest up 1-00:00:00 0/2/0/2 nehalem[01-02]
The principal queue for all users is the long queue, with 26 nodes and 1 day time limit. It is the default queue, if none is specified in the job.
Dedicated queues cmsp, esp, esp1, esp_guest and gpu are available to specific Argo users, upon authorization.
testing queue is small, comprising of two nodes, and a short time limit of 6h.
serial queue is specific for serial jobs, while it has a very long time limit of 7 days. Two nodes are in the serial queue.
Generally for all queues the nodes are NOT shared among jobs. The exceptions are queues serial and gpu.
Overall, Argo is a heterogeneous cluster, with nodes belonging to various generations of Intel CPU microarchitectures. Numerous are nodes of the sandybridge and ivybridge architecture, followed by broadwell. We have kept, for historical reasons, several nodes of older microarchitectures like nehalem and westmere, so you can run code on them, for testing and comparison. Memory size also varies.
For each node we list it's microarchitecture, memory size, and other features in sinfo output:
$ sinfo -N -o "%.20N %.15C %.15P %.40b" NODELIST CPUS(A/I/O/T) PARTITION ACTIVE_FEATURES node01 20/0/0/20 cmsp omnipart,128gb,broadwell-ep,e5-2640v4 node02 20/0/0/20 cmsp omnipart,128gb,broadwell-ep,e5-2640v4 ... node21 0/12/0/12 long infiniband,32gb,sandybridge-ep,e5-2620 node22 0/12/0/12 long infiniband,32gb,sandybridge-ep,e5-2620 ... node61 0/12/0/12 esp infiniband,32gb,sandybridge-ep,e5-2620 node62 0/12/0/12 esp infiniband,32gb,sandybridge-ep,e5-2620 ... node101 20/0/0/20 esp1 infiniband,64gb,ivybridge-ep,e5-2680v2 node102 20/0/0/20 esp1 infiniband,64gb,ivybridge-ep,e5-2680v2 ... node131 20/0/0/20 long* infiniband,64gb,ivybridge-ep,e5-2680v2 node132 20/0/0/20 long* infiniband,64gb,ivybridge-ep,e5-2680v2 ... node139 0/16/0/16 long* infiniband,32gb,sandybridge-ep,e5-2650 node140 16/0/0/16 long* infiniband,32gb,sandybridge-ep,e5-2650
Within each queue, nodes are homogeneous in terms of all of their features. The ony exception is the "long" queue.
All nodes are networked together with 1 Gbps ethernet links spanning multiple switches.
Access to storage is also done through the Gigabit ethernet network.
Nodes within each queue are also networked together in a low-latency fabric for MPI communication, thanks to Infiniband (IB) or Omni-Path technology.
The two IB switches perform at 40 Gbps (QDR), while the Omni-Path switch supports 100 Gbps .
Table below summarises queue, nodes and their characteristics
Queue/Partition |
Access Policy | Notes |
---|---|---|
long | All users |
- Allows allocations of a maximum of 10 nodes for running parallel jobs. |
testing | All users | |
serial | All users |
- ONLY FOR single core, cpu (or task) jobs; Parallel or MPI jobs will NOT WORK. - Up-to a maximum of 7 running independent serial jobs are allowed. - Resources are over-subscribed (the nodes are shared among jobs and users). |
nehalem | All users | |
westmere | All users | |
OTHER Dedicated Queues |
||
cmsp | Special authorization needed | - Omni-Path connectivity is provided. |
esp |
Special authorization needed | |
esp1 |
Special authorization needed | |
gpu
|
Special authorization needed |
- several GPU Accelerators are available of the type:
Nvidia Tesla K40
Nvidia Tesla P100
- resources are over-subscribed (the nodes are shared among jobs.).
|
Queue/Partition |
Max walltime (h) | Node range | Micro-architecture | Cores |
Ram per core (GB/c) | Total nodes/cores |
Ram per node (GB) |
---|---|---|---|---|---|---|---|
long | 24:00 |
node[139-148] node[131-138],[149-156] node[21-32] |
Sandybridge Ivybridge Sandybridge |
16(8x2) 20(10x2) 12(6x2) |
2 3.2 2.7 |
10 / 160 16 / 320 12/144 |
32 64 32 |
testing | 6:00 | testing[01-02] | Nehalem |
8(4x2) |
1.5 | 1 / 8 | 12 |
nehalem | 6:00 | nehalem[01-02] | Nehalem |
8(4x2) |
3 | 2 / 16 | 24 |
westmere | 6:00 | westmere01 | Westmere | 12(6x2) | 2 | 1 / 12 | 24 |
cmsp | 24:00 |
node[01-16] node[161-184] |
Broadwell |
20(10x2) |
6.4 9.4 |
16 / 320 24/480 |
128 188 |
esp | 24:00 | node[61-96] | Sandybridge |
12(6x2) |
2.7 | 36 / 432 | 32 |
esp1 | 24:00 | node[101-128] | Ivybridge |
20(10x2) |
3.2 | 28 / 560 | 64 |
gpu | 24:00 |
gpu01 gpu02 |
Broadwell + 2* gp100 Sandybridge+ 2* k40c |
20(10x2) + GPUs 16(8x2) + GPUs |
6.4 2 |
1 / 120 1 /16 |
128 32 |
Rich software stack of MPI, Compilers, Libraries and Applications is installed on Argo. Several versions of a given software component are present if necessary. In order to choose the desired version , use the module command.
The list of available software components follows
% module avail
]]>------------------------------------------------------- /opt-ictp/ohpc/modulefiles/mpi --------------------------------------------------------gnu-openmpi/boost/1.63.0 gnu-openmpi/netcdf/4.4.1.1 intel-mpi/2016 openmpi/1.6.5/intel/2013gnu-openmpi/fftw/3.3.4 gnu-openmpi/phdf5/1.8.17 openmpi/1.10.2/intel/2013 openmpi/gnugnu-openmpi/netcdf-cxx/4.3.0 gnu-openmpi/scalapack/2.0.2 openmpi/1.10.2/intel/2016 openmpi/intelgnu-openmpi/netcdf-fortran/4.4.4 gnu-openmpi/scipy/0.19.0 openmpi/1.10.2/intel/2017 (D)------------------------------------------------------- /opt-ictp/ohpc/modulefiles/libs -------------------------------------------------------boost/1.55 gmp/4.3.2/default-compiler hdf5/intel (D) mpc/0.9/default-compiler netcdf/4.3.1/intel/2013boost/1.60 (D) gmp/default-compiler hdf5/1.8.12/intel mpc/default-compiler netcdf/4.4.1/intel/2017cln/1.3.2 grib_api/intel (D) hdf5/1.8.19/intel mpfr/3.0.0/default-compiler plasma/2.6.0esmf/7.0.0 grib_api/1.12.0/intel iml/1.0.3/gnu/4.4.0 mpfr/default-compilergabriel/1.2 grib_api/1.23.1/intel intel-mkl/2017 netcdf/intel (D)---------------------------------------------------- /opt-ictp/ohpc/modulefiles/compilers -----------------------------------------------------cuda/5.5 default-compiler gcc/4.5.2 gnat/2014 intel/2013 (D) nocomp opencl/amd/2.8 pgi/16.10 (D)cuda/6.5 g++_addon/4.9.0 gcc/4.6.2 gnu/5.4.0 intel/2016 opencl-amd opencl/intel/1.2 pgi/18.4cuda/9.1 (D) gcc/4.4.7 gcc/4.9.0 (D) gnu7/7.2.0 intel/2017 opencl-intel pgi/10.9------------------------------------------------------- /opt-ictp/ohpc/modulefiles/apps -------------------------------------------------------PHCpack/2.3.95 espresso/5.2.1 gnu/numpy/1.11.1 ncl/6.3.0/gnu/4.4.7R/2.15 (D) espresso/6.2.1 gnu/openblas/0.2.19 nco/intel (D)R/3.1.2 ferret/v6.842 gnu/openmpi/1.10.6 nco/4.4.2/intelR/3.3.1 gdal/1.7.3 grads/binary nco/4.6.8/intelR/3.5.1 gdal/2.2.4 gromacs/5.0-rc1 nwchem/6.5alps/2.1.1 gdal/2.3.0 (D) gromacs/2016.3 (D) proj/4.7.0anaconda3/5.1.0 gdl/0.9.4/intel lhapdf/5.8.6/default-compiler python/espcdo/1.9.0 gengetopt/2.22/default-compiler maple/14 python/2.7.13 (D)cp2k/2.4 (D) glpk/4.45/gnu/4.4.0 matlab/r2011 root/5.30.04/default-compilercp2k/2.6.1 gnu/gsl/2.2.1 ncl/5.2.1/gnu/4.4.7 root/6.08.06espresso/5.1 (D) gnu/hdf5/1.8.17 ncl/6.0.0/gnu/4.4.7---------------------------------------------------------- /opt/ohpc/pub/modulefiles ----------------------------------------------------------autotools (L) cmake/3.9.2 gnu7/7.2.0 llvm5/5.0.0 papi/5.5.1 prun/1.1clustershell/1.8 gnu/5.4.0 llvm4/4.0.1 ohpc (L) pmix/1.2.3 prun/1.2 (L,D)
Mount point |
Function |
Implemented on |
Shared amongst nodes |
Note |
---|---|---|---|---|
/home/<username> |
User’s Home directory on Argo |
NFS on high-performance storage |
yes |
Quotas apply, based on groups and applications. |
/local_scratch |
scratch |
Internal Hard Disk on the Worker node |
no |
Size is ~200GB per node. Space is not guaranteed. Data is automatically cleaned 48 hours after a job finishes. |
Only for members of the ESP group!
Mount point |
Function |
Implemented on |
Shared amongst nodes |
Note |
---|---|---|---|---|
/home/netapp-clima |
|
NFS on high-performance storage |
yes |
Quotas apply |
/home/netapp-clima-users1 | NFS on high-performance storage | yes | Quotas apply | |
/home/netapp-clima-scratch |
scratch |
NFS on high-performance storage |
yes |
Data is automatically cleaned |
/home/netapp-clima-shared |
|
NFS on high-performance storage |
yes |
|
/home/netapp-clima-cordex |
|
NFS on high-performance storage |
yes |
|
/home/netapp-clima-dods |
|
NFS on high-performance storage |
yes |
|
/home/clima-archive |
|
NFS on archive storage |
yes |
|
/home/clima-archive2 |
|
NFS on archive storage |
yes |
|
/home/clima-archive3 |
|
NFS on archive storage |
yes |
|
/home/tompkins-archive |
|
NFS on archive storage |
yes |
|
ARGO cluster uses Slurm (https://slurm.schedmd.com/) for the workload management, including the queue management.
Note: until May 2018, Argo was Torque based. To assist users with their old job scripts, we still support a limited set of Torque commands by using Torque/PBS wrappers for transition from Torque/PBS to Slurm.
If you wish to quickly find new Slurm counterparts for Torque commands you know, please see any of the Rosetta Stone of Workload Managers, Translate PBS/Torque to SLURM or Slurm vs Moab/Torque.
The rest of this documentation will concentrate on Slurm environment.
$ sbatch jobscript.sh
A simple jobscript could be:
$ cat jobscript.sh #!/bin/bash #SBATCH -p testing # partition (queue) #SBATCH -N 1 # number of nodes ~/fortran_code/test.x
Each line beginning with #SBATCH will be interpreted by the queue manager as options to give to the sbatch command.
$ cat jobscript.sh #!/bin/bash # #SBATCH --job-name=my-test # job name #SBATCH -p testing # partition (queue) #SBATCH -N 1 # number of nodes #SBATCH -n 2 # number of cores #SBATCH -t 0-2:00 # time (D-HH:MM) #SBATCH -o slurm.%j.out # STDOUT #SBATCH -e slurm.%j.err # STDERR #SBATCH --mail-type=ALL # I want e-mail alerting ############# ### This job's working directory echo \"Working directory is $SLURM_SUBMIT_DIR\" cd $SLURM_SUBMIT_DIR echo Running on host `hostname` echo Time is `date` echo Directory is `pwd` # Run your executable /home/username/fortran_codes/test_f90.x
To check the queue for all the jobs:
$ squeue
To check the queue for all jobs belonging to one user:
$ squeue -u <user-id>
To check the queue for a given job id:
$ squeue -j <job-id>
$ sacct -j <job-id>
$ scancel -j <job-id>
$ srun -p long -N 1 --pty bash
$ showfreeor
$ sinfo -o '%10P %.5a %15l %6D %8t %15C %N' | egrep -v 'drain|down|alloc'
PARTITION AVAIL TIMELIMIT NODES STATE CPUS(A/I/O/T) NODELISTesp up 1-00:00:00 11 idle 0/132/0/132 node[74-77,90-96]esp1 up 1-00:00:00 1 idle 0/20/0/20 node102gpu up 1-00:00:00 2 idle 0/36/0/36 gpu[01-02]serial up 7-00:00:00 2 idle 0/32/0/32 serial[01-02]testing up 6:00:00 1 idle 0/16/0/16 testing02
#SBATCH --hint=nomultithread
The module command will let you load and unload the environment needed to run specific applications or use specific libraries. It is especially useful when you need to have different versions of the same software/library or the same software/library compiled against different compilers.
In our cluster, for instance, we have both GNU and Intel compilers, therefore, in order to use MPI when we compile using one of those compilers we need to have two different versions of OpenMPI, but instead of having binaries name like mpicc-1.4.3-intel-11.1 or similar we use module to update the PATH environment variable, so that after loading the specific module for openmpi/intel we will just run mpicc and mpirun.
HOWTO
The syntax of the module command is the following:
module <subcommand> [<arguments>]
<subcommand> may or may not require one or more <arguments>
A partial list of available subcommands follows:
A detailed list of all subcommands is shown by running the command module help
The output of the module avail command could be something like:
$ module avail
---------------- /opt-ictp/ohpc/modulefiles/mpi ----------------
gnu-openmpi/boost/1.63.0
gnu-openmpi/fftw/3.3.4
gnu-openmpi/netcdf-cxx/4.3.0
gnu-openmpi/netcdf-fortran/4.4.4
gnu-openmpi/netcdf/4.4.1.1
gnu-openmpi/phdf5/1.8.17
gnu-openmpi/scalapack/2.0.2
gnu-openmpi/scipy/0.19.0
intel-mpi/2016
openmpi/1.10.2/intel/2013
openmpi/1.10.2/intel/2016
openmpi/1.10.2/intel/2017 (D)
openmpi/1.6.5/intel/2013
openmpi/gnu
openmpi/intel
--------------- /opt-ictp/ohpc/modulefiles/libs ----------------
boost/1.55
boost/1.60 (D)
cln/1.3.2
esmf/7.0.0
gabriel/1.2
gmp/4.3.2/default-compiler
gmp/default-compiler
grib_api/intel (D)
grib_api/1.12.0/intel
grib_api/1.23.1/intel
hdf5/intel (D)
hdf5/1.8.12/intel
hdf5/1.8.19/intel
iml/1.0.3/gnu/4.4.0
intel-mkl/2017
mpc/0.9/default-compiler
mpc/default-compiler
mpfr/3.0.0/default-compiler
mpfr/default-compiler
netcdf/intel (D)
netcdf/4.3.1/intel/2013
netcdf/4.4.1/intel/2017
plasma/2.6.0
------------- /opt-ictp/ohpc/modulefiles/compilers -------------
cuda/5.5 intel/2013 (D)
cuda/6.5 intel/2016
cuda/9.1 (D) intel/2017
default-compiler nocomp
g++_addon/4.9.0 opencl-amd
gcc/4.4.7 opencl-intel
gcc/4.5.2 opencl/amd/2.8
gcc/4.6.2 opencl/intel/1.2
gcc/4.9.0 (D) pgi/10.9
gnat/2014 pgi/16.10 (D)
gnu/5.4.0 pgi/18.4
gnu7/7.2.0
--------------- /opt-ictp/ohpc/modulefiles/apps ----------------
PHCpack/2.3.95
R/2.15 (D)
R/3.1.2
R/3.3.1
R/3.5.1
alps/2.1.1
anaconda3/5.1.0
cdo/1.9.0
cp2k/2.4 (D)
cp2k/2.6.1
espresso/5.1 (D)
espresso/5.2.1
espresso/6.2.1
ferret/v6.842
gdal/1.7.3
gdal/2.2.4
gdal/2.3.0
gdal/2.3.1 (D)
gdl/0.9.4/intel
gengetopt/2.22/default-compiler
glpk/4.45/gnu/4.4.0
gnu/gsl/2.2.1
gnu/hdf5/1.8.17
gnu/numpy/1.11.1
gnu/openblas/0.2.19
gnu/openmpi/1.10.6
grads/binary
gromacs/2016.3
lhapdf/5.8.6/default-compiler
maple/14
matlab/r2011
ncl/5.2.1/gnu/4.4.7
ncl/6.0.0/gnu/4.4.7
ncl/6.3.0/gnu/4.4.7
nco/intel (D)
nco/4.4.2/intel
nco/4.6.8/intel
nwchem/6.5
proj/4.7.0
proj/5.1.0 (D)
python/esp
python/2.7.13 (D)
root/5.30.04/default-compiler
root/6.08.06
------------------ /opt/ohpc/pub/modulefiles -------------------
autotools (L) gnu7/7.2.0 papi/5.5.1
clustershell/1.8 llvm4/4.0.1 pmix/1.2.3
cmake/3.9.2 llvm5/5.0.0 prun/1.1
gnu/5.4.0 ohpc (L) prun/1.2 (L,D)
Where:
D: Default Module
L: Module is loaded
To load, e.g., the openmpi module for the intel compiler you can then run module load openmpi/intel. If you then run the command module list you will see:
$ module load openmpi/intel
$ module list
Currently Loaded Modulefiles:
1) intel/2013 2) openmpi/intel
as expected, the intel module has been loaded automatically since openmpi requires it.
]]>
The service uses the SparkleShare client software from http://www.sparkleshare.org/ You may also obtain a copy of the client software using the links below.
ICTP Desktop in Linux: To install run the command sudo ictp-install sparkleshare
Ubuntu and similar distributions such as LinuxMint: To install use the command sudo apt-get install sparkleshare
Other Linux versions may have to compile the source code available from http://www.sparkleshare.org/
Please NOTE that for synchronisation to work correctly, the names of files, documents, directories, folders and shared-folders MUST NOT include the following characters:
<
>
:
|
?
*
\
/
Also, "hidden" files (files that begin with a dot character) are not synchronised.
After, installation, you will need to manually start the client software. Once it is running (indicated by an icon on the task-bar) following the steps below to:
For ICTP account holders. Simply login to the admin page and create a share (project) to begin. The following procedures are available only after you Login to the Sharebox admin page. Before anyone can access your share, you will have to grant them access. Access control is based on a "Client ID" , which is unique for each device. That is the "Client ID" for a laptop is different from the that of a desktop, even if they are both used exclusively by the same individual. This implies that you will require a separate authorization for your home computer, different from your laptop and different from your desktop/office computer.
Thank you and enjoy the sharebox.ictp.it service.
]]>