Santis¶

Santis is the main Climate and Weather Platform cluster that provides compute nodes and file systems for GPU-enabled climate and weather workloads.

Cluster specification¶

Compute nodes¶

Santis consists of 302 Grace-Hopper nodes.

The number of nodes can vary as nodes are added or removed from other clusters on Alps. See the Slurm documentation for information on how to check the number of nodes.

There are two login nodes, santis-ln00[1,2], which are repurposed compute nodes from the pool of 302 GH200 nodes. You will be assigned to one of the two login nodes when you ssh onto the system, from where you can edit files, compile applications and launch batch jobs. This leaves 300 nodes available for compute jobs.

Storage and file systems¶

Santis uses the CWP filesystems and storage policies.

Getting started¶

Logging into Santis¶

To connect to Santis via SSH, first refer to the ssh guide.

Host santis
    HostName santis.alps.cscs.ch
    ProxyJump ela
    User cscsusername
    IdentityFile ~/.ssh/cscs-key
    IdentitiesOnly yes

Software¶

uenv¶

Santis provides uenv to deliver programming environments and application software for the climate and weather community. Please refer to the uenv documentation for detailed information on how to use the uenv tools on the system.

# list all images available for daint
uenv image find @daint

# download an image for daint
uenv image pull namd/3.0:v3@daint

# start the uenv
uenv start namd/3.0:v3@daint

Containers¶

Santis supports container workloads using the container engine.

To build images, see the guide to building container images on Alps.

Running jobs on Santis¶

Slurm¶

Santis uses Slurm as the workload manager, which is used to launch and monitor compute-intensive workloads.

There are four Slurm partitions on the system:

Node sharing and GPU requests¶

All GH200 partitions on Santis use node sharing. A job no longer receives a full node by default. Instead, resources are allocated at the granularity of one GH200 chip:

• request 1 GPU → receive 1 chip (72 cores, ~217 GB RAM)
• request 4 GPUs → receive the whole node

Jobs must request GPUs explicitly, for example:

sbatch --gres=gpu:2 ...
# or
sbatch --gpus-per-node=2 ...

If no GPU is requested, the job may receive only a minimal CPU allocation and will not run on an exclusive node.

Multi-GPU jobs with P2P/IPC¶

Because device isolation is enforced via cgroups, multi-GPU jobs that rely on GPU P2P/IPC (for example MPI across GPUs on the same node) must add to the srun command (NB: this does not work being specified in the #SBATCH definition block):

--gres-flags=allow-task-sharing

This keeps all GPUs allocated to a job visible to all tasks of that job while preserving per-task CUDA_VISIBLE_DEVICES bindings. Without this flag, intra-node GPU-GPU communication will fail.

CPU Power Capping¶

This power capping feature allows to optimize power distribution on the GH200 nodes in favour of the GPUs for applications using CPU and GPU simultaneously. You can find more details and instructions how to activate the power capping in your runs here.

Low-priority overflow partition¶

The low partition is available to all users for work that should only run when the higher-priority partitions have idle capacity. It is intended for:

• quota-exhausted projects that still need some resources before the next quarterly reset
• work that does not need fast turnaround

Jobs in low have lower scheduling priority and are not preempted.

Concurrent job limit¶

Each user may have at most 10 jobs running at the same time across normal, debug, and low. Submissions are not limited; additional jobs queue until one of the running jobs finishes.

See the Slurm documentation for instructions on how to run jobs on the Grace-Hopper nodes.

FirecREST¶

Santis can also be accessed using FirecREST at the https://api.cscs.ch/cw/firecrest/v2 API endpoint.

Maintenance and status¶

Scheduled maintenance¶

One Wednesday per month is reserved for planned maintenance (usually around the middle of the month, 8-12 CET), with services potentially unavailable during this timeframe. If the queues must be drained (redeployment of node images, rebooting of compute nodes, etc) then a Slurm reservation will be in place that will prevent jobs from running into the maintenance window.

Exceptional and non-disruptive updates may happen outside this time frame and will be announced via the CSCS status page.

Change log¶

Known issues¶