Machine learning platform¶

The Machine Learning Platform (MLP) provides compute, storage and expertise to the machine learning communities accessing the Alps Research Infrastructure.

Getting started¶

Getting access¶

Project administrators (PIs and deputy PIs) of projects on the MLP can to invite users to join their project, before they can use the project’s resources on Alps. This is performed using the project management tool.

Once invited to a project, you will receive an email, which you need to create an account and configure multi-factor authentication (MFA).

Systems¶

The main cluster provided by the MLP is Clariden, a large Grace-Hopper GPU system on Alps.

File Systems and Storage¶

The following file systems are mounted on the MLP clusters Clariden and Bristen:

In the paths above, <organization> is the organization your project belongs to (for example swissai) and <project> is your project’s name.

How storage works on the MLP¶

The file systems are tuned for different roles, and a typical job moves data between them rather than keeping everything in one place:

• Home (/users/$USER) holds your code, scripts, small configuration files and software. It is small, and not meant for datasets or job output.
• Scratch ($SCRATCH) is a large, fast workspace for running jobs: you stage input data in, the job reads and writes here, and you copy results out when it finishes. Scratch is temporary — a cleanup policy removes files that have not been accessed recently, and there are no backups.
• Project store (/capstor/store) is persistent, backed-up storage for the datasets and results you want to keep. It has no cleanup policy.
• Project data cache (/iopsstor/datacache) is a fast, project-level working area on Iopsstor for datasets shared across the whole project. Like scratch it is not backed up, but unlike scratch it has no cleanup policy — the project owns its data lifecycle. See project data cache.

The golden rule is to use scratch only while jobs are running, and move anything you want to keep to the project store afterwards — see transferring data between scratch and store for how to do this efficiently.

The same principle applies to datacache, only without a deadline: there is no cleanup policy forcing you to act, but because it is not backed up, the durable copy of anything you want to keep still belongs on the project store.

Scratch is available on more than one file system, on hardware tuned for different access patterns; choosing the right one matters for performance, as explained under file system suitability.

Home¶

Every user has a home path ($HOME) mounted at /users/$USER on the Vadret filesystem. The home directory has 50 GB of capacity, and is intended for configuration, small software packages and scripts.

Scratch¶

Scratch filesystems provide temporary storage for high-performance I/O for executing jobs. Use scratch to store datasets that will be accessed by jobs, and for job output. Scratch is per user - each user gets separate scratch path and quota.

• The environment variable SCRATCH=/iopsstor/scratch/cscs/$USER is set automatically when you log into a system of the ML platform, and can be used as a shortcut to access scratch.
• There is an additional scratch path mounted on Capstor at /capstor/scratch/cscs/$USER.
• There is a further scratch path on Ritom (a VAST file system) at /ritom/scratch/cscs/$USER.

Scratch Usage Recommendations¶

Use Iopsstor scratch ($SCRATCH) for:

• Training and validation datasets that are read frequently and non-sequentially.
• Workloads that perform many small, random I/O operations.

Use Capstor scratch (/capstor/scratch/cscs/$USER) for:

• Storing model checkpoints.
• Outputs from simulations or training jobs that involve large, contiguous I/O.

After your job completes, remember to transfer any important results to your project store, for example with rclone on the xfer queue.

Project store¶

The project store (/capstor/store/cscs/<organization>/<project>) is persistent, backed-up storage for datasets, shared code and configuration scripts that need to be accessed from different vClusters. It has no cleanup policy, and is per project: each project gets a folder with a project quota on capacity and inodes. Hard limits prevent writing once the quota is reached; you can check usage with the quota command on a login node or ela.

Writing to the project store directly from jobs is not recommended: stage data to scratch or the project data cache for a run, and keep the durable copy here.

Project data cache¶

datacache is a project-level working area on the fast Iopsstor file system, mounted at /iopsstor/datacache/cscs/<organization>/<project>. It gives the whole project one shared, high-performance copy of the datasets its members actively work on, so users no longer stage a private copy of the same data on their own scratch.

Like scratch, it is fast NVMe storage and is not backed up. Unlike scratch, it has no cleanup policy: files are never deleted automatically, and the project owns its data lifecycle and space hygiene within a project quota on capacity and inodes, as for the project store.

Because nothing here is backed up, datacache is a manual cache: CSCS never copies its contents anywhere else, so the durable copy of anything you keep must live on the project store (/capstor/store, which is backed up). A typical workflow keeps the authoritative dataset on the project store, stages it once into datacache where the whole project reads it at NVMe speed during training, and moves anything worth preserving back to the project store afterwards.