Batch system¶

A batch system queues user-submitted jobs and schedules their execution on compute nodes, ensuring fair and efficient distribution of shared resources among all users. On HoreKa 2, users have direct access to the login nodes only; all compute nodes are accessible exclusively through the batch system Slurm.

Slurm is an open-source, fault-tolerant, and highly scalable job scheduler for Linux clusters. It serves three key functions:

1. Allocating exclusive and/or shared access to compute nodes for a defined duration.
2. Providing a framework for launching, executing, and monitoring work on allocated nodes.
3. Managing a queue of pending jobs to resolve competing resource requests fairly.

All compute access on HoreKa 2 goes through Slurm — whether as a batch job (a script with commands and resource requirements, run asynchronously) or as an interactive job (a live session on allocated compute nodes).

The amount of available resources is tied to the respective compute project; Slurm enforces these limits and handles the accounting of consumed resources.

HoreKa 2 partitions¶

Slurm organizes compute nodes into partitions (informally also called queues) that group similar hardware and enforce different access policies and resource limits. You must specify the appropriate partition when submitting a job to target the desired hardware.

Regular partitions¶

SMT is enabled on all CPU nodes. Your application may benefit from it, but using more threads than physical cores can significantly impact performance depending on the workload. The large partition is intended for memory-intensive workloads that cannot fit on standard nodes.

As compute intensive tasks must not run on login nodes, development partitions exist to provide fast, usually interactive access to compute nodes — ideal for compiling software, testing workflows, or any short task that requires real compute resources.

Development partitions¶

The dev- partitions are intended for short, interactive sessions — developing, compiling, and testing code — where immediate resource access matters. Use them for quick compute tasks that would be disruptive on the login nodes.

Agent partition¶

The agent partition is a dedicated single-node single-core partition for AI agent workloads. It provides access to one HoreKa Onyx node and is intended for agentic AI workflows that require direct, persistent compute access — for example by connecting an IDE or AI coding assistant via an SSH tunnel to the allocated node (see Code Editors). Jobs are limited to a single task and may run for up to three days.

Slurm Usage¶

A typical workflow looks like this:

1. Check available resources — use sinfo_t_idle to see which nodes are currently idle, and scontrol show partition <name> to inspect partition limits and configuration.
2. Submit a job — for quick tests or interactive development, start an interactive session with salloc on a dev- partition. For production runs, write a job script and submit it with sbatch.
3. Monitor your jobs — track the status of your jobs in the queue with squeue -u $USER. Use squeue --start -j <jobid> to see when a pending job is expected to start.
4. Inspect and cancel — if something looks wrong, use scontrol show job <jobid> for details or scancel <jobid> to cancel the job.
5. Review completed jobs — after a job finishes, sacct -j <jobid> shows runtime, exit status, and resource usage.

The official Slurm documentation is quite exhaustive, so this page focuses on the most important commands and use cases. All commands are also available as manpages on the cluster, e.g. man sbatch.

Quick Reference¶

Submit Jobs¶

Submitting a job means requesting resources — such as CPUs, GPUs, or memory — from the cluster, so that Slurm can schedule and run your computation on the appropriate compute nodes.

Two commands are used to submit and run jobs in Slurm.

• sbatch submits a batch job script: the job is queued and runs asynchronously once resources are available, with output written to a file — this is the standard way to run production jobs.
• salloc allocates compute resources interactively, giving you a shell on the allocated nodes for development and testing.

Command Parameters¶

These options apply to both sbatch and salloc. They can be specified as command-line parameters or, for sbatch, as #SBATCH pragmas inside the job script.

On HoreKa 2, two parameters are mandatory — all others have sensible default values:

• --partition — selects the target hardware (see partitions above).
• --time — sets the maximum wall clock time; the job is cancelled if it exceeds this limit.

The following parameters are available, grouped by category. Most have a short form (e.g. -t) and a long form (e.g. --time) — both are equivalent. Command-line parameters always take precedence over #SBATCH pragmas in the script.

Examples¶

Please see Slurm Examples.

Monitor and Manage Jobs¶

Use squeue to inspect the job queue, scontrol for detailed job state, sacct for completed job accounting, and scancel to cancel a job.

Enter a running job¶

In some cases, you might want to access the resources of an existing allocation, e.g. in order to when troubleshoot a non-interactive job. SSH into the node with the running job is not possible. Instead, accessing the node can be done using Slurm's srun command. After allocating resources, e.g. with sbatch or salloc, you can access the resources of this allocation with

$ srun --jobid=<jobid> --pty bash

It creates a new job step and executes bash in pseudo terminal mode on task zero of this job step, i.e. it gives you an interactive bash on the previously allocated job. The job id is printed immediately after a job allocation and can be retrieved later with the squeue command.

If all the allocated CPU resources are already used, srun will prohibit the new job step the access to the resources. However, the argument --overlap can be passed to srun to allow job steps to overlap on the resources. In case you need to access specific nodes within the job allocation, you can do so with the argument --nodelist=<NODELIST>.

Example:

$ srun --jobid=11748 --nodelist=hk2n1132 --overlap --pty bash

In order to get the nodelist of a certain job, just type squeue -j <jobid> -o "%N"

Keep in mind, that your subsequent job will be cancelled if the original allocation is released. That can happen if a task running in the allocation is finished or cancelled.

If you plan to debug and develop code on compute nodes via an IDE such as VS Code or similar, use the SSH tunnel to compute nodes instead.

Explore Resources¶

Use sinfo_t_idle to see which nodes are currently idle, and scontrol show partition <name> to see the configured limits for a specific partition.

Environment variables¶

When Slurm starts a job, it automatically sets a number of environment variables inside the job environment. They give your job script access to runtime information — such as the assigned job ID, the list of allocated nodes, or the number of tasks.
The following environment variables are available within batch jobs while they are running:

Energy measurement¶

Slurm allows for the measurement of consumed energy on a job basis. This is automatically done for every job startet on HoreKa.

The energy consumed can be checked after a job has finished with either scontrol show job in the energy field or with sacct using the --format option and the field ConsumedEnergy e.g.:

$ sacct --format User,Account,JobID,JobName,ConsumedEnergy,NodeList,Elapsed,State

Furthermore a job feedback is appended to every job outputfile. Here both energy consumed in Joule / Watthours and the average node power draw are displayed.

To display the energy consumed during a running job sstat can be used e.g.:

$ sstat <jobid> --format JobID,ConsumedEnergy,NodeList

Further Reading¶

• Slurm Tutorials
• Slurm command/option summary (2 pages)
• Slurm Commands

Switching Between Projects¶

If you are a member of more than one compute project, you must actively switch between them. On HoreKa 2, $PROJECT is derived from the Slurm default account — changing the default account is therefore sufficient to update both.

Your current default account is printed on the login screen or can be checked via:

sacctmgr show user -n -P format=DefaultAccount

To see all accounts available to you:

sacctmgr show associations user=$USER -n -P format=Account

The recommended way to switch the project is:

sacctmgr modify user $USER set defaultaccount=<project_group>

For all changes to take full effect, you need to log out and log in again.

To bill an individual job to a different project without changing your default account, use the -A / --account option:

sbatch --account=<project_group> myjob.sh

This overrides the default account for that job only and does not affect $PROJECT or your Slurm default account.