HoreKa 2 / HAICORE 3.0¶

Welcome to the Tier-2 High Performance Computing system Hochleistungsrechner Karlsruhe 2 (HoreKa 2) at KIT.

HoreKa 2 is an innovative hybrid system with more than 42,000 processor cores, 250 terabytes of main memory and 300 NVIDIA B200 GPUs. The CPU partition is called HoreKa Onyx, while the GPU partitions are called HoreKa Jade (NVIDIA GB200), HoreKa Teal (NVIDIA H100) and HoreKa Ruby (NVIDIA H200). HoreKa Blue provides CPU-only nodes with extra large memory.

HoreKa is housed in a dedicated computer building on KIT's North Campus, featuring a specialized infrastructure designed to manage the high power density and cooling requirements of modern HPC systems. This includes advanced cooling solutions to ensure the stable and efficient operation of the high-performance CPU and GPU nodes.

Compute Hardware¶

HoreKa 2 / HAICORE 3.0 is designed as a hybrid system, consisting of separate hardware "islands" for x86 and ARM architectures. This allows the system to leverage the strengths of both architectures: the broad software compatibility of x86 and the high energy efficiency and specialized performance of ARM (specifically the NVIDIA Grace CPUs).

Partition Description¶

• HoreKa Onyx nodes provide CPU-only compute power. In operation since 08/2026.
• HoreKa Jade nodes feature the NVIDIA GB200 NVL4 platform with 2 Grace CPUs and 4 Blackwell GPUs each. In operation since 09/2026.
• HoreKa Blue nodes provide CPU-only compute power. The nodes are equipped with a huge amount of main memory. In operation since 02/2023.
• HoreKa Teal nodes are equipped with 4x NVIDIA H100 GPUs. In operation since 05/2024. One node is equipped with 8x NVIDIA H200 GPUs, in operation since 10/2025.
• HoreKa Ruby nodes are equipped with 4x NVIDIA H200 GPUs. In operation since 08/2025.

All nodes run Red Hat Enterprise Linux (RHEL) 9.x and are managed by the Slurm batch system.

Login Nodes¶

Separate login nodes are provided for x86 and ARM architectures, so software can be compiled and tested on the same architecture it will run on. Login nodes are the only nodes directly accessible to users and are intended for interactive work, file management, and software development. See Login for hostnames and access instructions.

Compute Nodes¶

Compute nodes are not directly accessible. Jobs must be submitted via the Slurm batch system, which schedules and executes them based on priority and resource availability. A single job can span hundreds of nodes and tens of thousands of CPU cores.

System Components¶

Interconnect¶

All nodes are connected via InfiniBand NDR (up to 400 Gbit/s per port depending on node type), providing high throughput and very low latency (~1 µs). This makes the network well suited for communication-intensive workloads and collective MPI operations.

Storage¶

HoreKa 2 uses a multi-level storage concept. The primary storage is based on IBM Spectrum Scale (GPFS) parallel file systems, providing globally accessible home ($HOME) and project ($PROJECT) directories, as well as workspace storage. See Storage system for an overview of the available file systems and usage model.

Each node also has local NVMe SSD storage accessible as $TMPDIR during a job. For jobs spanning multiple nodes, a temporary BeeGFS on Demand (BeeOND) file system can be requested. See Storage for details and quotas.

Facility Infrastructure¶

The HoreKa 2 facility is designed for dense, energy-efficient high-performance computing. The dedicated computer building on KIT’s North Campus houses the cluster in a purpose-built environment optimized for high power density and thermal management. For details on the building, cooling concept, and energy performance, see Infrastructure.