HoreKa 2 / HAICORE 3.0¶
Scheduled Downtime: Shutdown of all HPC Clusters: 20.11.2026 – 04.12.2026
From 23 November 2026 until 4 December 2026 (expected), the computing center building (449.3) at KIT Campus North will be completely disconnected from the power grid due to mandatory inspections and maintenance work on the electrical infrastructure. This will affect all HPC systems operated at KIT. Jobs will only be running until noon on Friday, November 20, 2026. The following clusters will be unavailable during this period:
- bwUniCluster 3.0
- HoreKa 2
- HoreKa
- HAICORE@KIT
- Future Technologies Partition
During this period, access to your data will not be possible. Compute jobs will not run while the systems are offline. Workspace expiration and automatic deletion will be suspended for the duration of the downtime; therefore, no ws_extend request is required.
To avoid additional service interruptions in the near future, we will combine this building-related outage with our own maintenance activities, including operating system updates, software upgrades, file system maintenance, completion of the final phase of the power busbar upgrade, and preparations of the rack infrastructure for the planned GPU-based expansion of the HPC systems in 2027. Most importantly, this will include the final phase of bringing the Jade partition of HoreKa 2 into operation.
Once the maintenance work has been completed, the HPC systems will be brought back into operation one cluster at a time. The restart of all systems is planned to be completed by 4 December 2026.
Please be aware that, following the downtime and commissioning of the Jade partition, the second mandatory migration phase is active. This means that HoreKa (legacy) will only be accessible for logging in and accessing the file systems.
HoreKa 2 alias Hochleistungsrechner Karlsruhe 2 is an innovative hybrid system with more than 42,000 processor cores, 250 terabytes of main memory and 300 NVIDIA B200 GPUs. The CPU partition is called HoreKa Onyx, while the GPU partitions are called HoreKa Jade (NVIDIA GB200), HoreKa Teal (NVIDIA H100) and HoreKa Ruby (NVIDIA H200). HoreKa Blue provides CPU-only nodes with extra large memory.
HoreKa 2 is housed in a dedicated computer building on KIT's North Campus, featuring a specialized infrastructure designed to manage the high power density and cooling requirements of modern HPC systems. This includes advanced cooling solutions to ensure the stable and efficient operation of the high-performance CPU and GPU nodes.
Compute Hardware¶
HoreKa 2 / HAICORE 3.0 is designed as a hybrid system, consisting of separate hardware "islands" for x86 and ARM architectures. This allows the system to leverage the strengths of both architectures: the broad software compatibility of x86 and the high energy efficiency and specialized performance of ARM (specifically the NVIDIA Grace CPUs).
Compute nodes are not directly accessible. Jobs must be submitted via the Slurm batch system, which schedules and executes them based on priority and resource availability. A single job can span hundreds of nodes and tens of thousands of CPU cores.
| Node Type | HoreKa Onyx Standard-CPU |
HoreKa Blue Extra Large Memory |
HoreKa Jade Accelerated GB200 NVL4 |
HoreKa Teal Accelerated 4x H100 |
HoreKa Teal Accelerated 8x H200 |
HoreKa Ruby Accelerated 4x H200 |
|---|---|---|---|---|---|---|
| Status | operational | operational | coming soon  |
operational | operational | operational |
| No. of nodes | 164 | 8 | 75 | 22 | 1 | 13 |
| Availability in partition | cpu, dev-cpu |
large |
gpu-b200, dev-gpu-b200 |
gpu-h100, dev-gpu-h100 |
gpu-h200-8 |
gpu-h200 |
| CPUs | AMD EPYC 9655 | Intel Xeon Platinum 8368 | NVIDIA Grace | AMD EPYC 9354 | Intel Xeon Platinum 8568Y+ | AMD EPYC 9454 |
| CPU Sockets per node | 2 | 2 | 2 | 2 | 2 | 2 |
| CPU Cores / Threads per node | 192 / 384 | 76 / 152 | 144 / 288 | 64 / 128 | 96 / 192 | 96 / 192 |
| Main memory | 768 GiB | 4096 GiB | 960 GiB | 768 GiB | 2048 GiB | 768 GiB |
| Accelerators | - | - | 4x NVIDIA B200 | 4x NVIDIA H100-94 | 8x NVIDIA HGX H200 | 4x NVIDIA HGX H200 |
| Memory per accelerator | - | - | 186 GB | 94 GB | 141 GB | 141 GB |
| Local disks | 1,92 TB NVMe SSD | 7 x 3,84 TB NVMe SSD | 15,36 TB NVMe SSD | 2 x 3,84 TB NVMe SSD | 8 x 3,84 TB NVMe SSD | 15,36 TB NVMe SSD |
| Interconnect | InfiniBand NDR400 Shared I/O (200 Gbit per node) |
InfiniBand HDR200 | 2x InfiniBand NDR200 Dual Port (4x 200 Gbit per node) |
4x InfiniBand NDR200 | 8x InfiniBand NDR400 | 4x InfiniBand NDR400 |
Partition Description¶
- HoreKa Onyx nodes provide CPU-only compute power. In operation since 08/2026.
- HoreKa Jade nodes feature the NVIDIA GB200 NVL4 platform with 2 Grace CPUs and 4 Blackwell GPUs each. Operation will start in December 2026.
- HoreKa Blue nodes provide CPU-only compute power. The nodes are equipped with a huge amount of main memory. In operation since 02/2023.
- HoreKa Teal nodes are equipped with 4x NVIDIA H100 GPUs. In operation since 05/2024. One node is equipped with 8x NVIDIA H200 GPUs, in operation since 10/2025.
- HoreKa Ruby nodes are equipped with 4x NVIDIA H200 GPUs. In operation since 08/2025.
All nodes run Red Hat Enterprise Linux (RHEL) 9.x and are managed by the Slurm batch system.
Login Nodes¶
Separate login nodes are provided for x86 and ARM architectures, so software can be compiled and tested on the same architecture it will run on. Login nodes are the only nodes directly accessible to users and are intended for interactive work, file management, and software development. See Login for hostnames and access instructions.
| Node Type | x86_64-Login | ARM-Login |
|---|---|---|
| Status | operational | coming soon |
| No. of nodes | 2 | 2 |
| CPUs | AMD EPYC 9655 | Grace Superchip |
| CPU Sockets per node | 1 | 2 |
| CPU Cores / Threads per node | 96 / 192 | 144 / 144 |
| Main memory | 512 GiB | 480 GiB |
| Accelerators | - | - |
| Interconnect | InfiniBand NDR200 |
InfiniBand NDR200 |
System Components¶
Interconnect¶
All nodes are connected via InfiniBand NDR (up to 400 Gbit/s per port depending on node type), providing high throughput and very low latency (~1 µs). This makes the network well suited for communication-intensive workloads and collective MPI operations.
Storage¶
HoreKa 2 uses a multi-level storage concept. The primary storage is based on IBM Spectrum Scale (GPFS) parallel file systems, providing globally accessible home ($HOME) and project ($PROJECT) directories, as well as workspace storage. See Storage for an overview of the available file systems and usage model.
Each node also has local NVMe SSD storage accessible as $TMPDIR during a job. For jobs spanning multiple nodes, a temporary BeeGFS on Demand (BeeOND) file system can be requested. See Storage for details and quotas.
Infrastructure¶
The HoreKa 2 facility is designed for dense, energy-efficient high-performance computing. The dedicated computer building on KIT’s North Campus houses the cluster in a purpose-built environment optimized for high power density and thermal management.