Skip to content

Intel

Work in Progress

This page is not yet complete. Content is being added gradually. Functionality may be incomplete or subject to change. Thank you for your patience!

The FTP hosts a diverse set of Intel compute nodes spanning three distinct hardware families: Sapphire Rapids CPUs (with and without HBM), the Ponte Vecchio (PVC) GPU accelerator, and the Gaudi AI accelerator series. This range allows users to evaluate Intel's full HPC and AI portfolio under realistic cluster conditions.

All Intel nodes are installed in the x86 cluster and are accessible via the x86 login node.

Hardware Overview

Intel Sapphire Rapids CPU

Sapphire Rapids is Intel's 4th-generation Xeon Scalable processor. Two variants are available:

Node type Description Queue
Sapphire Rapids Standard DDR5 memory configuration intel-spr
Sapphire Rapids + HBM CPU with integrated High Bandwidth Memory (HBM2e) intel-spr-hbm

The HBM variant allows users to evaluate memory-bandwidth-bound workloads directly on the CPU package without a discrete accelerator.

Intel Ponte Vecchio (GPU Max)

Ponte Vecchio (PVC) is Intel's datacenter GPU accelerator, also marketed as Intel Data Center GPU Max. It is based on the Xe-HPC microarchitecture and targets HPC and AI workloads. PVC nodes pair a Sapphire Rapids CPU with one or more PVC cards:

Node type Description Queue
Sapphire Rapids + PVC Intel Sapphire Rapids + Ponte Vecchio GPU intel-spr-pvc

PVC supports SYCL/DPC++, OpenCL, and Level Zero as primary programming interfaces. OpenMP offload is also supported.

Intel Gaudi AI Accelerator

Gaudi is Intel's dedicated AI training and inference accelerator. Two generations are available in the FTP:

Node type Description Queue
Gaudi 2 Intel Gaudi 2 AI accelerator gaudi2
Gaudi 3 Intel Gaudi 3 AI accelerator gaudi3

Gaudi nodes operate in exclusive mode. The Gaudi software stack includes the Intel Gaudi Software Suite with PyTorch integration.

Hardware Overview

The system management interface tool hl-smi aids in the management and monitoring of the Gaudi accelerators.

Running hl-smi without an Options argument set displays a summary table of the detected Gaudi devices:

+-----------------------------------------------------------------------------+
| HL-SMI Version:                              hl-1.19.1-fw-57.2.2.0          |
| Driver Version:                                     1.19.1-6f47ddd          |
|-------------------------------+----------------------+----------------------+
| AIP  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncor-Events|
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | AIP-Util  Compute M. |
|===============================+======================+======================|
|   0  HL-225              N/A  | 0000:19:00.0     N/A |                   0  |
| N/A   27C   N/A  82W /  600W  |   768MiB /  98304MiB |     0%           N/A |
|-------------------------------+----------------------+----------------------+
|   1  HL-225              N/A  | 0000:b3:00.0     N/A |                   0  |
| N/A   26C   N/A  73W /  600W  |   768MiB /  98304MiB |     0%           N/A |
|-------------------------------+----------------------+----------------------+
|   2  HL-225              N/A  | 0000:1a:00.0     N/A |                   0  |
| N/A   28C   N/A  75W /  600W  |   768MiB /  98304MiB |     0%           N/A |
|-------------------------------+----------------------+----------------------+
|   3  HL-225              N/A  | 0000:b4:00.0     N/A |                   0  |
| N/A   28C   N/A  79W /  600W  |   768MiB /  98304MiB |     0%           N/A |
|-------------------------------+----------------------+----------------------+
|   4  HL-225              N/A  | 0000:43:00.0     N/A |                   0  |
| N/A   28C   N/A  81W /  600W  |   768MiB /  98304MiB |     0%           N/A |
|-------------------------------+----------------------+----------------------+
|   5  HL-225              N/A  | 0000:cc:00.0     N/A |                   0  |
| N/A   28C   N/A  73W /  600W  |   768MiB /  98304MiB |     0%           N/A |
|-------------------------------+----------------------+----------------------+
|   6  HL-225              N/A  | 0000:44:00.0     N/A |                   0  |
| N/A   27C   N/A  82W /  600W  |   768MiB /  98304MiB |     0%           N/A |
|-------------------------------+----------------------+----------------------+
|   7  HL-225              N/A  | 0000:cd:00.0     N/A |                   0  |
| N/A   28C   N/A  77W /  600W  |   768MiB /  98304MiB |     0%           N/A |
|-------------------------------+----------------------+----------------------+
| Compute Processes:                                               AIP Memory |
|  AIP       PID   Type   Process name                             Usage      |
|=============================================================================|
|   0        N/A   N/A    N/A                                      N/A        |
|   1        N/A   N/A    N/A                                      N/A        |
|   2        N/A   N/A    N/A                                      N/A        |
|   3        N/A   N/A    N/A                                      N/A        |
|   4        N/A   N/A    N/A                                      N/A        |
|   5        N/A   N/A    N/A                                      N/A        |
|   6        N/A   N/A    N/A                                      N/A        |
|   7        N/A   N/A    N/A                                      N/A        |
+=============================================================================+

Using Gaudi Accelerators

Intel provides a custom Intel Gaudi PyTorch environment which is optimized for Intel Gaudi AI accelerator. The necessary software is preinstalled on the nodes and can be loaded using the Lmod module system. In order to avoid ambiguities, it is advisable to load architecture-specific modules on the compute nodes, working in an allocation created with salloc on the requested node:

salloc -p gaudi2 -t 01:00:00

salloc -p gaudi3 -t 01:00:00

We can then load the pytorch module:

module purge
module load toolkit/gaudi-torch

Afterwards a python program can be started via:

python torch_example.py

Code examples and more information can be found here.

Software for Gaudi

From our experience, the least error prone way to get a functional software stack for Gaudi is to use containers. These are provided by Intel directly. Here is a small working example to encorporate a Docker image from Intel, use the enroot runtime to launch it and to run a hello world example script found here here.

# Login to FTP
$ ssh <user>@ftp-x86-login.scc.kit.edu

# Allocate Gaudi 3 Node
$ salloc -p gaudi3 -t 01:00:00   

# Get Docker container
$ enroot import docker://vault.habana.ai#gaudi-docker/1.21.1/ubuntu24.04/habanalabs/pytorch-installer-2.6.0:latest
$ enroot create gaudi-docker+1.21.1+ubuntu24.04+habanalabs+pytorch-installer-2.6.0+latest.sqsh

# Download sample code
$ mkdir -p gaudi && cd gaudi
$ git clone -b 1.21.0 https://github.com/HabanaAI/Model-References.git
$ cd Model-References/PyTorch/examples/computer_vision/hello_world/

# Start container
$ enroot start --rw -m .:/mnt gaudi-docker+1.21.1+ubuntu24.04+habanalabs+pytorch-installer-2.6.0 bash
$ cd /mnt

# Run sample case
$ mpirun -n 8 --bind-to core --map-by socket:PE=6 \
      --rank-by core --report-bindings \
      --allow-run-as-root \
      -x PT_HPU_LAZY_MODE=0 python -W ignore mnist.py \
      --batch-size=64 --epochs=1 \
      --lr=1.0 --gamma=0.7 \
      --hpu --autocast --use-torch-compile
In a separate terminal on the same compute node, you can check the activity of the hardware:

# Login to FTP
$ ssh <user>@ftp-x86-login.scc.kit.edu

# Login to active compute node
$ srun --jobid $(squeue --format=%i -h|head -n 1) --pty bash

# Monitor the activity
$ hl-smi -l 1
You should see something like this: 
+-----------------------------------------------------------------------------+
| HL-SMI Version:                              hl-1.21.1-fw-59.2.3.0          |
| Driver Version:                                     1.21.0-ca59b5a          |
| Nic Driver Version:                                 1.21.0-732bcf3          |
|-------------------------------+----------------------+----------------------+
| AIP  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncor-Events|
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | AIP-Util  Compute M. |
|===============================+======================+======================|
|   0  HL-325L             N/A  | 0000:17:00.0     N/A |                   0  |
| N/A   45C   P0  222W /  900W  |131072MiB / 131072MiB |     0%          100% |
|-------------------------------+----------------------+----------------------+
|   1  HL-325L             N/A  | 0000:97:00.0     N/A |                   0  |
| N/A   44C   P0  223W /  900W  |131072MiB / 131072MiB |     0%          100% |
|-------------------------------+----------------------+----------------------+
|   2  HL-325L             N/A  | 0000:2c:00.0     N/A |                   0  |
| N/A   39C   P0  219W /  900W  |131072MiB / 131072MiB |     0%          100% |
|-------------------------------+----------------------+----------------------+
|   3  HL-325L             N/A  | 0000:ba:00.0     N/A |                   0  |
| N/A   39C   P0  220W /  900W  |131072MiB / 131072MiB |     0%          100% |
|-------------------------------+----------------------+----------------------+
|   4  HL-325L             N/A  | 0000:3d:00.0     N/A |                   0  |
| N/A   41C   P0  218W /  900W  |131072MiB / 131072MiB |     0%          100% |
|-------------------------------+----------------------+----------------------+
|   5  HL-325L             N/A  | 0000:a9:00.0     N/A |                   0  |
| N/A   43C   P0  222W /  900W  |131072MiB / 131072MiB |     0%          100% |
|-------------------------------+----------------------+----------------------+
|   6  HL-325L             N/A  | 0000:4e:00.0     N/A |                   0  |
| N/A   45C   P0  217W /  900W  |131072MiB / 131072MiB |     0%          100% |
|-------------------------------+----------------------+----------------------+
|   7  HL-325L             N/A  | 0000:cb:00.0     N/A |                   0  |
| N/A   43C   P0  224W /  900W  |131072MiB / 131072MiB |     0%          100% |
|-------------------------------+----------------------+----------------------+
| Compute Processes:                                               AIP Memory |
|  AIP       PID   Type   Process name                             Usage      |
|=============================================================================|
|   0       1661819     C   python                                  130400MiB  
|   1       1661823     C   python                                  130400MiB  
|   2       1661820     C   python                                  130400MiB  
|   3       1661825     C   python                                  130400MiB  
|   4       1661821     C   python                                  130400MiB  
|   5       1661824     C   python                                  130400MiB  
|   6       1661822     C   python                                  130400MiB  
|   7       1661826     C   python                                  130400MiB  
+=============================================================================+

Access

Intel nodes are available to all FTP users. See the FTP overview for how to request access.

Using Intel Nodes

Connect to the x86 login node and submit jobs to the desired partition:

ssh username@ftp-x86-login.scc.kit.edu

Use sbatch or salloc to access a partition, e.g.:

sbatch -p intel-spr-pvc script.sh

or, for interactive use:

salloc -p gaudi2 -t 30

Software for Intel

Intel nodes support the oneAPI toolchain, including the Intel oneAPI Base Toolkit and HPC Toolkit. Key components include the icx/icpx compilers, oneMKL, oneDNN, and the SYCL/DPC++ programming model for GPU offload.

Hello World Example