Skip to content

Security Vulnerabilities

This page describes how NHR@KIT responds when a security vulnerability that affects the HPC clusters is disclosed.

Current Situation

The security situation is currently tense as the number of disclosed vulnerabilities has significantly increased in the past months.

This includes an exceptional amount of vulnerabilities enabling local privilege escalation (LPE). By exploiting a LPE, any user with access to a system can gain administrative (root) privileges. On a multi-user system like the HPC clusters, this means that an adversary can potentially read, modify or delete data, disrupt running jobs or destabilize the entire system.

In most cases, the only remediation for these vulnerabilities is an update of the affected components. As LPEs often are the result of a vulnerability in the Linux kernel, a complete fix includes a reboot of the affected systems. As a consequence, this unfortunately means that we might be forced to temporarily limit our services on short-notice at any time in the current situation.

Warning

Maintenance caused by security vulnerabilities cannot always be announced in advance. You can take the following measure to minimize the impact of such downtimes on your work:

  • Use checkpointing in long-running jobs.
  • Do not rely on a job running to completion without interruption.
  • Keep copies of important data outside the cluster.

Measures

Depending on the severity of the vulnerability and the availability of a mitigation, an update or public exploit, we may take the following measures. They are listed in increasing order of severity.

  1. Update or mitigation: Most updates or mitigations are deployed without service disruption. Nonetheless, in case critical system components are affected you may experience short disruptions, e.g. outage of the login nodes. Where possible, we try to limit those disruptions to our regular reboot window on Thursday morning between 7:00 and 9:00 am.
  2. Login restriction: For severe vulnerabilities like LPEs: If no update or mitigation is available at time of disclosure, access to the cluster may be disabled to prevent exploitation of the vulnerability.
  3. Interruption of running jobs: If the remediation of a vulnerability requires an immediate restart of critical system services, running jobs may be interrupted.

We are aware that this affects the overall usability of our systems. Therefore we decide on a case-by-case basis which measures are appropriate to the risk of each vulnerability.

Communication

  • Service announcements: Information on operational disruptions is published at https://www.scc.kit.edu/dienste/meldungen.php.
  • Mailing lists: Depending on which HPC system is affected, the HPC operators also send e-mails via the corresponding mailing list.
  • Maintenance log: Completed security-related downtimes and the changes made are documented on the Maintenance page.

Important

Please read the e-mails sent by the HPC operators. They contain information about downtimes, required actions and the current state of the affected systems.

For questions, contact the support.