Storage¶

A central aspect in the design of HoreKa 2 has been the enormous amount of data generated and used by scientific research projects. A multi-level data storage concept guarantees high-throughput processing of data using several different storage systems.

The core of this design is two large-scale, parallel file systems based on IBM Storage Scale (also known as GPFS) used for globally visible user data. Individual home and project directories are automatically created for each user on the home file system, with $HOME and $PROJECT pointing to these directories. Each user can also create so-called workspaces on the work file system.

Other storage locations include a temporary directory called $TMPDIR located on the local solid state disks (SSDs) of a node, visible only on that individual node during job runtime. For temporary storage accessible from all nodes of a batch job, users can request a per-job BeeGFS On Demand (BeeOND) file system, accessible only from the job's nodes during its runtime.

Users with LSDF access can additionally use the LSDF's file systems directly from HoreKa 2.

File systems¶

The characteristics of the file systems are shown in the following table:

• global: all nodes see the same file system
• local: each node has its own local file system
• job local: only available within the currently running job
• permanent: data is stored permanently (across job runs and reboots)
• limited: data is stored across job runs and reboots, but will be deleted at some time
• job walltime: files are removed at end of the batch job

File system details:

• $HOME — permanent home directory with quota per user
• $PROJECT — shared project storage with quota per project group
• Workspaces — temporary scratch space with limited lifetime
• $TMPDIR — fast local NVMe SSD storage per node
• LSDF Online Storage — large-scale data facility for LSDF users
• BeeOND — private BeeGFS on-demand file system per job

Selecting the appropriate file system¶

In general, you should separate your data and store it on the appropriate file system.

• Permanently required data like software or important results should be stored below $HOME or $PROJECT, but capacity limits (so-called "quotas") apply. Permanent data which is not needed for months or exceeds the capacity restrictions should be sent to other large scale (e.g. to the LSDF) or archive (e.g. bwDataArchive) storage systems and deleted from the home file system.
• Temporary data which is only needed on a single node, which does not exceed the disk space shown in the table above and which is only needed during job runtime should be stored below $TMPDIR.
• Temporary data which is only needed during job runtime and which needs to be accessed from all nodes of a batch job should be stored on BeeOND.
• Scratch data which can be easily recomputed or which is the result of one job and input for another job should be stored below so-called workspaces. The lifetime of data in workspaces is limited and depends on the lifetime of the workspace.

Snapshots and backup¶

In case you inadvertently deleted some of your data, want to go back to a previous version or compare your data with a previous version you can use so-called snapshots. Snapshots are a point-in-time copy of your data.

• For $HOME and $PROJECT there are snapshots of the last 7 days, of the last 4 weeks and of the last 6 months. Snapshots are located below /hfs2/data/.snapshots/.
• For the workspaces there are snapshots of the last 7 days and of the last 4 weeks. Snapshots are located below /hfs2/work/.snapshots/.

There are also regular backups of all data of the project directories, whereas ACLs and extended attributes will not be saved by the backup. Please open a support ticket if you need us to restore backup data.

Quotas¶

Quota limits restrict the capacity and the number of inodes (number of files and diretories) of a user or group on a file system or on a subdirectory tree of the file system. There are soft and hard quota limits. If the hard limit is reached an I/O error is created and the application aborts if additional data should be stored. If the soft limit is reached during the grace period (7 days) only a warning message is created, afterwards this also causes an I/O error.

$HOME¶

On $HOME the capacity limit is 50 / 55 GiB (soft / hard) and the inode limit is 2 / 3 million (soft / hard). Check the current quota usage and limits of your user account with

/usr/lpp/mmfs/bin/mmlsquota -u $(whoami) --block-size G -C hk2n.scc.kit.edu hfs2-home:home

$PROJECT¶

For each project the capacity limit is 10 / 11 TiB (soft / hard) and the inode limit is 10 / 11 million (soft / hard). Check the current quota usage and limits of your project group with

/usr/lpp/mmfs/bin/mmlsquota -j $PROJECT_GROUP --block-size G -C hk2n.scc.kit.edu hfs2-home

Workspaces¶

On the work file system the capacity limit is 250 / 270 TiB (soft / hard) and the inode limit is 50 / 55 million (soft / hard). This includes the data of all your workspaces which are stored on this file system. Check the current quota usage and limits of your account on the workspace file system:

/usr/lpp/mmfs/bin/mmlsquota -u $(whoami) --block-size G -C hk2n.scc.kit.edu hfs2-work

Data sharing¶

There are two recommended ways to share data with other users or project members:

• $PROJECT — for permanent shared data within a project group. Access is controlled via standard Unix group permissions, since all project members share the same group. See $PROJECT.
• Workspaces — for temporary scratch data that needs to be shared. Access control lists (ACLs) allow fine-grained permissions for specific users or groups beyond the project group. See Workspaces.

Performance tuning¶

Hints on file system performance tuning can be found in the file system performance tuning section.

Deprovisioning¶

User data from expired projects is still available for 3 months after the project ends. Data in the home directory is retained for 6 months after the last project ends.