BeeOND¶

BeeOND (BeeGFS On-Demand) provides a private parallel file system per job, created automatically at job startup and deleted after job completion, accessible from all nodes of the job.

BeeOND/BeeGFS can be used like any other parallel file system. Tools like cp or rsync can be used to copy data in and out.

Using Slurm to request BeeOND file system creation¶

Creation and deletion of a BeeOND file system is integrated into the prolog and epilog script of the batch system Slurm. You can request it with the batch option --constraint <flag> where <flag> is BEEOND, BEEOND_4MDS or BEEOND_MAXMDS. In addition, it can only be used if compute nodes are exclusively used, i.e the batch option --exclusive is required. Meaning of the available <flag> options:

• BEEOND: One metadata server is started on the first node. This is typically the normal use case.
• BEEOND_4MDS: 4 metadata servers are started within your job. If you have less than 4 nodes, less metadata servers are started.
• BEEOND_MAXMDS: On every node of your job a metadata server is started.

After your job has started, you can find the private on-demand file system on the /mnt/odfs/$SLURM_JOB_ID directory. Below this mount point there are five pre-configured directories:

## for small files (stripe count = 1)
$ /mnt/odfs/$SLURM_JOB_ID/stripe_1
## for medium sized files (stripe count = 4)
$ /mnt/odfs/$SLURM_JOB_ID/stripe_default or /mnt/odfs/$SLURM_JOB_ID/stripe_4
## for large files or when using MPI-IO (stripe count = 8, 16 or 32)
$ /mnt/odfs/$SLURM_JOB_ID/stripe_8, /mnt/odfs/$SLURM_JOB_ID/stripe_16 or /mnt/odfs/$SLURM_JOB_ID/stripe_32

If you request less nodes than stripe count, the stripe count will be set to the number of nodes, e.g. if you request 8 nodes, the directory with stripe count 16 will be only used with a stripe count 8.

Note that you should always use the directory with the greatest stripe count for very large files, otherwise the individual SSDs which are used as stripe will fill up. E.g. if your largest file is 5.1 TB, then you have to use at least a stripe count of 4 (4 x 1500 GB).

The capacity of the private file system depends on the number of nodes. For each node you will get 1500 GB. If you request 20 nodes for your job, your private file system has a capacity of 30 TB (20 * 1500 GB).

Example job script:

#!/bin/bash
# very simple example on how to use BeeOND
#SBATCH -N 10
#SBATCH --constraint=BEEOND
#SBATCH --exclusive

# create a workspace
ws_allocate myresults-$SLURM_JOB_ID 30
RESULTDIR=$(ws_find myresults-$SLURM_JOB_ID)

# set ENV variable to on-demand file system
ODFSDIR=/mnt/odfs/$SLURM_JOB_ID/stripe_default/

# start application and write results to on-demand file system
mpirun -nolocal myapplication -o $ODFSDIR/results

# copy back data after your application has completed
rsync -av ${ODFSDIR}/results/ ${RESULTDIR}/

In case you want to transfer many input file to BeeOND, see Usage example below $TMPDIR which explains how this can be done efficiently.