Parallel File System Best Practices¶

Parallel file systems provide high aggregate I/O bandwidth to many compute nodes simultaneously. However, application performance depends strongly on the I/O access pattern.

The following recommendations can help improve data throughput and metadata performance. They apply primarily to the IBM Storage Scale (GPFS) file systems used for $HOME, $PROJECT, and workspaces. Some recommendations also apply to LSDF and BeeOND.

Choosing the Appropriate File System¶

Select the storage system according to the lifetime, access pattern, and locality of the data:

• use $HOME or $PROJECT for persistent data such as configuration files, software, and important or final results

• use workspaces for data that must remain available across multiple jobs

• use the node-local $TMPDIR for temporary data accessed by a single compute node, particularly for access patterns that are unfavorable for shared parallel file systems, such as large numbers of small files, frequent metadata operations, small or random I/O operations, or repeated access to the same data

• consider BeeOND for temporary data that must be accessed by multiple compute nodes and exhibits similar unfavorable access patterns. BeeOND can also be useful when the data does not fit on the local SSD of a single node or when copying the same dataset to $TMPDIR on every node would be inefficient

Improving Throughput Performance¶

Parallel file systems generally perform best with large, contiguous I/O operations. To improve throughput:

• aggregate small reads and writes into larger I/O operations
• access large files sequentially whenever possible
• distribute I/O across several compute nodes when high aggregate bandwidth is required
• avoid routing all I/O through a single process if several processes can perform I/O in parallel
• avoid many processes repeatedly modifying the same small or overlapping regions of a shared file

Predictable, contiguous access allows GPFS to make better use of mechanisms such as prefetching and write-behind and is therefore generally preferable to highly irregular or random I/O.

File system bandwidth is aggregate bandwidth. A single compute node is limited by its network connection and other per-client resources. Applications requiring high throughput should therefore perform I/O from several nodes in parallel.

Shared Files and File-per-Process I/O¶

Both shared-file and file-per-process approaches can perform well, but both can become inefficient at large scale.

Creating one file per process is simple, but applications using thousands of processes can create excessive metadata load. Conversely, many processes performing small, uncoordinated writes to one shared file can cause contention.

For large parallel datasets, consider using parallel I/O interfaces and libraries such as MPI-IO, parallel HDF5, PnetCDF, or ADIOS2. These libraries can coordinate and aggregate accesses so that the file system receives fewer and larger requests.

See Parallel I/O for details about the available I/O models and libraries.

Avoiding Unnecessary Synchronous I/O¶

Explicit synchronization can prevent the file system from buffering and aggregating writes. Therefore:

• avoid calling fsync() after every small write unless the application requires this durability guarantee

• keep files open while they are actively being used instead of repeatedly opening and closing them

• buffer frequently generated small records and write them in larger groups

Direct I/O using O_DIRECT is not automatically faster. Normal buffered I/O allows GPFS to use caching, prefetching, and write-behind. Use direct I/O only when it benefits the application's access pattern and has been verified by benchmarking.

IBM Storage Scale (GPFS) Data Distribution¶

GPFS automatically distributes file data across the underlying storage infrastructure. Users therefore do not need to configure file striping manually.

This differs from file systems such as BeeGFS or Lustre, where users may need to consider the stripe count of a file to determine how many storage targets are used.

Improving Metadata Performance¶

Metadata operations include creating, deleting, opening, closing, renaming, and querying files and directories. Large numbers of such operations can become a bottleneck even when only small amounts of data are transferred.

To improve metadata performance:

• avoid creating very large numbers of files, particularly when the files are short-lived or contain only small amounts of data

• avoid unnecessary file creation, deletion, opening, and closing

• avoid many processes concurrently creating or deleting files in the same directory; where appropriate, distribute files across separate subdirectories, for example per process or compute node

• use node-local or job-local temporary storage for temporary workloads involving many files or frequent metadata operations

Commands that recursively traverse large directory trees, such as find, du, ls -l, or recursive permission changes, can generate substantial metadata traffic and should not be run unnecessarily on directories containing very large numbers of files.

BeeOND¶

BeeOND is based on BeeGFS and uses a different data-placement mechanism from GPFS

BeeGFS files have a stripe count that determines how many storage targets are used. BeeOND therefore provides directories with different predefined stripe counts.

Use a higher stripe count for large files that require high throughput. Small files generally do not benefit from being distributed over many storage targets.

See the BeeOND documentation for the available stripe directories and recommendations.

Checkpoint and Temporary I/O¶

Checkpointing can generate short periods of very high file system load. Where possible:

• write a small number of large checkpoint files instead of many small files
• use parallel I/O instead of collecting all checkpoint data on one process
• avoid unnecessary synchronous writes
• consider asynchronous I/O if computation can safely overlap checkpoint output
• remove obsolete checkpoints when they are no longer needed

If checkpoint or intermediate data is required only during the current job, consider whether $TMPDIR or BeeOND is more appropriate than persistent storage.

Identifying I/O Bottlenecks with JobMon¶

JobMon provides file system performance metrics for your batch jobs without requiring application instrumentation. It is a useful first step for determining whether I/O may be affecting application performance before using more detailed profiling tools.

For the GPFS-based file systems like $HOME and Workspace, JobMon provides time series for:

• read and write throughput
• read and write operations per second
• metadata operations per second

When investigating an application, compare these metrics with the CPU and GPU utilization over the same time period. I/O-bound phases often appear as increased file system activity accompanied by reduced compute utilization.

Different patterns can indicate different optimization opportunities:

• high throughput with relatively few I/O operations usually indicates large I/O requests and is generally desirable
• many I/O operations with comparatively low throughput can indicate small reads or writes that may benefit from aggregation
• high metadata operation rates can indicate workloads creating, opening, closing, or deleting large numbers of files
• I/O activity concentrated on a single node can indicate that all application I/O is being routed through one process or node
• periodic I/O bursts with reduced compute utilization can indicate synchronous checkpointing that might benefit from parallel or asynchronous I/O

For multi-node jobs, inspect both aggregated values and individual nodes. This can help distinguish balanced parallel I/O from workloads in which only a small number of nodes perform the file system access.

JobMon provides compute-node-level file system activity but does not identify individual files, I/O calls, or source-code locations. Use application-level profiling or dedicated I/O tracing tools when more detailed information is required.

See JobMon - Job Performance Metrics for further information.

Summary¶

For good parallel file system performance:

• use the storage system appropriate for the data lifetime and access pattern
• perform large, contiguous I/O operations instead of many small operations
• use several clients to achieve high aggregate throughput
• use parallel I/O libraries for large shared datasets
• avoid unnecessary synchronization and contended access to shared file regions
• avoid creating large numbers of small files on global parallel file systems
• use $TMPDIR for temporary node-local workloads with unfavorable parallel file system access patterns; consider BeeOND when the data must be shared across multiple nodes