Parallel I/O¶

Large-scale scientific applications often need to efficiently store and retrieve large amounts of data. Common I/O workloads include simulation input and output, checkpoint/restart, data analysis, and producer-consumer workflows.

For simulations that run for many hours or days, checkpointing is essential to avoid losing substantial computing time in the event of a system or application failure.

Data is typically stored in files, which are streams of bytes presented by the file system as named objects. The main challenge is achieving high performance and scalability while maintaining data consistency and integrity.

I/O stack¶

The I/O stack is composed of two fundamental parts: the software stack and the hardware stack.

The hardware stack includes the storage nodes, the storage network, I/O nodes, and the I/O devices where data is physically stored.

The software stack includes the parallel file system and operating-system interfaces that abstract hardware complexities and provide file-access interfaces such as POSIX. Above this layer, I/O middleware such as MPI-IO coordinates file access from multiple processes. High-level libraries built on top of POSIX I/O, MPI-IO, or other transport mechanisms provide a more user-friendly way to store structured scientific data.

Scientific applications can perform I/O at different levels of the software stack, but higher-level interfaces often provide better portability, data organization, and opportunities for coordinated parallel access.

POSIX I/O¶

POSIX, or Portable Operating System Interface, is a family of standards that defines operating-system interfaces and semantics used by applications. For file I/O, this includes operations such as open(), read(), write(), lseek(), and close().

POSIX does not define how an operating system or file system accesses the underlying storage hardware.

Parallel file systems such as Lustre and IBM Storage Scale (GPFS) provide POSIX-compatible file access while distributing data and metadata across multiple storage servers. POSIX compatibility improves source-level portability and provides well-defined file-access semantics.

However, strict consistency requirements, metadata operations, locking, and poorly coordinated access patterns can introduce overhead on large parallel file systems.

Direct POSIX I/O can perform well when applications issue large, contiguous, and well-aligned requests. However, naive file-per-process approaches, large numbers of small files, and many small or uncoordinated accesses often scale poorly.

For structured parallel I/O, middleware such as MPI-IO or high-level parallel I/O libraries should therefore be considered.

MPI-IO¶

MPI-IO is the parallel I/O interface defined by the MPI standard. It allows multiple MPI processes to access a shared file using independent or collective operations.

MPI-IO supports:

• explicit file offsets
• individual and shared file pointers
• non-contiguous memory and file layouts
• MPI derived datatypes
• file views
• independent and collective I/O
• nonblocking I/O
• configurable consistency and atomicity semantics
• performance hints through MPI_Info

MPI-IO can be used directly by applications, but it is also commonly used underneath high-level parallel I/O libraries such as parallel HDF5 and PnetCDF.

I/O models¶

Scientific data often has a global logical structure, such as a multidimensional grid or mesh, while the corresponding data is distributed across MPI processes.

MPI-IO file views and derived datatypes allow each process to describe the part of the global file that corresponds to its local data. This can avoid explicit application-side rearrangement for many access patterns.

Depending on the use case, different file-access models can be used:

1. File per process
2. Shared file
3. Single I/O process
4. Independent I/O
5. Collective I/O

File per process¶

The file per process approach is straightforward to implement, as each process writes its own data to a separate file. No coordination between processes is required during output, making it a relatively simple implementation.

For input, processes can read the files independently. This is simplest when the same process decomposition is used for reading and writing.

This method can achieve high aggregate throughput for suitable workloads. However, the large number of files generated can put significant stress on the file system due to the increased number of metadata operations. Managing and transferring the resulting datasets can also become difficult when thousands or millions of files are involved.

A further disadvantage is that the persistent data layout can become coupled to the MPI decomposition. For example, restarting a simulation with a different number of processes may require redistribution of data from files that were created according to the original process layout.

Shared file¶

In contrast to the file per process method, a shared-file approach stores data from multiple processes in a common file.

During output, multiple processes can contribute their local data to different regions of the file. During input, processes can read the regions corresponding to their current decomposition. The read decomposition does not necessarily have to match the decomposition that was used when the file was written.

MPI-IO and high-level parallel I/O libraries provide interfaces for implementing this access model.

A shared-file approach does not require a parallel file system for correctness, but a parallel file system is normally required to obtain scalable I/O performance across many compute nodes.

Single I/O process (Centralized I/O)¶

With centralized I/O, one designated process performs file operations on behalf of the other processes.

For output, the designated process writes its own data as well as data received from other processes. For input, it reads the required data from the file and distributes it to the processes that need it.

This model is simple and can be appropriate for small datasets, metadata, or legacy file formats that do not support parallel access. However, it generally does not scale well for large HPC workloads.

The I/O bandwidth is limited by a single process or compute node, while additional communication is required to gather data before writing or distribute data after reading. The designated I/O process may also require substantial additional memory for buffering data exchanged with the other processes.

Independent I/O¶

With independent I/O, each process issues its file operations independently of the other processes.

For output, processes typically write to the file regions corresponding to their local data. For input, each process independently reads the regions it requires.

If processes access disjoint regions, application-level synchronization is normally not required for those accesses. However, fragmented requests, poorly aligned accesses, overlapping regions, or file-system lock granularity can lead to contention and lower performance.

Independent I/O can be effective when each process performs sufficiently large and contiguous accesses. For highly fragmented distributed access patterns, collective I/O may allow the MPI-IO implementation or high-level I/O library to generate more efficient storage requests.

High-level I/O libraries such as parallel HDF5 and PnetCDF provide both independent and collective access modes. ADIOS2 manages the physical access strategy according to the selected engine and its configuration.

Collective I/O¶

With collective I/O, all participating MPI processes call the I/O operation collectively.

For both reads and writes, the MPI-IO implementation can reorganize and aggregate requests so that fragmented application accesses are transformed into fewer, larger, and better-aligned file-system operations.

This optimization often uses selected processes, called I/O aggregators. During writes, aggregators collect data from other processes before performing larger file operations. During reads, aggregators can read larger regions of the file and redistribute the requested portions to the corresponding processes.

This strategy is commonly known as two-phase I/O.

Collective I/O therefore introduces communication between MPI processes in exchange for potentially more efficient access to the storage system. Whether it is faster than independent I/O depends on the access pattern, data layout, MPI implementation, and storage system.

High-level libraries can provide collective operations without requiring applications to implement the aggregation and redistribution explicitly.

Choosing an I/O strategy¶

The appropriate I/O strategy depends on the amount of data, the process count, the data model, and how the data is written and later read.

As a general starting point:

• Centralized I/O using a single designated I/O process can be sufficient for small datasets, metadata, or legacy formats that do not support parallel access.

• File-per-process I/O can be simple and effective at moderate scale, especially when processes can read and write their corresponding files independently. However, it may become problematic when it creates very large numbers of files or when the process decomposition changes between writing and reading.

• Independent I/O is suitable when processes access sufficiently large, mostly disjoint file regions and do not benefit significantly from collective coordination.

• Collective I/O is often a good starting point when many MPI processes access a shared structured dataset. It allows the I/O implementation to aggregate and reorganize distributed read or write requests.

• For large structured distributed datasets, prefer MPI-IO or a high-level parallel I/O library over implementing shared-file access manually with POSIX.

• Use high-level libraries when portable metadata, self-describing datasets, postprocessing compatibility, compression, or streaming capabilities are required.

When designing persistent data formats, consider both output and subsequent input. Restart or analysis may use a different number of MPI processes, a different domain decomposition, or different subsets of the data than the original simulation.

The persistent file layout should therefore not be unnecessarily tied to a particular MPI decomposition.

For filesystem-level considerations such as metadata scalability, temporary storage, and storage-system selection, see File system best practices.

MPI-IO example¶

The following minimal example writes one equally sized contiguous part of a distributed array per MPI process into a shared file:

MPI_File fh;

MPI_File_open(MPI_COMM_WORLD,
              "field.dat",
              MPI_MODE_CREATE | MPI_MODE_WRONLY,
              MPI_INFO_NULL,
              &fh);

MPI_File_set_size(fh, 0);

MPI_Offset offset =
    (MPI_Offset)rank * local_size * sizeof(float);

MPI_File_write_at_all(fh,
                      offset,
                      data,
                      local_size,
                      MPI_FLOAT,
                      MPI_STATUS_IGNORE);

MPI_File_close(&fh);

MPI_File_write_at_all() is a collective operation. Each process provides the file offset corresponding to its portion of the global dataset, while the MPI-IO implementation can coordinate and aggregate the underlying I/O requests.

More complex distributed layouts can be described using MPI derived datatypes and MPI_File_set_view().

Recommended resources for more details about MPI-IO:

1. High Performance Parallel I/O by Prabhat and Quincey Koziol
2. MPI 5.0 Standard, Chapter 15: I/O

High-level parallel I/O libraries¶

High-level I/O libraries such as NetCDF, HDF5, and ADIOS2 provide a more user-friendly way of working with structured data in scientific applications.

These libraries enable the representation of multidimensional and typed data together with metadata such as names and attributes. They also simplify visualization and analysis and provide portable file formats that can be read on different systems.

The available versions depend on the software environment of the cluster. Use the module system to inspect the currently available installations.

HDF5¶

Hierarchical Data Format (HDF5) is a widely used library for developing large-scale scientific codes.

It provides a portable and self-describing file format in which data can be organized into groups and datasets, with attributes and links used to describe and connect objects. Groups and datasets are conceptually similar to directories and files in a Unix-like file system.

For more details about HDF5 please refer to the documentation.

Parallel HDF5 supports parallel access to an HDF5 file through MPI. The MPI-enabled version of HDF5 uses MPI-IO underneath its parallel file driver.

Serial and MPI-enabled HDF5 installations may both be available. The currently available versions can be queried through the module system, for example:

$ module avail HDF5

With a serial HDF5 build, file-per-process and centralized I/O approaches can be used. An MPI-enabled HDF5 build allows multiple MPI processes to access a shared HDF5 file.

HDF5 is a feature-rich library but has a relatively steep learning curve.

APIs and language bindings are available for several programming languages, including C, C++, Fortran, Java, and third-party environments such as Python. Parallel HDF5 requires an MPI-enabled HDF5 build and, for language bindings, corresponding MPI support in the binding or application environment.

When possible, use higher-level wrappers or abstractions that match the data model of the application, as directly programming against the low-level HDF5 C API can be complex and error-prone.

NetCDF¶

NetCDF, short for Network Common Data Form, is widely used in climatology, meteorology, and oceanography applications.

It provides portable, self-describing formats and APIs for storing multidimensional scientific data.

The classic netCDF data model consists primarily of dimensions, variables, and attributes. NetCDF-4 extends this model with features such as hierarchical groups and user-defined types and typically uses HDF5 as its storage layer.

PnetCDF is a separate parallel I/O library for MPI applications that provides parallel access to the classic netCDF CDF-1, CDF-2, and CDF-5 file formats.

PnetCDF is implemented on top of MPI-IO and is particularly suitable for applications that use large distributed arrays and the classic netCDF data model.

Available NetCDF and PnetCDF installations can be queried through the module system, for example:

$ module avail netCDF
$ module avail PnetCDF

NetCDF and PnetCDF provide bindings for several programming languages, most commonly C and Fortran, with additional interfaces available in languages such as C++ and Python.

Parallel I/O support depends on the library, file format, and MPI-enabled software stack used by the application.

ADIOS2¶

ADIOS2 is a framework for high-performance data management that is used in fields such as weather forecasting, molecular dynamics simulations, and computational fluid dynamics.

It provides self-describing data formats and supports efficient file-based storage as well as in situ¹ and in transit² visualization and analysis workflows.

ADIOS2 engine¶

A key component of ADIOS2 is the engine, which abstracts how data is transported or stored.

Engines can broadly be categorized into file-based and streaming engines, while applications use a largely common I/O API. I/O operations are often organized into steps, which can correspond to simulation time steps or iterations.

One of the key features of ADIOS2 is the ability to configure or change engines without recompiling the application, for example through an XML configuration file, provided the selected engine is available in the installed ADIOS2 build and is not overridden by the application.

This makes it easier to evaluate different data transports without extensive code modifications.

For persistent simulation output and checkpointing, BP engines are commonly used. Streaming engines such as SST support producer-consumer workflows in which data can be consumed while the simulation is still running.

Other engines are available for more specialized workflows; refer to the ADIOS2 documentation for the complete list.

Available ADIOS2 installations can be queried through the module system, for example:

$ module avail ADIOS2

ADIOS2 capabilities of particular interest for HPC include:

• flexible file-based and streaming engines
• support for in situ and in transit data workflows
• compression operators, including lossy and lossless compression methods depending on the installed build
• GPU-aware I/O for supported engines and GPU backends
• configurable I/O aggregation and substream/subfile strategies
• deferred data movement
• support for network-based streaming workflows
• portable self-describing persistent data formats

There is much more to know about ADIOS2; please refer to the documentation for additional details.

Choosing the right library¶

Choosing between parallel HDF5, PnetCDF, and ADIOS2 depends on the specific needs of the application and the requirements of the data storage and management workflow.

• Parallel HDF5 offers a rich hierarchical data model and a mature ecosystem, making it suitable for applications that require complex scientific data organization, broad tool compatibility, and features such as chunked datasets and attributes. Achieving good parallel performance may require careful choices of dataset layout, chunking, collective operations, and metadata access.

• PnetCDF provides a comparatively simple array-oriented data model and efficient MPI-IO access to classic netCDF formats. It is commonly used in climatology, meteorology, oceanography, and other applications that work naturally with large distributed arrays and do not require the full netCDF-4/HDF5 feature set.

• ADIOS2 is designed for high-performance data management at scale and supports both persistent file I/O and streaming workflows. Its engine abstraction, aggregation mechanisms, compression operators, and in situ/in transit capabilities make it particularly useful for applications that need flexible data movement or high-throughput simulation output.

When choosing between the three libraries, consider:

• the application's data model
• read and write access patterns
• required file-format compatibility
• analysis and visualization tools
• expected process count
• restart and postprocessing workflows
• whether compression is required
• whether streaming or in situ workflows are required

Library choice should not be based on a general assumption that one library is inherently faster than another. Performance depends strongly on the workload, configuration, MPI implementation, and storage system.

I/O optimization¶

Application-level parallel I/O performance depends primarily on selecting an appropriate I/O model and configuring the I/O library according to the application's data layout and access pattern.

General filesystem-level recommendations, including request sizes, metadata behavior, storage selection, caching, and filesystem benchmarking, are covered in File system best practices.

General application-level recommendations include:

1. I/O model: Select an I/O model that matches the application's access pattern and scale. Avoid unnecessary serialization and excessive coupling between the MPI decomposition and persistent file layout.
2. Collective I/O: Use collective I/O when coordination and aggregation can transform fragmented accesses into more efficient storage requests. Collective operations are not automatically faster for every workload and should be benchmarked.
3. Read and write access patterns: Design the persistent data layout for both output and subsequent input. A layout that is efficient to write may not be efficient to read if restart or analysis uses a different process count, domain decomposition, or subset of the data. Avoid unnecessarily encoding the original MPI decomposition into the persistent format.
4. Aggregation: Tune the number of MPI-IO aggregators, ADIOS2 aggregators/substreams, or equivalent library-specific parameters when available. The optimal configuration depends on the workload and storage system.
5. Compression: Compression can reduce the amount of data transferred to storage, but it consumes compute resources. Measure the trade-off using representative data.
6. Overlap I/O and computation: Nonblocking or asynchronous I/O interfaces may allow data movement to overlap with useful computation, while deferred interfaces can allow operations to be grouped and scheduled more efficiently. Neither mechanism by itself guarantees asynchronous progress to persistent storage; actual overlap depends on the I/O library, implementation, engine, and storage backend.

Tuning parallel HDF5¶

Some tuning tips specific to parallel HDF5:

1. Dataset layout and chunking: Choose contiguous or chunked storage according to the expected access pattern. For chunked datasets, choose chunk dimensions that align with common read/write selections and avoid unnecessary sharing of chunks between MPI ranks.
2. Collective data I/O: Use collective dataset transfers when the access pattern benefits from request aggregation. Independent I/O may still be preferable for some workloads.
3. Collective metadata operations: For metadata-intensive workloads, collective metadata operations can improve scalability by reducing redundant metadata access. Use these modes consistently across participating MPI processes and benchmark their effect for the application.
4. File-space management: File-space allocation and alignment choices can affect fragmentation and I/O behavior. Select these settings according to the expected dataset layout and storage system.
5. Avoid unnecessary fill-value initialization: Creating large datasets, especially chunked datasets, may trigger storage allocation or initialization. If fill values are not required, configure the dataset so that unnecessary initialization is avoided.
6. Avoid unnecessary datatype conversions: Datatype conversions add processing during I/O. Use native datatypes for memory buffers and avoid unnecessary conversions in performance-critical paths, while choosing file datatypes according to portability and data-format requirements.

Tuning PnetCDF¶

Some tuning tips specific to PnetCDF:

1. Collective operations: Prefer collective APIs for regular parallel access patterns when the additional coordination allows the library to generate larger and more efficient MPI-IO requests.
2. Nonblocking I/O and request aggregation: Use PnetCDF's nonblocking APIs to post multiple operations before completing them with a wait operation. This allows PnetCDF to combine small requests into larger MPI-IO operations.
3. I/O hints: PnetCDF and MPI-IO hints can influence variable alignment, collective buffering, aggregator selection, and other implementation-specific optimizations. Their effect depends on the MPI implementation, file system, and workload.
4. File and variable alignment: On some parallel file systems, aligning large variables or file regions with the underlying storage layout can improve performance.
5. Benchmark library-specific settings: Aggregator counts, MPI hints, and other low-level parameters should be tuned using representative workloads rather than fixed rules.

Tuning ADIOS2¶

ADIOS2 provides several mechanisms that can influence data movement and storage behavior. The engine and its parameters should be selected according to the application's access pattern and workflow.

1. Engine: Choose an engine that matches the workload, for example a BP engine for persistent checkpoint/output data or a streaming engine for producer-consumer workflows.
2. Aggregation: Tune engine-specific aggregation or substream/subfile parameters when supported. The optimal setting depends on the number of ranks, nodes, storage targets, and workload.
3. Deferred operations: Where appropriate, use deferred Put and Get operations so that ADIOS2 can schedule and combine data movement more effectively.
4. In situ analysis: Analyze data while it is still in memory when this avoids unnecessary persistent-storage traffic and matches the workflow requirements.
5. Compression operators: Compression can reduce the amount of data written or transferred, but the trade-off between compression cost and I/O savings should be measured with representative data.
6. GPU-aware I/O: For supported engines and GPU backends, pass device-resident buffers directly to ADIOS2 to avoid explicit application-side staging through host memory.