$TMPDIR¶
$TMPDIR is a node-local temporary directory on the fast NVMe SSD of each compute node, intended for temporary files accessed only during job runtime from a single node. The environment variable $TMPDIR is set automatically by the batch system at job start and points to this directory. Note that different tasks of a parallel application use different directories when they do not share the same node — although $TMPDIR points to the same path name on different nodes of a batch job, the physical location and content differ per node.
This directory should be used for temporary files being accessed from the local node during job runtime. It should also be used if you read the same data many times from a single node, e.g. if you are doing AI training. In this case you should copy the data at the beginning of your batch job to $TMPDIR and read the data from there.
The $TMPDIR directory is located on an extremely fast NVMe SSD disks. Performance on small files is much better than on the parallel file systems. The size of $TMPDIR depends on the node type, for details see Compute Hardware.
Each time a batch job is started, a subdirectory is created on the SSD of each node and assigned to the job. $TMPDIR is set to the name of this subdirectory, i.e. it is unique for each job. At the end of the job the subdirectory is removed.
On login nodes, $TMPDIR also points to a fast directory on a local NVMe SSD disk but this directory is not unique. It is recommended to create your own unique subdirectory on these nodes. This directory should be used for the installation of software packages — unpack, compile, and link in a subdirectory of $TMPDIR, then install (e.g. make install) into $HOME or $PROJECT.
Attention
Local storage space is currently not managed as a resource via SLURM. Therefore, if several users or jobs want to use the local hard disk on the same node, the desired storage space may not be sufficient. If you want to work with large amounts of data on $TMPDIR which is close to the capacity of the local SSDs, you should allocate the compute node exclusively with the --exclusive flag.
Do not use /tmp or /scratch
Do not use /tmp or /scratch — use $TMPDIR instead. An automatic cleanup on /tmp or /scratch is not possible because another job could still be using data below these directories, which can cause issues for you and other users. $TMPDIR is created when the job starts and removed when the job completes, i.e. cleanup is automatic.
Usage example¶
If you have a dataset with many files which is frequently used by batch jobs, create a compressed archive on a workspace. This archive can be extracted on $TMPDIR inside your batch jobs. Such an archive can be read efficiently from a parallel file system since it is a single large file.
On a login node, create such an archive:
# Create a workspace to store the archive
ws_allocate data-ssd 60
# Create the archive from a local dataset folder (example)
tar -cvzf $(ws_find data-ssd)/dataset.tgz dataset/
Inside a batch job, extract the archive on $TMPDIR, read input data from $TMPDIR, store results on $TMPDIR, and save the results on a workspace:
#!/bin/bash
# very simple example on how to use local $TMPDIR
#SBATCH -N 1
#SBATCH -t 24:00:00
# Extract compressed input dataset on local SSD
tar -C $TMPDIR/ -xvzf $(ws_find data-ssd)/dataset.tgz
# The application reads data from dataset on $TMPDIR and writes results to $TMPDIR
myapp -input $TMPDIR/dataset/myinput.csv -outputdir $TMPDIR/results
# Before job completes save results on a workspace
rsync -av $TMPDIR/results $(ws_find data-ssd)/results-${SLURM_JOB_ID}/