Data Transfer¶

Data transfer to and from NHR@KIT systems is performed via the login nodes. All common file transfer protocols are supported. For large data transfers, consider using rsync for better performance and resumability.

For an overview of available storage locations on the cluster, see Storage.

Long-running transfers

For long-running data transfers, use tmux to keep the session alive even if your connection drops.

rsync¶

rsync is a powerful tool for synchronizing files between two locations. Unlike scp, it only transfers files that have changed, making it ideal for recurring backups or large dataset updates.

Basic usage¶

Synchronize a local directory to the cluster:

$ rsync -avz /path/to/local/dir/ user@hk2-x86.scc.kit.edu:~/destination/

Synchronize from the cluster to your local machine:

$ rsync -avz user@hk2-x86.scc.kit.edu:~/source/ /path/to/local/destination/

Options¶

Option	Description
`-a`	Archive mode (preserves permissions, timestamps, symlinks)
`-v`	Verbose output
`-z`	Compress data during transfer
`-n`	Dry run (show what would be transferred without doing it)
`--delete`	Delete files in destination that no longer exist in source
`--exclude=PATTERN`	Exclude files matching a pattern
`--progress`	Show transfer progress

Examples¶

Dry run to preview what will be transferred:

$ rsync -avzn /path/to/local/dir/ user@hk2-x86.scc.kit.edu:~/destination/

Exclude certain file types:

$ rsync -avz --exclude='*.tmp' --exclude='*.log' /path/to/local/dir/ user@hk2-x86.scc.kit.edu:~/destination/

Resume an interrupted transfer:

$ rsync -avz --partial /path/to/local/dir/ user@hk2-x86.scc.kit.edu:~/destination/

Trailing slash matters

Note the trailing slash in source/. Without it, rsync copies the directory itself into the destination. With it, rsync copies the directory's contents.

scp¶

scp (secure copy) is a simple way to copy files between your local machine and the cluster. It uses SSH for data transfer and provides the same security as ssh.

Basic usage¶

Copy a file from your local machine to your home directory on the cluster:

$ scp /path/to/local/file.txt user@hk2-x86.scc.kit.edu:~/file.txt

Copy a file from the cluster to your local machine:

$ scp user@hk2-x86.scc.kit.edu:~/file.txt /path/to/local/destination/

Copy a directory recursively:

$ scp -r /path/to/local/directory user@hk2-x86.scc.kit.edu:~/destination/

Options¶

Option	Description
`-r`	Copy directories recursively
`-p`	Preserve file permissions and timestamps
`-C`	Enable compression during transfer
`-v`	Verbose mode (show progress)
`-P port`	Specify a different SSH port
`-i keyfile`	Use a specific SSH private key

Examples¶

Copy with compression (useful over slow network links):

$ scp -C largefile.dat user@hk2-x86.scc.kit.edu:/

Preserve permissions when copying a directory:

$ scp -rp /path/to/local/dir user@hk2-x86.scc.kit.edu:/

Further Solutions¶

These alternatives may be useful in specific scenarios.

sftp¶

sftp (SSH File Transfer Protocol) provides an interactive file transfer session similar to the classic ftp client, but with the security of SSH.

Connecting¶

$ sftp user@hk2-x86.scc.kit.edu

You will be prompted for your password or can use SSH keys for passwordless authentication.

Common commands¶

Command	Description
`help`	Show available commands
`ls`	List files in the current directory
`cd directory`	Change the remote directory
`pwd`	Print working directory (remote)
`lpwd`	Print working directory (local)
`get remoteFile [localFile]`	Download a file
`put localFile [remoteFile]`	Upload a file
`mget pattern`	Download multiple files matching a pattern
`mput pattern`	Upload multiple files matching a pattern
`mkdir directory`	Create a remote directory
`rm file`	Delete a remote file
`bye` / `quit`	Exit sftp

Examples¶

Download a file:

sftp> get data.csv

Upload a file:

sftp> put analysis.py

Download all files matching a pattern:

sftp> mget *.dat

sshfs¶

sshfs mounts a remote directory over SSH, allowing you to browse and edit files as if they were local:

$ sshfs user@hk2-x86.scc.kit.edu:/home/user ~/cluster

Unmount when done:

$ fusermount -u /mnt/cluster

Warning

sshfs is useful for interactive work but not recommended for high-performance or parallel I/O workloads. For large data transfers, use scp or rsync instead.

Graphical Tools¶

For a graphical interface on Windows, see the MobaXterm file transfer section of this documentation. Drag and drop should be avoided for large or many files.

As an alternative, files can be uploaded and downloaded via the JupyterLab file browser when you use the Jupyter service. However, this is not recommended for large or many files.