Transferring Files

There are several ways to move files between your machine, cloud storage, and the supercomputer. The following is provided as a guide on how to do so, but note that we only support use of the programs that we provide or that are centrally installed on the supercomputer (namely Globus, rclone, scp, and sftp). Support for other software listed here is outside of our purview--refer to said software's documentation with questions.

Backups

We don't back up scratch directories, but you can do so if you want. Using Kopia to back up to Box regularly via cron is our recommended backup workflow.

Globus

Globus is ideal if moving large data sets, such as millions of files, or very large files. Globus file transfer jobs can be initiated from either a cloud-based GUI or the command line. Globus supports moving files between various supercomputing facilities, national labs, and personal computers. Globus also supports sharing datasets with individuals inside and outside of BYU.

Rclone

Rclone is our recommended tool for transferring files between the supercomputer and cloud storage.

SCP / SFTP

scp is useful when you know where your files are and simply want to copy them to/from the supercomputer. To do so, open a terminal/command prompt and enter something like (replacing username with your username, of course):

# From supercomputer to local computer:
scp username@ssh.rc.byu.edu:/home/netid/myfile.txt Documents/
# From local computer to supercomputer:
scp Documents/myfile.txt username@ssh.rc.byu.edu:/home/netid/

Linux and macOS computers will likely have scp out of the box; Windows users can download PuTTY and use pscp in the place of scp, or use scp within the Windows Subsystem for Linux. rsync is a more powerful alternative to scp if it's available for your operating system.

The interface for sftp is similar to FTP programs, but the data is transferred encrypted over ssh. See the manual for information about various commands.

SCP / SFTP Graphical Interfaces

Most programs that are specifically designed for transferring files between local and remote storage work somewhat like Norton Commander, with one column for local storage and one column for remote storage to facilitate moving files between them. FileZilla works on all the major operating systems; to get it to behave more nicely, follow this guide (replacing addresses and usernames appropriately). If you are using Windows, WinSCP is a bit more intuitive and easy to set up than FileZilla. Cyberduck is also commonly used, but SSHFS or SFTP Net Drive in combination with your file manager achieves the same effect.

Local Mounting

If you want to browse files on the supercomputer using your file manager, you can use SSHFS for Linux or SFTP Net Drive for Windows; ExpanDrive and Mountain Duck work on both Windows and MacOS (ExpanDrive also works on Linux), but are paid. This allows you to transparently interact with the files there as if they were on your local machine--you could open a remote text file with Notepad, or save the csv file you're working on in Excel directly to the supercomputer.

Linux

Most graphical file browsers on Linux allow you to navigate to paths like sftp://username@ssh.rc.byu.edu/home/netid. Sometimes you may use ssh: or fish: (KDE option, but sftp: works) instead of sftp:, but that's an uncommon need. You may need to type Ctrl+L to open up the URL bar where you can type the path.

sshfs allows you to mount supercomputer storage locally from the command line. Install sshfs with your package manager if it isn't included with your distro by default. To mount the supercomputer storage, open a terminal, choose an empty directory (called sc for the sake of this example), and use the following:

sshfs -o follow_symlinks username@ssh.rc.byu.edu:/home/netid sc

After entering your password and verification code, you should be able to navigate to sc in Finder/Nautilus/Dolphin/Thunar/etc. and view files on the supercomputer.

To unmount, use fusermount -u sc.

Windows

Windows users can use SFTP Net Drive, which is free for academic use, to mount supercomputer storage as a Windows drive. Download and install it, open it, then fill out the needed information:

• Profile: create a new one
• Server: ssh.rc.byu.edu
• Username: your username
• Authentication: Keyboard-interactive
• Drive Letter: the default ("Last Available") works

Upon pressing "Connect" you will be asked for your password and verification code, and File Explorer will open to your supercomputer storage.

As of the writing of this article, one can't use sshfs within the Windows Subsystem for Linux and SSHFS-Win still doesn't work with two-factor authentication. ExpanDrive and Mountain Duck both work but are not free.

MacOS

Unfortunately, SSHFS has problems on MacOS even though it is technically available. From what we've found, ExpanDrive and Mountain Duck (both paid) are the only viable options.