Globus is ideal if moving large data sets, such as millions of files, or very large files. Globus file transfer jobs can be initiated from either a cloud-based GUI or the command line. Globus supports moving files between various supercomputing facilities, national labs, and personal computers. Globus also supports sharing datasets with individuals inside and outside of BYU.
Rclone is our recommended tool for transferring files between the supercomputer and cloud storage.
SCP / SFTP
scp is useful when you know where your files are and simply want to copy them to/from the supercomputer. To do so, open a terminal/command prompt and enter something like (replacing
username with your username, of course):
# From supercomputer to local computer: scp email@example.com:/fslhome/username/myfile.txt Documents/ # From local computer to supercomputer: scp Documents/myfile.txt firstname.lastname@example.org:/fslhome/username/
Linux and macOS computers will likely have scp out of the box; Windows users can download PuTTY and use
pscp in the place of
scp, or use
scp within the Windows Subsystem for Linux.
The interface for
sftp is similar to FTP programs, but the data is transferred encrypted over ssh. See the manual for information about various commands.
SCP / SFTP Graphical Interfaces
Most programs that are specifically designed for transferring files between local and remote storage work somewhat like Norton Commander, with one column for local storage and one column for remote storage to facilitate moving files between them. FileZilla works on all the major operating systems; to get it to behave more nicely, follow this guide (replacing addresses and usernames appropriately). If you are using Windows, WinSCP is a bit more intuitive and easy to set up than FileZilla. Cyberduck is also commonly used, but SSHFS or SFTP Net Drive in combination with your file manager achieves the same effect.
If you want to browse files on the supercomputer using your file manager, you can use SSHFS for Linux or SFTP Net Drive for Windows; ExpanDrive and Mountain Duck work on both Windows and MacOS (ExpanDrive also works on Linux), but are paid. This allows you to transparently interact with the files there as if they were on your local machine--you could open a remote text file with Notepad, or save the csv file you're working on in Excel directly to the supercomputer.
Most graphical file browsers on Linux allow you to navigate to paths like
sftp://email@example.com/fslhome/username. Sometimes you may use
fish: (KDE option, but
sftp: works) instead of
sftp:, but that's an uncommon need. You may need to type
L to open up the URL bar where you can type the path.
sshfs allows you to mount supercomputer storage locally from the command line. Install sshfs with your package manager if it isn't included with your distro by default. To mount the supercomputer storage, open a terminal, choose an empty directory (called
sc for the sake of this example), and use the following:
sshfs -o follow_symlinks firstname.lastname@example.org:/fslhome/username sc
After entering your password and verification code, you should be able to navigate to
sc in Finder/Nautilus/Dolphin/Thunar/etc. and view files on the supercomputer.
To unmount, use
fusermount -u sc.
Windows users can use SFTP Net Drive, which is free for academic use, to mount supercomputer storage as a Windows drive. Download and install it, open it, then fill out the needed information:
- Profile: create a new one
- Server: ssh.rc.byu.edu
- Username: your username
- Authentication: Keyboard-interactive
- Drive Letter: the default ("Last Available") works
Upon pressing "Connect" you will be asked for your password and verification code, and File Explorer will open to your supercomputer storage.
As of the writing of this article, one can't use
sshfs within the Windows Subsystem for Linux and SSHFS-Win still doesn't work with two-factor authentication. ExpanDrive and Mountain Duck both work but are not free.
Unfortunately, SSHFS has problems on MacOS even though it is technically available. From what we've found, ExpanDrive and Mountain Duck (both paid) are the only viable options.
Last changed on Thu Jul 7 17:30:13 2022