Backing Up to an External Server With Borg

We now recommend kopia for backups; this page is unmaintained.

borg is a deduplicating, compressing, encrypting backup program that can work locally or over SSH. You will need either a Linux server running SSHD or an account with a cloud provider that provides SSH access, such as rsync.net or BorgBase, in order to back up your compute data with borg. In this example, we will assume that you are backing up to a server called myserver.byu.edu, with username name.

Preparation

This tutorial assumes that you have passwordless SSH access from the supercomputer to the machine you're backing up to. Usually, you can set this up by logging into the supercomputer and running something like:

ssh-copy-id name@myserver.byu.edu

You'll also need to make sure that borg is installed on the machine that you will be backing up to; it is possible to back up remotely without borg on the destination machine, but makes life harder and won't be covered in this article.

Initializing a repository

First, initalize the repository. In this example we'll use a directory called ORCBackup in your home directory on the destination machine:

borg init --encryption=repokey name@myserver.byu.edu:~/ORCBackup

You will be asked for a password; choose something strong and unique.

You'll also want to make sure that there is extra space so that if you run out, recovery is possible:

borg config name@myserver.byu.edu:~/ORCBackup additional_free_space 4G

Backing up

Backing up is dead simple. For example, to back up your compute directory to an archive named my_compute:

borg create name@myserver.byu.edu:~/ORCBackup::my_compute ~/compute/

Automating backups

The following script would back up your compute directory to the my_compute archive and prune the backups, keeping 4 weekly and 6 monthly backups. Make sure to change BORG_PASSPHRASE:

#!/bin/bash -l

# Prepare for backup
module load borg
export BORG_PASSPHRASE='definitely use a better password than this'

# Back up
borg --error create name@myserver.byu.edu:~/ORCBackup::my_compute ~/compute/

# Prune
borg --error prune --keep-weekly 4 --keep-monthly 6 name@myserver.byu.edu:~/ORCBackup::my_compute

You can use BORG_PASSCOMMAND instead of BORG_PASSPHRASE if you don't want the password stored in the script.

To run this script (~/backup-borg.sh in this example) weekly, add an entry such as this to your crontab, replacing M with a minute (0-59), H with an hour (0-23), and W with a day of the week (0-6):

MAILTO=your_email@address.com
M H * * W bash -l ~/backup-borg.sh

This will send you an email if the job fails (or if any output is written).

Checking your backups

To check your backups, mount the repository:

borg mount name@myserver.byu.edu:~/ORCBackup /tmp/borg_mount

Navigate into /tmp/borg_mount, ensure that your files are as they should be (or restore anything that might have been lost), then unmount the backup:

borg umount /tmp/borg_mount

Restoring from backup

To restore from backup, create an empty directory, navigate to it, and use borg extract:

mkdir ~/compute/restore
cd ~/compute/restore
borg extract username@myserver.byu.edu:/home/username/backups/ORCBackup::my_compute

This will put all of the files from the backup into ~/compute/restore; if you only want some of the files, you can specify paths to the directories or files that you want, and/or use the --exclude option:

borg extract --exclude '*.o' username@myserver.byu.edu:/home/username/backups/ORCBackup::my_compute \
             /fslhome/$USER/compute/stuff_i_want_restored