Containers

Apptainer, Charliecloud, and Enroot are available as modules on the supercomputer, any of which can be used to build and run containers. You can also install any container runtime that can be built without root privileges. Note that this precludes the installation of container runtimes that use setuid binaries like standard Docker.

Using Apptainer

We recommend Apptainer for most users due to its ubiquity and versatility. A full tutorial on using Apptainer is far beyond the scope of this page, but below is an example of a common use case on the supercomputer: you want to download a container, modify it, share it with your colleagues, and have them run it on some of their data.

Downloading a Container

You'll usually be able to find a container that's at least close to what you want on Docker Hub. In this tutorial, we'll use the Alpine container:

module load apptainer
apptainer build --sandbox /tmp/alpine docker://alpine

If you don't need to modify the container, you can download directly to an image file and skip the next step:

module load apptainer
apptainer pull alpine.sif docker://alpine

Tweaking the Container

The container is now stored at /tmp/alpine, which you can navigate to and peruse the root filesystem. To get a shell there for any modifications you'd like to make, use apptainer shell:

apptainer shell --writable --cleanenv /tmp/alpine # to enter the container
apk add --update --no-cache python3 # within the container; replace with your modifications

You may need --fakeroot if you need to do root things in the container, for example using some package managers.

You can also use a definition file instead of downloading the image and getting an interactive shell. You won't be able to solve problems interactively that way, but it gives you a reproducible method of creating the container.

Once you've made any tweaks, you can pack up the container for use and distribution:

apptainer build alpine-with-python.sif /tmp/alpine

You can move it to a shared group directory or transfer it to a colleague if you'd like to distribute a custom container.

Using the container

Whether you want to run a shell or a script within the container, you'll probably want to start with a fresh environment and mount your storage within the container:

apptainer shell --cleanenv --bind /grphome,/nobackup alpine-with-python.sif # interactive shell
apptainer exec  --cleanenv --bind /grphome,/nobackup alpine-with-python.sif /script/within/the/container.py # non-interactive

When running in a container, remember that you're using a different operating system--only the kernel is shared between the host and container operating systems. As such, in most cases you shouldn't use binaries from outside of the container within the container--sometimes they'll work correctly, sometimes they'll fail, and sometimes they'll silently produce corrupt results.

Slurm Integration

Slurm is configured to allow the easy use of containers within a job--just add --container C to your job submission arguments, where C is a directory with a valid Linux filesystem, a SquashFS file containing such a directory, or a Singularity image file. The job will then be run within the container, bind-mounting your home, group, scratch, and archive directories. As an example, you can run cat /etc/os-release in a container (alpine in this example) that's packaged in alpine.sqfs with:

sbatch --time 1 --mem 1G --container alpine.sqfs --wrap "cat /etc/os-release"

You can also run interactive jobs within containers:

salloc -t 60 --mem 4G --container ubuntu.sif

If the shell that you use (check with echo $SHELL) isn't available in the container, make sure to specify a shell that is when launching an interactive job:

SHELL=/bin/sh salloc -t 10 --mem 256M --container images/alpine/

If you want to customize the behavior of Slurm's container integration, you can create an executable script at ~/.slurm-oci-run, which will be used to run the container. There are example scripts at /apps/slurm-oci/examples, which you can use as a basis for your run script if you wish. The script will parse the command line arguments, which are ordered thus:

1. Container path: the C in --container C; it can be in any format supported by the container environment as long as it's on central storage
2. Root FS: in all cases we've observed, this is identical to the container path
3. Spool directory: the directory containing the sbatch run script, which resides in /var/spool; you'll need to mount it in the container to access the batch script
4. User name: your username
5. User ID: your user ID (the result of id -u)
6. Node name: the host name of the compute node the job is running on
7. Job ID: the compute job's ID
8. Step ID: the job's step ID
9. Task ID: the job's task ID
10. Subsequent arguments: the command to run in the container

...then launch the container, running the command specified by arguments 10 on.