Conda Environments

What are They?

At a high level, an environment is just a self-contained directory with its own packages. Environment managers typically handle the directory's location and its contents, so you don't need to worry about what goes where. You can create an environment for one workflow, another for a different workflow, and delete either without touching system software. sudo, or root permission, is not required. Reproducibility follows naturally: export an environment file, share it, and anyone can recreate the same setup on their machine or in their account.

Conda is our preferred package and environment manager. It is not limited to just Python packages. When conda needs to install new packages, it looks for them in channels. conda-forge is the only channel most of our users need. It is an open-source channel with broad platform coverage and frequent updates. Some of our bioinformatic and GPU users might need other channels discussed below.

Miniforge3 is the installer we use for conda itself. In the past, we've used Miniconda3, but it defaults to private channels that we are not licensed to use. Miniforge3 defaults to conda-forge.

How to Use

When conda installs packages, it makes sure that they all work well together. This often removes the dependency hell that pip can put you in. As conda gained popularity, mamba was written as an even faster solver. Since mamba is mostly a drop-in replacement for conda commands, we'll use it in this tutorial. If the mamba command fails to work, you can always fall back to conda.

To gain access to it, run:

module load miniforge3

If you want to always have these tools available at login, run module save to add it to your default collection. Though not always necessary, we recommend explicitly adding module load miniforge3 before any conda/mamba commands in job scripts.

To create a new environment and activate it, run:

mamba create -n my_new_env
mamba activate my_new_env

If you need a Python environment with scipy and numpy:

mamba install scipy numpy # does the installation
mamba list # shows you what's installed

If you need a specific version of numpy, you can look it up and install/update it like so:

mamba search numpy
mamba install numpy=2.3.0 # or whatever version

If you are unable to find your package, try searching different channels. These edge cases can often be found on bioconda or nvidia. You can also search all channels for a specific package on anaconda.org. To search on the command line:

mamba search blast -c bioconda # note: blast is installed on the supercomputer as a module

If you frequently find yourself adding -c channel_name to your command line, you can permanently search there after running:

conda config --add channels channel_name # note: mamba does not work here, so we fall back to conda

This will make your search/installation time longer since mamba now looks through both the channel_name and conda-forge channels for packages.

While not recommended, you can run pip install with an environment active.

For a full list of commands, run conda --help and mamba --help. Here are a few other noteworthy ones:

mamba uninstall package_name # removes a package from the environment
mamba update package_name # updates a package, --all updates everything
mamba env remove -n environment_name # deletes the entire environment
mamba env export > env.yml # exports the current environment to a yml file
mamba create -f env.yml # creates environment from yml file. Be sure to change the name on the first line of the yml file if desired

Group Environments

WARNING: By creating a group environment, every group member will have permission to alter the environment. Please make sure all group members know how to properly use conda environments.

By default, conda environments are created in ~/.conda/envs. Sometimes you might want to use the exact same environment as your fellow researchers. After all, exporting and creating new environments can get tiresome and duplicating environments unnecessarily eats up storage space.

You can change where conda looks and creates your environments. First, run:

mkdir -p ~/groups/grp_name/.conda/envs

to create a home for your environments. If this already exists, you don't need to run it again. Then, tell conda to always look there:

conda config --add envs_dirs ~/groups/grp_name/.conda/envs

You can run cat ~/.condarc to see it listed under envs_dirs. Multiple directories can be specified, but whichever one is at the top will be where new environments are created. Feel free to use # to comment out directories where conda should not look. You can delete the line or run

conda config --remove envs_dirs ~/groups/grp_name/.conda/envs

to remove a directory as well.

All group members will need to run the add command to be able to activate the environment.

R Environments

Many R packages are already available in conda-forge. Typically, you need to prepend r- to the package name before you can find them. Installing R itself is just:

mamba create -n my_r_env r

Check out our R page for more information on how to run it.