Conda and Mamba | Department of Statistics (2024)

Conda is a package management system which can be used to create Python and R development environments on many different platforms. You can install it on your own device, and it is often available at many High Performance Computing centers.

It's common that installing packages using Conda is slow or fails because Conda is unable to resolve dependencies. To get around this, we suggest the use of Mamba.

Mamba is a drop-in replacement for Conda that is generally faster and better at resolving dependencies.

In the examples below, we show the syntax for Conda, but you can simply replace "conda" with "mamba" on the SCF (or if you've installed Mamba on your own machine), e.g., "mamba install pytorch" rather than "conda install pytorch".

Installing software using Conda/Mamba

We have information on using Conda (and Mamba) to install software on the SCF.

Using Conda/Mamba

Once conda is installed, you can use it to search for and install other things.

(base) mylaptop:~$ conda search pytorchLoading channels: done# Name Version Build Channelpytorch 1.3.1 cpu_py27h0c87eb2_0 pkgs/mainpytorch 1.3.1 cpu_py36h0c87eb2_0 pkgs/mainpytorch 1.3.1 cpu_py37h0c87eb2_0 pkgs/mainpytorch 1.4.0 cpu_py36hf9bb1df_0 pkgs/mainpytorch 1.4.0 cpu_py37hf9bb1df_0 pkgs/mainpytorch 1.4.0 cpu_py38hf9bb1df_0 pkgs/mainpytorch 1.6.0 cpu_py37hd70000b_0 pkgs/mainpytorch 1.6.0 cpu_py38hd70000b_0 pkgs/main

Here we have searched for `pytorch` and conda has found several versions of it. There are also multiple build per version. For example above you can see that the latest version is 1.6.0 and there are builds for python3.7 and python3.8. If we ask conda to install version 1.6.0, it will check to make sure that it can proceed, and will tell you what it will do:

(base) mylaptop:~$ conda install pytorch=1.6.0Collecting package metadata (current_repodata.json): doneSolving environment: done## Package Plan ## environment location: /Users/me/miniconda3 added / updated specs: - pytorch=1.6.0The following packages will be downloaded: package | build ---------------------------|----------------- _pytorch_select-0.1 | cpu_0 169 KB blas-1.0 | mkl 6 KB ca-certificates-2020.6.24 | 0 125 KB certifi-2020.6.20 | py38_0 156 KB conda-4.8.4 | py38_0 2.9 MB intel-openmp-2019.4 | 233 887 KB libmklml-2019.0.5 | 0 16.2 MB llvm-openmp-10.0.0 | h28b9765_0 236 KB mkl-2019.4 | 233 101.9 MB mkl-service-2.3.0 | py38hfbe908c_0 42 KB mkl_fft-1.1.0 | py38hc64f4ea_0 138 KB mkl_random-1.1.1 | py38h959d312_0 290 KB ninja-1.10.0 | py38h879752b_0 102 KB numpy-1.19.1 | py38h3b9f5b6_0 21 KB numpy-base-1.19.1 | py38hcfb5961_0 4.1 MB pytorch-1.6.0 |cpu_py38hd70000b_0 34.0 MB ------------------------------------------------------------ Total: 161.2 MBThe following NEW packages will be INSTALLED: _pytorch_select pkgs/main/osx-64::_pytorch_select-0.1-cpu_0 blas pkgs/main/osx-64::blas-1.0-mkl intel-openmp pkgs/main/osx-64::intel-openmp-2019.4-233 libmklml pkgs/main/osx-64::libmklml-2019.0.5-0 llvm-openmp pkgs/main/osx-64::llvm-openmp-10.0.0-h28b9765_0 mkl pkgs/main/osx-64::mkl-2019.4-233 mkl-service pkgs/main/osx-64::mkl-service-2.3.0-py38hfbe908c_0 mkl_fft pkgs/main/osx-64::mkl_fft-1.1.0-py38hc64f4ea_0 mkl_random pkgs/main/osx-64::mkl_random-1.1.1-py38h959d312_0 ninja pkgs/main/osx-64::ninja-1.10.0-py38h879752b_0 numpy pkgs/main/osx-64::numpy-1.19.1-py38h3b9f5b6_0 numpy-base pkgs/main/osx-64::numpy-base-1.19.1-py38hcfb5961_0 pytorch pkgs/main/osx-64::pytorch-1.6.0-cpu_py38hd70000b_0The following packages will be UPDATED: ca-certificates 2020.1.1-0 --> 2020.6.24-0 certifi 2020.4.5.1-py38_0 --> 2020.6.20-py38_0 conda 4.8.3-py38_0 --> 4.8.4-py38_0Proceed ([y]/n)?

Typing `y`, or accepting the default choice (surrounded by brackets []) of `y` by pressing return, will let conda proceed.

Environments

Conda will install all of your packages into your default environment. However there are various situations in which you may want to create a separate environment, These include (1) if you are working on multiple projects simultaneously, (2) if you want to be able to eventually share your environment with a collaborator, and (3) if some package versions conflict with others in your default environment.

Creating

To create a new environment:

(base) mylaptop:~$ conda create --name myproject2

You can also create new environments with an entirely different version of python:

(base) mylaptop:~$ conda create --name myproject2 python=3.8

Note that under some situations, your environment might still make use of the default environment for the Python executable or certain packages already available in the default environment. See here for details and tips on creating a completely isolated environment.

To enter into your newly created environment:

(base) mylaptop:~$ source activate myproject2

The shell prompt will be updated to reflect the active environment name.

You can also activate an environment with `conda activate myproject2`, but see below for some cautionary notes before doing that.

Environment locations

If you have a scratch directory on the SCF, your Conda environments by default will be stored in `/scratch/users/<username>/conda/envs`. If you don't have a scratch directory, they will be in the usual `~/.conda/envs` directory. The purpose of this is to reduce backup sizes, as scratch it not backed up and Conda environments can be reproduced based on saving an environment recipe using `conda env export`.

Source packages that Conda caches for later use will be stored in `/tmp/pkgs` on each machine.

Conda activate versus source activate

When you first try to use `conda activate`, Conda will prompt you to run `conda init` to initialize conda for the shell you are using. You can do this, but note that it will modify your `.bashrc` so that Conda commands are available whenever you log in. As part of this, Conda will put you in the base Conda environment automatically when a new shell starts. Note that on the SCF, this will prevent you from accessing the Python version and related packages that the SCF provides. To avoid this, we recommend running `conda config --set auto_activate_base False` after running `conda init` on the SCF.

Using `source activate` (and `source deactivate`) instead of `conda activate` (and `conda deactivate`) are deprecated but still work, and you may wish to use them in some situations (such as on the SCF) to avoid the behavior discussed above.

Reproducing an environment

You can create a snapshot of your current environment which can then be reproduced by collaborators, or even just yourself if you want to setup the same environment on another device. To export an environment:

(base) mylaptop:~$ source activate myproject2(myproject2) mylaptop:~$ conda env export --from-history > environment.yml

Transmit the `environment.yml` file to the other person or device. One can then create a new environment from that file:

(base) mylaptop:~$ conda env create -f environment.yml

Installing Mamba/Conda on your own computer

On the SCF, we use Mamba as a drop-in replacement for Conda.

For your own computer, we recommend using the Miniforge distribution as your Python installation rather than Anaconda. Miniforge by default uses the conda-forge channel, a popular channel providing a wide variety of up-to-date packages.You can optionally have it use Mamba rather than Conda.

Mamba for Mac ARM64-based machines (Apple Silicon M1 and M2 Macs)

With the M1 and M2 Macs, Apple is now using its own chips, referred to as Apple Silicon. These have a different architecture, ARM64, than standard x86-64 chips, such as those produced by Intel and AMD.

To take full advantage of the new chips, you can use Miniforge as it provides versions for ARM64.

Non-ARM64 programs, including non-ARM64 Miniforge installations, will work on the new Apple Silicon-based machines by automatically making use of Apple's Rosetta2 system to translate machine code from ARM64 to x86_64, but you can expect some decrease in performance.

Using Anaconda/Miniconda

While we recommend installing Miniforge, you can also install Anaconda or a minimal installation called Miniconda.

Miniconda

Download Miniconda. There are installers for Linux, macOS, and Windows. Miniconda is a much smaller version of Anaconda which includes more packages out of the box as well as a graphical user interfaces to the installer. Miniconda is better if you don't want to install too many unnecessary packages right at the start.
Run the installer. This will prompt you a couple of times, and it is usually sensible to accept the defaults. Conda alters your shell initialization, and updates your executable path to find the base installation. For example, Miniconda will be installed into ~/miniconda3/ by default, so ~/miniconda3/bin/ will be added to your PATH variable. Conda may update your shell prompt as well with status information.
Open a new terminal window. This will start up a new shell which will be aware of the new Conda tools. For example, on a Mac:
```
(base) mylaptop:~$ type -a pythonpython is /Users/me/miniconda3/bin/pythonpython is /usr/bin/python
```

Anaconda for Mac ARM64-based machines (Apple Silicon M1 and M2 Macs)

To take full advantage of the new chips, you can use Anaconda with ARM64 support. You can either install Anaconda using the 64 -Bit (M1) installer, or update your existing conda installation:

conda install anaconda=2022.05

Non-ARM64 programs, including standard Miniconda and Anaconda installations, will work on the new Apple Silicon-based machines by automatically making use of Apple's Rosetta2 system to translate machine code from ARM64 to x86_64, but you can expect some decrease in performance.