Installing Python libraries with Miniforge and *conda

If you need a version of Python, other than the ones that are pre-installed on our HPC system, then you can install it for your account, without admin privileges, with the help of Conda.

Conda is an open-source package manager and environment management system, and is probably the most popular package manager for Python and R. The most common ways to set up a conda environment are to use the Anaconda or Miniconda installers, these currently require a license for commercial use. Following confusion over the license terms in late 2024 we recommend using miniforge rather than miniconda where possible.

Miniforge installation

If possible use Miniforge rather than Anaconda or Miniconda. It provides the open source conda (and mamba) package / environment management tools and the conda-forge repositories and does not require a license.

Connect to ALICE, open a terminal (if using NoMachine) and download the installation script to your home directpry with the wget command:

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh -P ~

Now navigate to your home directory (cd ~) and execute the install script:

bash Miniforge3-Linux-x86_64.sh

Follow the instructions and press 'Enter' to read the License Agreement, then 'q' to go to the last page and finally type 'yes' and press 'Enter' if you accept the terms. You will then be asked for the directory where you want Miniconda to be installed. The default location is ~/miniforge3 and you will have to press 'Enter' to confirm.

The installer will prompt:

installation finished.
Do you wish to update your shell profile to automatically initialize conda?
This will activate conda on startup and change the command prompt when activated.
If you'd prefer that conda's base environment not be activated on startup,
   run the following command when conda is activated:

conda config --set auto_activate_base false

You can undo this by running `conda init --reverse $SHELL`? [yes|no]
[no] >>> yes

Enter yes, then close and re-open the terminal for the changes to take effect. You will now see the word (base) just before your username, which means that the base environment is activated. You can now delete the downloaded shell script with

rm Miniforge3-Linux-x86_64.sh

conda config --set auto_activate_base false

If you experience problems with NoMachine then follow the instructions here.

Managing conda environments

To create a conda environment with a specific Python version type

conda create -n ENVNAME python=3.8

where ENVNAME is a name of your choice. This command will install the latest Python in the 3.8 branch. To verify that the environment was successfully installed type

conda env list

which lists all your environments. If you followed the guide till here, you should see two environments; the base and the one you created in the previous step. To activate it run

conda activate ENVNAME

To install a python package on that environment, e.g. numpy, type

conda install numpy

while the environment is activated, and if you want a specific version of that package you can use

conda install numpy=1.18.1

These commands with look for the package in the default channels. If you want to also specify a channel you can use the -c flag, e.g.

conda install -c conda-forge numpy

You can search for packages here. If a package is unavailable through conda, but is available in the Python Package Index (Pypi) repository, then you can install it with the pip command (e.g. pip install pycuda). It is strongly advised to install packages with pip only after installing all the packages that can be installed with conda first. 'pip' will need to be installed into the conda environment first:

conda activate ENVNAME
conda install pip

Using conda in an HPC batch job

Now, if you want to use a conda environment in a non-interactive job, you have to activate it by adding the following code in your submit script, before executing your code (change the path to the location where you have installed Minforge):

source ~/miniforge3/bin/activate ENVNAME

Removing an environment

If an environment is no longer required, it should be deleted to save file system space:

First, deactivate it with:

conda deactivate

and then type:

conda remove -n ENVNAME --all

which will remove the environment and all installed packages.

Migrating from Miniconda / Anaconda to Miniforge

If you have existing Miniconda / Anaconda environments, these should be migrated to Miniforge.

Simple environments

For simple environments, these can be moved from the previous Anaconda / Miniconda installs and reinstalled using Miniforge - for example:

 # move old conda install
 cd ~
 mv miniconda3 miniconda3-old
 mv .condarc .condarc-old
 # install Miniforge
 wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh -P ~
 bash Miniforge3-Linux-x86_64.sh
 # edit ~/.bashrc to remove any old Anaconda / Miniconda related setup
 # now recreate the environment(s), using the same commands used when they were installed before:
 conda create -n .... .....

Complex environments

For more complex environments with many installed packages, first export an environment definition file from Miniconda / Anaconda for each environment. For example, to create an environment file for an environment called 'env1':

 conda activate env1
 conda env export --no-builds > ~/conda-env1.yaml

Once the new environment definition files have been created, remove the old conda install and install Miniforge:

 # move old conda install
 cd ~
 mv miniconda3 miniconda3-old
 mv .condarc .condarc-old
 # install Miniforge
 wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh -P ~
 bash Miniforge3-Linux-x86_64.sh
 # edit ~/.bashrc to remove any old Anaconda / Miniconda related setup

Now, for each environment, re-install it from the yaml file:

 conda env create --name env1 --file conda-env1.yaml

Once the new environment(s) is / are working as expected, delete the miniconda3-old directory:

cd ~
rm -rf miniconda3-old