Available Software on HPC3

Examples in this guide use the following color schema:

very important message || links to documents and sites || command name || what you need to type

1. Overview

HPC3 serves researchers from all corners of the UCI campus. At the heart of the value that RCIC adds to HPC3, is wide range of application software that is pre-compiled and ready for use. There are literally hundreds of domain-specific, community-supported, applications maintained on the cluster.

Some software is quick and easy to compile, install, and make available to users. Other applications have numerous (often undocumented) software requirements, unique build methods, and conflicting requirements when compared to other applications. The size of the problem is quite large, and grows with each new application.

RCIC uses a multi-factor approach to provide a stable and flexible environment.

  • Most software is accessed only after loading the appropriate environment module

  • Nearly all installed software is packaged as an RPM with encoded dependencies.

  • Adoption of a declarative, repeatable build methodology (yaml2rpm), designed by RCIC that generates re-usable RPMS that can installed on any compatible CentOS system.

  • A Software Map (PDF) shows a snapshot of the collections of software, software modules they provide, and build dependencies. Users of HPC3 need to login to see the most recent software versions and availability.

For the most part, users need only know (or discover) the already-available environment module and load it, prior to using their application.

2. Modules difference

There are a few meanings for modules

  • Language modules are collections of related variables, functions and subroutines that perform a set of specific programming tasks. Simply put they are files consisting Perl/Python/R code. To access a language module or search what is available, you need to use its language: Perl/Python/R.

  • Environment modules are used to activate a specific software that a user wants to use. See below how to identify and use them.

3. Environment Modules

Environment modules are one way to control which versions of software are active (Lua modules provides similar functionality, but it not installed on HPC3). Typically, a user only needs to know which module(s) is (are) needed to gain access to their application(s). Modules are quite simple in concept.

loading a module modifies environment variables (e.g., PATH for executables or LD_LIBRARY_PATH for shared object libraries and others). It is used to activate the software.

unloading a module reverses the action of load: unsets the environment variables set by loading. It is used to deactivate the software.

3.1. Using modules

Suppose you want access to gcc version 8.4.0, the following shows which version of gcc is active prior to module loading, after module load, and after the unloading.

[user@login-x:~]$ module list  (1)
No Modulefiles Currently Loaded.
[user@login-x:~]$ gcc --version | grep ^gcc (2)
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
[user@login-x:~]$ module load gcc/8.4.0 (3)
[user@login-x:~]$ module list
Currently Loaded Modulefiles:
  1) gcc/8.4.0
[user@login-x:~]$ gcc --version | grep ^gcc
gcc (GCC) 8.4.0
[user@login-x:~]$ module unload gcc/8.4.0 (4)
[user@login-x:~]$ gcc --version | grep ^gcc
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
1 determine which modules are loaded
2 print the version of active gcc
3 make version 8.4.0 active
4 unload the module and restore

3.2. Available modules

A very large fraction of software applications on HPC3 have a module associated with it.

most modules have specific version numbers. If not specified, the latest found is used. You should include the version number when loading a module. This will ensure that you always get the version you intend.

Modulefiles are tcl scripts that, when executed, affect your unix environment variables. We try to broadly classify software by what they do so that it is a bit easier to find a module you are interested in. The following shows the partial output of running module avail command and has just the first line after each category (there are 100s of modules not shown here).

Sample output of module avail on HPC3, Categories highlighted in red
[user@login-x:~]$ module avail
--------------------------- /usr/share/Modules/modulefiles ---------------------------
dot         module-git  module-info modules     null        use.own

---------------------------------- /etc/modulefiles ----------------------------------

--------------------- /opt/rcic/Modules/modulefiles/AI-LEARNING ----------------------
pytorch/1.4.0.a0 tensorflow/2.0.0

----------------------- /opt/rcic/Modules/modulefiles/BIOTOOLS -----------------------
bandage/0.8.1      fastqc/0.11.9      ncbi-vdb/2.10.2    star/2.7.3a

---------------------- /opt/rcic/Modules/modulefiles/CHEMISTRY -----------------------
amber/19.11/gcc.8.4.0                  mdtraj/1.9.3

---------------------- /opt/rcic/Modules/modulefiles/COMPILERS -----------------------
clang/10.0.0               intel-tbb/20190605         openmpi/1.10.7/gcc.4.8.5

---------------------- /opt/rcic/Modules/modulefiles/LANGUAGES -----------------------
anaconda/2019.10 java/11          julia/1.4.0      python/2.7.17    tcl/8.6.9

---------------------- /opt/rcic/Modules/modulefiles/LIBRARIES -----------------------
eigen/3.3.7       ffmpeg/4.1.3      OpenBLAS/0.3.6

----------------------- /opt/rcic/Modules/modulefiles/PHYSICS ------------------------

------------------------ /opt/rcic/Modules/modulefiles/TOOLS -------------------------
boost/1.66.0/gcc.8.4.0                 hdf5/1.10.5/intel.2020u1-openmpi.4.0.3

3.3. Searching for modules

You might not know the name of a particular module or versions available. You can use module keyword or module avail <partialname> to find some information. Keyword will find the keyword if it shows up anywhere in the module definition:

[user@login-x:~]$ module keyword hmmer
hmmer/3.3            : Name_ hmmer
hmmer/3.3            :                 hmmer_3.3
repeatmasker/4.1.0   :                 in RepeatMasker are performed by one of several popular search engines including nhmmer,
repeatmasker/4.1.0   :                 hmmer/3.3
repeatmasker/4.1.0   :                 hmmer/3.3

The above output shows in the first column found module names (with their associated versions) that contain a keyword, and in the second column shows the text line of the module file where a keyword was found.

To turn on case insensitive module parameter evaluation, add -i which to your commands. The following shows that there are two versions of bwa module available, bwa/0.7.17 and bwa/0.7.8:

[user@login-x:~]$ module keyword -i BWA
bwa/0.7.17           : Name_ bwa
bwa/0.7.17           :                 bwa_0.7.17
bwa/0.7.8            : Name_ bwa
bwa/0.7.8            :                 bwa_0.7.8

You can look for partial words on just the module name (case sensitive). The following example looks for hdf modules available on HPC3:

[user@login-x:~]$ module avail hdf
---------------------- /opt/rcic/Modules/modulefiles/LIBRARIES ----------------------

------------------------ /opt/rcic/Modules/modulefiles/TOOLS ------------------------
hdf5/1.10.5/gcc.4.8.5                  hdf5/1.10.5/gcc.8.4.0-openmpi.4.0.3
hdf5/1.10.5/gcc.4.8.5-openmpi.1.10.7   hdf5/1.10.5/intel.2020u1
hdf5/1.10.5/gcc.8.4.0                  hdf5/1.10.5/intel.2020u1-openmpi.4.0.3

Here is an example that explains how to search for cmake. The first command showed no output, which means there is no module named "cmake". However, search for a keyword shows that module foundation provides cmake. Follow up with module display to find more info:

[user@login-x:~]$ module avail cmake
[user@login-x:~]$ module keyword cmake
foundation           :                 updated tools: cmake(3.20.5), curl(7.66.0),
[user@login-x:~]$ module display foundation

module-whatis	 Category TOOLS
module-whatis	 Name_ foundation
module-whatis	 Version_ 1.1
module-whatis	 Description__ This modules provides a setup for using up to date versions
module-whatis	                 of commonly used tools for building software (tools located in /opt/foundation).
module-whatis	                 Rather than individual modules, a single module provides access to the following
module-whatis	                 updated tools: cmake(3.20.5), curl(7.66.0),
module-whatis	                 git(2.23.0), git-lfs(2.11.0), ninja(1.9.0), and
module-whatis	                 swig(4.0.1).
module-whatis	 Prerequisites rcic-module-support
<--- output cut for brevity --->

To find more details about how to execute module commands see a manual page for module:

[user@login-x:~]$ man module

3.4. Module Naming

you will notice in the available modules listing that module names have four different forms:

Table 1. Formats of module file names
Naming schema Example

(1) name


(2) name/version


(3) name/version/compiler.compiler_version


(4) name/version/compiler.compiler_version-mpi.mpi_version



(5) name/version/compiler.compiler_version-cuda.cuda_version


(1) simple module with only a name without version. (2) python module for a specific version of python. (3) boost version compiled with a specific compiler version. (4) hdf5 version 1.10 built with in the Intel compiler version 2020u1 and openmpi version 4.0.3, and a second line shows a version built with the gcc compiler version 8.4.0. (5) namd module built with gcc compiler and is cuda enabled. Note, when a module lists cuda in its name this means a module provides a software that needs to be run on GPU-enabled nodes (any of GPU partitions).

The naming schema makes it more apparent what version is available and what are key differences among different versions. This also shows the potential combinatorial number of variants of any software. We do not attempt to build every variant of compiler x mpi for these kinds of software. We build what is needed.

3.5. Prerequisite modules

RCIC has built additional code to automatically load and unload prerequisite modules. Our goal is that if you want an application and it has several other software modules that must be loaded, then they will be added automatically when loading and removed automatically when unloading. Here are three examples:

  1. Loading module with a few prerequisites

    [user@login-x:~]$ module list
    No Modulefiles Currently Loaded.
    [user@login-x:~]$ module load hdf5/1.10.5/gcc.8.4.0-openmpi.4.0.3
    [user@login-x:~]$ module list
    Currently Loaded Modulefiles:
      1) gcc/8.4.0                             3) hdf5/1.10.5/gcc.8.4.0-openmpi.4.0.3
      2) openmpi/4.0.3/gcc.8.4.0
    [user@login-x:~]$ module unload hdf5/1.10.5/gcc.8.4.0-openmpi.4.0.3
    [user@login-x:~]$ module list
    No Modulefiles Currently Loaded.
  2. Loading module with many prerequisites

    [user@login-x:~]$ module list
    No Modulefiles Currently Loaded.
    [user@login-x:~]$ module load unicycler/0.4.8
    [user@login-x:~]$ module list
    Currently Loaded Modulefiles:
      1) python/3.8.0             7) perl/5.30.0             13) htslib/1.10.2
      2) gcc/8.4.0                8) intel-tbb/20190605      14) samtools/1.10
      3) SPAdes/3.14.0            9) bowtie2/2.4.1           15) gsl/2.6/gcc.8.4.0
      4) racon/1.4.13            10) boost/1.66.0/gcc.8.4.0  16) bcftools/1.10.2
      5) java/1.8.0              11) edirect/1.0             17) unicycler/0.4.8
      6) pilon/1.23              12) ncbi-blast/2.10.0
    [user@login-x:~]$ module unload unicycler/0.4.8
    [user@login-x:~]$ module list
    No Modulefiles Currently Loaded.
  3. Module loading/unloading with pre-loaded prerequisites

    Our modifications to modules has "smart" unloading where when a prerequisite is already loaded, unloading the higher-level module will leave the prerequisite intact. Here’s the Example 1 but with gcc/8.4.0 already loaded by the user.

    [user@login-x:~]$ module list
    No Modulefiles Currently Loaded.
    {pronmpt} module load gcc/8.4.0 (1)
    [user@login-x:~]$ module list
    Currently Loaded Modulefiles:
      1) gcc/8.4.0
    [user@login-x:~]$ module load hdf5/1.10.5/gcc.8.4.0-openmpi.4.0.3 (2)
    [user@login-x:~]$ module list
    Currently Loaded Modulefiles:
      1) gcc/8.4.0                             3) hdf5/1.10.5/gcc.8.4.0-openmpi.4.0.3
      2) openmpi/4.0.3/gcc.8.4.0
    [user@login-x:~]$ module unload hdf5/1.10.5/gcc.8.4.0-openmpi.4.0.3 (3)
    [user@login-x:~]$ module list (4)
    Currently Loaded Modulefiles:
      1) gcc/8.4.0
    1 manually preload gcc/8.4.0 as a prereq
    2 load hdf5 as in the simple example
    3 unload hdf5
    4 check that gcc is still loaded
Always unload modules in the reverse order of loading. Last-loaded should be first unloaded. Not doing this can result in an expected or broken environment

3.6. Modules limitations

Caveat emptor - Environment modules do their job, but have limitations. You can easily render your environment into a completely broken mess if you randomly unload modules. For example, the complex example automatically loaded 16 prerequisite modules. Nothing prevents you from doing a module unload gcc/8.4.0. You won’t see any errors or complaints until you attempt to run unicycler. Needed libraries will not be available. Please heed the tip above

Some modules overwrite environment variables, clang is an example of this. Clang requires llvm and gcc, but overwrites the CC environment variable. If you you preload gcc and then load clang, that will work just fine. However, if you then unload clang, the CC environment variable will NOT be restored to CC=gcc. Instead the environment variable will no longer be defined.

When a module changes an environment variable, it does not record the previous version of it.

3.7. Module details

There are two commands that are useful to find out about modules:

3.7.1. module whatis

module whatis gives you some information from specified modulefile. RCIC-authored modules follow a uniform formatting and it is best illustrated by the example:

Example module whatis output
[user@login-x:~]$ module whatis hdf5/1.10.5/gcc.8.4.0
hdf5/1.10.5/gcc.8.4.0: Category TOOLS (1)
hdf5/1.10.5/gcc.8.4.0: Name_ hdf5
hdf5/1.10.5/gcc.8.4.0: Version_ 1.10.5
hdf5/1.10.5/gcc.8.4.0: Description_ HDF5 is a data model, library, and file format for storing and managing data. It supports (2)
hdf5/1.10.5/gcc.8.4.0:                 an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for
hdf5/1.10.5/gcc.8.4.0:                 high volume and complex data. HDF5 is portable and is extensible, allowing applications to
hdf5/1.10.5/gcc.8.4.0:                 evolve in their use of HDF5. The HDF5 Technology suite includes tools and applications for
hdf5/1.10.5/gcc.8.4.0:                 managing, manipulating, viewing, and analyzing data in the HDF5 format.
hdf5/1.10.5/gcc.8.4.0:                 Environment var: HDF5_HOME = /opt/apps/hdf5/1.10.5/gcc/8.4.0
hdf5/1.10.5/gcc.8.4.0: Load modules java/1.8.0 (3)
hdf5/1.10.5/gcc.8.4.0:                 gcc/8.4.0
hdf5/1.10.5/gcc.8.4.0: Prerequisites__ java8-module (4)
hdf5/1.10.5/gcc.8.4.0:                 gcc_8.4.0-module
hdf5/1.10.5/gcc.8.4.0:                 rcic-module-support
hdf5/1.10.5/gcc.8.4.0:                 hdf5_1.10.5_gcc_8.4.0
1 Name, Category, Version for this particular module
2 A long description of what this software does
3 Prerequisite modules that will be loaded, if not already present
4 RPM Package requirements to have this module installed

3.7.2. module display

module display shows additional information, including the full path of the modulefile and the environment variables changes the modulefile will make when loaded.

Example module display output
[user@login-x:~]$ module display hdf5/1.10.5/gcc.8.4.0

module-whatis	 Category TOOLS
module-whatis	 Name_ hdf5
module-whatis	 Version_ 1.10.5
module-whatis	 Description_ HDF5 is a data model, library, and file format for storing and managing data. It supports
module-whatis	                 an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for
module-whatis	                 high volume and complex data. HDF5 is portable and is extensible, allowing applications to
module-whatis	                 evolve in their use of HDF5. The HDF5 Technology suite includes tools and applications for
module-whatis	                 managing, manipulating, viewing, and analyzing data in the HDF5 format.
module-whatis	                 Environment var: HDF5_HOME = /opt/apps/hdf5/1.10.5/gcc/8.4.0
module-whatis	 Load modules java/1.8.0
module-whatis	                 gcc/8.4.0
module-whatis	 Prerequisites java8-module
module-whatis	                 gcc_8.4.0-module
module-whatis	                 rcic-module-support
module-whatis	                 hdf5_1.10.5_gcc_8.4.0
prereq	 java/1.8.0 (1)
prereq	 gcc/8.4.0
setenv		 HDF5_HOME /opt/apps/hdf5/1.10.5/gcc/8.4.0 (2)
setenv		 hdf5PREFIX /opt/apps/hdf5/1.10.5/gcc/8.4.0
setenv		 hdf5CPPFLAGS -I/opt/apps/hdf5/1.10.5/gcc/8.4.0/include
setenv		 hdf5LDFLAGS -L/opt/apps/hdf5/1.10.5/gcc/8.4.0/lib -Wl,-rpath,/opt/apps/hdf5/1.10.5/gcc/8.4.0/lib
prepend-path	 PATH /opt/apps/hdf5/1.10.5/gcc/8.4.0/bin (3)
prepend-path	 MANPATH /opt/apps/hdf5/1.10.5/gcc/8.4.0/share/man
prepend-path	 LD_LIBRARY_PATH /opt/apps/hdf5/1.10.5/gcc/8.4.0/lib
prepend-path	 PKG_CONFIG_PATH /opt/apps/hdf5/1.10.5/gcc/8.4.0/lib/pkgconfig
prepend-path	 -d   CPPFLAGS -I/opt/apps/hdf5/1.10.5/gcc/8.4.0/include
prepend-path	 -d   LDFLAGS -L/opt/apps/hdf5/1.10.5/gcc/8.4.0/lib -Wl,-rpath,/opt/apps/hdf5/1.10.5/gcc/8.4.0/lib
1 Specification of pre-requisite modules
2 Environment vars set
3 PATH-like variable that are augmented with additional information.

4. User installed sw and environment modules

you don’t need to create a new environment module if you are installing a language package (module) for Python, R, or Perl, or adding packages with conda. Simply use respective Python/R/Perl/conda module (use module avail command to see what is installed) to access the desired language and follow your package installation instructions. Please see the install guides that explain how to create conda environments, and to install R, Python and Perl packages.

Users can install additional software and add environment module for it either for themselves or for their groups. There are a few basic steps:

  1. Compile and install your desired software in your user/group designated area. Make sure that for group access the directories and files permissions are set correctly. Note, the software installation and a module file installation are two separate tasks and the installation location of a module file is different from the location where the software is installed. Verify that the software is working according to the instructions.

    DO NOT install your software in $HOME/modulefiles/
  2. Create an environment module file for your new software. The environment module file is a text file in a specific format that provides information about the software and sets needed environment for using it.

    • We suggest to use one of the available system modules files as a template. Run command module display for one of the available modules, the output shows the full path to the module file which can be copied and modified according to your needs. In addition, take a look at this modulefile-template file that can be used as a template. In general, you will need to specify:

      • software description, name and version

      • environment variables your software needs, for example PATH, LD_LIBRARY_PATH

      • modules that you used to compile your softare (if any)

    • Please follow the naming schema for the module file name and choose where to put it:

      • If you install the new software and its module for yourself, we suggest to use $HOME/modulefiles/ directory to store your new module files. It is searched by module commands by default and in this case your job is done and no extra steps are needed.

      • If you install for the group, see If you install for the group instructions below.

  3. Verify your installed module file is working. Run module avail command. If your environment module file is installed correctly then your new module name will show at the end of the output.

    If no valid module files are present in $HOME/modulefiles/, the module name will NOT be shown when running module commands, or will produce an error. Review steps above and correct any errors.
  4. If you install for the group. You need to enable module commands to look at your new environment module file. This is done via adding pathnames to the text file $HOME/.usermodulepath. Initially, this file does not exist, simply create it using your favorite text editor using the following rules:

    • you can put multiple paths, type each path on a separate line

    • comment lines start with a #. Here is an example $HOME/.usermodulespath file:

      Example User-defined paths to search for additional modules
      [user@login-x:~]$ cat $HOME/.usermodulespath
      # Put a directory path per line to search for additional modules

      Let say, you (1) installed a new version of gcc software 8.4.1 for the group in /dfs3/panteater-lab/project1/sw/ and (2) created a directory /dfs3/panteater-lab/modulefiles/ to hold the module files. This directory can be used for any other module files that will be installed for the group. (3) Now, you need to create a /dfs3/panteater-lab/modulefiles/gcc/ directory where you will add a module file named 8.4.1 that defines your new module for your installed gcc version 8.4.1.

    • After you modify the contents of the file $HOME/.usermodulespath, for the changes to take effect please start a new bash shell:

      [user@login-x:~]$ . ~/.bashrc
      If you expect other users to use your publicly available module file, they will need to create $HOME/.usermodulespath file with the same contents as yours.
    • If your installed module file is correct you can run module commands to display and load your module as shown below:

      Loading a user-defined module
      [user@login-x:~]$ module list
      No Modulefiles Currently Loaded.
      [user@login-x:~]$ module avail
          .... truncated
      ------------------------ /opt/rcic/Modules/modulefiles/TOOLS -------------------------
      fftw/3.3.8                             netcdf-c/4.7.0/intel.2020u1
      --------------------------- /dfs3/panteater-lab/modulefiles ---------------------------
      [user@login-x:~]$ module load gcc/8.4.1
      [user@login-x:~]$ module list
      Currently Loaded Modulefiles:
        1) gcc/8.4.1
      [user@login-x:~]$ module unload gcc/8.4.1

5. Software Stack

The deployed software stack is generally added to on weekly basis. This is in response to a near-continuous stream of requests. If you want to request new software, please see the howto.

The complete stack is fairly complex and captures dependencies. For the most part, we build RPMs (the native package format for CentOS/RedHat systems). Exceptions generally fall around very large footprint software like the Intel Compiler, MATLAB, and (e.g., FreeSurfer).

Why RPMS? For us, it’s one extra step that comes a wide variety of benefits.

  • The package manager (yum) will not allow two different RPMs to claim ownership of the same file (or have conflicting permissions on a shared directory path).

  • Dependencies, when properly encoded, help ensure that removal of a particular package won’t leave other software in a "broken" state.

  • Ease of transitioning to container definitions. One of our goals is to allow users to easily take away subsets of the HPC3 software stack in the form of a singularity or Docker container.

  • It only takes a little bit of work to enable multiple versions of same software to be co-installed.

  • Use native OS tools for admins/users to get information about installed software.