Skip to content

Navigating Lmod Modules

One of the key differences between ARGO and Hopper is that Hopper uses Lmod modules for software provisioning while Argo uses Environment Modules. These differences will take some time to get used to.

Basic Lmod Usage

The table below summarizes the most commonly used Lmod commands. Please note that you can use ml as an alias or shortcut to module

Module Command Description
ml avail List available modules
ml list Show modules currently loaded
ml load/add package Load a selected module*
ml +package Load a selected module*
ml unload/rm package Unload a previously loaded module
ml -package Unload a previously loaded module
ml swap package1 package2 Unload package1 and load package*
ml purge Unload all loaded modules
ml reset Reset loaded modules to system defaults
ml display/show package Display the contents of a selected module
ml spider List all modules (not just available ones)
ml spider package Display description of a selected module
ml keyword key Search for available modules by keyword
ml, module help Display help, usage information for modules
ml use path Add path to the MODULPATH (module search path)
ml unuse path Remove path from the MODULPATH (module search path)

Info

* We have enabled the autoswap feature in Lmod such that loading a package while a conflicting package is loaded will automatically swap the modules. For example, tying to load python/3.8.6 while python/3.7.6 is loaded will automatically swap python/3.7.6 for python/3.8.6. Without the autoswap feature, you would have had to manually unload python/3.7.6 and load python/3.8.6

Click on the following clip to get the basic look and feel of Lmod in Hopper.

asciicast

Flat vs. Hierarchical Structure

ARGO

ARGO uses a flat module scheme that displays all module available on the system at any time. When you search a package via module avail <package_name>, you will see modules matching <package_name> regardless of their dependencies.

Hopper

Hopper uses a hierarchical module scheme that displays only modules that are compatible with the particular compiler and MPI library you have loaded at a given time to avoid incompatibilities. Therefore, searching a package via module avail <package_name> will not necessarily show you all available versions of that package. A more comprehensive way of searching for packages is using the module spider <package_name> command. That will report all available packages matching <package_name>.

Info

On Hopper, you cannot load more than one version of the same application. For example, you can not load python/3.6.8 and python/3.8.6 at the same time. Loading one will automatically unload the other. In the rare cases where it is absolutely necessary to load more than one compiler or MPI library at a time, you can set export LMOD_EXPERT=1 to enable that feature.

When you list the available modules on Hopper, you will see applications grouped as follows

  • Core - essential system modules
  • Independent - packages without particular compiler or MPI library dependence. These are often applications packaged as pre-compiled static binaries
  • <COMPILER> - packages built using the given <COMPILER>
  • <Compiler>_<MPI-library> - packages built using the given <COMPILER> and <MPI-library>

As will be discussed below, the default compiler and MPI library on Hopper are GNU/9.3.0 and OpenMPI/4.0.4. Therefore, you will see these groups of modules when executing module avail

  • Core
  • Independent
  • GNU-9.3.0
  • GNU-9.3.0_OpenMPI-4.0.4

Base/Default Modules

ARGO

No modules are loaded by default at login on Argo unless you explicitly load them via your startup scripts (~/.bashrc, ~/.profile, ~/.bash_profile).

$ module list

No Modulefiles Currently Loaded.

Hopper

Since modules available to you depend on the compiler and MPI library loaded in your environment, Hopper loads a set of default modules including a default compiler (gnu9/9.3.0) and MPI library (openmpi4/4.0.4).

The default module is hosts/hopper which sets up your environment for Hopper.

$ echo $LMOD_SYSTEM_DEFAULT_MODULES

hosts/hopper

$ ml show hosts/hopper

----------------------------------------------------------------------------------------
   /etc/modulefiles/Core/hosts/hopper:
----------------------------------------------------------------------------------------
prepend_path("MODULEPATH","/opt/ohpc/pub/modulefiles")
prepend_path("MODULEPATH","/opt/sw/other/modulefiles")
prepend_path("PATH","/opt/ohpc/pub/bin")
prepend_path("MANPATH","/usr/local/share/man:/usr/share/man/overrides:/usr/share/man/en:/usr/share/man")
try_load("use.own")
try_load("autotools")
try_load("prun")
try_load("gnu9")
try_load("openmpi4")
help([[Setup default login environment for Hopper
]])

As you can see in the /etc/modulefiles/Core/hosts/hopper above, the hosts/hopper module loads

  • the default compiler (gnu9),
  • the default MPI and associated libraries (openmpi4, libfabric, ucx),
  • the build tools (autotools),
  • wrappers and libraries for running parallel jobs (prun)
  • a module to load users' private modules (use.own)
$ module list

Currently Loaded Modules:
  1) autotools   2) prun/2.0   3) gnu9/9.3.0   4) ucx/1.8.0   5) libfabric/1.10.1   6) openmpi4/4.0.4   7) ohpc 8) hosts/hopper

You can revert back to the default modules at any time by executing ml reset or ml restore

$ ml restore

Resetting modules to system default. Reseting $MODULEPATH back to system default. All extra directories will be removed from $MODULEPATH.

$ ml list

Currently Loaded Modules:
  1) use.own   2) autotools   3) prun/2.0   4) gnu9/9.3.0   5) ucx/1.8.0   6) libfabric/1.10.1   7) openmpi4/4.0.4   8) hosts/hopper

Most Common Development Environments

ARGO

You can pick and choose any development environments (compilers, libraries, tools ... etc) from the available modules.

Hopper

Because Lmod is hierarchical, software built using particular development tools is grouped together and made available to you when you have those tools loaded.

The default development environment on Hopper is composed on the GNU/9.3 compiler and OpenMPI/4.0.4 MPI library. We have built an extensive list of applications based for this GNU-9.3.0_OpenMPI_4.0.4 environment. If you want to build your own software, we encourage you to built it on top of the GNU-9.3.0_OpenMPI_4.0.4 or Intel-2020.2_IMPI-2020.2

Since Hopper uses Intel CPUs and Intel provides highly optimized compilers (icc, icpc, ifort) and libraries (IMPI, MKL) for improved performance on these CPUs, it is usually recommended that you use software built using Intel tools or that you build your software using those same Intel toolkits.

To switch to Intel toolkits, you can simply load the Intel compiler and MPI library. The autoswap feature in Lmod will swap out the existing compilers and MPI libraries with Intel ones. Also, the Intel-IMPI version of any software built using the existing toolkits will also be reloaded or those modules will be inactive.

Please click on the clip below to see these concepts in action.

asciicast

Avoiding Environment Conflicts Between Hopper and Argo

Warning

If you load modules or set up other environmental variables using your startup scripts on Argo, you will likely get errors and warning messages when logging into Hopper because those modules do not exist on Hopper with the same name. To avoid these issues, you can wrap some of the logic in your startup scripts to behave differently based on the cluster. Such logic would look like this:

# load the proper set of modules based on the cluster
export CLUSTER=`sacctmgr -n  show cluster format=Cluster|xargs`
export CNODE=`hostname -s`
if [ ${CLUSTER} == "argo" ]; then
  module load <ARGO_MODULE_NAME...>
  source <ARGO_FILE...>
  export ARGO_ENVIRONMENT=...
  ...
elif [ ${CLUSTER} == "hopper" ] && [ ${CNODE} == "dgx-a100-01" ]; then
  module load <DGX_MODULE_NAME...>
  source <DGX_FILE...>
  export DGX_ENVIRONMENT=...
  ...
else
  module load <HOPPER_MODULE_NAME...>
  source <HOPPER_FILE...>
  export HOPPER\_ENVIRONMENT=...
  ...
fi

Module naming

ARGO

Modules are generally named as <package_name>/<package_version> on ARGO

Hopper

Depending on the source of the package, Lmod modules on Hopper can have longer names and aliases.

  • Spack-built packages are named <package_name>/<package_version>-<two-character-hash>
  • Some packages have useful aliases such as <package_name>/<package_version>-<two-character-hash> (mixed-precision)
  • Some important packages such as compilers, MPI and math libraries have global aliases appearing at the top when you execute module avail

Info

Applications that are built using Spack have module names appended with a two character hash such as r/4.0.3-hb and r/4.0.3-ta. The hashes are needed to differentiate between multiple installations of the same version of software and they do not have a particular meaning otherwise. In the aforementioned example, both modules refer to R version 4.0.3, but r/4.0.3-hb uses OpenBLAS math library while r/4.0.3-ta uses R's own math libraries.

Searching for Modules

Info

In LMod running on Hopper, module spider searches the whole module tree to find matching modules whereas module avail will only search modules built with your current compiler and MPI library.

How you search for modules in Hopper is very different from Argo and we will use an example to demonstrate this key difference

Let's take the package nwchem to demonstrate differences in how you search for modules in the two systems.

ARGO

You can easily see that there are two versions of nwchem.

$ module avail nwchem
------------------------ /cm/shared/modulefiles ----------

nwchem/intel/6.8.1 nwchem/intel/7.0.2

Hopper

You can initially see that there is one version of nwchem built using GNU/9.3.0 compiler and OpenMPI/4.0.4 MPI library.

$ module avail nwchem

---------------------- GNU-9.3.0_OpenMPI-4.0.4 -----------------
   nwchem/7.0.2-m4

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
However, you are not seeing any other versions that are built with other compilers and MPI libraries. That's where module spider comes in. 
$ ml spider nwchem

------------------------------
  nwchem:
------------------------------
     Versions:
        nwchem/6.8.1-ip
        nwchem/7.0.2-m4
        nwchem/7.0.2-mr

--------------------------------
  For detailed information about a specific "nwchem" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.

You can now see there are three versions of nwchem on Hopper.

If you randomly try to load one of these modules, you may get an error message like this:

$ module load nwchem/6.8.1-ip

LMod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: "nwchem/6.8.1-ip"
   Try: "module spider nwchem/6.8.1-ip" to see how to load the module(s).

To see how you can load any one of them, you can run module spider on any particular version.

$ module spider nwchem/6.8.1-ip

---------------------------------
  nwchem: nwchem/6.8.1-ip
---------------------------------

    You will need to load all module(s) on any one of the lines below before the "nwchem/6.8.1-ip" module is available to load.

      intel/2020.2  impi/2020.2

    Help:
      High-performance computational chemistry software

The output above is telling us that to load this module, you would need the compiler and MPI library it was built with, namely intel/2020.2 and impi/2020.2.

$ module load intel/2020.2 impi/2020.2

LMod is automatically replacing "gnu9/9.3.0" with "intel/2020.2".

LMod is automatically replacing "openmpi4/4.0.4" with "impi/2020.2".

$ module load nwchem/6.8.1-ip

$ module list

Currently Loaded Modules:
  1) use.own   2) slurm-tools/1.0   3) autotools   4) prun/2.0   5) ohpc   6) intel/2020.2   7) impi/2020.2   8) nwchem/6.8.1-ip

Location of Software and Modules

ARGO

The location of applications and modules that make them available to users is as follows

  • applications - /cm/shared/apps
  • module file - /cm/shared/modulefiles

Hopper

On Hopper, the location of applications and module files is structured based on the following

  • source of the application - does it come from OpenHPC repos, Spack recipes or manual build
  • compiler and MPI library dependence
  • CPU architecture - what CPU architecture is it built on. In our case, we have both Intel Cascade Lake and AMD EPYC CPUs
  • Operating system - what OS is it built on. We run both CentOS 8 and Ubuntu 20.04

Based on the above criteria, applications and module files are placed at these locations

  • OpenHPC provided
    • appilications - /opt/ohpc/pub
    • module files -
      • independent - /opt/ohpc/pub/modulefiles
      • compiler/MPI depdendent - /opt/ohpc/pub/moduledeps
  • Spack-built
    • appilications - /opt/sw/spack/apps/
    • module files - /opt/sw/spack/moduledeps
  • Manually built
    • appilications - /opt/sw/other
    • module files -
      • independent - /opt/sw/other/modulefiles
      • compiler/MPI depdendent - /opt/sw/other/moduledeps

The best way to find the location of an application and module files is to use module show <package_name>.

asciicast