Create and manage custom modules

Introduction

CHTC uses the module system, which allows users to load and manage their software environments on the HPC system. Experienced HPC users who use a range of software environments can create and manage their own custom modules, which makes it easier to switch between different environments for different types of jobs.

⚠️ Caution: You may break your HPC environment if you do not know what you are doing.

To create custom modules, we recommend that you have strong foundational knowledge in:

  • File systems (e.g., /home vs. /scratch, directory structure)
  • Software installation (e.g., Linux package management, libraries, using compilers)
  • Environment variables (e.g., PATH, LD_LIBRARY_PATH)

If you are unfamiliar with these concepts, we recommend asking for help before you try creating custom modules on your own.

Table of Contents

Why create custom modules?

Custom modules are a convenient way to “load” and “unload” software environments on the HPC cluster, allowing you to switch between different software or versions of software without having to manually modify your environment variables.

For example, loading and unloading the matlab module modifies LD_LIBRARY_PATH and PATH environment variables:

[netid@spark-login ~]$ env | grep -i matlab

[netid@spark-login ~]$ module load matlab

[netid@spark-login ~]$ env | grep -i matlab

__LMOD_REF_COUNT_PATH=/software/chtc/manual/el9/installs/matlab/R2024a:1;/software/chtc/manual/el9/installs/matlab/R2024a/bin/glnxa64:1;/software/chtc/manual/el9/installs/matlab/R2024a/bin:1;/home/netid/spack/bin:1;/home/netid/.local/bin:1;/home/netid/bin:1;/usr/local/bin:1;/usr/bin:1;/usr/local/sbin:1;/usr/sbin:1;/opt/puppetlabs/bin:1
LOADEDMODULES=matlab/R2024a
LD_LIBRARY_PATH=/software/chtc/manual/el9/installs/matlab/R2024a/extern/bin/glnxa64
PATH=/software/chtc/manual/el9/installs/matlab/R2024a:/software/chtc/manual/el9/installs/matlab/R2024a/bin/glnxa64:/software/chtc/manual/el9/installs/matlab/R2024a/bin:/home/netid/spack/bin:/home/netid/.local/bin:/home/netid/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
_LMFILES_=/software/chtc/manual/el9/modules/matlab/R2024a
__LMOD_REF_COUNT_LD_LIBRARY_PATH=/software/chtc/manual/el9/installs/matlab/R2024a/extern/bin/glnxa64:1

[netid@spark-login ~]$ module unload matlab

[netid@spark-login ~]$ env | grep -i matlab

Custom modules may be a good option if you

  • Use different software stacks in different jobs
  • Manage custom installations for a group
  • Want to keep your default environment variables simple and uncluttered

Before you start

Make sure that you already have your software installed on the system.

📝 Follow along

In this guide, we’ll be using Julia as an example.

If you’d like to follow along, use the commands below to install Julia 1.12.1 in your /home directory on spark-login.

cd $HOME
wget https://julialang-s3.julialang.org/bin/linux/x64/1.12/julia-1.12.1-linux-x86_64.tar.gz
tar -xzf julia-1.12.1-linux-x86_64.tar.gz

You should now have a julia-1.12.1 directory in your /home directory.

Create your own modules

Create directories for your module files

In your /home directory, create a directory for your custom module files.

mkdir -p ~/custom_modules

⚠️ Only put module files in this directory!

You can only put module files in this directory. Do not put any other types of files here, or it may break your module setup.

Next, create a subdirectory for the software you want to create a module for. Do not put the version number in the directory name. For example:

mkdir -p ~/custom_modules/julia

Create a custom module file

In the subdirectory for your software, create a new file with a name in the following format:

<version>.lua

For example, for Julia 1.12.1, you should create a file called 1.12.1.lua.

The module file is written in the Lua programming language. The module file contains information about our module and how to load it.

Below is an example module file, 1.12.1.lua:

help([[
Julia 1.12.1
]])
whatis("Name: Julia")
whatis("Version: 1.12.1")
whatis("Keywords: Julia")
whatis("URL: https://julialang.org/")
whatis("Description: The Julia programming language and CLI.")

prepend_path("PATH","/home/NetID/julia-1.12.1/bin")
Module file options What it does
prepend_path("<PATH>","<value>") Prepends <value> to the <PATH> variable. Use this to point to the location of executables, or load libraries into LD_LIBRARY_PATH.
setenv("<variable>", "<value>") Sets the environment variable <variable> to the given <value>
whatis() Provides information about the module to users. Use this to also provide help text or simple instructions.
help([[]]) Printed when a user uses the command module help <module name>

Visit the official documentation on more guidance for writing module files.

Use your custom modules

To use your custom module, you need to tell the module system where to look for your custom module files. Use the module use command to point to the parent directory (not the software directory) for your custom modules.

module use ~/custom_modules

Now when you use module avail, you should see your custom module listed. You can now use module load to load your custom module!

💡 Tip

To save yourself from using the module use command every time you log into the HPC system, append the module use command to your .bashrc file.