Install Software Using Spack

CHTC uses Spack (https://github.com/spack/spack) for installing and managing software packages on the HPC cluster for all users to use, via the module command. Recently, Spack has developed a feature that allows for users to integrate their local installation of Spack with the system-wide installation. This means that when a user installs software with their local installation of Spack, they can automatically incorporate the system-wide packages to satisfy their software’s dependencies (similar to Conda and Miniconda).

This guide describes how to install and manage software using Spack, including how to install and use a specific compiler.

This guide assumes you or your group has already set up your local installation of Spack. If you have not installed Spack, follow the instructions in Setting Up Spack on HPC.

Contents

  1. Installing Software Using Spack
  2. Using Software Installed in Spack
  3. Installing and Using a Specific Compiler

1. Installing Software Using Spack

Once your local installation of Spack has been properly configured, you are now ready to install software using Spack.

Check the documentation for the program you want to install to see if they have instructions for installation using Spack. Even if your program can’t be installed using Spack, you can still use it to install the dependencies that your program needs.

Note: For a group installation of Spack, you will not be able to modify or remove the packages installed by a different user. We recommend that you consult with the rest of your group for permission before proceeding.

A. Start an Interactive Job

Before creating a Spack environment or installing packages using Spack, first start an interactive Slurm job:

srun --mpi=pmix -n4 -N1 -t 240 -p int --pty bash

For more information on interactive Slurm jobs, see our guide Submitting and Managing Jobs Using SLURM.

When creating an environment, Spack automatically detects the hardware of the machine being used at the time and configures the packages as such. Since the login server uses newer hardware than the execute servers, creating an environment on the login server (not in an interactive job) is a bad idea.

B. Creating and Using a Spack Environment

Software installations with Spack should be done inside of a Spack environment, to help manage the shell and the paths to access programs and libraries needed for a particular software installation.

To create a Spack environment, run the command

spack env create yourEnvironmentName

where you should replace yourEnvironmentName with your desired name for the environment. You can then activate the environment with

spack env activate yourEnvironmentName

You will need to activate the environment when you wish to use the software that was installed in that environment.

You can see a list of your available environments using

spack env list

and you can see which environment you are currently using with

spack env status

To deactivate the environment, run

spack env deactivate

or close the terminal session.

C. Finding Program Packages in Spack

Once inside an active Spack environment, you can run the following command to see what packages are installed in the current environment

spack find

For a new environment, this will show that there are no programs installed. The output of this command will update after you install program packages in the environment.

To search for packages to install using Spack, use the command

spack list nameOfProgram

where you should replace nameOfProgram with the program that you are interested in finding. Spack will search for the package and print out a list of all the packages that match that name. (The first time you run this command may take several minutes while Spack downloads a current list of packages that can be installed.)

To learn more about an available package, use the exact name of the program and run

spack info exactNameOfProgram

This will print out information about the program, including a short description of the program, a link to the developer’s website, and the available versions of the program and its dependencies.

D. Adding Package Specifications to the Environment

Once you find the packages that you want to install, add their specifications to the environment using

spack add exactNameOfProgram

Spack will automatically decide which version of the program to use at installation time based on the other packages that you’ve added.

If you want a specific version of a package, you can specify it by appending @= to the end of the package name, followed by the version number. For example,

spack add python@=3.10

will tell the environment that you want to install version 3.10 of Python. There are additional ways of defining specifications for package versions, the compiler to be used, and dependencies. The documentation for Spack provides the details on how this is done.

If you need to install a compiler, or need to use a specific compiler to install the desired packages, see section 3. Installing and Using a Specific Compiler.

E. Installing Packages in an Environment

Once you have identified the package(s) you would like to install and have added the specifications to your environment,

i. Create the local scratch directory

Using the default configuration from Setting Up Spack on HPC, Spack will try to use the machine’s local disk space for staging and compiling files before transferring the finished results to the final installation directory. Using this space will greatly improve the speed of the installation process. Create the local directory with the command

mkdir /local/yourNetID/spack_build

where you should replace yourNetID with your NetID. At the end of the session, remember to delete this directory so that other people can use the disk space in their jobs.

If the directory already exists, that means you forgot to remove it after one of your previous Spack installation sessions. Simply remove the directory and make it again.

rm -rf /local/yourNetID/spack_build
mkdir /local/yourNetID/spack_build

ii. Check the programs/packages to be installed

If you’ve added the installation specifications to the environment, then you can check the installation plan using the command

spack spec -lI

(the first letter after the hyphen is a lowercase “L” and the second letter is an uppercase “i”).

This command identifies what dependencies Spack needs in order to install your desired packages along with how it will obtain them. Assuming their are no problems, then it will print a list of the packages and their dependencies, where entries that begin with a green [+] have already been installed somewhere in your local Spack installation, while those that begin with a green [^] are referencing the system installation, and those beginning with a gray - will need to be downloaded and installed.

Most users should see a bunch of packages with a green [^] in the first column. If you do not, then there are several possible explanations:

If you are satisfied with the results, then you can proceed to install the programs.

iii. Install the environment packages

Assuming that you are in an interactive Slurm session and have activated the desired environment containing the package specifications, you can run

spack install -j 4

to install the packages inside of the Spack environment, where the number that comes after -j needs to match the number that you noted from when you started the interactive session (the one after -n when you ran the srun command for the interactive session). You can also add the -v option to have the installation be verbose, which will cause Spack to print the compile and make outputs in addition to the standard Spack output.

Depending on the number and complexity of the programs you are installing, and how much can be bootstrapped from the system installation, the installation step can take anywhere from several minutes to several hours.

If something goes wrong or your connection is interrupted, the installation process can be resumed at a later time without having to start over from the beginning. Make sure that you are in an interactive Slurm session and that you have activated the Spack environment, then simply rerun the spack install command again.

iv. Finishing the installation

After the installation has successfully finished, you should be able to see that the programs have been installed by running

spack find

which should list the programs under the compiler heading used for installing the programs.

You may need to deactivate and reactivate the environment in order to properly use the programs that have been installed.

spack env deactivate
spack env activate yourEnvironmentName

Once you are satisfied that the programs have been installed properly, you can remove packages that are build-only (not used for running the packages you installed) using the command

spack gc

Finally, remove the local build directory that Spack used during the installation with

rm -rf /local/yourNetID/spack_build

and then enter exit to end the interactive session.

To use the packages that you installed, follow the instructions in the next section, 2. Using Software Installed in Spack. If you want to create custom modules using the installed packages, see our guide Creating Custom Modules Using Spack.

F. Removing an Environment and Uninstalling Unneeded Packages

You may find it necessary to remove a Spack environment, or packages installed using Spack. To uninstall a package, simply run

spack uninstall yourPackageName

where you should replace yourPackageName with the name of the package that you want to remove. This command will only work for packages that you ‘added’ to the Spack environment, as described above.

To remove an environment, first make sure that you have deactivated the environment with

spack env deactivate

and then run

spack env rm yourEnvironmentName

where you should replace yourEnvironmentName with the name of the environment that you want to remove. Note that this will not necessarily remove the packages that were installed in the environment! After the environment has been removed, you can uninstall the packages that are no longer needed using the command

spack gc

2. Using Software Installed in Spack

If your account is configured correctly for using Spack, and the software has been installed inside of a Spack environment, then to use the software all you need to do is activate the corresponding environment. Simply use the command

spack env activate yourEnvironmentName

and Spack will update your shell accordingly. (Remember that you can see the available Spack environments by running the command spack env list). Once the environment has been activated, you should be able to use the packages just as normal. You can confirm you are using a command installed using Spack by running

which nameOfYourCommand

where you replace nameOfYourCommand with the name of the command. The command will output a path, and you should see something like spack/var/spack/environments/yourEnvironmentName/ in that path.

For submitting jobs using Slurm, you will need to make sure that you activate the Spack environment near the beginning of your sbatch file before the srun command. For example,

#!/bin/sh
# This file is called submit-script.sh
#SBATCH --partition=shared       # default "shared", if not specified
#SBATCH --time=0-04:30:00        # run time in days-hh:mm:ss
#SBATCH --nodes=1                # require 1 nodes
#SBATCH --ntasks-per-node=64     # cpus per node (by default, "ntasks"="cpus")
#SBATCH --mem-per-cpu=4000       # RAM per node in megabytes
#SBATCH --error=job.%J.err
#SBATCH --output=job.%J.out

# v---Remember to activate your Spack environment!! 
spack env activate yourEnvironmentName

srun --mpi=pmix -n 64 /home/username/mpiprogram

When you submit this job to Slurm and it executes the commands in the sbatch file, it will first activate the Spack environment, and then your program will be run using the programs that are installed inside that environment.

Some programs include explicit module load commands in their execution, which may override the paths provided by the Spack environment. If your program appears to use the system versions of the packages instead of the versions installed in your Spack environment, you may need to remove or modify these explicit commands. Consult your program’s documentation for how to do so. You may want to create your own custom modules and modify your program to explicitly load your custom modules. See Creating Custom Modules Using Spack for more information on how to create your own modules using Spack.

3. Installing and Using a Specific Compiler

By default, Spack will attempt to compile packages it installs using one of the system compilers, most likely with GCC version 11.3.0. Some programs, however, may need to be compiled using a specific compiler, or require that their dependencies be built using a specific compiler. While this is possible using Spack, the process for installing and using a compiler is a bit more complicated than that for installing “regular” packages as was described above.

In brief, you will first create a separate environment for installing the compiler. Then you will add that compiler to the list of available compilers that Spack can use. Finally, you can install your desired packages as in a new environment, but you will need to specify which compiler to use.

A. Install the Compiler in its Own Environment

i. Identify the compiler and version

The first step is to identify the compiler and version you need for your program. Consult your program’s documentation for the requirements that it has. Then follow the instructions in C. Finding Program Packages in Spack to find the package name and confirm the version is available.

ii. Create the compiler’s environment

Next, create and activate an environment for installing the desired compiler. For example,

spack env create compilerName_compilerVersion
spack env activate compilerName_compilerVersion

where you should replace compilerName and compilerVersion with the name and version of the desired compiler.

iii. Add the compiler specification to its environment

Once you’ve activated the environment, add the exact specification for the compiler to the Spack environment with

spack add compilerName@=compilerVersion

where you need to replace compilerName and compilerVersion with the name and version of the compiler that you identified above.

iv. Install the compiler in its environment

Next, follow the instructions in E. Installing Packages in an Environment to install the desired compiler in this environment. Installing the compiler may take several hours, so consider increasing the number of threads to speed up the installation.

B. Add the Compiler to Spack

i. Identify the compiler’s installation path

After installing the compiler, you need to find its location. First, activate the compiler’s environment with spack env activate compilerName_compilerVersion. Next, use the following command to save the path to the compiler as the shell variable compilerPath:

compilerPath=$(spack location -i compilerName@=compilerVersion)

where you need to replace compilerName and compilerVersion with the name and version of the compiler that you installed. You can see print out the path using the command echo $compilerPath.

ii. Give the compiler’s path to Spack

Now that you know where the compiler is installed, deactivate the environment with spack env deactivate. Then run the following command to tell Spack to add the compiler to its list of available compilers:

spack compiler add $compilerPath

iii. Confirm compiler has been added to Spack

The command

spack compiler list

will print out the list of compilers that Spack can use, and should now show compilerName@compilerVersion in the results.

C. Install Packages Using the New Compiler

Once the compiler has been installed and recognized by Spack, you can now create and activate a new environment for installing your desired packages, following the instructions in Installing Software Using Spack.

To make sure the packages are installed using your desired compiler, you need to include the compiler when you add the package specification to the environment (D. Adding Package Specifications to the Environment). To include the compiler in the specification, you need to add the symbol % followed by the compiler name and version to the end of the spack add command. For example,

spack add python@=3.10 %gcc@=9.5.0

will use GCC version 9.5.0 to compile Python 3.10 when installing the package. As a general rule, you should use the same compiler for installing all of your packages within an environment, unless your program’s installation instructions say otherwise.