Using Licensed Software
This guide describes when and how to run jobs that use licensed software in CHTC’s high throughput compute (HTC) system.
To best understand the below information, users should already have an understanding of:
- Using the command line to: navigate within directories, create/copy/move/delete files and directories, and run their intended programs (aka "executables").
- The CHTC's Intro to Running HTCondor Jobs
Once you know that you need to use a licensed software program on our HTC system, you will need to do the following:
- View Licensed Software in the HTC System
- Submit Jobs Using Software Modules
A. CHTC's Licensed Software Policies on the HTC System
Our typical practice for software support in CHTC is for users to install and manage their own software installations. We have multiple guides to help users with common software programs and additional support is always available through CHTC's research computing facilitators.
However, certain software programs require paid licenses which can make it challenging for individual users to install the software and use the licenses correctly. As such, we provide support for software installation and use on our high throughput system. Installation of licensed programs is by request to and at the discretion of CHTC staff.
We always recommend using a free or open-source software alternative whenever possible, as certain software licenses restrict the amount of computing that can contribute to your research.
B. Viewing Licensed Software on the HTC System
Software with paid licenses that has been installed on the high throughput (HTC) system is accessible through software "modules", which are tools to access and activate a software installation. To see which software programs are available on the HTC system, run the following command on an HTC submit server:
[alice@submit]$ module avail
Note: you should never run a program directly on the submit server. Jobs that use licensed software/modules should always be submitted as HTCondor jobs as described below.
Note that not all software modules are available to all CHTC users. Some
ansys have a campus or shared license which makes them
available to all CHTC users. Other software, like
abaqus, is licensed to a specific group and is only available to
members of that group.
C. Submitting Jobs Using Licensed Software Modules
The following sections describe how to create a bash script executable and HTCondor submit file to run jobs that use software accessible via the modules.
1. Script For Running Jobs with Modules
To run a job that uses a licensed software installation on the HTC system, you need to write a script that loads the software module and then runs the program, like so:
#!/bin/bash # Commands to enable modules, and then load an appropriate software module export PATH . /etc/profile.d/modules.sh module load software # For Lumerical (the license requires a home directory) export HOME=$_CONDOR_SCRATCH_DIR # Command to run your software from the command line cmd -options input.file
software with the name of the software module you want to use,
found via the
module avail command described above. Replace
the final command with the syntax to run your software, with the
For example, to run a Comsol job, the script might look like this:
#!/bin/bash export PATH . /etc/profile.d/modules.sh module load COMSOL/5.4 comsol batch -inputfile test.mph -outputfile test-results.mph
2. Submit File Requirements
There are several important requirements to consider when writing a submit file for jobs that use our licensed software modules. They are shown in the sample submit file below and include:
Require access to the modules. To ensure that your job will have access to CHTC software modules you must include the following in your submit file.
requirements = (HasChtcSoftware == true)
Add a concurrency limit. For software with limited licenses, we have implemented concurrency limits, which control the number of jobs running at once in the HTC system. If your software is in the table below, use the concurrency limit name in your submit file like this:
concurrency_limits = LIMIT_NAME:num_of_licenses_used
Software Limit Name Limit ABAQUS ABAQUS 50 ANSYS ANSYS_RESEARCH 20 COMSOL (Physics) COMSOL_PHYSICS 2 Lumerical LUMERICAL 3
So if you were planning to run a job that used one ANSYS license, you would use:
concurrency_limits = ANSYS_RESEARCH:1
- Request accurate CPUs and memory. Run at least one test job and look at the log file produced by HTCondor to determine how much memory and disk space your jobs actually use. We recommend requesting the smallest number of CPUs where your job will finish in 1-2 days.
- The script you wrote above (shown as
run_job.shbelow) should be your submit file "executable", and any input files should be listed in transfer_input_files.
A sample submit file is given below:
# software.sub # A sample submit file for running a single job using software modules universe = vanilla log = job_$(Cluster).log output = job_$(Cluster).out error = job_$(Cluster).err # the executable should be the script you wrote above executable = run_job.sh # arguments = (if you want to pass any to the shell script) should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = (this should be a comma separate list of input files if needed) # Requirement for accessing new set of software modules requirements = ( HasChtcSoftware == true ) # If required, add the concurrency limit for your software and uncomment # concurrency_limits = LIMIT_NAME:num_of_licenses_used request_cpus = 1 request_memory = 2GB request_disk = 2GB queue
After the submit file is complete, you can submit your jobs using