Basic Scripting and Job Submission with Arguments
Table of Contents
Purpose
Many executables require arguments to perform tasks. This page shows you basic examples of how different programming languages take arguments, and how wrapper scripts can be written to pass arguments.
What is an argument?
In a computational workflow, how do you tell the computer what script to run, what data file to read, or which parameters to use? These inputs are typically passed as arguments, which your program is configured to read.
For example, on the command line, the sleep program takes in one argument, a non-negative number, then pauses for that number of seconds. If we run the following code,
sleep 60
the computer pauses for 60 seconds. In this example:
sleepis the executable.60is the argument.
What is a wrapper script, and why should I write one?
While the above example is simple, what if you need something more complex, like a workflow? Your workflow might need some pre- or post-processing, if/else statements, or iterations.
Wrapper scripts are a way to package simple computational workflows in one executable script, allowing computations to be run in noninteractive batches.
Using arguments in different programming languages
Let’s see how different programming languages might take an argument. Each tab contains different expressions of simple program, echo-next, that prints our next argument (we’ll use data.csv) to the terminal. Example wrapper scripts and HTCondor submit files are also included.
Code
Our executable written in shell, echo-next.sh:
#!/bin/bash
echo $1
We can use it on the command line after changing the file permissions to executable:
[user@login]$ chmod +x echo-next.sh
[user@login]$ ./echo-next.sh data.csv
data.csv
In bash,
$1references the first argument to a script. If you have more arguments,$2will refer to the second,$3to the third, etc. There are also ways to refer to the whole list of arguments if needed.
Wrapper script
We can now write a wrapper script! This script checks that one argument is passed, and also allows room for pre- and post-processing in this simple workflow.
Our wrapper script written in bash, wrapper.sh:
#!/bin/bash
# Check if an argument is provided
if [ $# -ne 1 ]; then
echo 'Please use one argument. Usage: wrapper.sh [arg]'
exit 1
fi
# Set filename variable to the first argument to the bash script
filename=$1
echo 'Pre-processing could go here.'
# Run code
./echo-next.sh ${filename}
echo 'Post-processing could go here.'
Ensure wrapper.sh is executable with chmod +x wrapper.sh before running it on the command line:
[user@login]$ chmod +x wrapper.sh
[user@login]$ ./wrapper.sh
Please use one argument. Usage: wrapper.sh [arg]
[user@login]$ ./wrapper.sh data.csv
Pre-processing could go here
data.csv
Post-processing could go here
Passing arguments with the HTCondor submit file
Now that we’ve understood how arguments work in shell script and how to write simple wrapper scripts that pass those arguments, we can pass arguments in HTCondor’s submit file by specifying the arguments attribute, as shown in this excerpt:
# Create custom variable called "data"
data = data.csv
# Specify the executable & arguments
executable = wrapper.sh
arguments = $(data)
Next steps
Practice with a more complex example, including submitting multiple jobs with different arguments: Practice: Passing Arguments from the Submit File to the Executable Script