Running Julia Jobs

Quickstart: Julia

Option A (recommended)

Build a container with Julia & packages installed inside:

Option B

Use a portable copy of Julia and create your own portable copy of your Julia packages:

Follow the instructions in our guide Run Julia Jobs.

This approach may be sensitive to the operating system of the execution point. We recommend building a container instead, but are keeping these instructions as a backup.

More information

No CHTC machine has Julia pre-installed, so you must configure a portable copy of Julia to work on the HTC system. Using a container as described above is the easiest way to accomplish this.

Executable

When using a container, you can use a .jl script as the submit file executable, provided that the first line (the “shebang”) in the .jl file is

#!/usr/bin/env julia

with the rest of the file containing the commands you want to run using Julia.

Alternatively, you can use a bash .sh script as the submit file executable, and in that file you can use the julia command:

#!/bin/bash

julia my-script.jl

In this case, remember to include your .jl file in the transfer_input_files line of your submit file.

Arguments

For more information on passing arguments to a Julia script, see the Julia documentation.

Option B: Create your own portable copy

Use a portable copy of Julia and create your own portable copy of your Julia packages

This approach may be sensitive to the operating system of the execution point. We recommend building a container instead, but are keeping these instructions as a backup.

Download the precompiled Julia software from https://julialang.org/downloads/. You will need the 64-bit, tarball compiled for general use on a Linux x86 system. The file name will resemble something like julia-#.#.#-linux-x86_64.tar.gz.
- Tip: use wget to download directly to your /home directory on the submit server, OR use transfer_input_files = url in your HTCondor submit files.
Submit an “interactive build” job to create a Julia project and install packages, else skip to the next step.
- For more details, see the section on installing Julia packages below: Installing Julia Packages

Submit a job that executes a Julia script using the Julia precompiled binary with base Julia and Standard Library.

 #!/bin/bash

 # extract Julia binaries tarball
 tar -xzf julia-#.#.#-linux-x86_64.tar.gz

 # add Julia binary to PATH
 export PATH=$_CONDOR_SCRATCH_DIR/julia-#.#.#/bin:$PATH

 # run Julia script
 julia my-script.jl

For more details on the job submission, see the section below: Submit Julia Jobs

Install Julia Packages

If your work requires additional Julia packages, you will need to peform a one-time installation of these packages within a Julia project. A copy of the project can then be saved for use in subsequent job submissions. For more details, please see Julia’s documentation at Julia Pkg.jl.

Create An Interactive Build Job Submit File

To install your Julia packages, first create an HTCondor submit for submitting an “interactive build” job which is a job that will run interactively on one of CHTC’s servers dedicated for building (aka compiling) software.

Using a text editor, create the following file, which can be named build.sub

# Julia build job submit file

universe = vanilla
log = julia-build.log

# In the latest version of HTCondor on CHTC, interactive jobs require an executable.
# If you do not have an existing executable, use a generic linux command like hostname as shown below.
executable = /usr/bin/hostname

# have job transfer a copy of precompiled Julia software
# be sure to match the name of the version 
# that you have downloaded to your home directory
transfer_input_files = julia-#.#.#-linux-x86_64.tar.gz

+IsBuildJob = true

request_cpus = 1
request_memory = 4GB
request_disk = 2GB

queue

The only thing you should need to change in the above file is the name of the Julia tarball file in the "transfer_input_files" line.

Submit Your Interactive Build Job

Once this submit file is created, submit the job using the following command:

[alice@submit]$ condor_submit -i build.sub

It may take a few minutes for the build job to start.

Install Julia Packages Interactively

Once the interactive jobs starts you should see the following inside the job’s working directory:

bash-4.2$ ls -F
julia-#.#.#-linux-x86_64.tar.gz   tmp/    var/

Run the following commands to extract the Julia software and add Julia to your PATH:

bash-4.2$ tar -xzf julia-#.#.#-linux-x86_64.tar.gz
bash-4.2$ export PATH=$_CONDOR_SCRATCH_DIR/julia-#.#.#/bin:$PATH

After these steps, you should be able to run Julia from the command line, e.g.

julia --version

Now create a project directory to install your packages (we’ve called it my-project/ below) and tell Julia its name:

bash-4.2$ mkdir my-project
bash-4.2$ export JULIA_DEPOT_PATH=$PWD/my-project

You can choose whatever name to use for this directory -- if you have different projects that you use for different jobs, you could use a more descriptive name than “my-project”.

We will now use Julia to install any needed packages to the project directory we created in the previous step.

Open Julia with the --project option set to the project directory:

bash-4.2$ julia --project=my-project

Once you’ve started up the Julia REPL (interpreter), start the Pkg REPL, used to install packages, by typing ]. Then install and test packages by using Julia’s add Package syntax.

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.5 (2019-09-09)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> ]
(my-project) pkg> add Package
(my-project) pkg> test Package

If you have multiple packages to install they can be combined into a single command, e.g. (my-project) pkg> add Package1 Package2 Package3.

If you encounter issues getting packages to install successfully, please contact us at chtc@cs.wisc.edu.

Once you are done, you can exit the Pkg REPL by typing the Delete key and then exit()

(my-project) pkg> 
julia> exit()

Save Installed Packages For Later Jobs

To use this project, and the associated installed packages, in subsequent jobs, we need to have HTCondor return some files to the submit server by converting the my-project/ directory to a tarball, before exiting the interactive job session:

bash-4.2$ tar -czf my-project.tar.gz my-project/
bash-4.2$ exit

After the job exits, you will be returned to your /home directory on the submit server (specifically where ever you were located when you submitted the interactive build job). A copy of packages.tar.gz will be present. Be sure to check the size of the project tarball before proceeding to subsequent job submissions. If the file is >100MB please contact us at chtc@cs.wisc.edu so that we can get you setup with access to our SQUID web proxy. More details are available on our SQUID guide: File Availability with SQUID

[alice@submit]$ ls 
build.sub     julia-#.#.#-linux-x86_64.tar.gz   julia-build.log
my-project.tar.gz
[alice@submit]$ ls -sh my-project.tar.gz

Submit Julia Jobs

To submit a job that runs a Julia script, create a bash script and HTCondor submit file following the examples in this section. These examples assume that you have downloaded a copy of Julia for Linux as a tar.gz file and if using packages, you have gone through the steps above to install them and create an additional tar.gz file of the installed packages.

Create Executable Bash Script

Your job will use a bash script as the HTCondor executable. This script will contain all the steps needed to unpack the Julia binaries and execute your Julia script (script.jl). Below are two example bash script, one which can be used to execute a script with base Julia, and one that will use packages installed in Julia project (see Install Julia Packages).

Example Bash Script For Base Julia Only

If your Julia script can run without additional packages (other than base Julia and the Julia Standard library) use the example script directly below.

#!/bin/bash

# julia-job.sh

# extract Julia tar.gz file
tar -xzf julia-#.#.#-linux-x86_64.tar.gz

# add Julia binary to PATH
export PATH=$_CONDOR_SCRATCH_DIR/julia-#.#.#/bin:$PATH

# run Julia script
julia script.jl

Example Bash Script For Julia With Installed Packages

#!/bin/bash

# julia-job.sh

# extract Julia tar.gz file and project tar.gz file
tar -xzf julia-#.#.#-linux-x86_64.tar.gz
tar -xzf my-project.tar.gz

# add Julia binary to PATH
export PATH=$_CONDOR_SCRATCH_DIR/julia-#.#.#/bin:$PATH
# add Julia packages to DEPOT variable
export JULIA_DEPOT_PATH=$_CONDOR_SCRATCH_DIR/my-project

# run Julia script
julia --project=my-project script.jl

Create HTCondor Submit File

After creating a bash script to run Julia, then create a submit file to submit the job to run.

More details about setting up a submit file, including a submit file template, can be found in our hello world example page at Run Your First CHTC Jobs.

# julia-job.sub

universe = vanilla

log = job_$(Cluster).log
error = job_$(Cluster)_$(Process).err
output = job_$(Cluster)_$(Process).out

executable = julia-job.sh

should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = julia-#.#.#-linux-x86_64.tar.gz, script.jl

request_cpus = 1
request_memory = 2GB
request_disk = 2GB

queue 1

If your Julia script needs to use packages installed for a project, be sure to include my-project.tar.gz as in input file in julia-job.sub. For project tar.gz files that are <100MB, you can follow the below example:

transfer_input_files = julia-#.#.#-linux-x86_64.tar.gz, script.jl, my-project.tar.gz

For project tar.gz files that are larger than 100MB, email a facilitator about using SQUID.

Modify the CPU/memory request lines to match what is needed by the job. Test a few jobs for disk space/memory usage in order to make sure your requests for a large batch are accurate! Disk space and memory usage can be found in the log file after the job completes.

Submit Your Julia Job

Once you have created an executable bash script and submit file, you can submit the job to run using the following command:

[alice@submit]$ condor_submit julia-job.sub