Running Julia Jobs on CHTC
This guide documents steps for submitting jobs that will execute Julia code, including preliminary steps for installing specific Julia packages that may be required for your work.
Table of Contents
Download the precompiled Julia software from https://julialang.org/downloads/. You will need the 64-bit, tarball compiled for general use on a Linux x86 system. The file name will resemble something like
- Tip: use
wgetto download directly to your
/homedirectory on the submit server, OR use
transfer_input_files = urlin your HTCondor submit files.
- Tip: use
Submit an “interactive build” job to create a Julia project and install packages, else skip to the next step.
- For more details, see the section on installing Julia packages below: Installing Julia Packages
Submit a job that executes a Julia script using the Julia precompiled binary with base Julia and Standard Library.
#!/bin/bash # extract Julia binaries tarball tar -xzf julia-#.#.#-linux-x86_64.tar.gz # add Julia binary to PATH export PATH=$_CONDOR_SCRATCH_DIR/julia-#.#.#/bin:$PATH # run Julia script julia my-script.jl
- For more details on the job submission, see the section below: Submit Julia Jobs
Install Julia Packages
If your work requires additional Julia packages, you will need to peform a one-time installation of these packages within a Julia project. A copy of the project can then be saved for use in subsequent job submissions. For more details, please see Julia’s documentation at Julia Pkg.jl.
Create An Interactive Build Job Submit File
To install your Julia packages, first create an HTCondor submit for submitting an “interactive build” job which is a job that will run interactively on one of CHTC’s servers dedicated for building (aka compiling) software.
Using a text editor, create the following file, which can be named
# Julia build job submit file universe = vanilla log = julia-build.log # have job transfer a copy of precompiled Julia software # be sure to match the name of the version # that you have downloaded to your home directory transfer_input_files = julia-#.#.#-linux-x86_64.tar.gz +IsBuildJob = true request_cpus = 1 request_memory = 4GB request_disk = 2GB queue
The only thing you should need to change in the above file is the name of the Julia tarball file in the "transfer_input_files" line.
Submit Your Interactive Build Job
Once this submit file is created, submit the job using the following command:
[alice@submit]$ condor_submit -i build.sub
It may take a few minutes for the build job to start.
Install Julia Packages Interactively
Once the interactive jobs starts you should see the following inside the job’s working directory:
bash-4.2$ ls -F julia-#.#.#-linux-x86_64.tar.gz tmp/ var/
Run the following commands
to extract the Julia software and add Julia to your
bash-4.2$ tar -xzf julia-#.#.#-linux-x86_64.tar.gz bash-4.2$ export PATH=$_CONDOR_SCRATCH_DIR/julia-#.#.#/bin:$PATH
After these steps, you should be able to run Julia from the command line, e.g.
Now create a project directory to install your packages (we’ve called
my-project/ below) and tell Julia its name:
bash-4.2$ mkdir my-project bash-4.2$ export JULIA_DEPOT_PATH=$PWD/my-project
You can choose whatever name to use for this directory -- if you have different projects that you use for different jobs, you could use a more descriptive name than “my-project”.
We will now use Julia to install any needed packages to the project directory we created in the previous step.
Open Julia with the
--project option set to the project directory:
bash-4.2$ julia --project=my-project
Once you’ve started up the Julia REPL (interpreter), start the Pkg REPL, used to
install packages, by typing
]. Then install and test packages by using
add Package syntax.
_ _ _ _(_)_ | Documentation: https://docs.julialang.org (_) | (_) (_) | _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 1.0.5 (2019-09-09) _/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release |__/ | julia> ] (my-project) pkg> add Package (my-project) pkg> test Package
If you have multiple packages to install they can be combined
into a single command, e.g.
(my-project) pkg> add Package1 Package2 Package3.
If you encounter issues getting packages to install successfully, please contact us at firstname.lastname@example.org.
Once you are done, you can exit the Pkg REPL by typing the Delete key and then
(my-project) pkg> julia> exit()
Save Installed Packages For Later Jobs
To use this project, and the associated installed packages, in
subsequent jobs, we need to have HTCondor return some files to
the submit server by converting the
to a tarball, before exiting the interactive job session:
bash-4.2$ tar -czf my-project.tar.gz my-project/ bash-4.2$ exit
After the job exits, you will be returned to your
/home directory on the
submit server (specifically where ever you were located when you submitted
the interactive build job). A copy of
packages.tar.gz will be present. Be
sure to check the size of the project tarball before proceeding to subsequent job
submissions. If the file is >100MB please contact us at email@example.com so
that we can get you setup with access to our SQUID web proxy. More details
are available on our SQUID guide: File Availability with SQUID
[alice@submit]$ ls build.sub julia-#.#.#-linux-x86_64.tar.gz julia-build.log my-project.tar.gz [alice@submit]$ ls -sh my-project.tar.gz
Submit Julia Jobs
To submit a job that runs a Julia script, create a bash
script and HTCondor submit file following the examples in this section.
These examples assume that you have downloaded a copy of Julia for Linux as a
file and if using packages, you have gone through the steps above to install them
and create an additional
tar.gz file of the installed packages.
Create Executable Bash Script
Your job will use a bash script as the HTCondor
executable. This script
will contain all the steps needed to unpack the Julia binaries and
execute your Julia script (
script.jl). Below are two example bash script,
one which can be used to execute a script with base Julia, and one that
will use packages installed in Julia project (see Install Julia Packages).
Example Bash Script For Base Julia Only
If your Julia script can run without additional packages (other than base Julia and the Julia Standard library) use the example script directly below.
#!/bin/bash # julia-job.sh # extract Julia tar.gz file tar -xzf julia-#.#.#-linux-x86_64.tar.gz # add Julia binary to PATH export PATH=$_CONDOR_SCRATCH_DIR/julia-#.#.#/bin:$PATH # run Julia script julia script.jl
Example Bash Script For Julia With Installed Packages
#!/bin/bash # julia-job.sh # extract Julia tar.gz file and project tar.gz file tar -xzf julia-#.#.#-linux-x86_64.tar.gz tar -xzf my-project.tar.gz # add Julia binary to PATH export PATH=$_CONDOR_SCRATCH_DIR/julia-#.#.#/bin:$PATH # add Julia packages to DEPOT variable export JULIA_DEPOT_PATH=$_CONDOR_SCRATCH_DIR/my-project # run Julia script julia --project=my-project script.jl
Create HTCondor Submit File
After creating a bash script to run Julia, then create a submit file to submit the job to run.
More details about setting up a submit file, including a submit file template, can be found in our hello world example page at Run Your First CHTC Jobs.
# julia-job.sub universe = vanilla log = job_$(Cluster).log error = job_$(Cluster)_$(Process).err output = job_$(Cluster)_$(Process).out executable = julia-job.sh should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = julia-#.#.#-linux-x86_64.tar.gz, script.jl request_cpus = 1 request_memory = 2GB request_disk = 2GB queue 1
If your Julia script needs to use packages installed for a project,
be sure to include
my-project.tar.gz as in input file in
For project tar.gz files that are <100MB, you can follow the below example:
transfer_input_files = julia-#.#.#-linux-x86_64.tar.gz, script.jl, my-project.tar.gz
For project tar.gz files that are larger than 100MB, email a facilitator about using SQUID.
Modify the CPU/memory request lines to match what is needed by the job. Test a few jobs for disk space/memory usage in order to make sure your requests for a large batch are accurate! Disk space and memory usage can be found in the log file after the job completes.
Submit Your Julia Job
Once you have created an executable bash script and submit file, you can submit the job to run using the following command:
[alice@submit]$ condor_submit julia-job.sub