Building R in HTC
Overview
This guide will provide instructions for compiling base R from source code and is intended for compiling versions of R that are currently not provided via the CHTC Squid Web Proxy
Before compiling R, please take a moment to review our R Jobs guide to see what versions of R are currently available.
To best understand the below information, you should already have an understanding of how to:
- navigate within directories
- create/copy/move/delete files and directories
- run your intended program
- use HTCondor to submit jobs
This guide details the steps needed to:
- Download R source code
- Submit an interactive job to a build server
- Compile base R from source code
- Install additional R packages
- Create a portable copy of your R installation that can be brought along with your jobs
- Use your custom R installation in your jobs
Compile R
1. Download R Source Code
Before running any commands on CHTC, use a browser to get the source code
for your desired version of R from CRAN
Under “Source Code for all Platforms”, find the R-#.#.#.tar.gz file for your
desired version of R and download it to your computer before transferring a copy
to your /home
directory on the submit server.
2. Submit An Interactive Job To A Build Server
Creating an R installation can be computationally intensive and should not be performed on the submit server. Instead, you will create your installation on a CHTC build server by using an interactive job.
The interactive job is essentially a job without an executable; you will be the one running the commands, instead (in this case, to install R). Like a regular HTCondor job, once you finish your R installation on the build server, the output files (your completed portable R installation) will be transferred back to the submit server (so that you can use the R installation for later jobs).
To submit an interactive job, create a submit file called build.sub
as shown below:
# build.sub
# Software build file
universe = vanilla
log = interactive.log
# In the latest version of HTCondor on CHTC, interactive jobs require an executable.
# If you do not have an existing executable, use a generic linux command like hostname as shown below.
executable = /usr/bin/hostname
# change the name of the file to be the name of your source code
transfer_input_files = R-#.#.#.tar.gz
+IsBuildJob = true
requirements = (OpSysMajorVer =?= 7)
request_cpus = 1
request_memory = 4GB
request_disk = 4GB
queue
Be sure to modify the above transfer_input_files
statement with the appropriate
name of the R source code that you downloaded in the previous ste.
Once this submit file is created, you will start the interactive job by running the following command:
[user@submit]$ condor_submit -i build.sub
The interactive build job should start in about a minute. Once it has started, the job has a time limit of four hours.
3. Compile Base R From Source Code
Once the interactive job starts, R can be compiled. The general workflow will be
to first run the R configuration script followed by make
and make install
.
To install R, first create a directory that will hold your R installation (e.g. R/
)
, then extract all of the source code files from R-#.#.#.tar.gz and move into the source code directory:
[user@build]$ mkdir R/
[user@build]$ tar xzf R-#.#.#.tar.gz
[user@build]$ cd R-#.#.#/
From the source code directory, run the following commands. The make
step may take a while to
complete! If building R version 4.x.x or higher you will need to modify the below configure
command to include the flag --with-pcre1
(e.g. ./configure --prefix=$_CONDOR_SCRATCH_DIR/R --with-pcre1
).
[user@build]$ touch doc/NEWS.2.rds
[user@build]$ touch doc/NEWS.3.rds
[user@build]$ ./configure --prefix=$_CONDOR_SCRATCH_DIR/R
[user@build]$ make
[user@build]$ make install
Then return the the top level directory:
[user@build]$ cd ../
If R has been installed successfully, you should be able to run the following command as a test:
[user@build]$ R/bin/R --version
The output of the above command should match the version number that you just installed!
4. Install Packages
After your R compilation has completed, if any additional R packages are required for your work,
please follow the steps in our R Jobs guide
starting with the export
commands shown in step
1.B.1. After creating a compressed tar
archive of your packages
directory return the the next steps in this guide.
5. Create A Portable Copy of Your R Installation
Once R has been successfully compiled (and after any packages have been installed), the final step is to create a compressed tar archive of your R installation. From the top level directory:
[user@build]$ tar czf my_R#.#.#.tar.gz R/
Be sure to check the size of your R tar archive:
[user@build]$ ls -lh my_R#.#.#.tar.gz
If the size of my_R#.#.#.tar.gz
is larger than ~100MB then your will need to use
the CHTC Squid Wed Proxy to host this file. See our
Squid Web Proxy guide for more details.
Now you have a portable copy of your R installation that can be brought along with your jobs.
The final step is to terminate your interactive job, and HTCondor will tranfer your R tar archive
back to your /home
directory:
[user@submit]$ exit
Once your interactive job terminates, you will be back in your /home
directory on the submit
server and should see a copy of your R tar archive:
[user@submit]$ ls
6. Use Your Custom R Installation In Your Jobs
To use the R installation that was built using the steps in this guide, please follow
the steps and examples described on our R Jobs guide, however
you will need to modify transfer_input_files
in your submit file to specify your R#.#.#.tar.gz
tar archive instead of transferring a copy of R from the Squid web proxy.