tRnslate: _translate chunks or inline R code in source files_

In tRnslate: Translate R Code in Source Files

knitr::opts_chunk$set(echo = TRUE, fig.align = 'center')

Author: Mario A. Martínez Araya
Date: 2021-07-07
Url: https://marioma.me/?i=soft
CRAN: https://cran.r-project.org/package=tRnslate

This R package is intended to translate chunks or inline R code in source files in general, replacing with its output if desired. It was first created to translate scripts for R package generation (to generate R code from R code) and then I used it for something similar within R and Bash scripts necessary for parallel computation using MPI. Then I packaged it independently since it was useful for other purposes as well.

Requirements

In principle any version of R should be useful but I have not tested them all.

Installation

The same than for any other R package. You can download the tar file from CRAN (tRnslate) and then install it using

R CMD INSTALL /path/to/tRnslate_0.0.3.tar.gz

or from R console

install.packages("tRnslate", lib = "path/to/R/library")

How to use it?

1. Templates with R code in chunks and inline

Imagine you have a template file of any kind called template.txt. Let us assume we will write a Bash script for submitting a parallel job using SLURM. Let us read the content from the template which has R code in chunks and inline:

# template with R code
T <- readLines(system.file("examples/template.txt", package = "tRnslate"))

In R we can write the content of the template to console:

cat(T, sep = "\n")

#!/bin/bash
@r # This is a chunk (only assignation)
@r if(.Platform$OS.type=="unix"){
@r     is_q <- system("clu=$(sinfo --version 2>&1) || clu=`echo -1`; echo $clu",intern = TRUE)
@r } else {
@r     is_q <- "-1"
@r }
@r s$intro <- ifelse(is_q=="-1", "<:NULL:>", s$intro)

<r@ s$intro @> --partition=<r@ s$partition @>
<r@ s$intro @> --nodes=<r@ s$nodes @>
<r@ s$intro @> --tasks-per-node=<r@ s$tasks @>
<r@ s$intro @> --mem=<r@ s$memory @>
<r@ s$intro @> --time=<r@ s$time @>
<r@ s$intro @> --nodes=<r@ s$nodes @>

@r # This is a chunk (only assignation)
@r # NOTE: remember, separate chunks with empty lines
@r array <- ifelse(s$array, paste(s$intro," --array=",s$array,sep=""), "")
<r@ array @>

@r # This is another chunk (only assignation)
@r if(.Platform$OS.type=="unix"){
@r     is_mod <- system("mod=$(module --dumpversion 2>&1) || mod=`echo -1`; echo $mod",intern = TRUE)
@r } else {
@r     is_mod <- "-1"
@r }

@r # And this a printing chunk
@r ifelse(is_mod=="-1", "# module environment not found", paste(s$modules))

@r # And this other printing chunk
@r if(is_q=="-1"){
@r     "# no slurm machine"
@r } else {
@r     s$workdir
@r }

@r # And the last chunk (printing)
@r # NOTE: that it also includes inline elements
<r@ system("which mpirun",intern = TRUE)@>/mpirun --mca mpi_warn_on_fork 0 -n <r@ s$nodes * s$tasks @> <r@ R.home("bin") @>/Rscript r-code-script.R

echo "Job submitted on $(date +%F) at $(date +%T)."

2. R code explanation

Lines starting with @r or @R followed by one space or tabular, define chunks of R code that is also interpreted and translated. The chunks of R code can be assignation or output chunks. Assignation chunks are those including <- for assigning an object, while output chunks print R output to the template. Thus several assignation chunks can be placed in adjacent lines, however assignation and output chunks must be separated by one empty line (the same for consecutive output chunks). Alternatively, inline R code can be entered using <r@ code @> or <R@ code @>. Inline R code with assignation does not produce output so is replaced by blank, while inline R code producing output will modify the resulting template.

KEEP IN MIND A FEW RULES

• R code in chunks and inline is evaluated from top to bottom and from left to right in each line.
• Code chunks allow R comments. In the example above I used them to explain a bit the code in the template.

• Code chunks can also be marked only using the opening @r, for example:

  @r # And this other printing chunk
  @  if(is_q=="-1"){
  @      "# no slurm machine"
  @  } else {
  @      s$workdir
  @  }
  

• Do not mix (actions of) assignation with printing in the same chunk. Note, that for an if clause, it is valid that if the condition is true then the chunk action prints and if the condition is false its action assigns.

• Each printing chunk must produce only one output object.
• Loops can be used in chunks but only with assignation (no printing).
• Separate chunks with an empty line.
• Chunks cannot contain inline R code (so far).
• Inline code does not allow comments in it.
• Tend to use inline code mainly for printing (although assignation is allowed).
• In inline code print objects explicitly using paste, print or similar commands. For instance, use <r@ paste("some random string") @> instead of <r@ "some random string" @> which will not be processed similarly as <r@ some random string @>.

3. Environment used to evaluate the R code

The R code in the template needs to be evaluated in an environment which can be specified by the user. This environment can contain objects which are called from the chunks or inline R code in the template. For example, in the previous template the R code calls an object s with several elements (partition, nodes, tasks, etc.) which are being used to replace the content in the template. For this template, to create the environment and object s we can do:

# NOTE: this is the environment that will be used later (see below)
renv <- new.env(parent = parent.frame())
# list with input arguments
renv$s <- list(
    intro = "#SBATCH",
    partition = "hpc01",
    nodes = 4,
    tasks = 10,
    memory = "2gb",
    time = "01:00:00",
    array = FALSE,
    modules = 'module load openmpi/chosen/module R/chosen/module',
    workdir = 'cd ${SLURM_SUBMIT_DIR}'
)

4. The translate_r_code command

Given the template above then we can "translate" its R code using the function translate_r_code as follows:

## Evaluate the R code
TT <- translate_r_code(T, envir = renv)

## See the output
cat(TT, sep="\n")

Resulting source file

Depending on the system where you execute the previous code, the resulting output will vary. For example, for a multicore PC with OpenMPI but without a dynamic environment modules manager such as environment-modules or Lmod and without a job scheduler such as SLURM then the output of cat(TT, sep="\n") will be something like this:

#!/bin/bash

# module environment not found

# no slurm machine

/usr/bin/mpirun/mpirun --mca mpi_warn_on_fork 0 -n 40 /usr/lib/R/bin/Rscript r-code-script.R

echo "Job submitted on $(date +%F) at $(date +%T)."

While for an HPC cluster having OpenMPI, environment-modules and SLURM the "translated" output file will be similar to:

#!/bin/bash

#SBATCH --partition=hpc01
#SBATCH --nodes=4
#SBATCH --tasks-per-node=10
#SBATCH --mem=2gb
#SBATCH --time=01:00:00
#SBATCH --nodes=4

module load openmpi/chosen/module R/chosen/module

cd ${SLURM_SUBMIT_DIR}

/usr/bin/mpirun/mpirun --mca mpi_warn_on_fork 0 -n 40 /usr/lib/R/bin/Rscript r-code-script.R

echo "Job submitted on $(date +%F) at $(date +%T)."

Additional rules could be added to control the lenght of the mpirun line, however as it is it works fine. Other source code can be generated following the same principles described before.

Limitations

As it is, translate_r_code has some limitations such as:

• For printing strings in inline code it is recommended to use always paste. Otherwise it will not evaluate R code.
• Inline code does not allow comments and requires using paste to print its content to template.

• Inline code cannot be placed within chunks of R code (almost certainly it will fail). For example:

  @r ## Chunk A with inline code. It will fail :(
  @r ## because there is still no object called <r@ path @>.
  @r a <- file.path(ifelse(condition, <r@ path @>, getwd()),"file.txt")
  

• Printing chunks must generate a unique output object (either vector, list, array, character, etc.). If multiple objects are being printed, then it will only print the last one or none.

• If a chunk aimed at printing includes an assignation, then the printing will be omitted.

• Tasks must be separated in independent chunks. Most R statements are allowed (for, while, if, {...}, etc.) but with some limitations:

  • Conditionals can be used within chunks but the action must not mix assignation with printing.
  • Loops can be used in chunks but only with assignation (no printing).

• And surely there are others. If you find any errors/bugs, please let me know.

Therefore, use this function carefully.

RECOMMENDATIONS

1. Never replace the content of a template writing the output to the same file.

2. Always check the content of the "translated" output before using it for other tasks.

3. Be cautious.

References

• Writing R Extensions
• Writing Vignettes
• SLURM documentation
• Environment modules documentation
• OpenMPI documentation

tRnslate documentation built on July 13, 2021, 5:08 p.m.