In AntonelliLab/outsider.devtools: Build 'outsider' Modules

In basics we introduced the essential elements that make up a module and then built a super simple module.

On this page, we will build a module for a slightly more complex program while introducing some more functionalities of outsider.

figlet

For this walkthrough, we will be creating a slightly more sophisticated module than om..echo by building a module for figlet, a program that takes textual input and returns ASCII art.

The program itself only takes text input and returns text output. We, however, will use outsider to create a module that can either take an input file or text and then return ASCII art to either the console or to an output file.

Not all command-line programs follow the same standards (different input/output file arguments, different help pages ...), so outsider comes with a series of functions to provide options for developers for how best they want to encapsulate the program in module form.

(For reference, this module does already exist in the wild: dombennett/om..figlet)

Walkthrough

Building

To get started, we will first need our module skeleton.

library(outsider.devtools)
flpth <- module_skeleton(repo_user = 'dombennett', program_name = 'figlet',
                         docker_user = 'dombennett', flpth = tempdir(),
                         service = 'github')
# folder name where module is stored
print(basename(flpth))

## [1] "om..figlet"

Next we will need to develop the Dockerfile! figlet is easy enough to install on Ubuntu Linux, we can just use the "apt-get" system to install it. (Here I'm editing the file from within R to make the process reproducible, you can edit the file directly with a text editor.)

dockerfile_text <- "
# select Ubuntu Linux image
FROM ubuntu:latest
# Use the 'apt-get' system to install figlet
RUN apt-get update && apt-get install -y figlet
# set the working_dir/
RUN mkdir /working_dir
WORKDIR /working_dir
"
# write to latest Dockerfile
cat(dockerfile_text, file = file.path(flpth, 'inst', 'dockerfiles', 'latest',
                                      'Dockerfile'))

Now, we have our Dockerfile set-up and our module skeleton, we must develop the R code. The skeleton code for our figlet-module is currently:

#' @name figlet
#' @title figlet
#' @description Run figlet
#' @param ... Arguments
#' @example /examples/example.R
#' @export
figlet <- function(...) {
  # convert the ... into a argument list
  arglist <- arglist_get(...)
  # create an outsider object: describe the arguments and program
  otsdr <- outsider_init(pkgnm = 'om..figlet', cmd = 'figlet',
                         arglist = arglist)
  # run the command
  run(otsdr)
}

The above function would work fine. It would pass any arguments from the user in R to the figlet program. But because figlet does not come with input/output arguments, we will need to hardcode these into our function and use some if statements and bash to create this functionality. Because this will require calling several commands via the terminal, we can create a shell script that we can then pass through outsider to our container.

function_text <- "
#' @name figlet
#' @title figlet
#' @description Run figlet with input and output files.
#' @param arglist Arguments for figlet
#' @param input_file Text input file
#' @param output_file ASCII art output file
#' @details If no input_file, will use arglist argument.
#' If no output_file, will print to console.
#' @example /examples/example.R
#' @export
figlet <- function(arglist = arglist_get(...), input_file = NULL,
                   output_file = NULL) {
  wd <- NULL
  # construct shell script from arglist
  arglist <- c('figlet', arglist)
  if (!is.null(input_file)) {
      # cat the input_file contents to figlet
      # (basename is used because on the container,
      #  filepaths cannot be used.)
      arglist <- c('cat', basename(input_file), '|', arglist)
  }
  if (!is.null(output_file)) {
      # write out the results of figlet to output_file
      arglist <- c(arglist, '>', basename(output_file))
      # determine where returned files should be sent
      wd <- dirpath_get(output_file)
  }
  # write arglist to temp file
  script <- file.path(tempdir(), 'script.sh')
  on.exit(file.remove(script))
  # ensure script is written in binary format
  script_cnntn <- file(script, 'wb')
  cmds <- paste(arglist, collapse = ' ')
  # debug print
  print(cmds)
  write(x = cmds, file = script_cnntn)
  close(script_cnntn)
  # initialise outsider container by specifying the command,
  # the arguments, the files to be sent, and the directory to where
  # returned files should be sent
  otsdr <- outsider_init(pkgnm = 'om..figlet', cmd = 'bash',
                         arglist = 'script.sh', wd = wd,
                         files_to_send = c(script, input_file))
  # run the command
  run(otsdr)
}
"
# write to R/functions.R
cat(function_text, file = file.path(flpth, 'R', 'functions.R'))

What does arglist do? This function takes R objects and converts them into character strings that then be passed to a command-line program. It can take as many arguments as you want. For example, with figlet you may want to convert the value of a to ASCII art. In this way, you would simply be able to pass a to the figlet function rather than spell it out again.

What's with the #'@? These are the roxygen tags. They determine the documentation of the function. See "Object documentation" for more details.

Ok, let's build the module!

module_build(flpth = flpth)

## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
## Running devtools::document() ...
## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

## Updating om..figlet documentation

## First time using roxygen2. Upgrading automatically...

## Updating roxygen version in /private/var/folders/x9/m8kwpxps2v93xk52zqzm5lkh0000gp/T/RtmpquGDgt/om..figlet/DESCRIPTION

## Writing NAMESPACE

## Loading om..figlet

## Writing NAMESPACE
## Writing figlet.Rd
## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
## Running devtools::install() ...
## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
##   
   checking for file ‘/private/var/folders/x9/m8kwpxps2v93xk52zqzm5lkh0000gp/T/RtmpquGDgt/om..figlet/DESCRIPTION’ ...

✓  checking for file ‘/private/var/folders/x9/m8kwpxps2v93xk52zqzm5lkh0000gp/T/RtmpquGDgt/om..figlet/DESCRIPTION’
## 

─  preparing ‘om..figlet’:
##    checking DESCRIPTION meta-information ...

✓  checking DESCRIPTION meta-information
## 

─  checking for LF line-endings in source and make files and shell scripts
## 

─  checking for empty or unneeded directories
## 

─  building ‘om..figlet_0.0.1.tar.gz’
## 


## 
Running /Library/Frameworks/R.framework/Resources/bin/R CMD INSTALL \
##   /var/folders/x9/m8kwpxps2v93xk52zqzm5lkh0000gp/T//RtmpquGDgt/om..figlet_0.0.1.tar.gz --install-tests 
## 
-
* installing to library ‘/Library/Frameworks/R.framework/Versions/3.6/Resources/library’
## 
|
* installing *source* package ‘om..figlet’ ...
## 
-
** using staged installation
## 
|
** R
## 
-
** inst
## 
|
** byte-compile and prepare package for lazy loading
## 
-

|
** help
## 
-
*** installing help indices
## 
|
** building package indices
## 
-
** testing if installed package can be loaded from temporary location
## 
|
** testing if installed package can be loaded from final location
## 
-
** testing if installed package keeps a record of temporary installation path
## 
|
* DONE (om..figlet)
## 
-

|


─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
## Running docker_build()
## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
## Command:
## docker build -t dombennett/om_figlet:latest /Library/Frameworks/R.framework/Versions/3.6/Resources/library/om..figlet/dockerfiles/latest
## .....................................................................................................................
## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
## Running devtools::build_readme() ...
## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

## Building om..figlet readme

Does it work?

library(outsider)
figlet <- module_import('figlet', repo = 'dombennett/om..figlet')
# without files
figlet(arglist = 'hello!')

## [1] "figlet hello!"

# with input file
input_file <- file.path(tempdir(), 'input_figlet.txt')
cat('This is from the input text file!', file = input_file)
figlet(arglist = '', input_file = input_file)

## [1] "cat input_figlet.txt | figlet "

# with output file
output_file <- file.path(tempdir(), 'output_figlet.txt')
figlet(arglist = 'Into the output file', output_file = output_file)

## [1] "figlet Into the output file > output_figlet.txt"

cat(readLines(con = output_file), sep = '\n')

##  ___       _          _   _                        _               _   
## |_ _|_ __ | |_ ___   | |_| |__   ___    ___  _   _| |_ _ __  _   _| |_ 
##  | || '_ \| __/ _ \  | __| '_ \ / _ \  / _ \| | | | __| '_ \| | | | __|
##  | || | | | || (_) | | |_| | | |  __/ | (_) | |_| | |_| |_) | |_| | |_ 
## |___|_| |_|\__\___/   \__|_| |_|\___|  \___/ \__,_|\__| .__/ \__,_|\__|
##                                                       |_|              
##   __ _ _      
##  / _(_) | ___ 
## | |_| | |/ _ \
## |  _| | |  __/
## |_| |_|_|\___|
## 

# from input file to output file... with font block!
figlet(arglist = c('-f', 'block'), input_file = input_file,
       output_file = output_file)

## [1] "cat input_figlet.txt | figlet -f block > output_figlet.txt"

cat(readLines(con = output_file), sep = '\n')

##                                                       
## _|_|_|_|_|  _|        _|                _|            
##     _|      _|_|_|          _|_|_|            _|_|_|  
##     _|      _|    _|  _|  _|_|          _|  _|_|      
##     _|      _|    _|  _|      _|_|      _|      _|_|  
##     _|      _|    _|  _|  _|_|_|        _|  _|_|_|    
##                                                       
##                                                       
##                                               
##     _|_|                                      
##   _|      _|  _|_|    _|_|    _|_|_|  _|_|    
## _|_|_|_|  _|_|      _|    _|  _|    _|    _|  
##   _|      _|        _|    _|  _|    _|    _|  
##   _|      _|          _|_|    _|    _|    _|  
##                                               
##                                               
##                                                                               
##   _|      _|                      _|                                  _|      
## _|_|_|_|  _|_|_|      _|_|            _|_|_|    _|_|_|    _|    _|  _|_|_|_|  
##   _|      _|    _|  _|_|_|_|      _|  _|    _|  _|    _|  _|    _|    _|      
##   _|      _|    _|  _|            _|  _|    _|  _|    _|  _|    _|    _|      
##     _|_|  _|    _|    _|_|_|      _|  _|    _|  _|_|_|      _|_|_|      _|_|  
##                                                 _|                            
##                                                 _|                            
##                                                                             
##   _|                            _|              _|_|  _|  _|            _|  
## _|_|_|_|    _|_|    _|    _|  _|_|_|_|        _|          _|    _|_|    _|  
##   _|      _|_|_|_|    _|_|      _|          _|_|_|_|  _|  _|  _|_|_|_|  _|  
##   _|      _|        _|    _|    _|            _|      _|  _|  _|            
##     _|_|    _|_|_|  _|    _|      _|_|        _|      _|  _|    _|_|_|  _|  
##                                                                             
## 

# clean-up
file.remove(c(input_file, output_file))

## [1] TRUE TRUE

Check, test and deploy

Ok! So we have a functioning module. But how do we know it's a functioning module? To check our module structure, configuration and functioning we have a few helpful functions.

# make sure the folder structure is correct
module_check(flpth = flpth)

## DESCRIPTION found ✓
## R folder with files found ✓
## inst found ✓
## inst/om.yml found ✓
## inst/dockerfiles found ✓
## inst/dockerfiles/latest with one Dockerfile found ✓

# check are the names of the module components
module_identities(flpth = flpth)

## R package name: 'om..figlet'
## URL: 'https://github.com/dombennett/om..figlet'
## Docker images: 'dombennett/om_figlet:latest'

# test that the module works
# module_test(flpth = flpth)

Running module_test we will find that there is an issue with the module. module_test checks the functioning of the code by running the example; our current example tries to call -h with figlet -- which is an invalid argument. To correct this we can simply updated our example.

example_text <- "
library(outsider)
figlet <- module_import('figlet', repo = 'dombennett/om..figlet')
figlet('hello!')
"
# write to examples/example.R
cat(example_text, file = file.path(flpth, 'examples', 'example.R'))
# re-build package
module_build(flpth = flpth, build_image = FALSE, verbose = FALSE)

## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
## Running devtools::document() ...
## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

## Updating om..figlet documentation

## Writing NAMESPACE

## Loading om..figlet

## Writing NAMESPACE
## Writing figlet.Rd
## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
## Running devtools::install() ...
## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

## 
## Attaching package: 'om..figlet'

## The following object is masked _by_ '.GlobalEnv':
## 
##     figlet

## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
## Running devtools::build_readme() ...
## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

## Building om..figlet readme

Now re-running the test ...

module_test(flpth = flpth)

## Running /Library/Frameworks/R.framework/Resources/bin/R CMD INSTALL \
##   /private/var/folders/x9/m8kwpxps2v93xk52zqzm5lkh0000gp/T/RtmpquGDgt/om..figlet --install-tests 
## * installing to library ‘/Library/Frameworks/R.framework/Versions/3.6/Resources/library’
## 
-
* installing *source* package ‘om..figlet’ ...
## 
|
** using staged installation
## 
-
** R
## 
|
** inst
## 
-
** byte-compile and prepare package for lazy loading
## 
|

-
** help
## 
|
*** installing help indices
## 
-
** building package indices
## 
|
** testing if installed package can be loaded from temporary location
## 
-
** testing if installed package can be loaded from final location
## 
|
** testing if installed package keeps a record of temporary installation path
## 
-
* DONE (om..figlet)
## 
|

-

## 
## Attaching package: 'om..figlet'

## The following object is masked _by_ '.GlobalEnv':
## 
##     figlet

## [1] "figlet hello!"

## Removing package from '/Library/Frameworks/R.framework/Versions/3.6/Resources/library'
## (as 'lib' is unspecified)

## Yikes! The module works! You are super-excellent!

Nice! The package passes. That means we can upload the package to GitHub and Docker-Hub using the same functions as we saw in "basic".

module_upload(flpth = flpth, code_sharing = TRUE, dockerhub = TRUE)

---

Delete it all

module_uninstall(repo = 'om..figlet')
unlink(x = flpth, recursive = TRUE, force = TRUE)

---

Next-up: Advanced

AntonelliLab/outsider.devtools documentation built on June 20, 2022, 4:36 a.m.