require("knitr")
temp_workspace <- tempfile(pattern = "containerit_temp")
dir.create(temp_workspace)
opts_knit$set(root.dir = temp_workspace)

1. Introduction

Even though R is designed for open and reproducible research, users who want to share their work with others are facing challenges. Sharing merely the R script or R Markdown document should warrant reproducibility, but many analyses rely on additional resources and specific third party software as well. An R script may produce unexpected results or errors when executed under a different version of R or another platform. Reproducibility is only assured by providing complete setup instructions and resources. Long-term reproducibility can be achieved by either regular maintenance of the code, i.e. keeping it always working with the latest package versions from CRAN. It can be supported by packages such as packrat and platforms such as MRAN, which provide means to capture a specific combination of R packages. An alternative to updating or managing packages explicitly is providing the full runtime environment in its original state, using virtual machines or software containers.

The R extension package containerit aims to facilitate the latter approach by making reproducible and archivable research with containers easier. The development is supported by the DFG-funded project Opening Reproducible Research (o2r, http://o2r.info). containerit relies on Docker and automatically generates a container manifest, or "recipe", with setup instructions to recreate a runtime environment based on a given R session, R script, R Markdown file or workspace directory. The resulting Dockerfile can not only be read and understood by humans, but also be interpreted by the Docker engine to create a software container containing all the R packages and their system dependencies. This way all requirements of an R workflow are packaged in an executable format.

The created Dockerfiles are based on the Rocker project (Rocker on Docker Hub, introduction). Using the stack of version-stable Rocker images, it is possible to match the container's R version with the local R installation or any R version the user requires. containerit executes the provided input workspace or file first locally on the host machine in order to detect all dependencies. For determining external software dependencies of attached packages, containerit relies (a) on the sysreqs database and makes use of the corresponding web API and R package, and (b) on internally defined rule sets for challenging configurations.

The Dockerfile created by containerit can then be used to build a Docker image. Running the image will start an R session that closely resembles the creating systems runtime environment. The image can be shared and archived and works anywhere with a compatible Docker version.

To build images and run containers, the package integrates with the stevedore package and adds a few convenience functions for interacting with Docker images and containers. For concrete details on reading, loading, or installing the exact versions of R packages including their system dependencies/libraries, this project focuses on the geospatial domain. containerit uses the package futile.logger to provide information to the user at a configurable level of detail, see futile.logger documentation.

The related package liftr creates Dockerfiles for rendering R Markdown documents, but relies on manually created metadata in the document's YAML header.

In the remainder of this vignette, we first introduce the main usage scenarios for containerit and document current challenges as well as directions for future work.

2. Creating a Dockerfile

2.1 Basics

The easiest way to generate a Dockerfile is to run an analysis in an interactive R session and create a Dockerfile for this session by loading containerit and calling the dockerfile()- method with default parameters. As shown in the example below, the result can be pretty-printed and written to a file. If no file argument is supplied to write(), the Dockerfile is written to the current working directory as ./Dockerfile, following the typical naming convention of Docker.

When packaging any resources, it is essential that the R working directory is the same as the build context, to which the Dockerfile refers. All resources must be located below this directory so that they can be referred to by relative paths (e.g. for copy instructions). This must also be considered when packaging R scripts that use relative paths, e.g. for reading a file or sourcing another R script.

2.2 Packaging an interactive session

library("containerit")

# do stuff, based on demo("krige")
library("gstat")
library("sp")

data(meuse)
coordinates(meuse) = ~x+y
data(meuse.grid)
gridded(meuse.grid) = ~x+y
v <- variogram(log(zinc)~1, meuse)
m <- fit.variogram(v, vgm(1, "Sph", 300, 1))
plot(v, model = m)

# create Dockerfile representation
dockerfile_object <- dockerfile()

The representation of a Dockerfile in R is an instance of the S4 class Dockerfile.

dockerfile_object

The printout below shows the rendered Dockerfile. Its instructions follow a pre-defined order:

  1. define the base image
  2. define the maintainer label
  3. install system dependencies and external software
  4. install the R packages themselves
  5. set the working directory
  6. copy instructions and metadata labels (see examples in later sections)
  7. CMD instruction (final line) defines the default command when running the container

Note that the maintainer label as well as the R version of the base image are detected from the runtime environment, if not set to different values manually.

print(dockerfile_object)

Instead of printing out to the console, you can also write to a file:

containerit::write(dockerfile_object, file = tempfile(fileext = "_dockerfile"))

There also is a convenience function to build the Dockerfile object with stevedore:

containerit::docker_build(dockerfile_object)

2.3 Packaging an external session

Packaging an interactive session has the disadvantage that unnecessary dependencies might be added to the Dockerfile and subsequently to the container. For instance the package futile.logger is a dependency of containerit, and it will be added to the container because it was loaded into the same session where the analysis was executed. It cannot be removed by default, because other packages in the session might use it as well (even unintentionally in case of generic methods). Therefore, it is safer not to tamper with the current session, but to run the analysis in an isolated vanilla session, which does not have containerit in it. The latter will batch-execute the commands in a separate instance of R and retrieves an object of class sessionInfo. The session info is then used as input to dockerfile(). This is also how dockerfile() works internally when packaging either expressions, scripts or R markdown files.

The following code creates a Dockerfile for a list of expressions in a vanilla session.

exp <- c(expression(library(sp)),
         expression(data(meuse)), 
         expression(mean(meuse[["zinc"]])))
dockerfile_object <- dockerfile(from = exp)
print(dockerfile_object)

2.4 Packaging an R script

R scripts are packaged by just supplying the file path or paths to the argument from of dockerfile(). They are automatically copied into the container's working directory. In order to run the R script on start-up, rather than an interactive R session, a CMD instruction can be added by providing the value of the helper function CMD_Rscript() as an argument to cmd.

# create simple script file
scriptFile <- tempfile(pattern = "containerit_", fileext = ".R")
writeLines(c('library(rgdal)',
             'nc <- rgdal::readOGR(system.file("shapes/", package="maptools"), "sids", verbose = FALSE)',
             'proj4string(nc) <- CRS("+proj=longlat +datum=NAD27")',
             'plot(nc)'), scriptFile)

# use a custom startup command
scriptCmd <- CMD_Rscript(basename(scriptFile))

# create Dockerfile for the script
dockerfile_object <- dockerfile(from = scriptFile, silent = TRUE, cmd = scriptCmd)
print(dockerfile_object)

2.5 Packaging an R Markdown file

Similarly to scripts, R Markdown files can be passed to the from argument. In the following example, a vignette from the Simple Features package sf is packaged in a container. To render the document at startup, the Dockerfile's CMD instruction must be changed. To do this, the cmd argument passed to dockerfile() is constructed using the function CMD_Render.

file.copy(from = system.file("doc/sf6.Rmd", package = "sf"),
          to = file.path(temp_workspace, "sf6.Rmd"),
          overwrite = TRUE)
dockerfile_object <- dockerfile(from = file.path(temp_workspace, "sf6.Rmd"),
                                silent = TRUE, 
                                cmd = CMD_Render("sf6.Rmd"))
print(dockerfile_object)

2.6 Packaging a workspace directory

A typical case expected to be interesting for containerit users is packaging a local directory with a collection of data and code files. If providing a directory path to the dockerfile() function, the package searches for the first occurence of an R script, or otherwise the first occurence of an R markdown file. It then proceeds to package this file along with all other resources in the directory, as shown in the next section.

2.7 Packaging from a stored sessionInfo

You can also save your sessionInfo() into an .RData file and create a Dockerfile form it. The sessionInfo object must have the one of the names sessionInfo, sessioninfo, or session_info. It must be the only object in the .RData file. This is useful when you want to re-create sessions on different machines.

suppressPackageStartupMessages(library("sf"))
sessionInfo <- sessionInfo()
save(sessionInfo, file = file.path(temp_workspace, "sessionInfo.RData"))
unloadNamespace("sf")
the_dockerfile <- dockerfile(from = file.path(temp_workspace, "sessionInfo.RData"))
print(the_dockerfile)

Two alternative packages provide their own version of a session info, both of which are supported as well. The following examples first load the packages fortunes and remotes, store session information objects, and then unload the packages again before creating Dockerfiles from the stored objects.

suppressPackageStartupMessages(library("devtools"))
suppressPackageStartupMessages(library("sessioninfo"))

suppressPackageStartupMessages(library("fortunes"))
suppressPackageStartupMessages(library("remotes"))

session_info <- devtools::session_info()
save(session_info, file = file.path(temp_workspace, "session_info_devtools.RData"))

sessioninfo = sessioninfo::session_info()
save(sessioninfo, file = file.path(temp_workspace, "session_info.RData"))

unloadNamespace("abc")
unloadNamespace("A3")

df_sessioninfo <- dockerfile(from = file.path(temp_workspace, "session_info.RData"))
df_devtools <- dockerfile(from = file.path(temp_workspace, "session_info_devtools.RData"))

sessioninfo::session_info()

print(df_sessioninfo)

devtools::session_info()

print(df_devtools)

2.8 Packaging a DESCRIPTION file or object

The DESCRIPTION file format contains basic information about R extension packages and their dependencies. containerit can also create Dockerfiles based on a DESCRIPTION file or ob description objects from the desc package. Unless specified otherwise, it uses the minimal R version mentioned in the Depends-field for the base image.

df_description <- dockerfile(from = system.file("DESCRIPTION", package = "dplyr"))
print(df_description)

Use this to quickly create a container that contains everything needed to run your package, including system dependencies and installing the actual package in a specific version:

df_description_sf <- dockerfile(from = desc::desc(package = "sf"),
                                 maintainer = "o2r",
                                 versioned_packages = TRUE)
print(df_description_sf)

3. Including resources

Analyses in R often rely on external files and resources that are located located in the workspace. When scripts or R markdown files are packaged, they are copied by default into the same location relative to the working directory. The argument copy influences how dockerfile() behaves in this matter. It can either have the values script (default behaviour), script_dir (copies the complete directory in which the input file is located), or a custom list of files and directories inside the current working directory

file.copy(from = system.file("simple_test_script_resources", 
                             package = "containerit"),
          to = temp_workspace, recursive = TRUE)
setwd(file.path(temp_workspace, "simple_test_script_resources"))
dockerfile_resources <- dockerfile(from = ".",
              copy = "script_dir",
              cmd = CMD_Rscript("simple_test.R"))
print(dockerfile_resources)

Including R objects works similar to resources, using the argument save_image. The argument can be set to TRUE to save all objects of the current workspace to an .RData file, which is then copied to the container's working directory and loaded on startup (based on save.image()).

the_dockerfile <- dockerfile(save_image = TRUE)
print(the_dockerfile)

Alternatively, a object names as well as other arguments can be passed as a list, which then are passed to the save() function.

require("fortunes")
rm(list = ls())
calculation <- 41 + 1
frtn <- fortunes::fortune()
original_sessionInfo <- sessionInfo()

the_dockerfile <- dockerfile(silent = TRUE,
                 save_image = list("original_sessionInfo", "frtn"))
print(the_dockerfile)

4. Advanced Dockerfile instructions

4.1 Image metadata

Metadata can be added to Docker images using Label instructions. Label instructions are key-value pairs of arbitrary content. A duplicate key overwrites existing ones. Although it is up to the user how many labels are created, it is recommended to bundle them into one Label instruction in the Dockerfile. Each use of the Label() function creates a separate instruction in the Dockerfile.

As shown in section 2, the maintainer label is set by default to the top as the dockerfile and contains the username of the current host system. The maintainer can be changed with the maintainer argument of dockerfile():

labeled_dockerfile <- dockerfile(from = clean_session(), maintainer = "Jon_Doe@example.com")

Labels can be applied to the existing Dockerfile object using the addInstructions() function, which adds any newly created instructions to the end of the Dockerfile but before the CMD statement. The Label() constructor can be used for creating labels of arbitrary content and works similar to creating named lists in R.

# A simple label that occupies one line:
label1 <- Label(key1 = "this", key2 = "that", otherKey = "content")
addInstruction(labeled_dockerfile) <- label1

#label with fixed namespace for all keys
label2 <- Label("name"="A name", "description" = "A description", label_ns = "my.label.ns.")

# A multiline label with one key/value pair per line
label3 <- Label("info.o2r.name" = "myProject_ImageName", "org.label-schema.name"="ImageName", 
                "yet.another_labelname"="true", multi_line = TRUE)
addInstruction(labeled_dockerfile) <- list(label2, label3)

Metadata according to the Label Schema conventions can be created with a function constructed by the helper factory LabelSchemaFactory().

Label_LabelSchema <- LabelSchemaFactory()
label <- Label_LabelSchema(name = "ImageName", description = "Description of the image", build_date = Sys.time())
addInstruction(labeled_dockerfile) <- label

You can also put session information, using either base R or devtools, into a label as plain text or as json:

addInstruction(labeled_dockerfile) <- Label_SessionInfo(session = clean_session())
addInstruction(labeled_dockerfile) <- Label_SessionInfo(session = devtools::session_info(), as_json = TRUE)

The resulting Dockerfile with all the labels:

print(labeled_dockerfile)

4.2 Exposing ports

If a Dockerfile contains a web service, specific ports of a running container might need to be exposed.

The EXPOSE instruction informs Docker that the container listens on the specified network ports at runtime. [..] It functions as a type of documentation between the person who builds the image and the person who runs the container, [..] To actually publish the port when running the container, use the -p flag on docker run to publish and map one or more ports [..] (Docker docs)

dockerfile_with_expose <- dockerfile()
my_port_1 <- Expose(8000)
my_port_2 <- Expose(port = "80/tcp", host = 8080)
addInstruction(dockerfile_with_expose) <- list(my_port_1, my_port_2)
print(dockerfile_with_expose)

4.3 Entrypoint and command

The ENTRYPOINT instruction allows using a container like a binary executable. The actually executed command is constructed from both ENTRYPOINT and CMD instructions - please check the Docker docs for details. The entrypoint parameter of the dockerfile() function supports both "forms" as shown below.

ep_exec <- Entrypoint(program = "R",
                 params = list(
                   "-e",
                   "pr <- plumber::plumb(commandArgs()[4]); pr$run(host='0.0.0.0', port=8080)"),
                 form = "exec")
dockerfile_with_entrypoint <- dockerfile(from = NULL,
                                         image = "trestletech/plumber",
                                         entrypoint = ep_exec,
                                         cmd = Cmd("schedule.R"))
print(dockerfile_with_entrypoint)
ep_shell <- Entrypoint(program = "R",
                 params = list(
                   "-e",
                   "pr <- plumber::plumb(commandArgs()[4]); pr$run(host='0.0.0.0', port=8080)"),
                 form = "shell")
dockerfile_with_entrypoint <- dockerfile(from = NULL,
                                         image = "trestletech/plumber",
                                         entrypoint = ep_shell,
                                         cmd = Cmd("schedule.R", form = "shell"))
print(dockerfile_with_entrypoint)

4.4 Comments

Comments can be added to a Dockerfile. By default they are the last instruction.

df_comment <- dockerfile(from = NULL, image = getImageForVersion("3.3.3"), silent = TRUE)
addInstruction(df_comment) <- Comment(text = "Don't forget about this!")
print(df_comment)

5. Custom base image

The dockerfile() function allows further customization regarding the R version or the used base image (cf. Rocker stack). Note that while choosing an R version for the Dockerfile explicitly is possible, the session to generate the required information (i.e. which packages are attached etc.) is still running the R version of the generating machine.

The following examples show usage of these options and the respective FROM statements in the Dockerfile.

df_custom <- dockerfile(from = NULL, image = getImageForVersion("3.3.3"), silent = TRUE)
print(df_custom@image)
df_custom <- dockerfile(from = NULL, image = "rocker/geospatial", silent = TRUE)
print(df_custom@image)
df_custom <- dockerfile(from = NULL, image = "rocker/verse:3.0.0", silent = TRUE)@image
print(df_custom@image)

When you change the base image, you might not want to re-install packages that are already in the base image. The helper function get_installed_packages() is used internally to filter the installed packages and can be activated with filter_baseimage_pkgs.

expressions <- c(expression(library("sp")))
session <- clean_session(expressions, echo = TRUE)
df_filtered <- dockerfile(from = session, maintainer = "o2r",
                   image = "rocker/geospatial:3.4.4",
                   filter_baseimage_pkgs = TRUE)
print(df_filtered)

6. CLI

A command line interface to the package functions is also available for Linux based on docopt.R. This allows integration into workflows and tools written in other programming languages than R.

You can make the command containerit available on your machine by linking the R script file delivered with the package as follows:

ln -s $(Rscript -e "cat(system.file(\"cli/container_it.R\", package=\"containerit\"))") /usr/local/bin/containerit

CLI Examples:

```{bash cli, eval=FALSE} containerit --help

# runs the first R markdown or R script file locally # prints Dockerfile without writing a file containerit dir -p --no-write

# Packages R-script # saves a workspace image (-i parameter) # Writes Dockerfile (overwrite with -f) # execute the script on start-up containerit file -ifp --cmd-R-file path/example.R

# Creates an empty R session with the given R commands # Set R version of the container to 3.3.0 containerit session -p -e "library(sp)" -e "demo(meuse, ask=FALSE)"

## 7. Challenges

We encountered several challenges during `containerit`'s development.
First and foremost, a well known limitation is that R packages don't define system dependencies and do not provide explicit versions for R package dependencies.
The `sysreqs` package is a promising approach towards handling system requirements, but so far lists package names but does not provide version information.
The [shinyapps-package-dependencies](https://github.com/rstudio/shinyapps-package-dependencies) demonstrate a (currently system dependent) alternative.
The high value of R might well lie in the fact that "packages currently on CRAN" should work well with each other.

An unmet challenge so far is the installation of specific versions of external libraries (see [issue](https://github.com/o2r-project/containerit/issues/46)).
A package like `sf` relies on well-tested and powerful system libraries, see `sf::sf_extSoftVersion()`, which ideally should be matched in the created container.

And of course users may do things that `containerit` cannot capture from the session state "after the analysis is completed", such as detaching packages or removing relevant files, and unknown side-effects might occur.

All software is presumed to be installed and run on the host system.
Although it is possible to use deviating versions of R or even create Dockerfiles using sessionInfo-objects created on a different host, this may lead to unexpected errors because the setup cannot be tested locally.

## 8. Conclusions and future work

`containerit` allows to create and customize Dockerfiles with minimal effort, which are suitable for packaging R analyses in the persistent runtime environment of a software container.
So far, we were able to reproduce complete R sessions regarding attached _but not "loaded only" packages and mitigate some challenges towards reproducible computational research.

Although we are able to package different versions of R, we still do not fully support the installation of specific versions of R packages and external software libraries, which R itself does not support.
This should be tested in the future by evaluating version-stable package repositories like MRAN and GRAN or utility packages such as packrat -- see the [GitHub issues](https://github.com/o2r-project/containerit/issues/new) for the status of these plans or provide your own ideas there.

Related to installing specific versions is support for other package repositories, such as Bioconductor, git, BitBucket, or even local files.
For now, it is recommended that users have all software up-to-date when building a software container, as the latest version is installed from CRAN during the image build, to have matching package versions between the creation runtime environment and the container.
All Dockerfiles and instructions are adjusted to the Rocker image stack and assume a Debian/Linux operating system.
As we are not yet supporting the build of Docker images from scratch, we are restricted to this setup.

The package is a first prototype available via GitHub.
While a publication on CRAN is a goal, it should be preceded by feedback from the user community and ideally be accompanied by related packages, such as `stevedore`, being available on CRAN, too.
The prototype of `containerit` was developed and tested only on Ubuntu/Linux, which should be extended before releasing a stable version on CRAN.

As part of the o2r project, it is planned to integrate `containerit` in a [web service](http://o2r.info/architecture) for creating archivable research in form of [Executable Research Compendia (ERC)](https://doi.org/10.1045/january2017-nuest).
Making `containerit` itself easier to use for end-users is a secondary but worthwhile goal, for example by building a graphical user interface for metadata creation.
Country locales are also not supported yet.
We may want to support other container OS (e.g. windows container or other Linux distributions) or even containerization solutions such as [Singularity](http://singularity.lbl.gov/) or the [Open Container Initiative](https://www.opencontainers.org/)'s (OCI) [Image Format](https://github.com/opencontainers/image-spec).

Feedback and contributions are highly welcome [on GitHub](https://github.com/o2r-project/containerit/issues) or [o2r_project](https://twitter.com/o2r_project) on Twitter.

## Metadata

```r
sessionInfo()


o2r-project/containerit documentation built on June 28, 2021, 2:46 p.m.