In pbiecek/archivist: Tools for Storing, Restoring and Searching for R Objects

What and why?

Everything in R is an object

What if we store all R objects that have been ever created?

Goals for the archivist package:

• keep trace of (every) created object,
• keep trace of relations between objects,
• keep meta-information/properties of objects,
• provide a way to seek objects that match some conditions,
• provide a way to restore specific objects.

What the archivist is doing?

The archivist is an R package for management of R objects outside R session. It stores binary copies of R objects in rda files and provides easy access for seeking and restoring these objects based on timestamps, classes or other properties

But, why anybody would like to store copies of all R objects?

Example 1: archivist + knitr

Enrich knitr reports with hooks for selected objects. Example.

archivist::aread("pbiecek/Eseje/arepo/e10f9d223df408fca73ea548456493f4")

Example 2: archivist + shiny

Enrich shiny applications with hooks to displayed plots. Example.

More about this example.

How does it work?

Basics

• Artifacts are stored in repositories.
• Use createLocalRepo() to initialize a new repo.
• Add new artifacts with saveToLocalRepo().
• Artifacts are identified by their md5hash.

library(archivist)

createLocalRepo("arepo", default = TRUE)

model <- lm(Sepal.Length ~ Petal.Length + Species, data=iris)
saveToLocalRepo(model)

library(ggplot2)

plot <- ggplot(iris, aes(Sepal.Length, Petal.Length)) + geom_point()
saveToLocalRepo(plot)

• Use aread() to read an artifact from a repository to an R session.
• You can read artifacts either from local folders or remote repositories.

# from the default local repository
aread("e700c6704890022a4e0277e11cd46488")

# from the github pbiecek/graphGallery repository
aread("pbiecek/graphGallery/18a98048f0584469483afb65294ce3ed")

• Use asearch() to seek for all artifacts with properties that fit some patterns.
• As a result you will get a list of R objects.
• You may use time intervals as patters as well.
• Below we ask for all lm models that are in a repository, apply BIC criteria and sort models.

models <- asearch("pbiecek/graphGallery", patterns = "class:lm")
modelsBIC <- sapply(models, BIC)
sort(modelsBIC)

The structure

• Repository is a folder with:
• a subfolder gallery with rda dumps of recorded objects,

• an SQLite database (file backpack.db).

• The database contains two tables, with artifacts and their properties and relations.

• Properties and relations are stored as Tags.
• Tags are being extracted from an artifact after each call to saveToLocalRepo().
• Use showLocalRepo() to examine tags for selected artifacts.
• Popular and useful tags are: format:..., name:..., class:..., date:....

• Different tags are extracted from different objects.
• The list is flexible, one may easily add new tags if needed.

What else you can do with the archivist?

Pedigree of an object

• Use %a% operator to record function calls.
• Use ahistory() to present the pedigree of an object.
• All partial intermediate results are also stored in a repo

library("dplyr")
iris %a%
 dplyr::filter(Sepal.Length < 6) %a%
 lm(Petal.Length~Species, data=.) %a%
 summary() -> tmp

ahistory(md5hash = "050e41ec3bc40b3004bc6bdd356acae7", format = "kable", alink = TRUE)

ahistory(tmp)

You need a village to raise an R object

• For some artifacts you need specific versions of packages (see ggplot2).
• For every object also the sessionInfo is stored.
• Use asession to check versions of packages loaded when artifact was added to the repo.
• Use restoreLibs to install packages in versions installed when artifact was added to the repo.

asession("050e41ec3bc40b3004bc6bdd356acae7")
# restoreLibs("pbiecek/graphGallery/arepo/600bda83cb840947976bd1ce3a11879d")

Other features

• Deeper integration with GitHub is available in archivist.github thanks to @MarcinKosinski (coauthor of the archivist).
• Use trace() to force adding an artifact to repo after each call of a specific function.
• The code below will adds all results of lm() to the repository allModels.

createLocalRepo("allModels", default = TRUE)
trace(lm, exit = quote(saveToLocalRepo(z, repoDir="allModels")))

• Use addHooksToPrint() to add archivist hooks to all objects of selected classes in a knitr report.

addHooksToPrint(class=c("ggplot", "data.frame"),
  repoDir = "arepo",
  repo = "Eseje", user = "pbiecek", subdir = "arepo")

Thank you!

Find more at pbiecek.github.io/archivist

pbiecek/archivist documentation built on May 25, 2021, 11:36 p.m.