In INWT/mctools: Tools for parallel and asynchronous programs

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

The problem

Here is the problem we'd like to solve.

• We would like to have all warnings to be errors so that we can deal with them.
• We use third party packages which sometimes throw warnings. These warnings do not make us happy, but they should not break in production. We know them and want to accept them.
• We use parallel::mclapply because we have some long running tasks and realize suppressWarnings and options(warn = 2) do not work as expected in parallel! Also, interactive sessions behave differently than batch mode. All this is very tricky to resolve.

options(warn = 0)
productionFunction <- function(x) {
  if (x == 1) suppressWarnings(warning("this may be okay"))
  ## just like *real* production code...
  else if (x == 2) stop("We should never end up in this branch...")
  else cbind(1:2, 1:3) # this is not okay and should be an error!
}

res <- parallel::mclapply(1:3, productionFunction, mc.cores = 2)
str(res)
options(warn = 2)
res1 <- parallel::mclapply(c(1, 3), productionFunction, mc.cores = 2)
str(res1)

Do you see the problem? Where was that error again? (Please notice, that if you try to reproduce these results in an interactive session, you won't even get a result back from mclapply.) Terrible indeed! I hear you. Fear not, there is a solution...

The salvation

Use mcMap as a drop in replacement for mclapply if you need more control over errors and warnings. Be aware that mcMap is a simple wrapper around mclapply and is not a novel parallel computing interface.

library(mctools)
options(warn = 2)

This is how I expected mclapply to behave in the first place:

mcMap(1:3, productionFunction)

White-list specific warnings if that is what you want:

mcMap(1:3, productionFunction, warningsWhitelist = "multiple")

Or fail when any of the processes encounter warnings:

mcMap(1:3, productionFunction, finallyStop = TRUE)

INWT/mctools documentation built on Sept. 24, 2021, 9:19 p.m.