batch_it: Execute a function in batches

In aedobbyn/dobtools: dobtools

Description

In the vector method, attempts to apply fun to inp that result in an error will return NAs for that batch. However, in the dataframe method, these will be returned as NULLs and not applied to the output.

Usage

batch_it(inp, fun, fun_2 = NULL, n_batches, grow_full_output = TRUE,
  verbose = TRUE, ...)

Arguments

inp	(vector, dataframe) A vector or dataframe of inputs.
fun	(function) A function to apply to the input. This is what is returned if grow_full_output is true.
fun_2	(function) Optional secondary function to apply to the result of fun(inp), conditional upon its successful execution. Good for e.g. sending results over a network or inserting them into a database. The result of fun_2(fun(inp)) is not returned.
n_batches	(integer) Number of batches to split inp into.
grow_full_output	(boolean) Should the result of each batch be combined with the ones before it and the whole result returned at the end?
verbose	(boolean) Should progress be messaged?
...	Further arguments.

Value

A dataframe or vector, depending on the type of inp.

Examples

add_one <- function(tbl) {
  tbl %>% purrr::map_dbl(~.x + 1)
}
batch_it(mtcars$mpg, fun = add_one, n_batches = 3)

print_it <- function(tbl) {
  tbl %>% purrr::map_dbl(~ print(.x))
}
# Print the result of each batch, but don't return the full output at the end
batch_it(mtcars$mpg, fun = add_one, fun_2 = print_it, n_batches = 8, grow_full_output = FALSE)

add_one_tbl <- function(tbl) {
  tbl %>% purrr::map_dfr(~.x + 1)
}
batch_it(mtcars, add_one_tbl, n_batches = 5)

aedobbyn/dobtools documentation built on May 28, 2019, 2:33 a.m.