batchApply: Apply a function to batches of data in an Andromeda table

In Andromeda: Asynchronous Disk-Based Representation of Massive Data

Apply a function to batches of data in an Andromeda table

Description

Apply a function to batches of data in an Andromeda table

Usage

batchApply(tbl, fun, ..., batchSize = 1e+05, progressBar = FALSE, safe = FALSE)

Arguments

tbl	An Andromeda table (or any other 'DBI' table).
fun	A function where the first argument is a data frame.
...	Additional parameters passed to fun.
batchSize	Number of rows to fetch at a time.
progressBar	Show a progress bar?
safe	Create a copy of tbl first? Allows writing to the same Andromeda as being read from.

Details

This function is similar to the lapply() function, in that it applies a function to sets of data. In this case, the data is batches of data from an Andromeda table. Each batch will be presented to the function as a data frame.

Value

Invisibly returns a list of objects, where each object is the output of the user-supplied function applied to a batch

See Also

groupApply()

Examples

andr <- andromeda(cars = cars)

fun <- function(x) {
  return(nrow(x))
}

result <- batchApply(andr$cars, fun, batchSize = 25)

result
# [[1]] 
# [1] 25 
# 
# [[2]] 
# [1] 25

close(andr)

Andromeda documentation built on Sept. 12, 2024, 7:41 a.m.