chunkapply: Apply a function stepwise to chunks of data

In rformassspectrometry/Spectra: Spectra Infrastructure for Mass Spectrometry Data

Apply a function stepwise to chunks of data

Description

chunkapply() splits x into chunks and applies the function FUN stepwise to each of these chunks. Depending on the object it is called, this function might reduce memory demand considerably, if for example only the full data for a single chunk needs to be loaded into memory at a time (e.g., for Spectra objects with on-disk or similar backends).

Usage

chunkapply(x, FUN, ..., chunkSize = 1000L, chunks = factor())

Arguments

x	object to which FUN should be applied. Can be any object that supports split.
FUN	the function to apply to x.
...	additional parameters to FUN.
chunkSize	integer(1) defining the size of each chunk into which x should be splitted.
chunks	optional factor or length equal to length(x) defining the chunks into which x should be splitted.

Value

Depending on FUN, but in most cases a vector/result object of length equal to length(x).

Author(s)

Johannes Rainer

Examples

## Apply a function (`sqrt`) to each element in `x`, processed in chunks of
## size 200.
x <- rnorm(n = 1000, mean = 500)
res <- chunkapply(x, sqrt, chunkSize = 200)
length(res)
head(res)

## For such a calculation the vectorized `sqrt` would however be recommended
system.time(sqrt(x))
system.time(chunkapply(x, sqrt, chunkSize = 200))

## Simple example splitting a numeric vector into chunks of 200 and
## aggregating the values within the chunk using the `mean`. Due to the
## `unsplit` the result has the same length than the input with the mean
## value repeated.
x <- 1:1000
res <- chunkapply(x, mean, chunkSize = 200)
length(res)
head(res)

rformassspectrometry/Spectra documentation built on Sept. 27, 2024, 5:43 a.m.