create_chunk_mapper: Create function that applies to each chunk if disk.frame

In disk.frame: Larger-than-RAM Disk-Based Data Manipulation Framework

Create function that applies to each chunk if disk.frame

Description

A function to make it easier to create functions like filter

Usage

create_chunk_mapper(chunk_fn, warning_msg = NULL, as.data.frame = FALSE)

Arguments

chunk_fn	The dplyr function to create a mapper for
warning_msg	The warning message to display when invoking the mapper
as.data.frame	force the input chunk of a data.frame; needed for dtplyr

Examples

filter = create_chunk_mapper(dplyr::filter)

#' example: creating a function that keeps only the first and last n row
first_and_last <- function(chunk, n, ...) {
  nr = nrow(chunk)
  print(nr-n+1:nr)
  chunk[c(1:n, (nr-n+1):nr), ]
}

#' create the function for use with disk.frame
first_and_last_df = create_chunk_mapper(first_and_last)

mtcars.df = as.disk.frame(mtcars)

#' the operation is lazy
lazy_mtcars.df = mtcars.df %>%
  first_and_last_df(2)

#' bring into R
collect(lazy_mtcars.df)

#' clean up
delete(mtcars.df)

disk.frame documentation built on Aug. 24, 2023, 5:09 p.m.