hdd_slice: Applies a function to slices of data to create a HDD data set
In hdd: Easy Manipulation of Out of Memory Data Sets

hdd_slice

R Documentation

Applies a function to slices of data to create a HDD data set

Description

This function is useful to apply complex R functions to large data sets (out of memory). It slices the input data, applies the function, then saves each chunk into a hard drive folder. This can then be a HDD data set.

Usage

hdd_slice(
  x,
  fun,
  dir,
  chunkMB = 500,
  rowsPerChunk,
  replace = FALSE,
  verbose = 1,
  ...
)

Arguments

`x`	A data set (data.frame, HDD).
`fun`	A function to be applied to slices of the data set. The function must return a data frame like object.
`dir`	The destination directory where the data is saved.
`chunkMB`	The size of the slices, default is 500MB. That is: the function `fun` is applied to each 500Mb of data `x`. If the function creates a lot of additional information, you may want this number to go down. On the other hand, if the function reduces the information you may want this number to go up. In the end it will depend on the amount of memory available.
`rowsPerChunk`	Integer, default is missing. Alternative to the argument `chunkMB`. If provided, the functions will be applied to chunks of `rowsPerChunk` of `x`.
`replace`	Whether all information on the destination directory should be erased beforehand. Default is `FALSE`.
`verbose`	Integer, defaults to 1. If greater than 0 then the progress is displayed.
`...`	Other parameters to be passed to `fun`.

Details

This function splits the original data into several slices and then apply a function to each of them, saving the results into a HDD data set.

You can perform merging operations with hdd_slice, but for regular merges not that you have the function hdd_merge that may prove more convenient (not need to write a ad hoc function).

Value

It doesn't return anything, the output is a "hard drive data" saved in the hard drive.

Author(s)

Laurent Berge

Examples


# Toy example with iris data.
# Say you want to perform a cartesian merge
# If the results of the function is out of memory
# you can use hdd_slice (not the case for this example)

# preparing the cartesian merge
iris_bis = iris
names(iris_bis) = c(paste0("x_", 1:4), "species_bis")


fun_cartesian = function(x){
	# Note that x is treated as a data.table
	# => we need the argument allow.cartesian
	merge(x, iris_bis, allow.cartesian = TRUE)
}

hdd_result = tempfile() # => folder where results are saved
hdd_slice(iris, fun_cartesian, dir = hdd_result, rowsPerChunk = 30)

# Let's look at the result
base_hdd = hdd(hdd_result)
summary(base_hdd)
head(base_hdd)

hdd documentation built on Aug. 25, 2023, 5:19 p.m.