hdd_merge: Merges data to a HDD file
In hdd: Easy Manipulation of Out of Memory Data Sets

hdd_merge

R Documentation

Merges data to a HDD file

Description

This function merges in-memory/HDD data to a HDD file.

Usage

hdd_merge(
  x,
  y,
  newfile,
  chunkMB,
  rowsPerChunk,
  all = FALSE,
  all.x = all,
  all.y = all,
  allow.cartesian = FALSE,
  replace = FALSE,
  verbose
)

Arguments

`x`	A HDD object or a `data.frame`.
`y`	A data set either a data.frame of a HDD object.
`newfile`	Destination of the result, i.e., a destination folder that will receive the HDD data.
`chunkMB`	Numeric, default is missing. If provided, the data 'x' is split in chunks of 'chunkMB' MB and the merge is applied chunkwise.
`rowsPerChunk`	Integer, default is missing. If provided, the data 'x' is split in chunks of 'rowsPerChunk' rows and the merge is applied chunkwise.
`all`	Default is `FALSE`.
`all.x`	Default is `all`.
`all.y`	Default is `all`.
`allow.cartesian`	Logical: whether to allow cartesian merge. Defaults to `FALSE`.
`replace`	Default is `FALSE`: if the destination folder already contains data, whether to replace it.
`verbose`	Numeric. Whether information on the advancement should be displayed. If equal to 0, nothing is displayed. By default it is equal to 1 if the size of `x` is greater than 1GB.

Details

If x (resp y) is a HDD object, then the merging will be operated chunkwise, with the original chunks of the objects. To change the size of the chunks for x: you can use the argument chunkMB or rowsPerChunk.

To change the chunk size of y, you can rewrite y with a new chunk size using write_hdd.

Note that the merging operation could also be achieved with hdd_slice (although it would require setting up an ad hoc function).

Value

This function does not return anything. It applies the merging between two potentially large (out of memory) data set and saves them on disk at the location of newfile, the destination folder which will be populated with .fst files representing chunks of the resulting merge.

To interact with the data (on disk) newly created, use the function hdd().

Author(s)

Laurent Berge

Examples


# Toy example with iris data

# Cartesian merge example
iris_bis = iris
names(iris_bis) = c(paste0("x_", 1:4), "species_bis")
# We must have a common key on which to merge
iris_bis$id = iris$id = 1

# merge, we chunk 'x' by 50 rows
hdd_path = tempfile()
hdd_merge(iris, iris_bis, newfile = hdd_path,
		  rowsPerChunk = 50, allow.cartesian = TRUE)

base_merged = hdd(hdd_path)
summary(base_merged)
print(base_merged)

hdd documentation built on Aug. 25, 2023, 5:19 p.m.