hdd_merge: Merges data to a HDD file

Description Usage Arguments Details Author(s) See Also Examples

View source: R/core.R

Description

This function merges in-memory/HDD data to a HDD file.

Usage

1
2
3
hdd_merge(x, y, newfile, chunkMB, rowsPerChunk, all = FALSE,
  all.x = all, all.y = all, allow.cartesian = FALSE,
  replace = FALSE, verbose)

Arguments

x

A HDD object or a data.frame.

y

A data set either a data.frame of a HDD object.

newfile

Destination of the result, i.e., a destination folder that will receive the HDD data.

chunkMB

Numeric, default is missing. If provided, the data 'x' is split in chunks of 'chunkMB' MB and the merge is applied chunkwise.

rowsPerChunk

Integer, default is missing. If provided, the data 'x' is split in chunks of 'rowsPerChunk' rows and the merge is applied chunkwise.

all

Default is FALSE.

all.x

Default is all.

all.y

Default is all.

allow.cartesian

Logical: whether to allow cartesian merge. Defaults to FALSE.

replace

Default is FALSE: if the destination folder already contains data, whether to replace it.

verbose

Numeric. Whether information on the advancement should be displayed. If equal to 0, nothing is displayed. By default it is equal to 1 if the size of x is greater than 1GB.

Details

If x (resp y) is a HDD object, then the merging will be operated chunkwise, with the original chunks of the objects. To change the size of the chunks for x: you can use the argument chunkMB or rowsPerChunk.

To change the chunk size of y, you can rewrite y with a new chunk size using write_hdd.

Note that the merging operation could also be achieved with hdd_slice (although it would require setting up a ad hoc function).

Author(s)

Laurent Berge

See Also

See hdd, sub-.hdd and cash-.hdd for the extraction and manipulation of out of memory data. For importation of HDD data sets from text files: see txt2hdd.

See hdd_slice to apply functions to chunks of data (and create HDD objects) and hdd_merge to merge large files.

To create/reshape HDD objects from memory or from other HDD objects, see write_hdd.

To display general information from HDD objects: origin, summary.hdd, print.hdd, dim.hdd and names.hdd.

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# Toy example with iris data

# Cartesian merge example
iris_bis = iris
names(iris_bis) = c(paste0("x_", 1:4), "species_bis")
# We must have a common key on which to merge
iris_bis$id = iris$id = 1

# merge, we chunk 'x' by 50 rows
hdd_path = tempfile()
hdd_merge(iris, iris_bis, newfile = hdd_path,
		  rowsPerChunk = 50, allow.cartesian = TRUE)

base_merged = hdd(hdd_path)
summary(base_merged)
print(base_merged)

hdd documentation built on Nov. 6, 2019, 5:07 p.m.