drPersist: Persist a Transformed 'ddo' or 'ddf' Object

In datadr: Divide and Recombine for Large, Complex Data

Description

Persist a transformed 'ddo' or 'ddf' object by making a deferred transformation permanent

Usage

drPersist(x, output = NULL, overwrite = FALSE, control = NULL)

Arguments

x	an object of class 'ddo' or 'ddf'
output	a "kvConnection" object indicating where the output data should reside (see localDiskConn, hdfsConn). If NULL (default), output will be an in-memory "ddo" object.
overwrite	logical; should existing output location be overwritten? (also can specify overwrite = "backup" to move the existing output to _bak)
control	parameters specifying how the backend should handle things (most-likely parameters to rhwatch in RHIPE) - see rhipeControl and localDiskControl

Details

When a transformation is added to a ddf/ddo via addTransform, the transformation is deferred until the some action is taken with the data (e.g. a call to recombine). See the documentation of addTransform for more information about the nature of transformations.

Calling drPersist() on the ddo/ddf makes the transformation permanent (persisted). In the case of a local disk connection (via localDiskConn) or HDFS connection (via hdfsConn), the transformed data are written to disk.

Value

a 'ddo' or 'ddf' object with the transformation evaluated on the data

Author(s)

Ryan Hafen

See Also

addTransform

Examples

bySpecies <- divide(iris, by = "Species")

# Create the transformation and add it to bySpecies
bySpeciesSepal <- addTransform(bySpecies, function(x) x[,c("Sepal.Length", "Sepal.Width")])

# Note the transformation is 'pending' a data action
bySpeciesSepal

# Make the tranformation permanent (persistent)
bySpeciesSepalPersisted <- drPersist(bySpeciesSepal)

# The transformation no longer pending--but a permanent part of the new ddo
bySpeciesSepalPersisted
bySpeciesSepalPersisted[[1]]

datadr documentation built on May 1, 2019, 8:06 p.m.