divide: Divide a Distributed Data Object
In tesseradata/datadr: Divide and Recombine for Large, Complex Data

Description Usage Arguments Details Value Author(s) References See Also Examples

Divide a ddo/ddf object into subsets based on different criteria

divide(data, by = NULL, spill = 1000000, filterFn = NULL, bsvFn = NULL,
  output = NULL, overwrite = FALSE, preTransFn = NULL,
  postTransFn = NULL, params = NULL, packages = NULL, control = NULL,
  update = FALSE, verbose = TRUE)

`data`	an object of class "ddf" or "ddo" - in the latter case, need to specify `preTransFn` to coerce each subset into a data frame
`by`	specification of how to divide the data - conditional (factor-level or shingles), random replicate, or near-exact replicate (to come) – see details
`spill`	integer telling the division method how many lines of data should be collected until spilling over into a new key-value pair
`filterFn`	a function that is applied to each candidate output key-value pair to determine whether it should be (if returns `TRUE`) part of the resulting division
`bsvFn`	a function to be applied to each subset that returns a list of between subset variables (BSVs)
`output`	a "kvConnection" object indicating where the output data should reside (see `localDiskConn`, `hdfsConn`). If `NULL` (default), output will be an in-memory "ddo" object.
`overwrite`	logical; should existing output location be overwritten? (also can specify `overwrite = "backup"` to move the existing output to _bak)
`preTransFn`	a transformation function (if desired) to applied to each subset prior to division - note: this is deprecated - instead use `addTransform` prior to calling divide
`postTransFn`	a transformation function (if desired) to apply to each post-division subset
`params`	a named list of objects external to the input data that are needed in the distributed computing (most should be taken care of automatically such that this is rarely necessary to specify)
`packages`	a vector of R package names that contain functions used in `fn` (most should be taken care of automatically such that this is rarely necessary to specify)
`control`	parameters specifying how the backend should handle things (most-likely parameters to `rhwatch` in RHIPE) - see `rhipeControl` and `localDiskControl`
`update`	should a MapReduce job be run to obtain additional attributes for the result data prior to returning?
`verbose`	logical - print messages about what is being done

The division methods this function will support include conditioning variable division for factors (implemented – see condDiv), conditioning variable division for numerical variables through shingles, random replicate (implemented – see rrDiv), and near-exact replicate. If by is a vector of variable names, the data will be divided by these variables. Alternatively, this can be specified by e.g. condDiv(c("var1", "var2")).

an object of class "ddf" if the resulting subsets are data frames. Otherwise, an object of class "ddo".

Ryan Hafen

recombine, ddo, ddf, condDiv, rrDiv

# divide iris data by Species by passing in a data frame
bySpecies <- divide(iris, by = "Species")
bySpecies

# divide iris data into random partitioning of ~30 rows per subset
irisRR <- divide(iris, by = rrDiv(30))
irisRR

# any ddf can be passed into divide:
irisRR2 <- divide(bySpecies, by = rrDiv(30))
irisRR2
bySpecies2 <- divide(irisRR2, by = "Species")
bySpecies2

# splitting on multiple columns
byEdSex <- divide(adult, by = c("education", "sex"))
byEdSex
byEdSex[[1]]

# splitting on a numeric variable
bySL <- ddf(iris) %>%
  addTransform(function(x) {
    x$slCut <- cut(x$Sepal.Length, 10)
    x
  }) %>%
  divide(by = "slCut")
bySL
bySL[[1]]