drQuantile: Sample Quantiles for 'ddf' Objects
In datadr: Divide and Recombine for Large, Complex Data

Description Usage Arguments Details Value Author(s) See Also Examples

Compute sample quantiles for 'ddf' objects

1
2
3

drQuantile(x, var, by = NULL, probs = seq(0, 1, 0.005), preTransFn = NULL,
  varTransFn = identity, varRange = NULL, nBins = 10000, tails = 100,
  params = NULL, packages = NULL, control = NULL, ...)

`x`	a 'ddf' object
`var`	the name of the variable to compute quantiles for
`by`	an optional variable name or vector of variable names by which to group quantile computations
`probs`	numeric vector of probabilities with values in [0-1]
`preTransFn`	a transformation function (if desired) to applied to each subset prior to computing quantiles (here it may be useful for adding a "by" variable that is not present) - note: this transformation should not modify `var` (use `varTransFn` for that) - also note: this is deprecated - instead use `addTransform` prior to calling divide
`varTransFn`	transformation to apply to variable prior to computing quantiles
`varRange`	range of x (can be left blank if summaries have been computed)
`nBins`	how many bins should the range of the variable be split into?
`tails`	how many exact values at each tail should be retained?
`params`	a named list of objects external to the input data that are needed in the distributed computing (most should be taken care of automatically such that this is rarely necessary to specify)
`packages`	a vector of R package names that contain functions used in `fn` (most should be taken care of automatically such that this is rarely necessary to specify)
`control`	parameters specifying how the backend should handle things (most-likely parameters to `rhwatch` in RHIPE) - see `rhipeControl` and `localDiskControl`
`...`	additional arguments

This division-agnostic quantile calculation algorithm takes the range of the variable of interest and splits it into nBins bins, tabulates counts for those bins, and reconstructs a quantile approximation from them. nBins should not get too large, but larger nBins gives more accuracy. If tails is positive, the first and last tails ordered values are attached to the quantile estimate - this is useful for long-tailed distributions or distributions with outliers for which you would like more detail in the tails.

data frame of quantiles q and their associated f-value fval. If by is specified, then also a variable group.

Ryan Hafen

updateAttributes

# break the iris data into k/v pairs
irisSplit <- list(
  list("1", iris[1:10,]), list("2", iris[11:110,]), list("3", iris[111:150,])
)
# represent it as ddf
irisSplit <- ddf(irisSplit, update = TRUE)

# approximate quantiles over the divided data set
probs <- seq(0, 1, 0.005)
iq <- drQuantile(irisSplit, var = "Sepal.Length", tails = 0, probs = probs)
plot(iq$fval, iq$q)

# compare to the all-data quantile "type 1" result
plot(probs, quantile(iris$Sepal.Length, probs = probs, type = 1))