rqdatatable
/rquery
is designed to have a number of different modes of use. The primary intended one the considered mode of building up a pipelines from a description of the tables to be acted on.
(Note the R
/rquery
version of this example can be found here, and the Python
/data_algebra
version can be found here.)
For our example, lets start with the following example data.
d <- data.frame( x = c(1, 2, 3, 4, 5, 6), y = c(2, 2, 2, 3, 7, 10), g = c('a', 'a', 'a', 'b', 'b' ,'b'), stringsAsFactors = FALSE ) knitr::kable(d)
For our task: let's find a row with the largest ratio of 'y' to 'x', per group 'g'.
The rquery
concept is to break this into small sub-goals and steps:
In the standard rquery
practice we build up our processing pipeline to follow our above plan. The translation involves some familiarity with the rquery
steps, including the row-numbering command row_number()
.
library(rqdatatable) library(rquery) ops <- local_td(d) %.>% # Describe table for later operations extend(., # add a new column ratio := y / x) %.>% extend(., # rank the rows by group and order simple_rank := row_number(), partitionby = 'g', orderby = 'ratio', reverse = 'ratio') %.>% extend(., # mark the rows we want choice := simple_rank == 1)
The ops
operator pipeline can than be used to process data.
d %.>% ops %.>% knitr::kable(.)
Another point is: this form documents check-able (and enforceable) pre and post conditions on the calculation. For example such a calculation documents what columns are required by the calculation, and which ones are produced.
# columns produced column_names(ops) # columns used columns_used(ops)
We can in fact make these conditions the explicit basis of an interpretation of these data transforms as category theory arrows.
arrow(ops)
Another way to use rquery
/rqdatatable
is in "immediate mode", where we send the data from pipeline stage to pipeline state.
d %.>% extend(., # add a new column ratio := y / x) %.>% extend(., # rank the rows by group and order simple_rank := row_number(), partitionby = 'g', orderby = 'ratio', reverse = 'ratio') %.>% extend(., # mark the rows we want choice := simple_rank == 1) %.>% knitr::kable(.)
Immediate mode skips the local_td()
step of building a specification for the data to later come, and runs directly on the data. The advantage of this mode is: it is quick for the user. A disadvantage of this mode is: the pipeline is not left for re-use, and there are possibly expensive data conversions (from data.frame
to data.table
) at each operator stage.
d %.>% wrap %.>% # wrap data in a description extend(., # add a new column ratio := y / x) %.>% extend(., # rank the rows by group and order simple_rank := row_number(), partitionby = 'g', orderby = 'ratio', reverse = 'ratio') %.>% extend(., # mark the rows we want choice := simple_rank == 1) %.>% ex %.>% # signal construction done, and execute knitr::kable(.)
The difference is: we use the wrap
to build a special operator collecting (and checking) pipeline, and then later ex
to say we are done specifying steps and to apply the operations to the data. Prior to the ex
step the operator pipeline is available as a field called underlying
and the set of wrapped data.frame
s is available as a field called data_map
.
Additional execution options are as follows.
We can use local_td
to convert tables to table descriptions and delay execution.
ops <- d %.>% local_td %.>% # capture table description extend(., # add a new column ratio := y / x) %.>% extend(., # rank the rows by group and order simple_rank := row_number(), partitionby = 'g', orderby = 'ratio', reverse = 'ratio') %.>% extend(., # mark the rows we want choice := simple_rank == 1)
We can then execute using the pipe.
# execute d %.>% ops %.>% knitr::kable(.)
Or by the ex_data_table()
method, which is useful for pipelines that contain many tables.
ex_data_table( # useful for multi table pipelines ops, tables = list('d' = d)) %.>% knitr::kable(.)
The wrapping of data as a different kind of rquery
pipeline is an example of using the "decorator pattern" (which can be considered as an object oriented variation of the functional monad pattern, ref). However, these are technical considerations that are the package developer's problem- not problems for the package users. Think of these terms as examples of things developers worry about so users don't have to worry about them.
set.seed(2019) n_rows <- 1000000 d_large <- data.frame( x = rnorm(n = n_rows), y = rnorm(n = n_rows), g = sample(paste0('v_', seq_len(n_rows/10)), size = n_rows, replace = TRUE), stringsAsFactors = FALSE ) f_compiled <- function(dat) { dat %.>% ops # use pre-compiled pipeline } f_immediate <- function(dat) { dat %.>% extend(., # add a new column ratio := y / x) %.>% extend(., # rank the rows by group and order simple_rank := row_number(), partitionby = 'g', orderby = 'ratio', reverse = 'ratio') %.>% extend(., # mark the rows we want choice := simple_rank == 1) } f_wrapped <- function(dat) { dat %.>% wrap %.>% # wrap data in a description extend(., # add a new column ratio := y / x) %.>% extend(., # rank the rows by group and order simple_rank := row_number(), partitionby = 'g', orderby = 'ratio', reverse = 'ratio') %.>% extend(., # mark the rows we want choice := simple_rank == 1) %.>% ex # signal construction done, and execute }
And we can also time data.table
itself (without the translation overhead, though we are adding in the time to convert the data.frame
).
library(data.table) f_data_table = function(dat) { dat <- data.table(dat) dat[ , ratio := y / x ][order(-ratio) , simple_rank := 1:.N, by = list(g) ][ , choice := simple_rank == 1] } f_data_table(d) %.>% knitr::kable(.)
library(microbenchmark) timings <- microbenchmark( rquery_compiled = f_compiled(d_large), rquery_immediate = f_immediate(d_large), rquery_wrapped = f_wrapped(d_large), data.table = f_data_table(d_large), times = 10L ) print(timings)
Notice, the speed differences are usually not that large for short pipelines. Then intent is: pipeline construction and data conversion steps should be cheap compared to the actual processing steps.
For those interested: we add dplyr
and dtplyr
to the timing comparisons here.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.