knitr::opts_chunk$set( collapse = TRUE, comment = "#>" )
Currently sparklyr.flint
supports a number of commonly used summarizers (e.g., count, sum, average, etc) that are implemented in the Flint time series library. Each summarizer can be either applied to a moving time window (e.g., in_past(5s)
) or groups of rows within a TimeSeriesRDD
having the same timestamps (which is known as a “cycle” in Flint nomenclature).
The following is a quick example of applying the sum summarizer to a moving time window:
library(sparklyr) library(sparklyr.flint) # Step 0: decide which Spark version to use, how to connect to Spark, etc spark_version <- "3.0.0" sc <- spark_connect(master = "local", version = spark_version) example_time_series <- data.frame( t = c(1, 3, 4, 6, 7, 10, 15, 16, 18, 19), v = c(4, -2, NA, 5, NA, 1, -4, 5, NA, 3) ) # Step 1: import example time series data into a Spark dataframe sdf <- copy_to(sc, example_time_series, overwrite = TRUE) # Step 2: specify how the Spark dataframe should be interpreted as a time series by Flint ts_rdd <- fromSDF(sdf, is_sorted = TRUE, time_unit = "SECONDS", time_column = "t") # Step 3: apply a Flint summarizer to the time series above sum <- summarize_sum(ts_rdd, column = "v", window = in_past("3s")) # Step 4: collect summarized result from Spark to R res <- ts_sum %>% collect() print(res)
## # A tibble: 10 x 3 ## time v v_sum ## <dttm> <dbl> <dbl> ## 1 1970-01-01 00:00:01 4 4 ## 2 1970-01-01 00:00:03 -2 2 ## 3 1970-01-01 00:00:04 NaN 2 ## 4 1970-01-01 00:00:06 5 3 ## 5 1970-01-01 00:00:07 NaN 5 ## 6 1970-01-01 00:00:10 1 1 ## 7 1970-01-01 00:00:15 -4 -4 ## 8 1970-01-01 00:00:16 5 1 ## 9 1970-01-01 00:00:18 NaN 1 ## 10 1970-01-01 00:00:19 3 8
From the result above, one can see as a result of specifying window = in_past("3s")
, for each time point t
from example_time_series
(i.e., t = 1
, t = 3
, t = 4
, t = 6
, and so on), Flint has created a row containing t
and the summation of all v
value(s) occurring within the time window of [t - 3, t]
, and the sums are stored in a new column named v_sum
.
Given a timestamp t
, the subset of rows in a TimeSeriesRDD
having that timestamp is known as a "cycle" in Flint.
If the window = "<time window specification>"
argument is omitted, then the summarizer function will look at all cycles in the TimeSeriesRDD
. In other words, it will group all rows by their timestamps and perform aggregation within each group.
For example:
ts_sum <- summarize_sum(ts_rdd, column = "v")
will return a TimeSeriesRDD
with a timestamp column named time
and a summation column named v_sum
. For each timestamp t
present in ts_rdd
, ts_sum
will contain a row with timestamp t
and v_sum
value equal to summation of all v
values occurring at t
.
Because all rows from ts_rdd
are already ordered internally by timestamps, aggregations on cycles can be performed efficiently in Flint without re-shuffling rows in the input TimeSeriesRDD
.
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.