In nathaneastwood/sparkts: Call Methods from the spark-ts Package

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#",
  fig.path = "tools/images/README-"
)
library(sparkts)

sparkts

The goal of sparkts is to provide a test bed of sparklyr extensions for the spark-ts framework which was modified from the spark-timeseries framework.

Installation

You can install sparkts from GitHub with:

# install.packages("devtools")
devtools::install_github("nathaneastwood/sparkts")

For details on how to set up for further developing the package, please see the development vignette.

Example

This is a basic example which shows you how to calculate the standard error for some time series data:

library(sparkts)

# Set up a spark connection
sc <- sparklyr::spark_connect(
  master = "local",
  version = "2.2.0",
  config = list(sparklyr.gateway.address = "127.0.0.1")
)

# Extract some data
std_data <- spark_read_json(
  sc,
  "std_data",
  path = system.file(
    "data_raw/StandardErrorDataIn.json",
    package = "sparkts"
  )
) %>%
  spark_dataframe()

# Call the method
p <- sdf_standard_error(
  sc = sc, data = std_data,
  x_col = "xColumn", y_col = "yColumn", z_col = "zColumn",
  new_column_name = "StandardError"
)

p %>% dplyr::collect()

# Disconnect from the spark connection
spark_disconnect(sc = sc)