sdf_melt: Call the melt method

In nathaneastwood/sparkts: Call Methods from the spark-ts Package

Description

This method will take a sequence of column names (strings) and unpivots them into two columns, the "variable_name" and its values.

Usage

sdf_melt(sc, data, id_variables, value_variables, variable_name, value_name)

Arguments

sc	A spark_connection.
data	A jobj: the Spark DataFrame on which to perform the function.
id_variables	list(string). Column(s) which are used as unique identifiers.
value_variables	list(string). Column(s) which are being unpivoted.
variable_name	c(string). The name of a new column, which holds all the value_variables names, defaulted to "variable".
value_name	c(string). The name of a new column, which holds all the values of value_variables column(s). Defaults to "value".

Value

Returns a jobj

Examples

## Not run: 
# Set up a spark connection
sc <- spark_connect(master = "local", version = "2.2.0")

# Extract some data
melt_data <- spark_read_json(
  sc,
  "melt_data",
  path = system.file(
    "data_raw/Melt.json",
    package = "sparkts"
  )
) %>%
  spark_dataframe()

# Call the method
p <- sdf_melt(
  sc = sc, data = melt_data, id_variables = c("identifier", "date"),
  value_variables = c("two", "one", "three", "four"),
  variable_name = "variable", value_name = "turnover"
)

#' # Return the data to R
p %>% dplyr::collect()

spark_disconnect(sc = sc)

## End(Not run)

nathaneastwood/sparkts documentation built on May 25, 2019, 10:34 p.m.