sdf_utils: Utility functions for importing a Spark data frame into a...

Description Arguments

Description

These functions provide an interface for specifying how a Spark data frame should be imported into a TimeSeriesRDD (e.g., which column represents time, whether rows are already ordered by time, and time unit being used, etc)

Arguments

sc

Spark connection

is_sorted

Whether the rows being imported are already sorted by time

time_unit

Time unit of the time column (must be one of the following values: "NANOSECONDS", "MICROSECONDS", "MILLISECONDS", "SECONDS", "MINUTES", "HOURS", "DAYS"

time_column

Name of the time column


sparklyr.flint documentation built on Jan. 11, 2022, 9:06 a.m.