spark_read_delta: Read a Delta file into a 'spark_tbl'.

In danzafar/tidyspark: A Tidy Interface to Spark

Description

Read a Delta file into a spark_tbl.

Usage

spark_read_delta(path, version = NULL, timestamp = NULL, ...)

Arguments

path	string, the path to the file. Needs to be accessible from the cluster.
version	numeric, the version of the Delta table. Can be obtained from the output of DESCRIBE HISTORY events. Alias of timestampAsOf.
timestamp	string, the time-based version of the Delta table to pull. Only date or timestamp strings are accepted. For example, "2019-01-01" and "2019-01-01T00:00:00.000Z". Alias of versionAsOf.
...	optional named arguments to the reader.

Details

Other options such as specifing a schema can be specified in the ... For more information on version and timestamp, see https://docs.databricks.com/delta/delta-batch.html#dataframereader-options

Value

a spark_tbl

Examples

## Not run: 
spark_session(sparkPackages = "io.delta:delta-core_2.11:0.5.0")

iris_tbl <- spark_tbl(iris)

iris_tbl %>%
  spark_write_delta("/tmp/iris_tbl")

spark_read_delta("/tmp/iris_tbl") %>%
  collect

## End(Not run)

danzafar/tidyspark documentation built on Sept. 30, 2020, 12:19 p.m.