dlt_readwrite_stream: Write and read streaming SparkDataFrame in delta format

In zero323/dlt: SparkR Delta Lake Interface

Write and read streaming SparkDataFrame in delta format

Description

Write and read streaming SparkDataFrame in delta format

Usage

dlt_write_stream(df, path, ...)

dlt_read_stream(path, ...)

## S4 method for signature 'SparkDataFrame,character'
dlt_write_stream(df, path, ...)

Arguments

df	streaming SparkDataFrame
path	character path to write the data.
...	additional arguments passed to writer

Functions

• dlt_write_stream(): Generic

• dlt_read_stream(): Loads data stored in delta format, returning the result as a streaming SparkDataFrame

• dlt_write_stream(df = SparkDataFrame, path = character): Writes streaming SparkDataFrame using delta format

Note

dlt_read_stream since 1.0.0

dlt_write_stream since 1.0.0

Examples

## Not run: 
set.seed(323)
input_path <- tempfile()
output_path <- tempfile()

# Dummy input
data.frame(
  id = 1:12,
  key = rep(c("a", "b", "c"), each = 4),
  value = rnorm(12)
) %>%
  createDataFrame() %>%
  dlt_write(input_path)

# Read data as stream and write back to output location
query <- dlt_read_stream(path = input_path) %>%
  dlt_write_stream(
    path = output_path, queryName = "test", trigger.once = TRUE,
    checkpointLocation = file.path(output_path, "_checkpoints", "test")
  )

awaitTermination(query)

## End(Not run)

zero323/dlt documentation built on Dec. 1, 2022, 11:25 p.m.