Parquetr: R6 Object to wrap Spark connection

In zapier/parquetr: Easy methods for reading/writing Parquet files on S3

Description

R6 Object to wrap Spark connection

Usage

Parquetr

Format

R6Class object.

Value

Object of R6Class with methods for reading/writing Parquet on S3.

Fields

bucket

A R6S3 function that generates a target bucket

df

A data.frame to process

location

character Of the location relative to the bucket root (allowing for the root prefix of spark-storage) to interact with

Methods

new(bucket)

This method is used to create object of this class with bucket as the target bucket.

write_parquet(df, location, mode = "overwrite", ...)

This method is used to create object of this class with bucket as the target bucket.

Examples

## Not run: 
temp_table <- uuid::UUIDgenerate()
local_spark <- sparkConnection$new()
local_spark$write_parquet(iris, temp_table)
local_spark$read_parquet(temp_table)
local_spark$delete_parquet(temp_table)

## End(Not run)

zapier/parquetr documentation built on Jan. 1, 2021, 1:50 p.m.