persist: Storage Functions

In danzafar/tidyspark: A Tidy Interface to Spark

Description

Functions to manage the storage of spark_tbl objects. persist is capable of various storage modes (see details), and cache is shorthand for "MEMORY_AND_DISK" storage level.

Usage

cache(.data)

persist(.data, newLevel)

unpersist(.data, blocking = TRUE)

storage_level(.data)

Arguments

.data	a spark_tbl
newLevel	storage level chosen for the persistence. See available options in the details.
blocking	boolean, whether to block until all blocks are deleted.

Details

For details of the supported storage levels, refer to rdd-persistence.

Value

a spark_tbl

Examples

## Not run: 
spark_session()
iris_tbl <- spark_tbl(iris)

storage_level(iris_tbl)

persist(iris_tbl, "MEMORY_AND_DISK")
storage_level(iris_tbl)

unpersist(iris_tbl)
storage_level(iris_tbl)

## End(Not run)

danzafar/tidyspark documentation built on Sept. 30, 2020, 12:19 p.m.