refresh: Refreshing Data

In catalog: Access the 'Spark Catalog' API via 'sparklyr'

Refreshing Data

Description

• recover_partitions(): Recovers all the partitions in the directory of a table and update the catalog. This only works for partitioned tables and not un-partitioned tables or views.

• refresh_by_path(): Invalidates and refreshes all the cached data (and the associated metadata) for any Dataset that contains the given data source path. Path matching is by prefix, i.e. "/" would invalidate everything that is cached.

• refresh_table(): Invalidates and refreshes all the cached data and metadata of the given table. For performance reasons, Spark SQL or the external data source library it uses might cache certain metadata about a table, such as the location of blocks. When those change outside of Spark SQL, users should call this function to invalidate the cache. If this table is cached as an InMemoryRelation, drop the original cached version and make the new version cached lazily.

Usage

recover_partitions(sc, table)

refresh_by_path(sc, path)

refresh_table(sc, table)

Arguments

sc	A spark_connection.
table	character(1). The name of the table.
path	character(1). The path to refresh.

Value

NULL, invisibly. These functions are mostly called for their side effects.

See Also

cache_table(), create_table(), get_table(), list_tables(), table_exists(), uncache_table()

catalog documentation built on Aug. 23, 2022, 1:07 a.m.