spark_read_warc: Reads a WARC File into Apache Spark

Description Usage Arguments Examples

View source: R/sparkwarc.R

Description

Reads a WARC (Web ARChive) file into Apache Spark using sparklyr.

Usage

1
2
spark_read_warc(sc, name, path, repartition = 0L, memory = TRUE,
  overwrite = TRUE, group = FALSE, parse = FALSE, ...)

Arguments

sc

An active spark_connection.

name

The name to assign to the newly generated table.

path

The path to the file. Needs to be accessible from the cluster. Supports the "hdfs://", "s3n://" and "file://" protocols.

repartition

The number of partitions used to distribute the generated table. Use 0 (the default) to avoid partitioning.

memory

Boolean; should the data be loaded eagerly into memory? (That is, should the table be cached?)

overwrite

Boolean; overwrite the table with the given name if it already exists?

group

TRUE to group by warc segment. Currently supported only in HDFS and uncompressed files.

parse

TRUE to parse warc into tags, attribute, value, etc.

...

Additional arguments reserved for future use.

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
library(sparklyr)
sc <- spark_connect(master = "spark://HOST:PORT")
df <- spark_read_warc(
  sc,
  system.file("samples/sample.warc", package = "sparkwarc"),
  repartition = FALSE,
  memory = FALSE,
  overwrite = FALSE
)

spark_disconnect(sc)

sparkwarc documentation built on May 30, 2017, 6:16 a.m.