get_warc: Get a WARC from the Common Crawl via AWS

Description Usage Arguments Value Examples

View source: R/warcs.R

Description

The WARC is cached in a directory specified by the cache argument to ccwarcs_options

Usage

1
2
get_warc(filename, offset, length, digest, include_headers = FALSE,
  .options = NULL)

Arguments

filename

AWS path to the WARC

offset

Starting byte offset for the chunk

length

Number of bytes in the chunk

digest

Common Crawl digest for requested chunk

include_headers

If TRUE, include the WARC and HTTP headers in the result. See below.

.options

An optional object of class ccwarcs_options

Value

HTML contents of the requested WARC, and optionally the WARC and HTTP headers. If include_headers = TRUE, the result is a list with elements headers and payload. Otherwise the result is a character string (vector) containing the HTML of the requested WARC.

Examples

1
2
3
4
5
6
7
8
9
## Not run: 
warc <- get_warc(filename = paste0("crawl-data/CC-MAIN-2019-35/segments/1566027317516.88/",
           "warc/CC-MAIN-20190822215308-20190823001308-00434.warc.gz"),
         offset = 938342027, length = 3019, digest = "ILB6S7TS5WMLJVJIUBRQA53XRK2I3DN7", 
         include_headers = TRUE)
cat(warc[[1]])
cat(warc[[2]])

## End(Not run)

jasonmtroos/ccwarcs documentation built on Nov. 4, 2019, 2:20 p.m.