warc_file: Create a new WARC file

In hrbrmstr/jwatr: Tools to Query and Create Web Archive Files Using the Java Web Archive Toolkit

Description

Create a new WARC file

Usage

warc_file(path, gzip = TRUE)

Arguments

path	filesystem path and base name for WARC file
gzip	if TRUE then the resultant WARC file will be comprised of individual gzip streams for each WARC record (the same format the Creative Commons and Internet Archive WARC files are produced). Otherwise, an uncompressed WARC file will be created.

Note

A .warc or .warc.gz extension will be added to path by this function.

Examples

## Not run: 
tf <- tempfile("test")
wf <- warc_file(tf)
warc_write_response(wf, "https://rud.is/b/")
warc_write_response(wf, GET("https://rud.is/b/"))
warc_write_response(wf, "https://www.rstudio.com/")
warc_write_response(wf, "https://www.r-project.org/")
warc_write_response(wf, "http://che.org.il/wp-content/uploads/2016/12/pdf-sample.pdf")

POST(
  url = "https://data.police.uk/api/crimes-street/all-crime",
  query = list( lat = "52.629729", lng = "-1.131592", date = "2017-01")
) -> uk_res

warc_write_response(wf, uk_res)
warc_write_response(wf, "https://journal.r-project.org/RLogo.png")

close_warc_file(wf)
unlink(tf)

## End(Not run)

hrbrmstr/jwatr documentation built on May 31, 2019, 1:15 p.m.