warc_GET: WARC-ify an httr::GET request

In hrbrmstr/jwatr: Tools to Query and Create Web Archive Files Using the Java Web Archive Toolkit

Description

Automagically writes out the result of an httr::GET request to an open WARC file connection enabling seamless recording/caching of the response for later re-use.

Usage

warc_GET(wobj, url = NULL, config = list(), ..., handle = NULL)

Arguments

wobj	WARC file object
url	the url of the page to retrieve
config	Additional configuration settings such as http authentication (authenticate), additional headers (add_headers), cookies (set_cookies) etc. See config for full details and list of helpers.
...	Further named parameters, such as query, path, etc, passed on to modify_url. Unnamed parameters will be combined with config.
handle	The handle to use with this request. If not supplied, will be retrieved and reused from the handle_pool based on the scheme, hostname and port of the url. By default httr requests to the same scheme/host/port combo. This substantially reduces connection time, and ensures that cookies are maintained over multiple requests to the same host. See handle_pool for more details.

Examples

## Not run: 
tf <- tempfile("test")
wf <- warc_file(tf)

warc_GET(wf, "https://data.police.uk/api/crimes-street/all-crime",
         query = list(lat=52.629729, lng=-1.131592, date="2017-01"))

warc_POST(
  wf,
  url = "https://data.police.uk/api/crimes-street/all-crime",
  query = list( lat = "52.629729", lng = "-1.131592", date = "2017-01")
) -> uk_res

close_warc_file(wf)
unlink(tf)

## End(Not run)

hrbrmstr/jwatr documentation built on May 31, 2019, 1:15 p.m.