multi_req_perform: Perform multiple requests in parallel

Description Usage Arguments Value Limitations Examples

View source: R/multi-req.R

Description

This variation on req_perform() performs multiple requests in parallel. Unlike req_perform() it always succeeds; it will never throw an error. Instead it will return error objects, which are your responsibility to handle.

Exercise caution when using this function; it's easy to pummel a server with many simultaneous requests. Only use it with hosts designed to serve many files at once.

Usage

1
multi_req_perform(reqs, paths = NULL, pool = NULL, cancel_on_error = FALSE)

Arguments

reqs

A list of requests.

paths

An optional list of paths, if you want to download the request bodies to disks. If supplied, must be the same length as reqs.

pool

Optionally, a curl pool made by curl::new_pool(). Supply this if you want to override the defaults for total concurrent connections (100) or concurrent connections per host (6).

cancel_on_error

Should all pending requests be cancelled when you hit an error. Set this to TRUE to stop all requests as soon as you hit an error. Responses that were never performed will have class httr2_cancelled in the result.

Value

A list the same length as reqs where each element is either a response or an error.

Limitations

In general, where req_perform() might make multiple requests due to retries or OAuth failures, multi_req_perform() will make only make 1.

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# Requesting these 4 pages one at a time would take four seconds:
reqs <- list(
  request("https://httpbin.org/delay/1"),
  request("https://httpbin.org/delay/1"),
  request("https://httpbin.org/delay/1"),
  request("https://httpbin.org/delay/1")
)
# But it's much faster if you request in parallel
system.time(resps <- multi_req_perform(reqs))

reqs <- list(
  request("https://httpbin.org/status/200"),
  request("https://httpbin.org/status/400"),
  request("FAILURE")
)
# multi_req_perform() will always succeed
resps <- multi_req_perform(reqs)
# you'll need to inspect the results to figure out which requests fails
fail <- vapply(resps, inherits, "error", FUN.VALUE = logical(1))
resps[fail]

httr2 documentation built on Sept. 28, 2021, 5:10 p.m.