In ropensci/canpaginate: Pagination Helper For Working With 'REST' 'APIs'

canpaginate

knitr::opts_chunk$set(
  comment = "#>",
  collapse = TRUE,
  warning = FALSE,
  message = FALSE
)

canpaginate helps you paginate with REST APIs

(p.s. named inspired from Ruby's will_paginate, though this pkgs focus is slightly diff. from that of will_paginate)

Why? The problems:

REST APIs are super variable in how they do pagination. There's no one way to do it. In my experience most nearly all APIs expect users to paginate via query parameters, while some APIS return pagination information in their body and others in their response headers.
For query parameters, some have a pair (one for how many records, another for what record to start at - but the names for these parameters are all over the place), while others have a single parameter for number of records, and others use a cursor format for continuing to next page.
On the user side, a very common pain point is "paging" through an API. That is, say you want 100 records, while the API only returns 10 per request. Thus, you need to make 10 requests of 10 records each. Those familiar with APIs can sort this problem out easily, but most people are not going to be able to do it easily.

install

devtools::install_github("ropensci/canpaginate")

library(canpaginate)

Set pagination options

query parameters

x <- can_paginate(
  by = "query_params",
  limit_param = "limit",
  offset_param = "offset",
  limit = 5000
)
x$by

link headers

x <- can_paginate(
  by = "link_headers",
  limit = 5000
)
x$limit

paginate

using options set via cp_init

(alternatively, set options in the call)

x <- can_paginate(
  by = "query_params",
  limit_param = "limit",
  offset_param = "offset",
  limit = 600
)
library(crul)
cli <- crul::HttpClient$new(url = "http://api.gbif.org")
res <- x$paginate(cli$get("v1/occurrence/search"))

Inspect results

res
res$status()
res$content()
res$parse()
res$as_df()

Verbose curl output

x$paginate(cli$get("get"), verbose = TRUE)

http library

Determined by what you pass in to paginate

only crul package will be supported, maybe others later

Meta

Please report any issues or bugs.
License: MIT
Get citation information for canpaginate in R doing citation(package = 'canpaginate')
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.