paginate: Paginate results from API

Description Usage Arguments Details

View source: R/paginate.R

Description

Ideas for improvement:

Usage

1
paginate(url, head = FALSE, limit = 10000, verbose = FALSE)

Arguments

url

API endpoint

head

Return the data (FALSE) or the number of records (TRUE)?

limit

Number of items per query

verbose

Whether to produce verbose output.

Details

- Paginated endpoints return a 'X-Total-Count' header record with the total number of records. This could be used to pre-allocate space and avoid the while() loop. First call to endpoint could have limit=0 to avoid retrieving any results. - Move the JSON processing outside of the loop. Just accumulate all of the JSON into a single document. Evaluate whether this is an improvement! - Use HEAD to find expected number of results. - Could avoid final API query by checking if the number of results returned was less than limit.


trundler documentation built on July 29, 2020, 1:06 a.m.