paginate: Paginate results from API

In trundler: Historical Retail Data from the 'Trundler' API

Description

Ideas for improvement:

Usage

paginate(url, head = FALSE, limit = 10000, verbose = FALSE)

Arguments

url	API endpoint
head	Return the data (FALSE) or the number of records (TRUE)?
limit	Number of items per query
verbose	Whether to produce verbose output.

Details

- Paginated endpoints return a 'X-Total-Count' header record with the total number of records. This could be used to pre-allocate space and avoid the while() loop. First call to endpoint could have limit=0 to avoid retrieving any results. - Move the JSON processing outside of the loop. Just accumulate all of the JSON into a single document. Evaluate whether this is an improvement! - Use HEAD to find expected number of results. - Could avoid final API query by checking if the number of results returned was less than limit.

trundler documentation built on July 29, 2020, 1:06 a.m.