mot_get_pages: Download multiple pages of MOT data

In ITSLeeds/motCarData: Download and analysis of MOT data

Description

Download one page of MOT data

Usage

mot_get_pages(
  pages = 2:11,
  key = Sys.getenv("MOT_key"),
  ncores = 8L,
  max_pages = 1000,
  dir = tempdir(),
  recombine = FALSE,
  get_comments = TRUE,
  ...
)

Arguments

pages	a range o page numberspage number
key	API Key
ncores	Now many cores to use
max_pages	Manximum number of pages to download per batch
dir	where to save batch files
recombine	logical should batches be combined togther
get_comments	logical, should comments be downloaded, default TRUE
...	extra variaibles passed to 'mot_get_page()'

Details

This function wraps 'mot_get_page()' for bulk downloading of data, by default it is multicore optimised and will run with 8 cores as this is the upper limit allowed by the API.

The function will create batches of pages defined by 'max_pages' and save them to disk. This prevents you running out of RAM. The default 1000 pages per batch is about 600 Mb in size. However you can reduce this significantly by setting 'get_comments' to false. The MOT test comments are by far the largest part of the data, and are not needed for many use cases. Without comments 1000 pages is around 300 Mb, so you could safely increase 'max_pages'.

ITSLeeds/motCarData documentation built on March 5, 2020, 3:17 a.m.