fetch_missing_pages_parallel: Download missing pages in parallel to local cache directory

In jefferis/gscraper: Basic web scraping - designed to generate local cache of remote website

Description

Download missing pages in parallel to local cache directory

Usage

  fetch_missing_pages_parallel(urls, cacheloc,
    fetchpagefn = fetch_page, njobs = 10,
    jobdir = tempfile("fmppjobs"), jobstem = "fmppjob",
    RunJobs = TRUE, ...)

Arguments

urls	Character vector of URLs
cacheloc	Path to local cache
fetchpagefn	Function that actually fetches the pages
njobs	Number of download jobs to run in parallel
jobdir	Directory that will contain download job scripts
jobstem	Filename prefix for shell scripts
RunJobs	Whether jobs are actualy run
...	Additional parameters passed to fetchpagefn

Details

parallel downloads are based on asynchronous shell scripts

Value

character vector of paths to job files

Author(s)

jefferis

See Also

Other fetchpage: fetch_page

jefferis/gscraper documentation built on Oct. 25, 2020, 12:08 a.m.