fetchR_parseR: Fetch a List of Url's.

In barob1n/crawlR: Async Web crawler for R.

Fetch a List of Url's.

Description

Fetch a List of Url's.

Usage

fetchR_parseR(
  out_dir = NULL,
  work_dir = NULL,
  fetch_list = NULL,
  crawl_delay = NULL,
  max_concurr = NULL,
  max_host = NULL,
  timeout = Inf,
  timeout_request = NULL,
  queue_scl = 1,
  comments = "",
  log_file = NULL,
  readability_content = F,
  parser = crawlR::parse_content_fetch
)

Arguments

out_dir	(Required) Current output directory.
work_dir	(Required) Current working directory.
fetch_list	(Required) Created by generateR.R.
crawl_delay	time (in seconds) for calls to the same host.
max_concurr	Max. total concurrent connections open at any given time.
max_host	Max. total concurrent connections per host at any given time.
timeout	Total (all requests) timeout
timeout_request	per request timeout
queue_scl	Scaler
comments	Some comments to print while running.
log_file	Name of log file. If null, writes to stdout().
readability_content	T
parser	parse func

Value

None.

barob1n/crawlR documentation built on May 23, 2023, 10:53 a.m.