collect_content: Fetch homepage HTML and text for domains

In rdomains: Get the Category of Content Hosted by a Domain

Fetch homepage HTML and text for domains

Description

Every requested domain comes back, in order, whether or not it was reachable. Failures carry a code from [fetch_error_codes()] rather than a bare 'NA'.

Usage

collect_content(
  domains = NULL,
  delay = 1,
  timeout = 10,
  max_bytes = 2 * 1024^2,
  obey_robots = TRUE,
  max_crawl_delay = 30,
  max_redirects = 5,
  user_agent = rdomains_user_agent()
)

Arguments

domains	Character vector of domains or URLs.
delay	Minimum seconds between requests to the same host. 'Crawl-delay' overrides this upward.
timeout	Per-request timeout, seconds.
max_bytes	Cap on the response body actually read.
obey_robots	Whether to fetch and honour robots.txt. Turning this off is discouraged and is your responsibility, not the package's.
max_crawl_delay	Skip a host that asks for a longer delay than this rather than sleeping on it.
max_redirects	Maximum redirect hops to follow. Every hop is re-validated, so a redirect cannot be used to reach an address the first check refused.
user_agent	Override the identifying user-agent.

Details

The crawler identifies itself as 'rdomains/<version>', obeys 'robots.txt' including 'Crawl-delay', spaces requests to the same host, caps the response body, and refuses to fetch hosts that resolve to private or link-local addresses.

Value

A tibble with one row per input: 'domain_name', 'status', 'stage', 'error_code', 'retryable', 'http_status', 'final_url', 'fetched_at', 'content_bytes', 'title', 'description', 'lang', 'text', 'n_tokens', 'page_state', 'block_vendor', 'robots_allowed', 'source_last_published'.

See Also

[fetch_report()] to summarise the run, [page_signals()] for what the page states are, [source_vintage()] for how a live fetch compares with the static lists.

Examples

## Not run: 
res <- collect_content(c("example.com", "wikipedia.org"))
fetch_report(res)

# retry only what is worth retrying
again <- collect_content(res$domain_name[res$retryable])

## End(Not run)

rdomains documentation built on Aug. 2, 2026, 9:07 a.m.