crawl_urls: Crawl URLs in a data frame.

In stephLH/webr: Fonctions to crawl web pages

Description

Crawl URLs in a data frame.

Usage

crawl_urls(urls, parallel = FALSE, progress_bar = FALSE, message = FALSE, ...)

Arguments

urls	URLs to crawl.
parallel	If TRUE then URLs are crawled using multiple CPU cores (using parallel and pbapply packages).
progress_bar	If TRUE then a progress bar is displayed (using pbapply package).
message	If TRUE then a message indicates how URLs are crawled.
...	Optional arguments from webr::download_url

Value

A data frame with one URL per row.

This data frame contains the following colmuns :
- url : initial URL
- crawl : adresse nettoyée
- url_response : returned URL
- date_crawl : download datetime
- status_code : status code return by HTTP request
- error : error message

stephLH/webr documentation built on Dec. 25, 2019, 2:54 p.m.