crawl_delays: Retrieve all agent crawl delay values in a 'robxp'...

In spiderbar: Parse and Test Robots Exclusion Protocol Files and Rules

Retrieve all agent crawl delay values in a robxp robots.txt object

Description

Retrieve all agent crawl delay values in a robxp robots.txt object

Usage

crawl_delays(obj)

Arguments

obj	robxp object

Value

data frame of agents and their crawl delays

Note

-1 will be returned for any listed agent without a crawl delay setting

Examples

gh <- paste0(readLines(system.file("extdata", "github-robots.txt",
             package="spiderbar")), collapse="\n")
gh_rt <- robxp(gh)
crawl_delays(gh_rt)

imdb <- paste0(readLines(system.file("extdata", "imdb-robots.txt",
               package="spiderbar")), collapse="\n")
imdb_rt <- robxp(imdb)
crawl_delays(imdb_rt)

spiderbar documentation built on Feb. 16, 2023, 7:39 p.m.