robxp: Parse a 'robots.txt' file & create a 'robxp' object

In spiderbar: Parse and Test Robots Exclusion Protocol Files and Rules

Parse a 'robots.txt' file & create a 'robxp' object

Description

This function takes in a single element character vector and parses it into a 'robxp' object.

Usage

robxp(x)

Arguments

x

either an atomic character vector containing a complete 'robots.txt“ file _or_ a length >1 character vector that will concatenated into a single string _or_ a 'connection' object that will be passed to [readLines()], the result of which will be concatenated into a single string and parsed and the connection will be closed.

Value

a classed object holding an external pointer to parsed robots.txt data

Examples

imdb <- paste0(readLines(system.file("extdata", "imdb-robots.txt",
               package="spiderbar")), collapse="\n")
rt <- robxp(imdb)

spiderbar documentation built on Feb. 16, 2023, 7:39 p.m.