Man pages for spiderbar
Parse and Test Robots Exclusion Protocol Files and Rules

can_fetchTest URL paths against a 'robxp' 'robots.txt' object
crawl_delaysRetrive all agent crawl delay values in a 'robxp'...
print.robxpCustom printer for 'robxp" objects
robxpParse a 'robots.txt' file & create a 'robxp' object
sitemapsRetrieve a character vector of sitemaps from a parsed...
spiderbarParse and Test Robots Exclusion Protocol Files and Rules
spiderbar documentation built on Sept. 25, 2017, 9:03 a.m.