Files in spiderbar

Parse and Test Robots Exclusion Protocol Files and Rules

MD5

NAMESPACE

DESCRIPTION

LICENSE

R/RcppExports.R R/crawl-delay.r R/can-fetch.r R/robxp.r R/spiderbar-package.R

src/Makevars

src/url.h

src/url.cpp

src/RcppExports.cpp

src/agent.cpp

src/directive.h

src/psl.h

src/utf8.h

src/psl.cpp

src/repmain.cpp

src/robots.cpp

src/utf8.cpp

src/agent.h

src/directive.cpp

src/punycode.h

src/robots.h

src/punycode.cpp

inst/extdata/cdc-robots.txt

inst/extdata/github-robots.txt

inst/extdata/imdb-robots.txt

inst/extdata/wikipedia-robots.txt

inst/tinytest/test_spiderbar.R tests/tinytest.R man/spiderbar.Rd man/crawl_delays.Rd man/robxp.Rd man/can_fetch.Rd man/sitemaps.Rd man/print.robxp.Rd

spiderbar documentation built on Feb. 16, 2023, 7:39 p.m.