Files in spiderbar
Parse and Test Robots Exclusion Protocol Files and Rules

README.md
LICENSE
NEWS.md
NAMESPACE
DESCRIPTION
MD5
R/RcppExports.R R/crawl-delay.r R/can-fetch.r R/spiderbar-package.R R/robxp.r
src/agent.h
src/robots.h
src/agent.cpp
src/psl.h
src/directive.h
src/utf8.h
src/psl.cpp
src/punycode.h
src/Makevars
src/punycode.cpp
src/url.cpp
src/url.h
src/repmain.cpp
src/utf8.cpp
src/robots.cpp
src/RcppExports.cpp
src/directive.cpp
man/crawl_delays.Rd man/spiderbar.Rd man/robxp.Rd man/can_fetch.Rd man/print.robxp.Rd man/sitemaps.Rd tests/test-all.R tests/testthat/test-spiderbar.R
inst/extdata/github-robots.txt
inst/extdata/imdb-robots.txt
inst/extdata/cdc-robots.txt
inst/extdata/wikipedia-robots.txt
spiderbar documentation built on Sept. 25, 2017, 9:03 a.m.