Files in robotstxt
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

MD5
NEWS.md
LICENSE
DESCRIPTION
README.md
NAMESPACE
inst/urls.txt
inst/doc/using_robotstxt.html
inst/doc/using_robotstxt.Rmd inst/doc/using_robotstxt.R
inst/robotstxts/crawl_delay.txt
inst/robotstxts/robots_yahoo.txt
inst/robotstxts/robots_pmeissner.txt
inst/robotstxts/robots_wikipedia.txt
inst/robotstxts/disallow_some_for_all.txt
inst/robotstxts/testing_comments.txt
inst/robotstxts/robots_cdc2.txt
inst/robotstxts/selfhtml_Example.txt
inst/robotstxts/robots_new_york_times.txt
inst/robotstxts/disallow_all_for_BadBot.txt
inst/robotstxts/robots_spiegel.txt
inst/robotstxts/robots_google.txt
inst/robotstxts/robots_bundestag.txt
inst/robotstxts/robots_facebook.txt
inst/robotstxts/empty.txt
inst/robotstxts/robots_facebook_unsupported.txt
inst/robotstxts/host.txt
inst/robotstxts/disallow_two_at_once.txt
inst/robotstxts/robots_cdc.txt
inst/robotstxts/allow_single_bot.txt
inst/robotstxts/disallow_all_for_all.txt
inst/robotstxts/robots_amazon.txt
inst/robotstxts/robots_wikipedia_20170706.txt
vignettes/using_robotstxt.Rmd R/tools.R R/rt_get_fields.R R/pipe.R R/paths_allowed.R R/parse_robotstxt.R R/get_robotstxt.R R/sanitize_path.R R/valid_robotstxt.R R/paths_allowed_worker_spiderbar.R R/rt_get_fields_worker.R R/robotstxt.R R/rt_get_useragent.R R/print_robotstxt.R R/remove_domain.R R/print_robotstxt_text.R R/guess_domain.R R/rt_get_comments.R R/rt_cache.R R/get_robotstxts.R R/get_robotstxt_http_get.R man/remove_domain.Rd man/paths_allowed_worker_spiderbar.Rd man/robotstxt.Rd man/get_robotstxts.Rd man/rt_get_useragent.Rd man/print.robotstxt.Rd man/sanitize_path.Rd man/rt_get_fields_worker.Rd man/guess_domain.Rd man/parse_robotstxt.Rd man/rt_cache.Rd man/rt_get_rtxt.Rd man/paths_allowed.Rd man/rt_list_rtxt.Rd man/is_valid_robotstxt.Rd man/get_robotstxt.Rd man/print.robotstxt_text.Rd man/pipe.Rd man/rt_get_fields.Rd man/named_list.Rd man/get_robotstxt_http_get.Rd man/rt_get_comments.Rd
build/vignette.rds
tests/testthat.R tests/testthat/test_parser.R tests/testthat/test_paths_allowed.R tests/testthat/test_tools.R tests/testthat/test_robotstxt.R
robotstxt documentation built on July 19, 2018, 9:03 a.m.