Files in robotstxt
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

MD5
LICENSE
NEWS.md
DESCRIPTION
README.md
NAMESPACE
vignettes/using_robotstxt.Rmd man/guess_domain.Rd man/paths_allowed.Rd man/get_robotstxt.Rd man/rt_get_fields_worker.Rd man/sanitize_permission_values.Rd man/print.robotstxt_text.Rd man/remove_domain.Rd man/named_list.Rd man/rt_get_comments.Rd man/rt_cache.Rd man/is_valid_robotstxt.Rd man/sanitize_permissions.Rd man/get_robotstxt_http_get.Rd man/sanitize_path.Rd man/parse_robotstxt.Rd man/print.robotstxt.Rd man/pipe.Rd man/robotstxt.Rd man/rt_get_useragent.Rd man/rt_get_fields.Rd man/rt_get_rtxt.Rd man/rt_list_rtxt.Rd man/path_allowed.Rd tests/testthat.R tests/testthat/test_permissions.R tests/testthat/test_parser.R tests/testthat/test_robotstxt.R
build/vignette.rds
inst/robotstxts/robots_yahoo.txt
inst/robotstxts/robots_cdc2.txt
inst/robotstxts/robots_pmeissner.txt
inst/robotstxts/robots_new_york_times.txt
inst/robotstxts/robots_google.txt
inst/robotstxts/robots_facebook_unsupported.txt
inst/robotstxts/allow_single_bot.txt
inst/robotstxts/robots_facebook.txt
inst/robotstxts/disallow_all_for_all.txt
inst/robotstxts/host.txt
inst/robotstxts/selfhtml_Example.txt
inst/robotstxts/disallow_some_for_all.txt
inst/robotstxts/empty.txt
inst/robotstxts/robots_wikipedia.txt
inst/robotstxts/robots_spiegel.txt
inst/robotstxts/robots_amazon.txt
inst/robotstxts/disallow_all_for_BadBot.txt
inst/robotstxts/disallow_two_at_once.txt
inst/robotstxts/crawl_delay.txt
inst/robotstxts/robots_wikipedia_20170706.txt
inst/robotstxts/testing_comments.txt
inst/robotstxts/robots_bundestag.txt
inst/robotstxts/robots_cdc.txt
inst/doc/using_robotstxt.R inst/doc/using_robotstxt.Rmd
inst/doc/using_robotstxt.html
R/parse_robotstxt.R R/remove_domain.R R/path_allowed.R R/print_robotstxt_text.R R/paths_allowed.R R/rt_get_useragent.R R/rt_get_fields_worker.R R/sanitize_permission_values.R R/robotstxt.R R/get_robotstxt.R R/guess_domain.R R/print_robotstxt.R R/rt_get_comments.R R/pipe.R R/sanitize_permissions.R R/tools.R R/rt_cache.R R/sanitize_path.R R/valid_robotstxt.R R/rt_get_fields.R R/get_robotstxt_http_get.R
robotstxt documentation built on Sept. 2, 2017, 1:13 a.m.