Files in robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

NEWS.md

MD5

README.md

NAMESPACE

DESCRIPTION

LICENSE

build/vignette.rds

R/paths_allowed_worker_spiderbar.R R/rt_cache.R R/fix_url.R R/rt_get_useragent.R R/rt_request_handler_defaults.R R/tools.R R/is_suspect_robotstxt.R R/parse_robotstxt.R R/http_was_redirected.R R/sanitize_path.R R/rt_get_fields.R R/list_merge.R R/print_robotstxt_text.R R/rt_get_comments.R R/rt_get_fields_worker.R R/remove_domain.R R/robotstxt.R R/request_handler_handler.R R/http_subdomain_changed.R R/http_domain_changed.R R/null_to_default.R R/is_valid_robotstxt.R R/parse_url.R R/guess_domain.R R/rt_request_handler.R R/paths_allowed.R R/get_robotstxt.R R/print_robotstxt.R R/get_robotstxt_http_get.R R/get_robotstxts.R R/pipe.R R/as_list.R

vignettes/style.css

vignettes/using_robotstxt.Rmd man/as.list.robotstxt_text.Rd man/rt_cache.Rd man/paths_allowed_worker_spiderbar.Rd man/http_subdomain_changed.Rd man/rt_list_rtxt.Rd man/get_robotstxts.Rd man/get_robotstxt_http_get.Rd man/sanitize_path.Rd man/is_valid_robotstxt.Rd man/pipe.Rd man/guess_domain.Rd man/rt_get_useragent.Rd man/list_merge.Rd man/fix_url.Rd man/is_suspect_robotstxt.Rd man/rt_request_handler.Rd man/rt_get_fields_worker.Rd man/rt_get_rtxt.Rd man/paths_allowed.Rd man/rt_get_comments.Rd man/print.robotstxt.Rd man/print.robotstxt_text.Rd man/rt_get_fields.Rd man/remove_domain.Rd man/http_was_redirected.Rd man/request_handler_handler.Rd man/get_robotstxt.Rd man/robotstxt.Rd man/http_domain_changed.Rd man/named_list.Rd man/parse_url.Rd man/parse_robotstxt.Rd man/null_to_defeault.Rd tests/testthat.R tests/testthat/test_issue50.R tests/testthat/test_http_event_handling.R tests/testthat/test_path_examples_from_rfc.R tests/testthat/test_parser.R tests/testthat/test_paths_allowed.R tests/testthat/test_attribute_handling.R tests/testthat/test_tools.R tests/testthat/test_get_robotstxt.R tests/testthat/test_robotstxt.R

inst/urls.txt

inst/doc/using_robotstxt.html

inst/doc/using_robotstxt.R inst/doc/using_robotstxt.Rmd

inst/robotstxts/robots_cdc2.txt

inst/robotstxts/robots_commented_token.txt

inst/robotstxts/robots_wikipedia.txt

inst/robotstxts/robots_pmeissner.txt

inst/robotstxts/robots_amazon.txt

inst/robotstxts/rbloggers.txt

inst/robotstxts/disallow_all_for_all.txt

inst/robotstxts/testing_comments.txt

inst/robotstxts/host.txt

inst/robotstxts/robots_wikipedia_20170706.txt

inst/robotstxts/disallow_some_for_all.txt

inst/robotstxts/robots_new_york_times.txt

inst/robotstxts/robots_spiegel.txt

inst/robotstxts/robots_cdc.txt

inst/robotstxts/robots_facebook_unsupported.txt

inst/robotstxts/robots_yahoo.txt

inst/robotstxts/disallow_two_at_once.txt

inst/robotstxts/crawl_delay.txt

inst/robotstxts/allow_single_bot.txt

inst/robotstxts/empty.txt

inst/robotstxts/disallow_all_for_BadBot.txt

inst/robotstxts/robots_facebook.txt

inst/robotstxts/selfhtml_Example.txt

inst/robotstxts/robots_google.txt

inst/robotstxts/robots_bundestag.txt

inst/http_requests/http_404.rds

inst/http_requests/http_client_error.rds

inst/http_requests/http_html_content.rds

inst/http_requests/http_server_error.rds

inst/http_requests/http_redirect_www.rds

inst/http_requests/http_ok_1.rds

inst/http_requests/http_ok_3.rds

inst/http_requests/http_ok_2.rds

inst/http_requests/http_ok_4.rds

inst/http_requests/http_domain_change.rds

robotstxt documentation built on Sept. 4, 2020, 1:08 a.m.