Files in petermeissner/robotstxt
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

.Rbuildignore
.gitignore
.travis.yml
DESCRIPTION
LICENSE
NAMESPACE
NEWS.md R/get_robotstxt.R R/get_robotstxt_http_get.R R/get_robotstxts.R R/guess_domain.R R/http_domain_changed.R R/http_was_redirected.R R/is_suspect_robotstxt.R R/is_valid_robotstxt.R R/list_merge.R R/null_to_default.R R/parse_robotstxt.R R/paths_allowed.R R/paths_allowed_worker_spiderbar.R R/pipe.R R/print_robotstxt.R R/print_robotstxt_text.R R/remove_domain.R R/request_handler_handler.R R/robotstxt.R R/rt_cache.R R/rt_get_comments.R R/rt_get_fields.R R/rt_get_fields_worker.R R/rt_get_useragent.R R/rt_request_handler.R R/rt_request_handler_defaults.R R/sanitize_path.R R/subdomain_changed_to_www.R R/tools.R README.Rmd README.md
_old_.travis.yml
autotest.Rexec
benchmarks/spiderbar_and_futures.r
codecov.yml
cran-comments.md dev.r
inst/http_requests/http_404.rds
inst/http_requests/http_client_error.rds
inst/http_requests/http_domain_change.rds
inst/http_requests/http_html_content.rds
inst/http_requests/http_ok_1.rds
inst/http_requests/http_ok_2.rds
inst/http_requests/http_ok_3.rds
inst/http_requests/http_ok_4.rds
inst/http_requests/http_redirect_www.rds
inst/http_requests/http_server_error.rds
inst/robotstxts/allow_single_bot.txt
inst/robotstxts/crawl_delay.txt
inst/robotstxts/disallow_all_for_BadBot.txt
inst/robotstxts/disallow_all_for_all.txt
inst/robotstxts/disallow_some_for_all.txt
inst/robotstxts/disallow_two_at_once.txt
inst/robotstxts/empty.txt
inst/robotstxts/host.txt
inst/robotstxts/robots_amazon.txt
inst/robotstxts/robots_bundestag.txt
inst/robotstxts/robots_cdc.txt
inst/robotstxts/robots_cdc2.txt
inst/robotstxts/robots_facebook.txt
inst/robotstxts/robots_facebook_unsupported.txt
inst/robotstxts/robots_google.txt
inst/robotstxts/robots_new_york_times.txt
inst/robotstxts/robots_pmeissner.txt
inst/robotstxts/robots_spiegel.txt
inst/robotstxts/robots_wikipedia.txt
inst/robotstxts/robots_wikipedia_20170706.txt
inst/robotstxts/robots_yahoo.txt
inst/robotstxts/selfhtml_Example.txt
inst/robotstxts/testing_comments.txt
inst/urls.txt
logo/github_footer.png
logo/robotstxt.jpeg
logo/robotstxt.jpeg~
logo/robotstxt.kra
logo/robotstxt.kra~
logo/robotstxt.png
logo/robotstxt.png~
man/get_robotstxt.Rd man/get_robotstxt_http_get.Rd man/get_robotstxts.Rd man/guess_domain.Rd man/http_domain_changed.Rd man/http_was_redirected.Rd man/is_suspect_robotstxt.Rd man/is_valid_robotstxt.Rd man/list_merge.Rd man/named_list.Rd man/null_to_defeault.Rd man/parse_robotstxt.Rd man/paths_allowed.Rd man/paths_allowed_worker_spiderbar.Rd man/pipe.Rd man/print.robotstxt.Rd man/print.robotstxt_text.Rd man/remove_domain.Rd man/request_handler_handler.Rd man/robotstxt.Rd man/rt_cache.Rd man/rt_get_comments.Rd man/rt_get_fields.Rd man/rt_get_fields_worker.Rd man/rt_get_rtxt.Rd man/rt_get_useragent.Rd man/rt_list_rtxt.Rd man/rt_request_handler.Rd man/sanitize_path.Rd man/subdomain_changed_to_www.Rd misc/spiderbar_issue_2_minimal_example.R
robotstxt.Rproj
tests/testthat.R tests/testthat/test_attribute_handling.R tests/testthat/test_get_robotstxt.R tests/testthat/test_http_event_handling.R tests/testthat/test_parser.R tests/testthat/test_paths_allowed.R tests/testthat/test_robotstxt.R tests/testthat/test_tools.R
vignettes/style.css
vignettes/using_robotstxt.Rmd
petermeissner/robotstxt documentation built on Jan. 14, 2019, 9:51 p.m.