Files in ropenscilabs/robotstxt
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

.Rbuildignore
.gitignore
.travis.yml
DESCRIPTION
LICENSE
NAMESPACE
NEWS.md R/get_robotstxt.R R/get_robotstxt_http_get.R R/get_robotstxts.R R/guess_domain.R R/parse_robotstxt.R R/path_allowed.R R/paths_allowed.R R/paths_allowed_worker_robotstxt.R R/paths_allowed_worker_spiderbar.R R/pipe.R R/print_robotstxt.R R/print_robotstxt_text.R R/remove_domain.R R/robotstxt.R R/rt_cache.R R/rt_get_comments.R R/rt_get_fields.R R/rt_get_fields_worker.R R/rt_get_useragent.R R/sanitize_path.R R/sanitize_permission_values.R R/sanitize_permissions.R R/tools.R R/valid_robotstxt.R README.Rmd README.md
_old_.travis.yml
autotest.Rexec
benchmarks/spiderbar_and_futures.r
codecov.yml
cran-comments.md dev.r
inst/robotstxts/allow_single_bot.txt
inst/robotstxts/crawl_delay.txt
inst/robotstxts/disallow_all_for_BadBot.txt
inst/robotstxts/disallow_all_for_all.txt
inst/robotstxts/disallow_some_for_all.txt
inst/robotstxts/disallow_two_at_once.txt
inst/robotstxts/empty.txt
inst/robotstxts/host.txt
inst/robotstxts/robots_amazon.txt
inst/robotstxts/robots_bundestag.txt
inst/robotstxts/robots_cdc.txt
inst/robotstxts/robots_cdc2.txt
inst/robotstxts/robots_facebook.txt
inst/robotstxts/robots_facebook_unsupported.txt
inst/robotstxts/robots_google.txt
inst/robotstxts/robots_new_york_times.txt
inst/robotstxts/robots_pmeissner.txt
inst/robotstxts/robots_spiegel.txt
inst/robotstxts/robots_wikipedia.txt
inst/robotstxts/robots_wikipedia_20170706.txt
inst/robotstxts/robots_yahoo.txt
inst/robotstxts/selfhtml_Example.txt
inst/robotstxts/testing_comments.txt
inst/urls.txt
logo/github_footer.png
logo/robotstxt.jpeg
logo/robotstxt.jpeg~
logo/robotstxt.kra
logo/robotstxt.kra~
logo/robotstxt.png
logo/robotstxt.png~
man/get_robotstxt.Rd man/get_robotstxt_http_get.Rd man/get_robotstxts.Rd man/guess_domain.Rd man/is_valid_robotstxt.Rd man/named_list.Rd man/parse_robotstxt.Rd man/path_allowed.Rd man/paths_allowed.Rd man/paths_allowed_worker_robotstxt.Rd man/paths_allowed_worker_spiderbar.Rd man/pipe.Rd man/print.robotstxt.Rd man/print.robotstxt_text.Rd man/remove_domain.Rd man/robotstxt.Rd man/rt_cache.Rd man/rt_get_comments.Rd man/rt_get_fields.Rd man/rt_get_fields_worker.Rd man/rt_get_rtxt.Rd man/rt_get_useragent.Rd man/rt_list_rtxt.Rd man/sanitize_path.Rd man/sanitize_permission_values.Rd man/sanitize_permissions.Rd misc/spiderbar_issue_2_minimal_example.R
robotstxt.Rproj
tests/testthat.R tests/testthat/test_parser.R tests/testthat/test_paths_allowed.R tests/testthat/test_permissions.R tests/testthat/test_robotstxt.R tests/testthat/test_tools.R vignettes/using_robotstxt.R vignettes/using_robotstxt.Rmd
vignettes/using_robotstxt.html
ropenscilabs/robotstxt documentation built on Nov. 14, 2017, 4:21 a.m.