Man pages for robotstxt
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

as.list.robotstxt_textMethod as.list() for class robotstxt_text
fix_urlfix_url
get_robotstxtdownloading robots.txt file
get_robotstxt_http_getstorage for http request response objects
get_robotstxtsfunction to get multiple robotstxt files
guess_domainfunction guessing domain from path
http_domain_changedhttp_domain_changed
http_subdomain_changedhttp_subdomain_changed
http_was_redirectedhttp_was_redirected
is_suspect_robotstxtis_suspect_robotstxt
is_valid_robotstxtfunction that checks if file is valid / parsable robots.txt...
list_mergeMerge a number of named lists in sequential order
named_listmake automatically named list
null_to_defeaultnull_to_defeault
parse_robotstxtfunction parsing robots.txt
parse_urlparse_url
paths_allowedcheck if a bot has permissions to access page(s)
paths_allowed_worker_spiderbarpaths_allowed_worker spiderbar flavor
pipere-export magrittr pipe operator
print.robotstxtprinting robotstxt
print.robotstxt_textprinting robotstxt_text
remove_domainfunction to remove domain from path
request_handler_handlerrequest_handler_handler
robotstxtGenerate a representations of a robots.txt file
rt_cacheget_robotstxt() cache
rt_get_commentsextracting comments from robots.txt
rt_get_fieldsextracting permissions from robots.txt
rt_get_fields_workerextracting robotstxt fields
rt_get_rtxtload robots.txt files saved along with the package
rt_get_useragentextracting HTTP useragents from robots.txt
rt_list_rtxtlist robots.txt files saved along with the package
rt_request_handlerrt_request_handler
sanitize_pathmaking paths uniform
robotstxt documentation built on Sept. 4, 2020, 1:08 a.m.