Man pages for robotstxt
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

get_robotstxtdownloading robots.txt file
get_robotstxt_http_getget_robotstxt() worker function to execute HTTP request
guess_domainfunction guessing domain from path
is_valid_robotstxtfunction that checks if file is valid / parsable robots.txt...
named_listmake automatically named list
parse_robotstxtfunction parsing robots.txt
path_allowedcheck if a bot has permissions to access page
paths_allowedcheck if a bot has permissions to access page(s)
pipere-export magrittr pipe operator
print.robotstxtprinting robotstxt
print.robotstxt_textprinting robotstxt_text
remove_domainfunction to remove domain from path
robotstxtGenerate a representations of a robots.txt file
rt_cacheget_robotstxt() cache
rt_get_commentsextracting comments from robots.txt
rt_get_fieldsextracting permissions from robots.txt
rt_get_fields_workerextracting robotstxt fields
rt_get_rtxtload robots.txt files saved along with the package
rt_get_useragentextracting HTTP useragents from robots.txt
rt_list_rtxtlist robots.txt files saved along with the package
sanitize_pathmaking paths uniform
sanitize_permissionstransforming permissions into regular expressions (whole...
sanitize_permission_valuestransforming permissions into regular expressions (values)
robotstxt documentation built on Sept. 2, 2017, 1:13 a.m.