Man pages for petermeissner/robotstxt
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

get_robotstxtdownloading robots.txt file
get_robotstxt_http_getget_robotstxt() worker function to execute HTTP request
get_robotstxtsfunction to get multiple robotstxt files
guess_domainfunction guessing domain from path
is_valid_robotstxtfunction that checks if file is valid / parsable robots.txt...
named_listmake automatically named list
parse_robotstxtfunction parsing robots.txt
paths_allowedcheck if a bot has permissions to access page(s)
paths_allowed_worker_spiderbarpaths_allowed_worker spiderbar flavor
pipere-export magrittr pipe operator
print.robotstxtprinting robotstxt
print.robotstxt_textprinting robotstxt_text
remove_domainfunction to remove domain from path
robotstxtGenerate a representations of a robots.txt file
rt_cacheget_robotstxt() cache
rt_get_commentsextracting comments from robots.txt
rt_get_fieldsextracting permissions from robots.txt
rt_get_fields_workerextracting robotstxt fields
rt_get_rtxtload robots.txt files saved along with the package
rt_get_useragentextracting HTTP useragents from robots.txt
rt_list_rtxtlist robots.txt files saved along with the package
sanitize_pathmaking paths uniform
petermeissner/robotstxt documentation built on Oct. 10, 2018, 1:44 a.m.