Man pages for ropenscilabs/robotstxt
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

get_robotstxtdownloading robots.txt file
get_robotstxt_http_getget_robotstxt() worker function to execute HTTP request
get_robotstxtsfunction to get multiple robotstxt files
guess_domainfunction guessing domain from path
http_domain_changedhttp_domain_changed
http_was_redirectedhttp_was_redirected
is_suspect_robotstxtis_suspect_robotstxt
is_valid_robotstxtfunction that checks if file is valid / parsable robots.txt...
list_mergeMerge a number of named lists in sequential order
named_listmake automatically named list
null_to_defeaultnull_to_defeault
parse_robotstxtfunction parsing robots.txt
paths_allowedcheck if a bot has permissions to access page(s)
paths_allowed_worker_spiderbarpaths_allowed_worker spiderbar flavor
pipere-export magrittr pipe operator
print.robotstxtprinting robotstxt
print.robotstxt_textprinting robotstxt_text
remove_domainfunction to remove domain from path
request_handler_handlerrequest_handler_handler
robotstxtGenerate a representations of a robots.txt file
rt_cacheget_robotstxt() cache
rt_get_commentsextracting comments from robots.txt
rt_get_fieldsextracting permissions from robots.txt
rt_get_fields_workerextracting robotstxt fields
rt_get_rtxtload robots.txt files saved along with the package
rt_get_useragentextracting HTTP useragents from robots.txt
rt_list_rtxtlist robots.txt files saved along with the package
rt_request_handlerrt_request_handler
sanitize_pathmaking paths uniform
subdomain_changed_to_wwwsubdomain_changed_to_www
ropenscilabs/robotstxt documentation built on Dec. 2, 2018, 10:13 a.m.