API for ropensci/robotstxt
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

Global functions
%>% Man page
get_robotstxt Man page
get_robotstxt_http_get Man page
get_robotstxts Man page
guess_domain Man page
http_domain_changed Man page
http_was_redirected Man page
is_suspect_robotstxt Man page
is_valid_robotstxt Man page
list_merge Man page
named_list Man page
null_to_defeault Man page
on_client_error_default Man page
on_domain_change_default Man page
on_file_type_mismatch_default Man page
on_not_found_default Man page
on_redirect_default Man page
on_server_error_default Man page
on_suspect_content_default Man page
parse_robotstxt Man page
paths_allowed Man page
paths_allowed_worker_spiderbar Man page
print.robotstxt Man page
print.robotstxt_text Man page
remove_domain Man page
request_handler_handler Man page
robotstxt Man page
rt_cache Man page
rt_get_comments Man page
rt_get_fields Man page
rt_get_fields_worker Man page
rt_get_rtxt Man page
rt_get_useragent Man page
rt_list_rtxt Man page
rt_request_handler Man page
sanitize_path Man page
subdomain_changed_to_www Man page
ropensci/robotstxt documentation built on May 14, 2019, 2:11 p.m.