API for robotstxt
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

Global functions
\%>\% Man page
get_robotstxt Man page Source code
get_robotstxt_http_get Man page Source code
guess_domain Man page Source code
is_valid_robotstxt Man page Source code
named_list Man page Source code
parse_robotstxt Man page Source code
path_allowed Man page Source code
paths_allowed Man page Source code
print.robotstxt Man page Source code
print.robotstxt_text Man page Source code
remove_domain Man page Source code
robotstxt Man page Source code
rt_cache Man page
rt_get_comments Man page Source code
rt_get_fields Man page Source code
rt_get_fields_worker Man page Source code
rt_get_rtxt Man page Source code
rt_get_useragent Man page Source code
rt_list_rtxt Man page Source code
sanitize_path Man page Source code
sanitize_permission_values Man page Source code
sanitize_permissions Man page Source code
robotstxt documentation built on Sept. 2, 2017, 1:13 a.m.