Man pages for spiderbar
Parse and Test Robots Exclusion Protocol Files and Rules
| can_fetch | Test URL paths against a 'robxp' 'robots.txt' object |
| crawl_delays | Retrieve all agent crawl delay values in a 'robxp'... |
| print.robxp | Custom printer for 'robxp" objects |
| robxp | Parse a 'robots.txt' file & create a 'robxp' object |
| sitemaps | Retrieve a character vector of sitemaps from a parsed... |
| spiderbar | Parse and Test Robots Exclusion Protocol Files and Rules |
