parse_robotstxt: Parse a robots.txt file
In ropensci/robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
View source: R/parse_robotstxt.R
parse_robotstxt R Documentation
Parse a robots.txt file
Description
Parse a robots.txt file
Usage
parse_robotstxt(txt)
Arguments
txt
content of the robots.txt file
Value
a named list with useragents, comments, permissions, sitemap
ropensci/robotstxt documentation built on Nov. 16, 2024, 1:08 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
View source: R/parse_robotstxt.R
| parse_robotstxt | R Documentation |
Parse a robots.txt file
Description
Parse a robots.txt file
Usage
parse_robotstxt(txt)
Arguments
txt |
content of the robots.txt file |
Value
a named list with useragents, comments, permissions, sitemap
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.