is_suspect_robotstxt: Check if file is valid / parsable robots.txt file
In petermeissner/robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
View source: R/is_suspect_robotstxt.R
is_suspect_robotstxt R Documentation
Check if file is valid / parsable robots.txt file
Description
Function that checks if file is valid / parsable robots.txt file
Usage
is_suspect_robotstxt(text)
Arguments
text
content of a robots.txt file provides as character vector
petermeissner/robotstxt documentation built on Nov. 17, 2024, 9:50 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
View source: R/is_suspect_robotstxt.R
| is_suspect_robotstxt | R Documentation |
Check if file is valid / parsable robots.txt file
Description
Function that checks if file is valid / parsable robots.txt file
Usage
is_suspect_robotstxt(text)
Arguments
text |
content of a robots.txt file provides as character vector |
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.