is_valid_robotstxt: function that checks if file is valid / parsable robots.txt...
In robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
View source: R/is_valid_robotstxt.R
Description
function that checks if file is valid / parsable robots.txt file
Usage
1
is_valid_robotstxt(text, check_strickt_ascii = FALSE)
Arguments
text
content of a robots.txt file provides as character vector
check_strickt_ascii
whether or not to check if content does adhere to the specification of RFC to use plain text aka ASCII
robotstxt documentation built on Sept. 4, 2020, 1:08 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
View source: R/is_valid_robotstxt.R
Description
function that checks if file is valid / parsable robots.txt file
Usage
1 | is_valid_robotstxt(text, check_strickt_ascii = FALSE)
|
Arguments
text |
content of a robots.txt file provides as character vector |
check_strickt_ascii |
whether or not to check if content does adhere to the specification of RFC to use plain text aka ASCII |
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.