rt_get_fields: extracting permissions from robots.txt
In ropenscilabs/robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
View source: R/rt_get_fields.R
rt_get_fields R Documentation
extracting permissions from robots.txt
Description
extracting permissions from robots.txt
Usage
rt_get_fields(txt, regex = "", invert = FALSE)
Arguments
txt
content of the robots.txt file
regex
regular expression specify field
invert
invert selection made via regex?
ropenscilabs/robotstxt documentation built on Jan. 21, 2023, 7:19 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
View source: R/rt_get_fields.R
| rt_get_fields | R Documentation |
extracting permissions from robots.txt
Description
extracting permissions from robots.txt
Usage
rt_get_fields(txt, regex = "", invert = FALSE)
Arguments
txt |
content of the robots.txt file |
regex |
regular expression specify field |
invert |
invert selection made via regex? |
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.