rt_get_fields: extracting permissions from robots.txt

Description Usage Arguments

View source: R/parse_robotstxt.R

Description

extracting permissions from robots.txt

Usage

1
rt_get_fields(txt, regex = "", invert = FALSE)

Arguments

txt

content of the robots.txt file

regex

regular expression specify field

invert

invert selection made via regex?


robotstxt documentation built on May 19, 2017, 7 a.m.

Search within the robotstxt package
Search all R packages, documentation and source code

Questions? Problems? Suggestions? Tweet to @rdrrHQ or email at ian@mutexlabs.com.

Please suggest features or report bugs in the GitHub issue tracker.

All documentation is copyright its authors; we didn't write any of that.