extracting permissions from robots.txt
rt_get_fields(txt, regex = "", invert = FALSE)
content of the robots.txt file
regular expression specify field
invert selection made via regex?
Questions? Problems? Suggestions? Tweet to @rdrrHQian@mutexlabs.com.
Please suggest features or report bugs with the GitHub issue tracker.
All documentation is copyright its authors; we didn't write any of that.