extracting permissions from robots.txt
rt_get_fields(txt, regex = "", invert = FALSE)
content of the robots.txt file
regular expression specify field
invert selection made via regex?
Want to suggest features or report bugs for rdrr.io? Use the GitHub issue tracker.
Questions? Problems? Suggestions? Tweet to @rdrrHQian@mutexlabs.com.
All documentation is copyright its authors; we didn't write any of that.