extracting robotstxt fields
rt_get_fields_worker(txt, type = "all", regex = NULL, invert = FALSE)
content of the robots.txt file
name or names of the fields to be returned, defaults to all
subsetting field names via regular expressions
Want to suggest features or report bugs for rdrr.io? Use the GitHub issue tracker.
Questions? Problems? Suggestions? Tweet to @rdrrHQian@mutexlabs.com.
All documentation is copyright its authors; we didn't write any of that.