extracting robotstxt fields
rt_get_fields_worker(txt, type = "all", regex = NULL, invert = FALSE)
content of the robots.txt file
name or names of the fields to be returned, defaults to all
subsetting field names via regular expressions
Questions? Problems? Suggestions? Tweet to @rdrrHQian@mutexlabs.com.
Please suggest features or report bugs with the GitHub issue tracker.
All documentation is copyright its authors; we didn't write any of that.