rt_get_fields_worker: extracting robotstxt fields

Description Usage Arguments

Description

extracting robotstxt fields

Usage

1
rt_get_fields_worker(txt, type = "all", regex = NULL, invert = FALSE)

Arguments

txt

content of the robots.txt file

type

name or names of the fields to be returned, defaults to all fields

regex

subsetting field names via regular expressions

invert

field selection



Search within the robotstxt package
Search all R packages, documentation and source code

Questions? Problems? Suggestions? or email at ian@mutexlabs.com.

Please suggest features or report bugs with the GitHub issue tracker.

All documentation is copyright its authors; we didn't write any of that.