rt_get_fields_worker: extracting robotstxt fields

Description

extracting robotstxt fields

Usage

1
rt_get_fields_worker(txt, type = "all", regex = NULL, invert = FALSE)

Arguments

txt

content of the robots.txt file

type

name or names of the fields to be returned, defaults to all fields

regex

subsetting field names via regular expressions

invert

field selection


Questions? Problems? Suggestions? or email at ian@mutexlabs.com.

All documentation is copyright its authors; we didn't write any of that.