rt_get_fields_worker: extracting robotstxt fields

Description Usage Arguments

View source: R/parse_robotstxt.R

Description

extracting robotstxt fields

Usage

1
rt_get_fields_worker(txt, type = "all", regex = NULL, invert = FALSE)

Arguments

txt

content of the robots.txt file

type

name or names of the fields to be returned, defaults to all fields

regex

subsetting field names via regular expressions

invert

field selection


robotstxt documentation built on May 29, 2017, 9:15 a.m.

Search within the robotstxt package
Search all R packages, documentation and source code