rt_get_fields: extracting permissions from robots.txt

View source: R/rt_get_fields.R

rt_get_fieldsR Documentation

extracting permissions from robots.txt

Description

extracting permissions from robots.txt

Usage

rt_get_fields(txt, regex = "", invert = FALSE)

Arguments

txt

content of the robots.txt file

regex

regular expression specify field

invert

invert selection made via regex?


robotstxt documentation built on Sept. 12, 2024, 7:36 a.m.