is_valid_robotstxt: function that checks if file is valid / parsable robots.txt...

In ropensci/robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

function that checks if file is valid / parsable robots.txt file

Description

function that checks if file is valid / parsable robots.txt file

Usage

is_valid_robotstxt(text, check_strickt_ascii = FALSE)

Arguments

text	content of a robots.txt file provides as character vector
check_strickt_ascii	whether or not to check if content does adhere to the specification of RFC to use plain text aka ASCII

ropensci/robotstxt documentation built on Jan. 7, 2023, 12:37 p.m.