is_suspect_robotstxt: is_suspect_robotstxt

View source: R/is_suspect_robotstxt.R

is_suspect_robotstxtR Documentation

is_suspect_robotstxt

Description

function that checks if file is valid / parsable robots.txt file

Usage

is_suspect_robotstxt(text)

Arguments

text

content of a robots.txt file provides as character vector


ropensci/robotstxt documentation built on Jan. 7, 2023, 12:37 p.m.