parse_robotstxt: Parse a robots.txt file

View source: R/parse_robotstxt.R

parse_robotstxtR Documentation

Parse a robots.txt file

Description

Parse a robots.txt file

Usage

parse_robotstxt(txt)

Arguments

txt

content of the robots.txt file

Value

a named list with useragents, comments, permissions, sitemap


ropenscilabs/robotstxt documentation built on Nov. 18, 2024, 8:14 a.m.