parse_robotstxt: function parsing robots.txt

View source: R/parse_robotstxt.R

parse_robotstxtR Documentation

function parsing robots.txt

Description

function parsing robots.txt

Usage

parse_robotstxt(txt)

Arguments

txt

content of the robots.txt file

Value

a named list with useragents, comments, permissions, sitemap


robotstxt documentation built on Sept. 12, 2024, 7:36 a.m.