parse_robotstxt: function parsing robots.txt

Description Usage Arguments Value

View source: R/parse_robotstxt.R

Description

function parsing robots.txt

Usage

1

Arguments

txt

content of the robots.txt file

Value

a named list with useragents, comments, permissions, sitemap


petermeissner/robotstxt documentation built on Jan. 14, 2019, 9:51 p.m.