robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
Version 0.4.0

Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, scrapers, ...) are allowed to access specific resources on a domain.

Package details

AuthorPeter Meissner [aut, cre], Oliver Keys [ctb], Rich Fitz John [ctb]
Date of publication2017-07-16 20:24:36 UTC
MaintainerPeter Meissner <>
LicenseMIT + file LICENSE
Package repositoryView on CRAN
Installation Install the latest version of this package by entering the following in R:

Try the robotstxt package in your browser

Any scripts or data that you put into this service are public.

robotstxt documentation built on July 17, 2017, 1:03 a.m.