spiderbar: Parse and Test Robots Exclusion Protocol Files and Rules
Version 0.2.1

The 'Robots Exclusion Protocol' documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' C++ library for processing these 'robots.txt' files.

Getting started

Package details

AuthorBob Rudis ([email protected]) [aut, cre], SEOmoz, Inc [aut]
Date of publication2017-11-17 12:40:19 UTC
MaintainerBob Rudis <[email protected]>
LicenseMIT + file LICENSE
URL https://github.com/hrbrmstr/spiderbar
Package repositoryView on CRAN
Installation Install the latest version of this package by entering the following in R:

Try the spiderbar package in your browser

Any scripts or data that you put into this service are public.

spiderbar documentation built on Nov. 18, 2017, 4:09 a.m.