spiderbar | R Documentation |
The 'Robots Exclusion Protocol' (https://www.robotstxt.org/orig.html) documents a set
of standards for allowing or excluding robot/spider crawling of different areas of
site content. Tools are provided which wrap The rep-cpp
https://github.com/seomoz/rep-cpp
C++ library for processing these 'robots.txt“ files.
Bob Rudis (bob@rud.is)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.