Nothing
robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
Getting started
Browse package contents
Package details |
|
|---|---|
| Author | Pedro Baltazar [aut, cre], Peter Meissner [aut], Kun Ren [aut, cph] (Author and copyright holder of list_merge.R.), Oliver Keys [ctb] (original release code review), Rich Fitz John [ctb] (original release code review) |
| Maintainer | Pedro Baltazar <pedrobtz@gmail.com> |
| License | MIT + file LICENSE |
| Version | 0.7.15 |
| URL | https://docs.ropensci.org/robotstxt/ https://github.com/ropensci/robotstxt |
| Package repository | View on CRAN |
| Installation |
Install the latest version of this package by entering the following in R:
|
Try the robotstxt package in your browser
Any scripts or data that you put into this service are public.
