get_robotstxt: downloading robots.txt file

Description Usage Arguments

Description

downloading robots.txt file

Usage

1
get_robotstxt(domain, warn = TRUE)

Arguments

domain

domain from which to download robots.txt file

warn

warn about being unable to download domain/robots.txt because of HTTP resposne status 404. If this happens,



Search within the robotstxt package
Search all R packages, documentation and source code

Questions? Problems? Suggestions? or email at ian@mutexlabs.com.

Please suggest features or report bugs with the GitHub issue tracker.

All documentation is copyright its authors; we didn't write any of that.