downloading robots.txt file
get_robotstxt(domain, warn = TRUE)
domain from which to download robots.txt file
warn about being unable to download domain/robots.txt because of
HTTP resposne status 404. If this happens,
Questions? Problems? Suggestions? Tweet to @rdrrHQian@mutexlabs.com.
Please suggest features or report bugs with the GitHub issue tracker.
All documentation is copyright its authors; we didn't write any of that.