downloading robots.txt file
get_robotstxt(domain, warn = TRUE)
domain from which to download robots.txt file
warn about being unable to download domain/robots.txt because of
HTTP resposne status 404. If this happens,
Want to suggest features or report bugs for rdrr.io? Use the GitHub issue tracker.
Questions? Problems? Suggestions? Tweet to @rdrrHQian@mutexlabs.com.
All documentation is copyright its authors; we didn't write any of that.