get_robotstxt: downloading robots.txt file

Description

downloading robots.txt file

Usage

1
get_robotstxt(domain, warn = TRUE)

Arguments

domain

domain from which to download robots.txt file

warn

warn about being unable to download domain/robots.txt because of HTTP resposne status 404. If this happens,


Questions? Problems? Suggestions? or email at ian@mutexlabs.com.

All documentation is copyright its authors; we didn't write any of that.