get_robotstxt: downloading robots.txt file

Description Usage Arguments

View source: R/parse_robotstxt.R

Description

downloading robots.txt file

Usage

1
get_robotstxt(domain, warn = TRUE)

Arguments

domain

domain from which to download robots.txt file

warn

warn about being unable to download domain/robots.txt because of HTTP resposne status 404. If this happens,


robotstxt documentation built on May 29, 2017, 9:15 a.m.

Search within the robotstxt package
Search all R packages, documentation and source code