get_robotstxt: downloading robots.txt file

Description Usage Arguments

View source: R/parse_robotstxt.R

Description

downloading robots.txt file

Usage

1
get_robotstxt(domain, warn = TRUE, force = FALSE, user_agent = NULL)

Arguments

domain

domain from which to download robots.txt file

warn

warn about being unable to download domain/robots.txt because of

force

if TRUE instead of using possible cached results te function will re-download the robotstxt file HTTP resposne status 404. If this happens,

user_agent

HTTP user-agent string to be used to retireve robots.txt file from domain


robotstxt documentation built on July 17, 2017, 1:03 a.m.