get_robotstxt: downloading robots.txt file

Description Usage Arguments

View source: R/get_robotstxt.R

Description

downloading robots.txt file

Usage

1
get_robotstxt(domain, warn = TRUE, force = FALSE, user_agent = NULL)

Arguments

domain

domain from which to download robots.txt file

warn

warn about being unable to download domain/robots.txt because of

force

if TRUE instead of using possible cached results the function will re-download the robotstxt file HTTP response status 404. If this happens,

user_agent

HTTP user-agent string to be used to retrieve robots.txt file from domain


robotstxt documentation built on Sept. 2, 2017, 1:13 a.m.