get_robotstxt_http_get: storage for http request response objects

In ropensci/robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

storage for http request response objects

Description

storage for http request response objects

get_robotstxt() worker function to execute HTTP request

Usage

rt_last_http

get_robotstxt_http_get(
  domain,
  user_agent = utils::sessionInfo()$R.version$version.string,
  ssl_verifypeer = 1
)

Arguments

domain	the domain to get tobots.txt. file for
user_agent	the user agent to use for HTTP request header
ssl_verifypeer	analog to CURL option https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html – and might help with robots.txt file retrieval in some cases

Format

An object of class environment of length 1.

ropensci/robotstxt documentation built on Jan. 7, 2023, 12:37 p.m.