get_robotstxt_http_get: Storage for http request response objects

In petermeissner/robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

Storage for http request response objects

Description

Storage for http request response objects

Execute HTTP request for get_robotstxt()

Usage

rt_last_http

get_robotstxt_http_get(
  domain,
  user_agent = utils::sessionInfo()$R.version$version.string,
  ssl_verifypeer = 1
)

Arguments

domain	the domain to get robots.txt. file for
user_agent	the user agent to use for HTTP request header
ssl_verifypeer	either 1 (default) or 0, if 0 it disables SSL peer verification, which might help with robots.txt file retrieval

Format

An object of class environment of length 1.

petermeissner/robotstxt documentation built on Sept. 4, 2024, 6:53 a.m.