get_robotstxt_http_get: get_robotstxt() worker function to execute HTTP request

Description Usage Arguments

View source: R/get_robotstxt_http_get.R

Description

get_robotstxt() worker function to execute HTTP request

Usage

1
2
3
get_robotstxt_http_get(domain,
  user_agent = utils::sessionInfo()$R.version$version.string,
  ssl_verifypeer = 1)

Arguments

domain

the domain to get tobots.txt. file for

user_agent

the user agent to use for HTTP request header

ssl_verifypeer

analog to CURL option https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html – and might help with robots.txt file retrieval in some cases


petermeissner/robotstxt documentation built on May 25, 2019, 1:56 a.m.