get_robotstxt_http_get: storage for http request response objects
In robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
View source: R/get_robotstxt_http_get.R
rt_last_http R Documentation
storage for http request response objects
Description
storage for http request response objects
get_robotstxt() worker function to execute HTTP request
Usage
rt_last_http
get_robotstxt_http_get(
domain,
user_agent = utils::sessionInfo()$R.version$version.string,
ssl_verifypeer = 1
)
Arguments
domain
the domain to get tobots.txt. file for
user_agent
the user agent to use for HTTP request header
ssl_verifypeer
either 1 (default) or 0, if 0 it disables SSL peer verification, which
might help with robots.txt file retrieval
Format
An object of class environment of length 1.
robotstxt documentation built on Sept. 12, 2024, 7:36 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
View source: R/get_robotstxt_http_get.R
| rt_last_http | R Documentation |
storage for http request response objects
Description
storage for http request response objects
get_robotstxt() worker function to execute HTTP request
Usage
rt_last_http
get_robotstxt_http_get(
domain,
user_agent = utils::sessionInfo()$R.version$version.string,
ssl_verifypeer = 1
)
Arguments
domain |
the domain to get tobots.txt. file for |
user_agent |
the user agent to use for HTTP request header |
ssl_verifypeer |
either 1 (default) or 0, if 0 it disables SSL peer verification, which might help with robots.txt file retrieval |
Format
An object of class environment of length 1.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.