View source: R/get_robotstxt_http_get.R
rt_last_http | R Documentation |
storage for http request response objects
get_robotstxt() worker function to execute HTTP request
rt_last_http get_robotstxt_http_get( domain, user_agent = utils::sessionInfo()$R.version$version.string, ssl_verifypeer = 1 )
domain |
the domain to get tobots.txt. file for |
user_agent |
the user agent to use for HTTP request header |
ssl_verifypeer |
analog to CURL option https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html – and might help with robots.txt file retrieval in some cases |
An object of class environment
of length 1.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.