rt_request_handler: rt_request_handler

Description Usage Arguments Format Value

View source: R/rt_request_handler.R

Description

A helper function for get_robotstxt() that will extract the robots.txt file from the HTTP request result object. furthermore it will inform get_robotstxt() if the request should be cached and which problems occured.

Usage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

Arguments

request

result of an HTTP request (e.g. httr::GET())

on_server_error

request state handler for any 5xx status

on_client_error

request state handler for any 4xx HTTP status that is not 404

on_not_found

request state handler for HTTP status 404

on_redirect

request state handler for any 3xx HTTP status

on_domain_change

request state handler for any 3xx HTTP status where domain did change as well

on_file_type_mismatch

request state handler for content type other than 'text/plain'

on_suspect_content

request state handler for content that seems to be something else than a robots.txt file (usually a JSON, XML or HTML)

warn

suppress warnings

encoding

The text encoding to assume if no encoding is provided in the headers of the response

Format

An object of class list of length 4.

Value

a list with three items following the following schema:
list( rtxt = "", problems = list( "redirect" = list( status_code = 301 ), "domain" = list(from_url = "...", to_url = "...") ) )


petermeissner/robotstxt documentation built on May 25, 2019, 1:56 a.m.