request_handler_handler: request_handler_handler

In robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

Description

Helper function to handle robotstxt handlers.

Usage

request_handler_handler(request, handler, res, info = TRUE, warn = TRUE)

Arguments

request	the request object returned by call to httr::GET()
handler	the handler either a character string entailing various options or a function producing a specific list, see return.
res	a list a list with elements '[handler names], ...', 'rtxt', and 'cache'
info	info to add to problems list
warn	if FALSE warnings and messages are suppressed

Value

a list with elements '[handler name]', 'rtxt', and 'cache'

robotstxt documentation built on Sept. 4, 2020, 1:08 a.m.