paths_allowed_worker_spiderbar: paths_allowed_worker spiderbar flavor

Description Usage Arguments

View source: R/paths_allowed_worker_spiderbar.R

Description

paths_allowed_worker spiderbar flavor

Usage

1
paths_allowed_worker_spiderbar(domain, bot, paths, robotstxt_list)

Arguments

domain

Domain for which paths should be checked. Defaults to "auto". If set to "auto" function will try to guess the domain by parsing the paths argument. Note however, that these are educated guesses which might utterly fail. To be on the save side, provide appropriate domains manually.

bot

name of the bot, defaults to "*"

paths

paths for which to check bot's permission, defaults to "/". Please, note that path to a folder should end with a trailing slash ("/").

robotstxt_list

either NULL – the default – or a list of character vectors with one vector per path to check


petermeissner/robotstxt documentation built on Jan. 14, 2019, 9:51 p.m.