paths_allowed: check if a bot has permissions to access page(s)

Description Usage Arguments See Also

View source: R/paths_allowed.R

Description

wrapper to path_allowed

Usage

1
2
paths_allowed(paths = "/", domain = "auto", bot = "*",
  user_agent = NULL)

Arguments

paths

paths for which to check bot's permission, defaults to "/"

domain

Domain for which paths should be checked. Defaults to "auto". If set to "auto" function will try to guess the domain by parsing the paths argument. Note however, that these are educated guesses which might utterly fail. To be on the save side, provide appropriate domains manually.

bot

name of the bot, defaults to "*"

user_agent

HTTP user-agent string to be used to retrieve robots.txt file from domain

See Also

path_allowed


robotstxt documentation built on Sept. 2, 2017, 1:13 a.m.