paths_allowed: check if a bot has permissions to access page(s)

In robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

Description

wrapper to path_allowed

Usage

paths_allowed(paths = "/", domain = "auto", bot = "*",
  user_agent = NULL)

Arguments

paths	paths for which to check bot's permission, defaults to "/"
domain	Domain for which paths should be checked. Defaults to "auto". If set to "auto" function will try to guess the domain by parsing the paths argument. Note however, that these are educated guesses which might utterly fail. To be on the save side, provide appropriate domains manually.
bot	name of the bot, defaults to "*"
user_agent	HTTP user-agent string to be used to retireve robots.txt file from domain

See Also

path_allowed

robotstxt documentation built on July 17, 2017, 1:03 a.m.

check if a bot has permissions to access page

robotstxt index

printing robotstxt