allowedByRobots: Function to check if URL is blocked by robots.txt
In seoR: SEO Related Analyses
Description
Usage
Arguments
Examples
View source: R/allowedByRobots.R
Description
This function checks if a given URL is blocked by the robots.txt file of the website.
Usage
1
allowedByRobots(url, bot = "googlebot")
Arguments
url
The url you want to check
bot
The bot you want to check the indexability with. Default is googlebot
allowedByRobots()
Examples
1
allowedByRobots("https://www.r-project.org/", bot = "googlebot")
seoR documentation built on Jan. 29, 2018, 5:05 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Description Usage Arguments Examples
View source: R/allowedByRobots.R
Description
This function checks if a given URL is blocked by the robots.txt file of the website.
Usage
1 | allowedByRobots(url, bot = "googlebot")
|
Arguments
url |
The url you want to check |
bot |
The bot you want to check the indexability with. Default is googlebot allowedByRobots() |
Examples
1 | allowedByRobots("https://www.r-project.org/", bot = "googlebot")
|
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.