isIndexable: Function to check if a URL is indexable

In dschmeh/seoR: Various Functions for 'SEO-related Analyses' in 'R'

Function to check if a URL is indexable

Description

This function checks if a given URL is indexable. Therfore it checks different rules: Status-Code: Does the URl gives a correct Status-Code (200) Canonical-Link: Is the canonical Link Element implementet, so the URL can be indexed? Meta-Robots: Is the Meta-Robots blocking the ULR? robots.txt: Is the robots.txt blocking the URL from crawling?

Usage

isIndexable(url, bot = "googlebot", errorType = TRUE)

Arguments

url	The url you want to check the indexability for
bot	The bot you want to check the indexability with. Default is googlebot
errorType	Logical. Retrieves the Error, why the URL can not be indexed. Attention not all rules at once are displayed. isIndexable()

Examples

isIndexable("https://www.r-project.org/", bot = "googlebot", errorType = TRUE)

dschmeh/seoR documentation built on Jan. 7, 2023, 12:19 a.m.