crawl_delays | R Documentation |
robxp
robots.txt
objectRetrieve all agent crawl delay values in a robxp
robots.txt
object
crawl_delays(obj)
obj |
|
data frame of agents and their crawl delays
-1
will be returned for any listed agent without a crawl delay setting
gh <- paste0(readLines(system.file("extdata", "github-robots.txt", package="spiderbar")), collapse="\n") gh_rt <- robxp(gh) crawl_delays(gh_rt) imdb <- paste0(readLines(system.file("extdata", "imdb-robots.txt", package="spiderbar")), collapse="\n") imdb_rt <- robxp(imdb) crawl_delays(imdb_rt)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.