can_fetch | R Documentation |
robxp
robots.txt
objectProvide a character vector of URL paths plus optional user agent and this function will return a logical vector indicating whether you have permission to fetch the content at the respective path.
can_fetch(obj, path = "/", user_agent = "*")
obj |
|
path |
path to test |
user_agent |
user agent to test |
logical vector indicating whether you have permission to fetch the content
gh <- paste0(readLines(system.file("extdata", "github-robots.txt", package="spiderbar")), collapse="\n") gh_rt <- robxp(gh) can_fetch(gh_rt, "/humans.txt", "*") # TRUE can_fetch(gh_rt, "/login", "*") # FALSE can_fetch(gh_rt, "/oembed", "CCBot") # FALSE can_fetch(gh_rt, c("/humans.txt", "/login", "/oembed"))
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.