R/rt_get_comments.R
In ropensci/robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
Defines functions
rt_get_comments
Documented in
rt_get_comments
#' Extract comments from robots.txt
#' @param txt content of the robots.txt file
#' @keywords internal
rt_get_comments <- function(txt){
txt <- unlist(stringr::str_split(txt, "\n"))
clines <- grep("#", txt)
ccontent <- stringr::str_extract(txt[clines], "#.*")
data.frame(line=clines, comment=ccontent, stringsAsFactors = FALSE)
}
ropensci/robotstxt documentation built on Sept. 3, 2024, 2:32 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Defines functions rt_get_comments
Documented in rt_get_comments
#' Extract comments from robots.txt
#' @param txt content of the robots.txt file
#' @keywords internal
rt_get_comments <- function(txt){
txt <- unlist(stringr::str_split(txt, "\n"))
clines <- grep("#", txt)
ccontent <- stringr::str_extract(txt[clines], "#.*")
data.frame(line=clines, comment=ccontent, stringsAsFactors = FALSE)
}
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.