R/rt_get_comments.R

Defines functions rt_get_comments

Documented in rt_get_comments

#' extracting comments from robots.txt
#' @param txt content of the robots.txt file
#' @keywords internal
rt_get_comments <- function(txt){
  txt      <- unlist(stringr::str_split(txt, "\n"))
  clines   <- grep("#", txt)
  ccontent <- stringr::str_extract(txt[clines], "#.*")
  data.frame(line=clines, comment=ccontent, stringsAsFactors = FALSE)
}

Try the robotstxt package in your browser

Any scripts or data that you put into this service are public.

robotstxt documentation built on Sept. 12, 2024, 7:36 a.m.