Description Usage Arguments Value
View source: R/parse_robotstxt.R
function parsing robots.txt
1 | parse_robotstxt(txt)
|
txt |
content of the robots.txt file |
a named list with useragents, comments, permissions, sitemap
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.