tests/testthat/_snaps/paths_allowed.md

In petermeissner/robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

paths_allowed() works also with 'downloaded' robots.txt files

Code
  domain_change <- readRDS(system.file("http_requests/http_domain_change.rds",
    package = "robotstxt"))
  suppressMessages(paths_allowed(paths = "https://github.io/index.html",
    rt_robotstxt_http_getter = function(...) {
      domain_change
    }, warn = FALSE))
Output
  [1] TRUE

Code
  domain_change <- readRDS(system.file("http_requests/http_domain_change.rds",
    package = "robotstxt"))
  suppressMessages(paths_allowed(paths = "https://github.io/index.html",
    rt_robotstxt_http_getter = function(...) {
      domain_change
    }))
Condition
  Warning in `request_handler_handler()`:
  Event: on_domain_change
  Warning in `request_handler_handler()`:
  Event: on_file_type_mismatch
  Warning in `request_handler_handler()`:
  Event: on_suspect_content
Output
  [1] TRUE

petermeissner/robotstxt documentation built on Nov. 17, 2024, 9:50 p.m.