R/canonicalization.R

In pagerankr: Modular Toolkit for PageRank Calculation

#' @title pagerankr URL Canonicalization Profile
#' @description The explicit set of `rurl` canonicalization arguments that
#'   determine a pagerankr node identity (scheme + host + path). Every knob that
#'   *shapes that key* is pinned here with an explicit value, so node keys never
#'   depend on `rurl`'s own defaults -- which have changed across `rurl`
#'   versions (e.g. `case_handling` flipped from `"keep"` to `"lower_host"`) and
#'   previously desynced the pagerankr <-> semantic join. Knobs that only govern
#'   components the key drops (port, query, fragment, userinfo) are deliberately
#'   left unpinned -- see the "Knobs deliberately left unpinned" note in
#'   @details.
#'
#' @details The canonical node key is **scheme + host + non-default port +
#'   path + contentful query**, with the path normalized under the WHATWG URL
#'   standard and percent-encoding preserved byte-for-byte. Fragment and
#'   userinfo are dropped by `get_clean_url` and identify no resource --
#'   userinfo under `credential_handling = "strip"`, which is pinned because
#'   the alternative (`"reject"`) yields `NA` rather than a key.
#'
#'   **The governing principle: parse, do not fold.** pagerankr has redirects
#'   and canonical tags as first-class inputs, and those are the site's own
#'   statement about which URLs are the same page. Canonicalization therefore
#'   normalizes only what the *standard* says is the same resource, and asserts
#'   nothing about site configuration. If `www.example.com/a` and
#'   `example.com/a` have no redirect and no canonical between them, they are
#'   two nodes -- and that is a finding, not a defect to be papered over. The
#'   knobs that would fold them (`www_handling`, `trailing_slash_handling`,
#'   `index_page_handling`, `protocol_handling`) are all pinned to their
#'   non-folding values for exactly this reason.
#'
#'   Two corollaries that are easy to get backwards:
#'   * A **non-default port is a different origin**, so it stays in the key.
#'     `rurl`'s default `port_handling = "exclude"` drops every port and would
#'     merge `host:8080` with `host`. `"strip_default"` removes only `:80` on
#'     http and `:443` on https, which the standard makes redundant.
#'   * An **IDN host and its punycode form are the same request on the wire**,
#'     so no redirect or canonical can ever fold them -- the graph has to.
#'     `host_encoding = "idna"` normalizes both to the punycode form.
#'
#'   **Why `whatwg`, and why `path_encoding = "keep"`.** These two are the only
#'   knobs here that do not simply mirror a `rurl` default, and they are the
#'   load-bearing pair. `path_encoding` is a *presentation* dial -- `rurl`'s own
#'   documentation says only `"keep"` preserves a profile's canonical identity
#'   path verbatim, and that `"encode"`/`"decode"` "may re-encode or decode
#'   reserved octets (so `%2F` may fold to a path-separating `/`)". pagerankr
#'   pinned `"decode"` for several releases and thereby merged `/a%2Fb` with
#'   `/a/b`: two different resources, one node. Identity semantics live on
#'   `url_standard` instead, which reaches a profile-internal path-identity axis
#'   no presentation dial can touch.
#'
#'   `"whatwg"` rather than `"rfc3986"` because pagerankr models what a search
#'   engine sees. Measured over the node-key fixture, `whatwg` additionally
#'   resolves percent-encoded dot segments (`%2e%2e` behaves as `..`), strips
#'   tab/newline from paths instead of failing the parse, percent-encodes
#'   literal spaces instead of failing the parse, and keeps `http://host` and
#'   `http://host/` as one node. `rfc3986` splits that last pair and returns
#'   `NA` for the whitespace classes, both of which are common in crawl
#'   exports. The cost is that `whatwg` preserves percent spellings, so
#'   `/a%7Eb` and `/a~b` are two nodes, as they are to a crawler.
#'
#'   The remaining knobs equal `rurl`'s current defaults and are pinned only to
#'   freeze them. (The anti-drift guarantee ultimately lives in the golden-key
#'   fixtures in `test-canonicalization.R`: `rurl` 3.0.0 re-keyed six of them by
#'   reordering decode after dot-segment removal *without touching any
#'   argument*, which the surface guard cannot see.)
#'
#'   All twenty of these arguments are accepted by both
#'   `rurl::get_clean_url()` (the cleaning path) and `rurl::safe_parse_url()`
#'   (the domain-filtering path), so one profile drives both and the two paths
#'   stay symmetrical.
#'
#'   **Knobs deliberately left unpinned.** `rurl::get_clean_url()` has grown
#'   options that govern components the node key does not carry, or that select
#'   a parsing route rather than a key component:
#'   * `source` (default `"all"`) -- the public-suffix source, reachable only
#'     through `www_handling`/`subdomain_levels_to_keep`, both pinned to values
#'     that do not consult it.
#'   * Added in `rurl` 2.7.0: `scheme_policy` (default `"infer"`),
#'     `scheme_acceptance` (default `"web"`), `engine` (default `NULL`) and
#'     `profile` (default `NULL`, an unrelated `rurl` concept that merely shares
#'     a name with this function). In particular `profile = "seo"` is **not**
#'     used and must not be: it bundles `protocol_handling = "https"`,
#'     `www_handling = "strip"`, `trailing_slash_handling = "strip"` and
#'     `index_page_handling = "strip"`, every one of which is a redirect class
#'     pagerankr resolves through its own auditable fold map
#'     ([build_fold_map()], applied by [pagerank()]). Folding those at
#'     canonicalization time makes the redirect row self-referential, which
#'     the fold map drops as a no-op -- the hop is erased before anything can
#'     audit or report it.
#'
#'   Under the scheme+host+path key these have no visible effect at their
#'   defaults, so pinning them would add noise without changing identity. They
#'   are intentionally **not** part of this profile; instead
#'   `test-canonicalization.R` guards them from two sides, so a `rurl` change
#'   is caught on the pagerankr side rather than silently changing node
#'   identity: a behavioral guard asserts that a canonical key really does drop
#'   the port, query and fragment (catching a default *flip*), and a surface
#'   guard reads `rurl`'s own formals and fails on any argument this profile
#'   has neither pinned nor listed above (catching an *addition*). A committed
#'   node-key probe covering the cross-platform parse-determinism risk surface
#'   pins the keys themselves.
#'
#'   The cross-repo contract requires **semantic** to pin the identical profile;
#'   change both repos together.
#'
#'   **Accepted divergence on un-canonicalizable input.** For a value `rurl`
#'   cannot parse (an unsupported scheme like `mailto:`/`tel:`, whitespace, a
#'   dotless bare token), `rurl` returns `NA`. pagerankr's [clean_url_columns()]
#'   keeps such a value as its raw self so it survives as an opaque graph node
#'   (see that function; PR #50), whereas semantic's `canonical_url()` returns
#'   `None` and drops it (FR-05 rurl byte-parity). This is intentional and does
#'   **not** break the `node_score` <-> `page` join: valid URLs still produce
#'   byte-identical keys on both sides (the actual contract), and in the
#'   semantic -> pagerankr bridge semantic canonicalizes and drops
#'   un-canonicalizable inputs *before* pagerankr sees the edges, so the raw
#'   fallback never fires on that path. It only affects pagerankr run standalone
#'   on raw crawl data, where such tokens become opaque nodes instead of being
#'   dropped.
#'
#' @return A named list of `rurl` canonicalization arguments.
#' @export
#' @examples
#' # The pinned profile that determines pagerankr node identity.
#' profile <- canonical_profile()
#' str(profile)
#'
#' # Key knobs that shape the scheme + host + path node key.
#' profile$case_handling      # "lower_host"
#' profile$path_normalization # "dot_segments"
#' profile$path_encoding      # "keep"    (presentation dial, held at identity)
#' profile$url_standard       # "whatwg"  (where identity semantics live)
canonical_profile <- function() {
  list(
    protocol_handling = "keep",
    case_handling = "lower_host",
    www_handling = "none",
    trailing_slash_handling = "none",
    index_page_handling = "keep",
    path_normalization = "dot_segments",
    scheme_relative_handling = "keep",
    subdomain_levels_to_keep = NULL,
    # An IDN host and its punycode form are the SAME request on the wire, so
    # no redirect or canonical can ever fold them -- the graph has to. "idna"
    # normalizes both to the punycode form, the stabler of the two byte forms.
    host_encoding = "idna",
    # `url_standard` selects the IDENTITY semantics; `path_encoding` is a
    # PRESENTATION dial that rurl documents as not identity-preserving. Pinning
    # "decode" as the identity function was the root cause of the %2F false
    # merge. "whatwg" + "keep" is the pair that gives node identity; see the
    # "Why whatwg, and why path_encoding = keep" note in @details.
    path_encoding = "keep",
    url_standard = "whatwg",
    # A non-default port is a different origin, so it belongs in the key;
    # `:80` on http and `:443` on https are redundant per spec and do not.
    # rurl's default "exclude" drops every port, merging `host:8080` with
    # `host` -- two origins, one node.
    port_handling = "strip_default",
    # Contentful params are part of the resource: `?color=red` and
    # `?color=blue` are two pages, and no redirect or canonical need exist
    # between them. Pure tracking noise is not -- no crawler treats
    # `?utm_source=x` as a distinct page. "filter" is the only value that
    # separates the two.
    query_handling = "filter",
    # Once the query is IN the key these stop being inert and start shaping
    # it, so they are pinned like every other key component. All six equal
    # rurl's defaults; `sort_params = FALSE` is the load-bearing one --
    # rurl's own key contract makes query "order and duplicates significant",
    # so matching it here keeps a later move to get_url_key() a no-op rather
    # than a third re-key.
    params_keep = NULL,
    params_drop = NULL,
    params_case_sensitive = FALSE,
    sort_params = FALSE,
    empty_param_handling = "keep",
    decode_plus = FALSE,
    # Userinfo names a requester, not a resource, so it is not part of the key
    # -- but this knob decides whether a credentialed URL yields a key AT ALL.
    # rurl's default "strip" drops the credentials and keys the URL normally;
    # "reject" returns NA, which [clean_url_columns()] would keep as its raw
    # self, turning every credentialed URL into an opaque node. That is a
    # node-identity consequence, so it is pinned rather than left to a default
    # that could flip. Added in rurl 3.0.1; "strip" is the current default, so
    # pinning it changes no key today.
    credential_handling = "strip"
  )
}

#' @title Merge User rurl Parameters Over the Canonical Profile
#' @description Returns the canonical profile with any user-supplied `rurl`
#'   parameters overriding individual keys. This is the single place pagerankr
#'   resolves the effective canonicalization arguments for both cleaning and
#'   filtering, so the two paths cannot drift apart.
#'
#' @param user_params A named list of `rurl` arguments to override the profile.
#'   Unknown keys are passed through (so new `rurl` arguments work without a
#'   code change here); `NULL` or empty yields the bare profile.
#' @return A named list of effective `rurl` canonicalization arguments.
#' @noRd
.resolve_rurl_params <- function(user_params = list()) {
  if (is.null(user_params) || length(user_params) == 0) {
    return(canonical_profile())
  }
  if (!is.list(user_params)) {
    stop("`rurl_params` must be a list.", call. = FALSE)
  }
  utils::modifyList(canonical_profile(), user_params)
}