evaluateShiftRecovery: Evaluate Shift Recovery Against Simulated Ground Truth

In bifrost: Branch-Level Inference Framework for Recognizing Optimal Shifts in Traits

Evaluate Shift Recovery Against Simulated Ground Truth

Description

Compare inferred shift locations against known simulated shift locations across multiple datasets using both strict node matching and fuzzy matching within a configurable node distance. This is a package-clean export of the manuscript's evaluate_shift_recovery() function.

Usage

evaluateShiftRecovery(
  simdata,
  simresults,
  fuzzy_distance = 2,
  weighted = TRUE,
  verbose = TRUE
)

Arguments

simdata	A list of simulated shifted datasets, typically the simdata component returned by runShiftRecoverySimulationStudy(). Each element must contain at least paintedTree and shiftNodes.
simresults	A list of search results corresponding to simdata, usually the results component returned by runShiftRecoverySimulationStudy(). Each element should contain shift_nodes_no_uncertainty and num_candidates. Results from searchOptimalConfiguration() also carry candidate_nodes, which allows candidate-aware specificity calculations; legacy result objects without that component retain the historical count-only calculation. If weighted = TRUE, ic_weights is also used when available.
fuzzy_distance	Integer node distance threshold used for fuzzy matching.
weighted	Logical; if TRUE, compute weighted precision/recall/F1 using the inferred-node IC weights from each search result when available.
verbose	Logical; if TRUE, print a compact summary of the strict, fuzzy, and weighted metrics.

Details

The evaluation uses three complementary summaries:

• strict metrics count only exact node matches,

• fuzzy metrics allow inferred shifts within fuzzy_distance nodes of a true shift, using a greedy one-to-one assignment, and

• weighted metrics apply IC weights only to inferred nodes, following the simulation-study implementation.

Search results carrying a non-empty error field are retained by the study wrappers for diagnosis but excluded from recovery counts and metrics. In otherwise complete, successful results, an explicitly present shift_nodes_no_uncertainty = NULL means no inferred shifts and contributes false negatives for the true shifts. A missing field remains incomplete and is excluded. Empty integer vectors are handled identically to explicit NULL.

For current search results, true-negative counts are calculated over the explicit candidate_nodes universe. True shifts excluded by the search's candidate filter still contribute false negatives, but are not subtracted from the number of candidate negatives. This prevents specificity and balanced accuracy from being distorted when a simulated shift is not an eligible search candidate. Supplied candidate-node vectors must contain unique, valid whole-number node IDs and agree with num_candidates. For compatibility, result objects that provide only num_candidates use the historical count-only formula.

F1 is calculated directly from counts as 2 * TP / (2 * TP + FP + FN). It is zero when no true shifts are recovered but false positives or false negatives exist, including successful searches with no inferred shifts. It remains NA when both true and inferred shifts are absent, or when no records can be evaluated. Precision and recall retain their own undefined cases. Weighted F1 uses 2 * weighted_TP / (weighted_TP + weighted_FP + n_true_shifts) and is NA if inferred-node IC weights are unavailable.

When a replicate has no evaluable candidate shifts (num_candidates == 0), recall-style quantities can still be computed from the true and inferred shifts, but specificity, false-positive rate, and balanced accuracy are returned as NA because there are no evaluable negatives.

Value

A list of class bifrost_shift_recovery_evaluation with five components:

strict

Strict precision, recall, F1, specificity, false-positive rate, and balanced accuracy.

fuzzy

The same metrics under fuzzy matching.

weighted

Weighted strict and fuzzy precision/recall/F1 summaries when weighted = TRUE; otherwise NULL.

counts

Aggregated contingency-table counts for the strict and fuzzy matching schemes.

n_evaluable_replicates

Number of replicate pairs included in the aggregated counts after failed or structurally incomplete records are excluded.

See Also

runShiftRecoverySimulationStudy(), simulateShiftedDataset()

Examples

set.seed(1)
tr <- ape::rtree(8)
tr <- phytools::paintSubTree(
  tr,
  node = ape::Ntip(tr) + 1L,
  state = "ancestral",
  anc.state = "ancestral"
)
true_node <- setdiff(unique(tr$edge[, 1]), ape::Ntip(tr) + 1L)[1]
simdata <- list(list(paintedTree = tr, shiftNodes = true_node))
simresults <- list(list(
  shift_nodes_no_uncertainty = true_node,
  num_candidates = 6L,
  ic_weights = data.frame(
    node = true_node,
    ic_weight_withshift = 0.8
  )
))

recovery <- evaluateShiftRecovery(
  simdata,
  simresults,
  fuzzy_distance = 2,
  verbose = FALSE
)
recovery$strict

bifrost documentation built on Nov. 5, 2026, 9:07 a.m.