hmda.rashomon.set: Select a Near-Optimal from a Model Grid

In HMDA: Holistic Multimodel Domain Analysis for Exploratory Machine Learning

Select a Near-Optimal from a Model Grid

Description

Selects a set of near-optimal models from a grid-analysis table. The function can either retain all models within a given percentage of the best model, or retain the top n models according to a selected performance metric.

Usage

hmda.rashomon.set(
  grid_analysis,
  metric = "aucpr",
  criterion = c("within_percent", "top_n"),
  distance_percentage = 0.01,
  top_n = 5,
  fallback_min_models = 2
)

Arguments

grid_analysis	A data frame containing model IDs and performance metrics, usually returned by hmda.grid.analysis().
metric	Character string specifying the metric used for model selection and weighting. Supported values include "aucpr", "auc", "logloss", and "r2".
criterion	Character string specifying the selection rule. Supported values are "within_percent" and "top_n".
distance_percentage	Numeric. The allowed distance from the best model when criterion = "within_percent". For higher-is-better metrics, models are retained if their performance is at least best * (1 - distance_percentage). For lower-is-better metrics, models are retained if their value is at most best * (1 + distance_percentage). Default is 0.01.
top_n	Integer. Number of top-ranked models to retain when criterion = "top_n". Default is 5.
fallback_min_models	Integer. Minimum number of models to retain. If the selected model based on within_perent set contains fewer models than this value, the function falls back to the best fallback_min_models models according to metric. Default is 2.

Details

The function is used to define the model set from which WMSHAP values are computed. In the "within_percent" criterion, the function retains models that are close to the best-performing model according to the selected metric.

For metrics where larger values indicate better performance, such as "aucpr", "auc", and "r2", models are ranked in decreasing order. For metrics where smaller values indicate better performance, such as "logloss", models are ranked in increasing order.

The argument fallback_min_models is useful when the "within_percent" criterion retains only a single model. In that case, WMSHAP would no longer be meaningfully multi-model. The fallback rule can be used to force a minimum number of retained models while recording whether this correction was necessary.

Value

A list with the following components:

model_ids

Character vector of the final selected model IDs. This includes the fallback models if the fallback rule was used.

pure_model_ids

Character vector of model IDs selected by the requested rule before applying the minimum-size fallback.

selected_model_table

A data frame containing the selected model IDs, the selection metric, the metric value, and the normalized model weight for each retained model.

all_model_performance

A named numeric vector containing the metric value for all valid models in grid_analysis. The names are the model IDs.

diagnostics

A one-row data frame describing the selection rule, the number of models retained before and after fallback, whether the fallback was used, the best and second-best metric values, the top-two metric gap, and the cutoff used for selection.

Examples

## Not run: 
  # Assume grid_analysis was returned by hmda.grid.analysis()

  # Retain all models within 1 percent of the best AUCPR model.
  rashomon_1pct <- hmda.rashomon.set(
    grid_analysis = grid_analysis,
    metric = "aucpr",
    criterion = "within_percent",
    distance_percentage = 0.01,
    fallback_min_models = 2
  )

  # Access the final retained model IDs.
  rashomon_1pct$model_ids

  # Inspect whether the fallback rule was used.
  rashomon_1pct$diagnostics

  # Retain the top 5 models according to AUCPR.
  rashomon_top5 <- hmda.rashomon.set(
    grid_analysis = grid_analysis,
    metric = "aucpr",
    criterion = "top_n",
    top_n = 5
  )

  rashomon_top5$selected_model_table

## End(Not run)

HMDA documentation built on Sept. 18, 2026, 5:06 p.m.