shapley.domain.test: Weighted permutation test for difference between two domains

In shapley: Weighted Mean SHAP and CI for Robust Feature Assessment in ML Grid

Weighted permutation test for difference between two domains

Description

Computes domain-level contribution ratios (via shapley.domain()) and tests whether two domains differ using a weighted paired permutation test across models.

Usage

shapley.domain.test(shapley, domains, n = 2000)

Arguments

shapley	Object of class "shapley", as returned by the 'shapley' function
domains	A named list of length 2. Each element is a character vector of feature names defining a domain; the two element names are the domain labels to be compared.
n	Integer, number of permutations (default 2000)

Value

A list with mean_wmshap_diff (observed weighted mean difference) and p_value.

Author(s)

E. F. Haghish

Examples

## Not run: 
# load the required libraries for building the base-learners and the ensemble models
library(h2o)            #shapley supports h2o models
library(autoEnsemble)   #autoEnsemble models, particularly useful under severe class imbalance
library(shapley)

# initiate the h2o server
h2o.init(ignore_config = TRUE, nthreads = 2, bind_to_localhost = FALSE, insecure = TRUE)

# upload data to h2o cloud
prostate_path <- system.file("extdata", "prostate.csv", package = "h2o")
prostate <- h2o.importFile(path = prostate_path, header = TRUE)

### H2O provides 2 types of grid search for tuning the models, which are
### AutoML and Grid. Below, I demonstrate how weighted mean shapley values
### can be computed for both types.

set.seed(10)

#######################################################
### PREPARE AutoML Grid (takes a couple of minutes)
#######################################################
# run AutoML to tune various models (GBM) for 60 seconds
y <- "CAPSULE"
prostate[,y] <- as.factor(prostate[,y])  #convert to factor for classification
aml <- h2o.automl(y = y, training_frame = prostate, max_runtime_secs = 120,
                 include_algos=c("GBM"),

                 # this setting ensures the models are comparable for building a meta learner
                 seed = 2023, nfolds = 10,
                 keep_cross_validation_predictions = TRUE)

### call 'shapley' function to compute the weighted mean and weighted confidence intervals
### of SHAP values across all trained models.
### Note that the 'newdata' should be the testing dataset!
result <- shapley(models = aml, newdata = prostate, plot = TRUE)

#######################################################
### Significance testing of contributions of two domains (or latent factors)
#######################################################
domains = list(Demographic = c("RACE", "AGE"),
               Cancer = c("VOL", "PSA", "GLEASON"))
shapley.domain.test(result, domains = domains, n=5000)

## End(Not run)

shapley documentation built on March 4, 2026, 9:06 a.m.