accuracy_metrics: Type I Error Accuracy Metrics for Method Selection

In DPComb: Discrete p-Value Combination Tests

Type I Error Accuracy Metrics for Method Selection

Description

Computes the two metrics recommended for comparing and selecting discrete p-value combination methods: the variance ratio \mathrm{Var}(Z)/\mathrm{Var}(Y) and the normalized (scaled) Wasserstein distance W_2(Z,\tilde{Y})/\mathrm{SD}(Y). Here Z is the adjusted discrete statistic, Y is the continuous reference null, and \tilde{Y} is the moment-matched continuous surrogate used in the testing procedure. A higher variance ratio (closer to 1) and a smaller normalized distance both indicate more accurate finite-sample Type I error control.

Usage

accuracy_metrics(p_support, method = "fisher_mean")

Arguments

p_support	A valid p-value support vector: nonnegative, nondecreasing, and ending at 1, characterizing the null distribution of the discrete p-value.
method	The combination method, one of "fisher_mean" (default), "fisher_median", "pearson", "george", "stouffer", or "edgington".

Details

The surrogate \tilde{Y} matches the first two moments of Z: a gamma distribution for Fisher's and Pearson's methods, and a normal distribution for George's, Stouffer's, and Edgington's methods. The Wasserstein distance is computed under the optimal (quantile) coupling,

W_2^2(Z,\tilde{Y}) = \sum_i \int_{\tilde{G}^{-1}(P(Z<z_i))}^{\tilde{G}^{-1}(P(Z\le z_i))} (z_i-y)^2 \tilde{g}(y)\,dy,

where \tilde{G} and \tilde{g} are the CDF and density of \tilde{Y}. The integral over each atom is evaluated numerically. For the non-i.i.d. case, the average variance ratio \sum_j \mathrm{Var}(Z_j)/(n\,\mathrm{Var}(Y)) can be obtained by averaging var_ratio across the component supports.

Value

A list with the following elements:

var_Z	The variance of the adjusted statistic Z.
var_Y	The variance of the continuous reference null Y, a known constant per method: 4 for Fisher and Pearson, \pi^2/3 for George, 1 for Stouffer, and 1/12 for Edgington.
var_ratio	The variance ratio \mathrm{Var}(Z)/\mathrm{Var}(Y), which lies in [0,1].
W2	The Wasserstein-2 distance W_2(Z,\tilde{Y}) between Z and the moment-matched surrogate \tilde{Y}.
norm_dist	The normalized distance W_2(Z,\tilde{Y})/\mathrm{SD}(Y).

References

Contador, Gonzalo and Wu, Zheyang (2026). Optimal Adjustment and Combination of Independent Discrete p-Values. Under revision at the Journal of Computational and Graphical Statistics.

Examples

# Distribution P_L (large mass at small p-values), cf. Table 4 of Contador and Wu (2026).
p_support <- c(0.40, (41:100) / 100)
methods <- c("fisher_mean", "pearson", "stouffer", "edgington", "george")
round(sapply(methods, function(m) unlist(accuracy_metrics(p_support, m))), 3)

DPComb documentation built on Aug. 22, 2026, 5:08 p.m.