perm_test_pv: Permutation Test

In DiscreteTests: Vectorised Computation of P-Values and Their Supports for Several Discrete Statistical Tests

Permutation Test

Description

perm_test_pv() performs a permutation test for the comparison of two independent samples using a user-supplied test statistic. It is vectorised over multiple test problems: by passing lists of sample vectors for x and y, several tests can be evaluated simultaneously. The function can compute exact p-values (by enumerating all permutations) or approximate p-values via Monte Carlo sampling. It is capable of returning the discrete p-value supports, i.e. all observable p-values under the null hypothesis.

Usage

perm_test_pv(
  x,
  y,
  statistic = c("diff_mean", "diff_median", "diff_t", "diff_welch", "diff_hl",
    "ratio_var", "ratio_sd"),
  mu = NULL,
  alternative = "two.sided",
  exact = NULL,
  max_exact_combs = 10L^7,
  MC_sims = 10L^5,
  seed = NULL,
  simple_output = FALSE
)

Arguments

x	numerical vector or list of numerical vectors of observations in the first group.
y	numerical vector or list of numerical vectors of observations in the second group.
statistic	single character giving the type of test statistic to be used; must be one of "diff_mean", "diff_median", "diff_t", "diff_welch", "diff_hl", "ratio_var" or "ratio_sd" (see Details).
mu	numerical vector giving the hypothesised values under the null; if is.null(mu) (the default) it is set to 0 for test statistics that are based on differences (diff_) or 1 for ratio-based tests statistics (ratio_).
alternative	character vector that indicates the alternative hypotheses; each value must be one of "two.sided" (the default), "less" or "greater".
exact	logical value that indicates whether p-values are to be calculated by exact computation (TRUE) or by simulation (FALSE); if NULL (the default) exact computation is performed if the number of possible sample combinations (see Details) is below or equal to max_exact_combs; otherwise the respective p-values are obtained by simulation.
max_exact_combs	maximum number of allowed combinations for exact computation of the permutation distribution (see details).
MC_sims	positive integer or NULL. The number of Monte Carlo permutations to draw when exact = FALSE. Ignored when exact = TRUE. Default is 10000L.
seed	single integer or NULL. Random seed passed to set.seed() before Monte Carlo sampling to ensure reproducibility. Use NULL to skip seed setting. Ignored when exact = TRUE.
simple_output	logical value that indicates whether an R6 class object, including the tests' parameters and support sets, i.e. all observable p-values under each null hypothesis, is to be returned (see below).

Details

Under the null hypothesis of exchangeability (i.e. both samples come from the same distribution), all \binom{n_x + n_y}{n_x} partitions of the pooled sample into groups of sizes n_x and n_y are equally likely. perm_test_pv() exploits this by:

1. Computing the observed test statistic T_\text{obs} from x and y using the function selected by statistic (see the Test Statistics section below).

2. Enumerating (if exact = TRUE) or randomly sampling (if exact = FALSE) permutations of the pooled sample.

3. Evaluating the selected test statistic on each permuted split.

4. Deriving the p-value as the fraction of permutation statistics that are at least as extreme as T_\text{obs}, where extreme is defined by alternative:

   "less"

   fraction with permuted statistic \le T_\text{obs}

   "greater"

   fraction with permuted statistic \ge T_\text{obs}

   "two.sided"

   fraction with |\text{permuted statistic}| \ge |T_\text{obs}|

Exact computation is only feasible for small pooled samples; the total number of permutations grows as \binom{n_x + n_y}{n_x}. If exact = NULL, exact computation is applied when the number of computations is below or equal to max_exact_combs (default 10{,}000{,}000). Otherwise, the distribution of the test statistics is approximated by Monte Carlo simulation. When exact = TRUE and the number of permutations exceeds max_exact_combs, an error is raised. Set exact = FALSE to use Monte Carlo sampling in such cases.

The parameters x, y, alternative, and MC_sims are vectorised via lists: if x and y are lists of the same length, each pair ⁠(x[[i]], y[[i]])⁠ defines one test. Scalars and single vectors are recycled to the required length.

Test Statistics

The statistic argument selects among seven built-in test statistics for comparing two groups. Let \bar{x}, \bar{y} denote the group means, s_x^2, s_y^2 the sample variances, and n_x, n_y the group sizes.

"diff_mean" — Difference of means T = \bar{x} - \bar{y} - \mu_0

The most common choice for location comparisons. Tests whether the population means differ by \mu_0 (default: 0).
Advantages: Intuitive interpretation; optimal power under normality and equal variances (Pitman efficiency 1 vs. the two-sample t-test).
Disadvantages: Sensitive to outliers; less powerful than "diff_hl" or "diff_t" under heavy-tailed distributions.

"diff_median" — Difference of medians T = \text{median}(x) - \text{median}(y) - \mu_0

Tests for a shift in the median rather than the mean (default: \mu_0 = 0).
Advantages: Robust to outliers and skewed distributions; directly interpretable in terms of the median.
Disadvantages: The sample median is a step function of the data, leading to a coarser permutation distribution with many ties; can have lower power than "diff_hl" for continuous data.

"diff_t" — Pooled two-sample t-statistic T = \frac{\bar{x} - \bar{y} - \mu_0}{s_p \sqrt{1/n_x + 1/n_y}} where s_p = \sqrt{\frac{(n_x-1)s_x^2 + (n_y-1)s_y^2}{n_x+n_y-2}}

Scales the mean difference by the pooled standard deviation, analogous to the classical Student t-test.
Advantages: Studentisation makes the statistic (approximately) scale-free; useful when group variances are expected to be equal; the permutation p-value is valid even without normality.
Disadvantages: Assumes equal variances (homoscedasticity); sensitive to outliers; no power gain over "diff_mean" in the permutation setting when sample sizes are equal.

"diff_welch" — Welch t-statistic T = \frac{\bar{x} - \bar{y} - \mu_0}{\sqrt{s_x^2/n_x + s_y^2/n_y}}

Scales the mean difference by the unpooled standard error, analogous to Welch's two-sample t-test. Unlike "diff_t", the variances of the two groups are estimated separately rather than pooled.
Advantages: Does not assume equal variances; retains more power than "diff_t" when group variances are unequal and sample sizes differ; in the permutation setting the p-value remains exactly valid regardless of the variance ratio, just as with "diff_t".
Disadvantages: No power gain over "diff_t" when variances are equal; estimating two separate variances instead of one pooled variance introduces additional estimation uncertainty, which can slightly reduce power compared to "diff_t" in balanced designs with equal variances.

"diff_hl" — Hodges–Lehmann estimator T = \text{median}_{i,j}\,(x_i - y_j) - \mu_0

Tests for a shift in the median of all n_x \cdot n_y pairwise differences between the two groups (default: \mu_0 = 0). It is the natural companion statistic to the Wilcoxon–Mann–Whitney test and estimates the location shift \Delta under a shift model.
Advantages: Highly robust to outliers; produces fewer ties in the permutation distribution than "diff_median".
Disadvantages: O(n_x \cdot n_y) computation; the interpretation as a median shift requires a location-shift model.

"ratio_var" — Ratio of variances T = s_x^2 / (s_y^2 \cdot \mu_0)

Tests for differences in spread (scale) rather than location. Under the null H_0\colon \sigma_x^2 / \sigma_y^2 = \mu_0 (default: \mu_0 = 1).
Advantages: Direct measure of relative variability; permutation validity does not require normality (unlike the classical F-test).
Disadvantages: Highly sensitive to outliers and non-normality (the variance itself is not robust); should be interpreted with caution if the distributions differ in shape as well as scale.

"ratio_sd" — Ratio of standard deviations T = s_x / (s_y \cdot \mu_0)

Equivalent to "ratio_var" on the standard deviation scale (T = \sqrt{s_x^2 / (s_y^2 \cdot \mu_0^2)}); tests the same null hypothesis H_0 \colon \sigma_x / \sigma_y = \mu_0 (default: \mu_0 = 1).
Advantages: Same unit as the original data, which can aid interpretation; monotone transformation of "ratio_var", so the p-values are identical (note that \mu_0 needs to be squared for the variance-based statistic to be equivalent).
Disadvantages: Same sensitivity to outliers as "ratio_var"; the monotone relationship means it carries no additional statistical information compared to "ratio_var".

Value

If simple.output = TRUE, a vector of computed p-values is returned. Otherwise, the output is a DiscreteTestResults R6 class object, which also includes the p-value supports and testing parameters. These have to be accessed by public methods, e.g. ⁠$get_pvalues()⁠.

References

Pitman, E. J. G. (1937). Significance tests which may be applied to samples from any populations. Supplement to the Journal of the Royal Statistical Society, 4(1), pp. 119–130. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.2307/2984124")}

Good, P. (2000). Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses. Second Edition. New York: Springer. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1007/978-1-4757-3235-1")}

See Also

mann_whitney_test_pv(), stats::wilcox.test()

Examples

set.seed(42)
x1 <- rnorm(8)
y1 <- rnorm(10, mean = 1)

# Two-sided test for difference of means = 0
results_ex <- perm_test_pv(x1, y1)
print(results_ex)
results_ex$get_pvalues()

# Hodges Lehmann statistic with Monte Carlo approximation
results_hl <- perm_test_pv(x1, y1, "diff_hl", exact = FALSE, seed = 1L)
results_hl$print()
results_hl$get_pvalues()

# Multiple tests simultaneously, one-sided alternative (mu = 0.5),
# using t-statistic, forced exact computation
xs <- list(rnorm(12), rnorm(9, 1))
ys <- list(rnorm(11, 1), rnorm(10, 2))
results_multi <- perm_test_pv(xs, ys, "diff_t", c(0.5, -1.5), "greater", TRUE)
results_multi$print()
results_multi$get_pvalues()

DiscreteTests documentation built on Sept. 2, 2026, 9:06 a.m.