confirm_ctreeMI: Confirm a Discovered Tree on Independent Data

In ctreeMI: Conditional Inference Trees with Stacked Multiple Imputation

Confirm a Discovered Tree on Independent Data

Description

Applies the terminal-node rules of a tree fitted by ctree_stacked() to an independently imputed confirmation sample, and tests whether the outcome differs across those subgroups using a test with valid error control under multiple imputation.

Usage

confirm_ctreeMI(
  tree,
  data,
  outcome_type = "auto",
  min_node = 5L,
  adjust = "holm",
  conf.level = 0.95,
  cluster = NULL,
  vcov_type = "HC1"
)

Arguments

tree	An object of class ctreeMI, fitted on the discovery data.
data	The confirmation data as a mids object from mice::mice() or a list of imputed data frames. Must have been imputed separately from the discovery data; see split_holdout().
outcome_type	Either "auto" (default), in which case numeric outcomes are modelled by linear regression and two-level factors by logistic regression, or a character vector of the same length as the number of outcomes with entries "continuous" or "binary".
min_node	Minimum number of confirmation observations, per imputation, that a terminal node must receive. Nodes below this are reported but excluded from the test. Default 5.
adjust	Method for adjusting the per-split p-values across the internal nodes of the tree, passed to stats::p.adjust(). Default "holm". Use "none" to report unadjusted values.
conf.level	Confidence level for the pooled contrast reported with each split. Default 0.95.
cluster	Optional name of a column in the confirmation data identifying a higher-level unit such as a school or district. When given, every model's covariance matrix is replaced by a cluster-robust estimate from sandwich::vcovCL(), so that within-cluster correlation is accounted for in every test and interval. Requires the sandwich package.
vcov_type	The small-sample adjustment passed to sandwich::vcovCL() when cluster is given. Default "HC1".

Details

Why a separate confirmation step

A tree uses the data to choose which variables to split on and where. Testing the resulting subgroups on the same data is invalid regardless of how the missing values were handled, because the partition was selected to maximise exactly the separation being tested. Independently of that, the node-level test of ctree_stacked() is not calibrated under outcome-conditioned imputation; see the "Node-level calibration" section of that help page.

Both problems are avoided by treating the discovery tree as a hypothesis about subgroup structure and testing it on data the tree has not seen. This function implements that step. The subgroups are fixed by the discovery tree, so the multiplicity of the search is not carried into the test, and the confirmation test pools across imputations by Rubin's rules rather than stacking, so the between-imputation variance that miscalibrates the stacked test is properly accounted for.

What is tested

Two families of test are run, both pooled across imputations by the multivariate Wald procedure of Li, Raghunathan and Rubin (1991). The pooling is implemented within this package rather than through mice::D1() so that a cluster-robust covariance can be supplied; with cluster = NULL the two agree.

The omnibus test fits, for each outcome, a model with terminal-node membership as the only predictor, and tests whether the outcome differs across nodes at all. This establishes that the partition carries information, not that every split is real: a tree with one genuine split and one spurious one will still yield a small p value.

The per-split tests address that directly. For each internal node of the discovery tree, the confirmation observations falling within that node are divided by the node's own split rule, and the outcome is tested for a difference between the resulting children. Because the parent node and its rule are fixed by the discovery tree, each is a pre-specified contrast, and the test is valid. A split whose children do not differ in independent data is one the discovery tree should not have made. The per-split p values are adjusted across internal nodes by the method in adjust.

Each split is also reported with the pooled difference between its children and a confidence interval, on the scale of the outcome for a continuous response and as a log odds ratio for a binary one. These are the quantities to report: the p-value says whether the split survived, the contrast says how much the subgroups differ and how precisely that is known. prune_unconfirmed() collapses the splits that did not survive.

A split below a non-confirmed split has no clear interpretation, since the partition it refines was not itself supported; the depth column allows this to be read off. Pooled node-level estimates are returned so that the pattern of differences can be examined directly.

Clustered data

When observations are nested in units, the members of a unit are not independent and a model-based variance understates uncertainty. Passing the unit identifier as cluster substitutes a cluster-robust covariance for every fitted model before pooling, so that the omnibus test, the per-split tests, the contrasts and the node estimates all reflect the effective sample size. The discovery split should have kept units intact; see split_holdout().

The reference distributions carry the usual small-sample adjustments: the Reiter (2007) adjustment for the multivariate test and the Barnard-Rubin (1999) adjustment for the scalar contrasts, both of which cap the degrees of freedom by the complete-data degrees of freedom. Unclustered, that is the residual degrees of freedom, and the tests then agree with mice::D1() and mice::pool(). Clustered, the complete-data degrees of freedom are taken as the number of clusters minus one, which is the conventional choice for cluster-robust inference and is what makes the tests more conservative as clusters become few.

Cluster-robust variance is nonetheless unreliable with very few clusters, and a warning is issued below twenty. The number of clusters behind each node and each split is reported so that this can be judged.

Node assignment under imputation

Each imputed confirmation dataset is passed through the discovery tree's split rules. Because imputed predictor values vary across imputations, an observation whose imputed value straddles a split point may be assigned to different nodes in different imputations. This is not an error; it reflects imputation uncertainty, and pooling by Rubin's rules accounts for it.

Value

An object of class "ctreeMI_confirm": a list with elements

test

A data frame with one row per outcome: outcome, F, df1, df2, p, and riv, the relative increase in variance due to nonresponse.

splits

A data frame with one row per internal node and outcome: node_id, depth, split_var, rule (the split as written), outcome, contrast with lower and upper (the pooled difference between the children and its confidence interval), F, df1, df2, p, p_adj, and n_confirm (mean observations in the parent node per imputation). NA where a child node fell below min_node in some imputation.

nodes

A data frame of pooled per-node estimates: node_id, outcome, estimate (mean or proportion), se, n_confirm (mean observations per imputation), n_clusters, and tested.

m

Number of imputations used.

n_confirm

Number of confirmation observations.

cluster, n_clusters, vcov_type

The cluster column, the number of clusters in the confirmation data, and the variance estimator used ("model" when unclustered).

excluded

Terminal-node ids excluded for having fewer than min_node observations, if any.

References

Barnard, J., and Rubin, D. B. (1999). Small-sample degrees of freedom with multiple imputation. Biometrika, 86, 948–955.

Li, K. H., Raghunathan, T. E., and Rubin, D. B. (1991). Large-sample significance levels from multiply imputed data using moment-based statistics and an F reference distribution. Journal of the American Statistical Association, 86, 1065–1073.

Reiter, J. P. (2007). Small-sample degrees of freedom for multi-component significance tests with multiple imputation for missing data. Biometrika, 94, 502–508.

See Also

split_holdout(), discover_confirm(), ctree_stacked()

Examples

## Not run: 
library(mice)
set.seed(7)
n <- 600
d <- data.frame(x1 = rnorm(n), x2 = rnorm(n))
d$y <- rnorm(n) + 1.2 * (d$x1 > 0)
d$x1[sample(n, 90)] <- NA

parts <- split_holdout(d, seed = 7)
imp_d <- mice(parts$discover, m = 20, printFlag = FALSE, seed = 1)
imp_c <- mice(parts$confirm,  m = 20, printFlag = FALSE, seed = 2)

tree <- ctree_stacked(y ~ x1 + x2, data = imp_d, verbose = FALSE)
confirm_ctreeMI(tree, imp_c)

## End(Not run)

ctreeMI documentation built on Oct. 12, 2026, 5:08 p.m.