In lmiratrix/blkvar: ATE and Treatment Variation Estimation for Blocked and Multisite RCTs

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
library( blkvar )
library( dplyr )
library( knitr )
options( digits=3 )

The blkvar package implements a variety of methods for estimating the average treatment impact in a blocked or multisite randomized trial. It also provides methods for estimating the amount of cross-site (or cross-block) treatment variation, defined as the variance of the individual block (or site) ATEs. It finally provides a variety of methods to implement blocked or multisite simulation studies.

Estimating the ATE

There are several classes of methods for estimating the ATE:

• Linear regression with block dummies
• Multilevel regression
• Design based estimators

For a running example we generate some data

df <- generate_multilevel_data_no_cov(n.bar = 50, J = 30, 
                     gamma.10 = 0, 
                     tau.11.star = 0.2 ^ 2, ICC = 0.65, 
                     variable.n = TRUE)
head( df )

To estimate the ATE using (almost) all the various methods provided we have

tbl <- compare_methods(Yobs, Z, sid, data=df )
knitr::kable( tbl )

The results are as follows:

• The hybrid_m and hybrid_p methods are from Pashley & Miratrix (2020), and are estimators that can account for singleton treated and control units in some of the blocks. plug_in_big is another version that uses the estimated variance across the big blocks for the small block variances.
• The DB are the 4 design-based estimators (see, e.g., the RCTYes software and documentation).
• FE are the "Fixed Effect" linear model estimators with different choices for uncertainty estimation (classic OLS, heteroskedastic robust, cluster robust, and the club sandwich cluster robust).
• FE-IPTW are reweighted fixed effect regressions to debias the model to target either site or person average ATE.
• RICC, FIRC, and RIRC are multilevel model estimators. RICC is the Random Intercept, Constant Coefficient, FIRC is Fixed Intercept, Random Coefficient, and RIRC is Random Intercept, Random Coefficient.

One can select which methods to use:

tbl <- compare_methods(Yobs, Z, sid, data=df, 
                       include_block = FALSE, 
                       include_MLM = FALSE,
                       include_DB = FALSE)
knitr::kable( tbl )

Adjusting for covariates

Covariate adjustment for individual level covariates is implemented for several methods (especially the linear regression based ones).

df$X1 = df$Y0 + rnorm( nrow( df ) ) # add a fake covariate
df$X2 = df$Y1 + rnorm( nrow( df ) ) # add another fake covariate

rs = compare_methods( Yobs, Z, sid, data=df, 
                      control_formula = ~X1 + X2 )
knitr::kable( rs )

Estimating cross site variation

We can estimate cross site variation using some of the multilevel modeling methods. See Weiss et al. (2017) for good discussion of the methods discussed. The estimate_ATE_FIRC method, for example, provides ATE_hat, the estimate of cross site variation. To call the method, list the outcome, the treatment assignment (as a 0-1 vector, and a site identifier).

ests <- estimate_ATE_FIRC( Yobs, Z, sid, data=df )
t( ests )

(We transpose with t() to make it a nice row instead of a list.)

This can also be done as

ests <- estimate_ATE_FIRC( df$Yobs, df$Z, df$sid )
t( ests )

It is preferred, however, to pass as a dataframe with variable names as follows:

ests.RIRC = estimate_ATE_RIRC( Yobs, Z, sid, data=df )
t( ests.RIRC )

These methods also have a pool option which uses a MLM with a single variance parameter for all individuals, rather than separate ones for the treatment and control groups (the latter option is still pooled across block, as is suggested by the literature).

We can also test for cross site variation using a $Q$-statistic approach:

ests.Qstat = analysis_Qstatistic( Yobs, Z, sid, data=df, alpha=0.05, calc_CI=TRUE )
t( ests.Qstat )

The confidence interval comes from inverting the test statistic for various values of cross-site variation.

We can compare all these estimators via

tbl = compare_methods_variation( Yobs, Z, sid, data=df, long_results=TRUE )
knitr::kable(tbl)

The $p$-values are testing for these estimates being nonzero.

References

Pashley, N. E., & Miratrix, L. W. (2020). Insights on variance estimation for blocked and matched pairs designs. arXiv preprint arXiv:1710.10342.

Weiss, M. J., Bloom, H. S., Verbitsky-Savitz, N., Gupta, H., Vigil, A. E., & Cullinan, D. N. (2017). How Much Do the Effects of Education and Training Programs Vary Across Sites? Evidence From Past Multisite Randomized Trials. Journal of Research on Educational Effectiveness, 10(4), 843–876. http://doi.org/10.1080/19345747.2017.1300719

lmiratrix/blkvar documentation built on Nov. 18, 2024, 1:27 p.m.