Bootstrapping for Propensity Score Analysis

Jason M. Bryer

Project website:

Bug Reports:



As the popularity of propensity score methods for estimating causal effects in observational studies increase, the choices for specific methods for has also increased. Rosenbaum (2012) has suggested that there are substantial benefits for testing a hypothesis more than once. With the wide availability of high power computers resampling methods such as bootstrapping (Efron, 1979) have become popular for providing more stable estimates of the sampling distribution. This paper introduces the PSAboot package for R that provides functions for bootstrapping propensity score methods. It deviates from traditional bootstrapping methods by allowing for different sampling specifications for treatment and control groups. Additionally, this framework will provide estimates using multiple methods for each bootstrap sample. Two examples are discussed: the classic National Work Demonstration and PSID (Lalonde, 1986) study and a study on tutoring effects on student grades.

Set the number of bootstrap samples. This should be set to at least 100 but kept small to reduce the execution time for CRAN submissions.

boot.M <- 10
if(boot.M < 100) {
    cat(paste0('**NOTE: This vignette uses ', boot.M, ' bootstrap samples. It is ',
        'generally recommended to use at least 100 bootstrap samples or more for final publication.**'))


The latest version of the PSAboot package can be downloaded from Github using the devtools package.


The PSAboot function will perform the actual bootstrapping. It has a number of parameters for to specify how the bootstrap samples are drawn.

Other parameters can be passed to methods using the ... parameter.


The methods parameter on the PSAboot function specifies the different propensity score methods that will be used. Specifically, for each bootstrap sample drawn, each function will be called. This allows for a comparison of methods across all bootstrap samples. Five methods are included, they are:

Defining Custom Methods

It is possible to define a custom method. Each method is a function with, at minimum, the following six parameters:

Each method must return a list with three elements:

For example, the boot.matching.1to3 function below wraps the built-in boot.matching method but sets the M parameter to 3, thereby performing 1-to-3 matching instead of the default 1-to-1 matching. This framework simplifies the process of using, and comparing, slight variations of different propensity score methods.

boot.matching.1to3 <- function(Tr, Y, X, X.trans, formu, ...) {
    return(boot.matching(Tr=Tr, Y=Y, X=X, X.trans=X.trans, formu=formu, M=3, ...))

The PSAboot function returns an object of class PSAboot. The following S3 methods are implemented: print, summary, plot, boxplot, and matrixplot.

Example One: National Work Demonstration and PSID

The lalonde (Lalonde, 1986) has become the de defacto teaching dataset in PSA since Dehejia and Wahba's (1999) re-examination of the National Supported Work Demonstration (NSW) and the Current Population Survey (CPS).

The lalonde data set is included in the MatchIt package. The crosstab shows that there are 429 control units and 185 treatment units.

data(lalonde, package='MatchIt')
lalonde.formu <- treat ~ age + I(age^2) + educ + I(educ^2) + black +
    hispan + married + nodegree + re74  + I(re74^2) + re75 + I(re75^2) +
    re74 + re75
boot.lalonde <- PSAboot(Tr=lalonde$treat, 

The summary function provides numeric results for each method including the overall estimate and confidence interval using the complete sample as well as the pooled estimates and confidence intervals with percentages of the number of confidence intervals that do not span zero.


The plot function plots the estimate (mean difference) for each bootstrap sample. The default is to sort from largest to smallest estimate for each method separately. That is, rows do not correspond across methods. The sort parameter can be set to none for no sorting or the name of any method to sort only based upon the results of that method. In these cases the rows then correspond to matching bootstrap samples. The blue points correspond to the the estimate for each bootstrap sample and the horizontal line to the confidence interval. Confidence intervals that do not span zero are colored red. The vertical blue line and green lines correspond to the overall pooled estimate and confidence for each method, respectively.


The hist function plots a histogram of the estimates across all bootstrap samples for each method.


The boxplot function depicts the distribution of estimates for each method along with confidence intervals in green. Additionally, the overall pooled estimate and confidence interval across all bootstrap samples and methods are represented by the vertical blue and green lines, respectively.


The matrixplot summarizes the estimates across methods for each bootstrap sample. The lower half of the matrix are scatter plots where each point represents the one bootstrap sample. The red line is a Loess regression line. The main diagonal depicts the distribution of effects and the upper half provides the correlation of estimates. In the following example we see that the ctree and Stratification methods have the strongest agreement with a correlation of 0.63 whereas the rpart and MatchIt methods have the least agreement with a correlation of 0.22.


Evaluating Balance

The strength of propensity score analysis relies on achieving good balance. Typically one would evaluate each covariate separately to ensure that sufficient balance has been achieved. We recommend Helmreich and Pruzek (2009) for a more complete discussion of visualizations to evaluate balance. Given the large number of samples and methods used, it is desirable to have a single metric to evaluate balance. Drawing on the principles of the multiple covariate balance assessment plot (Helmreich & Pruzek, 2009), the balance function will estimate the effect of each covariate before and after adjustment. Moreover, to provide a single metric for each sample and method, the mean standardized effect size will be used.

lalonde.bal <- balance(boot.lalonde)

The plot function provides density plots of the balance statistics across all bootstrap samples for each method. The mean overall balance is represented by the vertical black lines, the overall balance for the complete dataset is represented by the vertical blue line, and the adjusted balance is represented by the vertical red line. Although no specific guidelines are recommended in the literature, achieving a balance of less than 0.1 is desirable. In this example we see that PSA has reduce the bias in all methods although rpart, and to a lesser extent MatchIt, did not reduce the bias by as much as would be typically desired.


The boxplot function provides a more nuanced depiction of the balance by separating on the distribution of balance statistics by individual covariate. In addition to the boxplot of balance statistics, the mean balance statistic is represented by the blue point and the unadjusted balance statistic by the red point. We see that the largest imbalance was in the black, married, and re74 (real earnings in 1974) before adjustment. All of the estimates reduced the bias in these covariates although it is worth noting that the MatchIt method did not reduce the bias in the black covariate to a desirable level.

This figure, in conjunction with the density plot above, show that the relatively large mean balance for rpart is likely due to some outlier samples where the adjusted balance for educ, re75, and hispan are fairly large. It should be noted that with these exceptions, balance for each covariates are generally less than the unadjusted balance.


Example Two: Effectiveness of Tutoring on Course Grades

This example utilizes a dataset in the TriMatch package that provides student data in online English courses. The original study examined the effects of tutoring on English course grades. In the original study two treatments were defined, those who used tutoring services once and those who used tutoring services two or more times during the course term. The outcome of interest was course grade (0=F, 1=D, 2=C, 3=B, 4=A). For our purposes here we will consider only one treatment group: students who used tutoring services at least once during the course.

data(tutoring, package='TriMatch')
tutoring$treatbool <- tutoring$treat != 'Control'
covs <- tutoring[,c('Gender', 'Ethnicity', 'Military', 'ESL', 'EdMother', 'EdFather',
                    'Age', 'Employment', 'Income', 'Transfer', 'GPA')]

tutoring.boot <- PSAboot(Tr=tutoring$treatbool, 
                         control.sample.size=918, control.replace=TRUE,
                         treated.sample.size=224, treated.replace=TRUE,
tutoring.bal <- balance(tutoring.boot)


Bryer, J.M. (2013). TriMatch: Propensity Score Matching of Non-Binary Treatments. R package version 0.9.1.

Helmreich, J.E., Pruzek, R.M. (2009). PSAgraphics: An R Package to Support Propensity Score Analysis. Journal of Statistical Software 29(6), 1-23.

Ho, D.E., Imai, K., King, G., & Stuart, E.A. (2011). MatchIt: Nonparametric Preprocessing for Parametric Causal Inference. Journal of Statistical Software, (42) 8, 1-28. URL

Hothorn, T., Hornik, K., & Zeileis, A. (2006). Unbiased Recursive Partitioning: A Conditional Inference Framework. Journal of Computational and Graphical Statistics, 15(3), 651-674.

Lalonde, R. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review 76: 604-620.

Dehejia, R.H. and Wahba, S. (1999). Causal Effects in Nonexperimental Studies: Re-Evaluating the Evaluation of Training Programs. Journal of the American Statistical Association 94: 1053-1062.

R Core Team (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL

Rosenbaum, P.R. (2012). Testing one hypothesis twice in observational studies. Biometrika, 99, 4, 763-774.

Rosenbaum, P.R., & Rubin, D.B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 1, 41-55.

Sekhon, J.S. (2011). Multivariate and Propensity Score Matching Software with Automated Balance Optimization: The Matching Package for R. Journal of Statistical Software, 42(7), 1-52. URL

Therneau, T., Atkinson, B., & Ripley, B. (2013). rpart: Recursive Partitioning. R package version 4.1-3.

Try the PSAboot package in your browser

Any scripts or data that you put into this service are public.

PSAboot documentation built on May 29, 2017, 10:18 a.m.