Nothing
Pre-Testing in a DiD Setup using the did Package
In did: Treatment Effects with Multiple Periods and Groups
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>"
)
library(did)
# Source the currently version of the did package (based on our Dropbox)
# fldr <- here::here("R/")
# sapply(paste0(fldr,list.files(fldr)), source)
# Source simulation designs
source(here::here("vignettes/setup_sims.R"))
Introduction
This vignette provides a discussion of how to conduct pre-tests in DiD setups using the did package
-
One appealing feature of many DiD applications with multiple periods is that the researcher can pre-test the parallel trends assumptions.
-
The idea here is simple: although one can not always test whether parallel trends itself holds, one can check if it holds in periods before treated units actually become treated.
-
Importantly, this is just a pre-test; it is different from an actual test. Whether or not the parallel trends assumption holds in pre-treatment periods does not actually tell you if it holds in the current period (and this is when you need it to hold!). It is certainly possible for the identifying assumptions to hold in previous periods but not hold in current periods; it is also possible for identifying assumptions to be violated in previous periods but for them to hold in current periods. That being said, we view the pre-test as a piece of evidence on the credibility of the DiD design in a particular application.
-
In this vignette, we demonstrate that the approach used in the did package for pre-testing may work substantially better than the more common "event study regression".
Common Approaches to Pre-Testing in Applications
By far the most common approach to pre-testing in applications is to run an event-study regression. Here, the idea is to run a regression that includes leads and lags of the treatment dummy variable such as
$$
Y_{it} = \theta_t + \eta_i + \sum_{l=-\mathcal{T}}^{\mathcal{T}-1} D_{it}^l \mu_l + v_{it}
$$
where $D_{it}^l = 1$ if individual $i$ has been exposed to the treatment for $l$ periods in period $t$, and $D_{it}^l = 0$ otherwise. To be clear here, it is helpful to give some examples. Suppose individual $i$ becomes treated in period 3. Then,
-
$D_{it}^0 = 1$ when $t=3$ and is equal to 0 in other time periods
-
$D_{it}^2 = 1$ when $t=5$ and is equal to 0 in other time periods
-
$D_{it}^{-2} = 1$ when $t=1$ and is equal to 0 in other time periods.
And $\mu_l$ is interpreted as the effect of treatment for different lengths of exposure to the treatment. Typically, $\mu_{-1}$ is normalized to be equal to 0, and we follow that convention here. It is common to interpret estimated $\mu_l$'s with $l < 0$ as a way to pre-test the parallel trends assumption.
Pitfalls with Event Study Regressions
time.periods <- 4
reset.sim()
bett <- betu <- rep(0, time.periods)
te <- 0
set.seed(1814)
Best Case Scenario for Pre-Testing
First, let's start with a case where an event study regression is going to work well for pre-testing the parallel trends assumption
# generate dataset with 4 time periods
time.periods <- 4
# generate dynamic effects
te.e <- time.periods:1
# generate data set with these parameters
# (main thing: it generates a dataset that satisfies
# parallel trends in all periods...including pre-treatment)
data <- build_sim_dataset()
head(data)
The main thing to notice here:
- The dynamics are common across all groups. This is the case where an event-study regression will work.
Next, a bit more code
#-----------------------------------------------------------------------------
# modify the dataset a bit so that we can run an event study
#-----------------------------------------------------------------------------
# generate leads and lags of the treatment
Dtl <- sapply(-(time.periods-1):(time.periods-2), function(l) {
dtl <- 1*( (data$period == data$G + l) & (data$G > 0) )
dtl
})
Dtl <- as.data.frame(Dtl)
cnames1 <- paste0("Dtmin",(time.periods-1):1)
colnames(Dtl) <- c(cnames1, paste0("Dt",0:(time.periods-2)))
data <- cbind.data.frame(data, Dtl)
row.names(data) <- NULL
head(data)
#-----------------------------------------------------------------------------
# run the event study regression
#-----------------------------------------------------------------------------
# load plm package
library(plm)
# run event study regression
# normalize effect to be 0 in pre-treatment period
es <- plm(Y ~ Dtmin3 + Dtmin2 + Dt0 + Dt1 + Dt2,
data=data, model="within", effect="twoways",
index=c("id","period"))
summary(es)
#-----------------------------------------------------------------------------
# make an event study plot
#-----------------------------------------------------------------------------
# some housekeeping for making the plot
# add 0 at event time -1
coefs1 <- coef(es)
ses1 <- sqrt(diag(summary(es)$vcov))
idx.pre <- 1:(time.periods-2)
idx.post <- (time.periods-1):length(coefs1)
coefs <- c(coefs1[idx.pre], 0, coefs1[idx.post])
ses <- c(ses1[idx.pre], 0, ses1[idx.post])
exposure <- -(time.periods-1):(time.periods-2)
cmat <- data.frame(coefs=coefs, ses=ses, exposure=exposure)
library(ggplot2)
ggplot(data=cmat, mapping=aes(y=coefs, x=exposure)) +
geom_line(linetype="dashed") +
geom_point() +
geom_errorbar(aes(ymin=(coefs-1.96*ses), ymax=(coefs+1.96*ses)), width=0.2) +
ylim(c(-2,5)) +
theme_bw()
You will notice that everything looks good here. The pre-test performs well (the caveat to this is that the standard errors are "pointwise" and would be better to have uniform confidence bands though this does not seem to be standard practice in applications).
We can compare this to what happens using the did package:
# estimate group-group time average treatment effects
did_att_gt <- att_gt(yname="Y",
tname="period",
idname="id",
gname="G",
data=data,
bstrap=FALSE,
cband=FALSE)
summary(did_att_gt)
# plot them
ggdid(did_att_gt)
# aggregate them into event study plot
did_es <- aggte(did_att_gt, type="dynamic")
# plot the event study
ggdid(did_es)
Overall, everything looks good using either approach. (Just to keep things fair, we report pointwise confidence intervals for group-time average treatment effects, but it is easy to get uniform confidence bands by setting the options bstrap=TRUE, cband=TRUE to the call to att_gt.)
Pitfall: Selective Treatment Timing
Sun and Abraham (2021) point out a major limitation of event study regressions: when there is selective treatment timing the $\mu_l$ end up being weighted averages of treatment effects across different lengths of exposures.
Selective treatment timing means that individuals in different groups experience systematically different effects of participating in the treatment from individuals in other groups. For example, there would be selective treatment timing if individuals choose to be treated in earlier periods if they tend to experience larger benefits from participating in the treatment. This sort of selective treatment timing is likely to be present in many applications in economics / policy evaluation.
Contrary to event study regressions, pre-tests based on group-time average treatment effects (or based on group-time average treatment effects that are aggregated into an event study plot) are still valid even in the presence of selective treatment timing.
To see this in action, let's keep the same example as before, but add selective treatment timing.
reset.sim()
bett <- betu <- rep(0, time.periods)
te <- 0
set.seed(1814)
# generate dataset with 4 time periods
time.periods <- 4
# generate dynamic effects
te.e <- time.periods:1
# generate selective treatment timing
# (*** this is what is different here ***)
te.bet.ind <- time.periods:1 / (time.periods/2)
# generate data set with these parameters
# (main thing: it generates a dataset that satisfies
# parallel trends in all periods...including pre-treatment)
data <- build_sim_dataset()
# run through same code as in earlier example...omitted
#-----------------------------------------------------------------------------
# modify the dataset a bit so that we can run an event study
#-----------------------------------------------------------------------------
# generate leads and lags of the treatment
Dtl <- sapply(-(time.periods-1):(time.periods-2), function(l) {
dtl <- 1*( (data$period == data$G + l) & (data$G > 0) )
dtl
})
Dtl <- as.data.frame(Dtl)
cnames1 <- paste0("Dtmin",(time.periods-1):1)
colnames(Dtl) <- c(cnames1, paste0("Dt",0:(time.periods-2)))
data <- cbind.data.frame(data, Dtl)
row.names(data) <- NULL
#-----------------------------------------------------------------------------
# run the event study regression
#-----------------------------------------------------------------------------
# load plm package
library(plm)
# run event study regression
# normalize effect to be 0 in pre-treatment period
es <- plm(Y ~ Dtmin3 + Dtmin2 + Dt0 + Dt1 + Dt2,
data=data, model="within", effect="twoways",
index=c("id","period"))
summary(es)
#-----------------------------------------------------------------------------
# make an event study plot
#-----------------------------------------------------------------------------
# some housekeeping for making the plot
# add 0 at event time -1
coefs1 <- coef(es)
ses1 <- sqrt(diag(summary(es)$vcov))
idx.pre <- 1:(time.periods-2)
idx.post <- (time.periods-1):length(coefs1)
coefs <- c(coefs1[idx.pre], 0, coefs1[idx.post])
ses <- c(ses1[idx.pre], 0, ses1[idx.post])
exposure <- -(time.periods-1):(time.periods-2)
cmat <- data.frame(coefs=coefs, ses=ses, exposure=exposure)
# run through same code as before...omitted
# new event study plot
ggplot(data=cmat, mapping=aes(y=coefs, x=exposure)) +
geom_line(linetype="dashed") +
geom_point() +
geom_errorbar(aes(ymin=(coefs-1.96*ses), ymax=(coefs+1.96*ses)), width=0.2) +
ylim(c(-2,5)) +
theme_bw()
In contrast to the last case, it is clear that things have gone wrong here. Parallel trends holds in all time periods and for all groups here, but the event study regression incorrectly rejects that parallel trends holds -- this is due to the selective treatment timing.
We can compare this to what happens using the did package:
# estimate group-group time average treatment effects
did.att.gt <- att_gt(yname="Y",
tname="period",
idnam="id",
gname="G",
data=data
)
summary(did.att.gt)
# plot them
ggdid(did.att.gt)
# aggregate them into event study plot
did.es <- aggte(did.att.gt, type="dynamic")
# plot the event study
ggdid(did.es)
This is the correct performance (up to aforementioned caveats about multiple hypothesis testing).
Conditional Moment Tests
Another main use case for the did package is when the parallel trends assumptions holds after conditioning on some covariates. This is likely to be important in many applications. For example, to evaluate the effect of participating in a job training program on earnings, it is likely to be important to condition on an individual's education. This would be true if (i) the distribution of education is different for individuals that participate in job training relative to those that don't (this is very likely to hold as people that participate in job training tend to have less education than those who do not), and (ii) if the path of earnings (absent participating in job training) depends on an individual's education. See Heckman, Ichimura, and Todd (1998) and Abadie (2005) for more discussion.
Even when one includes covariates to estimate group-time average treatment effects, pre-tests based only on group-time average treatment effects can fail to detect some violations of the parallel trends assumption. To give an example, suppose that the only covariate is binary variable for an individual's sex. Pre-tests based on group-time average treatment effects could fail to detect violations of the conditional parallel trends assumption in cases where it is violated in one direction for men and in the other direction for women.
The did package contains an additional pre-test for the conditional parallel trends assumption in the conditional_did_pretest function.
# not run (this code can be substantially slower)
reset.sim()
set.seed(1814)
nt <- 1000
nu <- 1000
cdp <- conditional_did_pretest("Y", "period", "id", "G", xformla=~X, data=data)
cdp
References
-
-
-
-
Try the did package in your browser
Any scripts or data that you put into this service are public.
did documentation built on July 20, 2022, 5:06 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
knitr::opts_chunk$set( collapse = TRUE, comment = "#>" )
library(did) # Source the currently version of the did package (based on our Dropbox) # fldr <- here::here("R/") # sapply(paste0(fldr,list.files(fldr)), source) # Source simulation designs source(here::here("vignettes/setup_sims.R"))
Introduction
This vignette provides a discussion of how to conduct pre-tests in DiD setups using the did package
-
One appealing feature of many DiD applications with multiple periods is that the researcher can pre-test the parallel trends assumptions.
-
The idea here is simple: although one can not always test whether parallel trends itself holds, one can check if it holds in periods before treated units actually become treated.
-
Importantly, this is just a pre-test; it is different from an actual test. Whether or not the parallel trends assumption holds in pre-treatment periods does not actually tell you if it holds in the current period (and this is when you need it to hold!). It is certainly possible for the identifying assumptions to hold in previous periods but not hold in current periods; it is also possible for identifying assumptions to be violated in previous periods but for them to hold in current periods. That being said, we view the pre-test as a piece of evidence on the credibility of the DiD design in a particular application.
-
-
In this vignette, we demonstrate that the approach used in the did package for pre-testing may work substantially better than the more common "event study regression".
Common Approaches to Pre-Testing in Applications
By far the most common approach to pre-testing in applications is to run an event-study regression. Here, the idea is to run a regression that includes leads and lags of the treatment dummy variable such as $$ Y_{it} = \theta_t + \eta_i + \sum_{l=-\mathcal{T}}^{\mathcal{T}-1} D_{it}^l \mu_l + v_{it} $$
where $D_{it}^l = 1$ if individual $i$ has been exposed to the treatment for $l$ periods in period $t$, and $D_{it}^l = 0$ otherwise. To be clear here, it is helpful to give some examples. Suppose individual $i$ becomes treated in period 3. Then,
-
$D_{it}^0 = 1$ when $t=3$ and is equal to 0 in other time periods
-
$D_{it}^2 = 1$ when $t=5$ and is equal to 0 in other time periods
-
$D_{it}^{-2} = 1$ when $t=1$ and is equal to 0 in other time periods.
And $\mu_l$ is interpreted as the effect of treatment for different lengths of exposure to the treatment. Typically, $\mu_{-1}$ is normalized to be equal to 0, and we follow that convention here. It is common to interpret estimated $\mu_l$'s with $l < 0$ as a way to pre-test the parallel trends assumption.
Pitfalls with Event Study Regressions
time.periods <- 4 reset.sim() bett <- betu <- rep(0, time.periods) te <- 0 set.seed(1814)
Best Case Scenario for Pre-Testing
First, let's start with a case where an event study regression is going to work well for pre-testing the parallel trends assumption
# generate dataset with 4 time periods time.periods <- 4 # generate dynamic effects te.e <- time.periods:1 # generate data set with these parameters # (main thing: it generates a dataset that satisfies # parallel trends in all periods...including pre-treatment) data <- build_sim_dataset() head(data)
The main thing to notice here:
- The dynamics are common across all groups. This is the case where an event-study regression will work.
Next, a bit more code
#----------------------------------------------------------------------------- # modify the dataset a bit so that we can run an event study #----------------------------------------------------------------------------- # generate leads and lags of the treatment Dtl <- sapply(-(time.periods-1):(time.periods-2), function(l) { dtl <- 1*( (data$period == data$G + l) & (data$G > 0) ) dtl }) Dtl <- as.data.frame(Dtl) cnames1 <- paste0("Dtmin",(time.periods-1):1) colnames(Dtl) <- c(cnames1, paste0("Dt",0:(time.periods-2))) data <- cbind.data.frame(data, Dtl) row.names(data) <- NULL head(data) #----------------------------------------------------------------------------- # run the event study regression #----------------------------------------------------------------------------- # load plm package library(plm) # run event study regression # normalize effect to be 0 in pre-treatment period es <- plm(Y ~ Dtmin3 + Dtmin2 + Dt0 + Dt1 + Dt2, data=data, model="within", effect="twoways", index=c("id","period")) summary(es) #----------------------------------------------------------------------------- # make an event study plot #----------------------------------------------------------------------------- # some housekeeping for making the plot # add 0 at event time -1 coefs1 <- coef(es) ses1 <- sqrt(diag(summary(es)$vcov)) idx.pre <- 1:(time.periods-2) idx.post <- (time.periods-1):length(coefs1) coefs <- c(coefs1[idx.pre], 0, coefs1[idx.post]) ses <- c(ses1[idx.pre], 0, ses1[idx.post]) exposure <- -(time.periods-1):(time.periods-2) cmat <- data.frame(coefs=coefs, ses=ses, exposure=exposure) library(ggplot2) ggplot(data=cmat, mapping=aes(y=coefs, x=exposure)) + geom_line(linetype="dashed") + geom_point() + geom_errorbar(aes(ymin=(coefs-1.96*ses), ymax=(coefs+1.96*ses)), width=0.2) + ylim(c(-2,5)) + theme_bw()
You will notice that everything looks good here. The pre-test performs well (the caveat to this is that the standard errors are "pointwise" and would be better to have uniform confidence bands though this does not seem to be standard practice in applications).
We can compare this to what happens using the did package:
# estimate group-group time average treatment effects did_att_gt <- att_gt(yname="Y", tname="period", idname="id", gname="G", data=data, bstrap=FALSE, cband=FALSE) summary(did_att_gt) # plot them ggdid(did_att_gt)
# aggregate them into event study plot did_es <- aggte(did_att_gt, type="dynamic") # plot the event study ggdid(did_es)
Overall, everything looks good using either approach. (Just to keep things fair, we report pointwise confidence intervals for group-time average treatment effects, but it is easy to get uniform confidence bands by setting the options bstrap=TRUE, cband=TRUE to the call to att_gt.)
Pitfall: Selective Treatment Timing
Sun and Abraham (2021) point out a major limitation of event study regressions: when there is selective treatment timing the $\mu_l$ end up being weighted averages of treatment effects across different lengths of exposures.
Selective treatment timing means that individuals in different groups experience systematically different effects of participating in the treatment from individuals in other groups. For example, there would be selective treatment timing if individuals choose to be treated in earlier periods if they tend to experience larger benefits from participating in the treatment. This sort of selective treatment timing is likely to be present in many applications in economics / policy evaluation.
Contrary to event study regressions, pre-tests based on group-time average treatment effects (or based on group-time average treatment effects that are aggregated into an event study plot) are still valid even in the presence of selective treatment timing.
To see this in action, let's keep the same example as before, but add selective treatment timing.
reset.sim() bett <- betu <- rep(0, time.periods) te <- 0 set.seed(1814)
# generate dataset with 4 time periods time.periods <- 4 # generate dynamic effects te.e <- time.periods:1 # generate selective treatment timing # (*** this is what is different here ***) te.bet.ind <- time.periods:1 / (time.periods/2) # generate data set with these parameters # (main thing: it generates a dataset that satisfies # parallel trends in all periods...including pre-treatment) data <- build_sim_dataset()
# run through same code as in earlier example...omitted
#----------------------------------------------------------------------------- # modify the dataset a bit so that we can run an event study #----------------------------------------------------------------------------- # generate leads and lags of the treatment Dtl <- sapply(-(time.periods-1):(time.periods-2), function(l) { dtl <- 1*( (data$period == data$G + l) & (data$G > 0) ) dtl }) Dtl <- as.data.frame(Dtl) cnames1 <- paste0("Dtmin",(time.periods-1):1) colnames(Dtl) <- c(cnames1, paste0("Dt",0:(time.periods-2))) data <- cbind.data.frame(data, Dtl) row.names(data) <- NULL #----------------------------------------------------------------------------- # run the event study regression #----------------------------------------------------------------------------- # load plm package library(plm)
# run event study regression # normalize effect to be 0 in pre-treatment period es <- plm(Y ~ Dtmin3 + Dtmin2 + Dt0 + Dt1 + Dt2, data=data, model="within", effect="twoways", index=c("id","period")) summary(es)
#----------------------------------------------------------------------------- # make an event study plot #----------------------------------------------------------------------------- # some housekeeping for making the plot # add 0 at event time -1 coefs1 <- coef(es) ses1 <- sqrt(diag(summary(es)$vcov)) idx.pre <- 1:(time.periods-2) idx.post <- (time.periods-1):length(coefs1) coefs <- c(coefs1[idx.pre], 0, coefs1[idx.post]) ses <- c(ses1[idx.pre], 0, ses1[idx.post]) exposure <- -(time.periods-1):(time.periods-2) cmat <- data.frame(coefs=coefs, ses=ses, exposure=exposure)
# run through same code as before...omitted # new event study plot ggplot(data=cmat, mapping=aes(y=coefs, x=exposure)) + geom_line(linetype="dashed") + geom_point() + geom_errorbar(aes(ymin=(coefs-1.96*ses), ymax=(coefs+1.96*ses)), width=0.2) + ylim(c(-2,5)) + theme_bw()
In contrast to the last case, it is clear that things have gone wrong here. Parallel trends holds in all time periods and for all groups here, but the event study regression incorrectly rejects that parallel trends holds -- this is due to the selective treatment timing.
We can compare this to what happens using the did package:
# estimate group-group time average treatment effects did.att.gt <- att_gt(yname="Y", tname="period", idnam="id", gname="G", data=data ) summary(did.att.gt) # plot them ggdid(did.att.gt)
# aggregate them into event study plot did.es <- aggte(did.att.gt, type="dynamic") # plot the event study ggdid(did.es)
This is the correct performance (up to aforementioned caveats about multiple hypothesis testing).
Conditional Moment Tests
Another main use case for the did package is when the parallel trends assumptions holds after conditioning on some covariates. This is likely to be important in many applications. For example, to evaluate the effect of participating in a job training program on earnings, it is likely to be important to condition on an individual's education. This would be true if (i) the distribution of education is different for individuals that participate in job training relative to those that don't (this is very likely to hold as people that participate in job training tend to have less education than those who do not), and (ii) if the path of earnings (absent participating in job training) depends on an individual's education. See Heckman, Ichimura, and Todd (1998) and Abadie (2005) for more discussion.
Even when one includes covariates to estimate group-time average treatment effects, pre-tests based only on group-time average treatment effects can fail to detect some violations of the parallel trends assumption. To give an example, suppose that the only covariate is binary variable for an individual's sex. Pre-tests based on group-time average treatment effects could fail to detect violations of the conditional parallel trends assumption in cases where it is violated in one direction for men and in the other direction for women.
The did package contains an additional pre-test for the conditional parallel trends assumption in the conditional_did_pretest function.
# not run (this code can be substantially slower) reset.sim() set.seed(1814) nt <- 1000 nu <- 1000 cdp <- conditional_did_pretest("Y", "period", "id", "G", xformla=~X, data=data) cdp
References
Try the did package in your browser
Any scripts or data that you put into this service are public.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.