Repeated Cross Sections Data

DiD often works just fine with repeated cross sections data. \[ \begin{aligned} ATT &= \E[Y_{t=2}-Y_{t=1} | G=1] - \E[Y_{t=2}-Y_{t=1} | G=0] \hspace{150pt} \end{aligned} \]

Repeated Cross Sections Data

DiD often works just fine with repeated cross sections data. \[ \begin{aligned} ATT &= \E[Y_{t=2}-Y_{t=1} | G=1] - \E[Y_{t=2}-Y_{t=1} | G=0] \hspace{150pt}\\ &= \Big( \E[Y_{t=2} | G=1] - \E[Y_{t=1} | G=1] \Big) - \Big(\E[Y_{t=2} | G=0] - \E[Y_{t=1} | G=0]\Big) \end{aligned} \]

which you can compute with repeated cross sections data

“Repeated Cross Sections” in Min. Wage Ex.

attgt_rc <- did::att_gt(yname="lemp",
                     gname="G",
                     tname="year",
                     data=data2,
                     control_group="nevertreated",
                     base_period="universal",
                     panel=FALSE)
ggdid(aggte(attgt_rc, type="dynamic"))

“Repeated Cross Sections” in Min. Wage Ex.

Unbalanced Panel Data

Issues with unbalanced panel data seem to be very similar to typical panel data methods

• If not observing a unit in a particular time period (e.g., attrition) is “at random”, then everything should be fine

• If missing data is not a random, then this can be a major issue

“Unbalanced Panel Data” in Min. Wage Ex.

set.seed(123)
# randomly drop 100 observations
this_data <- data2[sample(1:nrow(data2), nrow(data2)-100),]
attgt_up <- did::att_gt(yname="lemp",
                        idname="id",
                        gname="G",
                        tname="year",
                        data=this_data,
                        control_group="nevertreated",
                        base_period="universal",
                        panel=TRUE,
                        allow_unbalanced_panel=TRUE)
ggdid(aggte(attgt_up, type="dynamic"))

“Unbalanced Panel Data” in Min. Wage Ex.

“Unbalanced Panel Data” in Min. Wage Ex.

attgt_upd <- did::att_gt(yname="lemp",
                         idname="id",
                         gname="G",
                         tname="year",
                         data=this_data,
                         control_group="nevertreated",
                         base_period="universal",
                         panel=TRUE,
                         allow_unbalanced_panel=FALSE)
ggdid(aggte(attgt_upd, type="dynamic"))

“Unbalanced Panel Data” in Min. Wage Ex.

Alternative Comparison Groups

So far, we have used the never-treated group as the comparison group. But there are other possibilities:

Min. Wage with Not-Yet-Treated Comp. Group

attgt_nyt <- did::att_gt(yname="lemp",
                         idname="id",
                         gname="G",
                         tname="year",
                         data=data2,
                         control_group="notyettreated",
                         base_period="universal")
ggdid(aggte(attgt_nyt, type="dynamic"))

Min. Wage with Not-Yet-Treated Comp. Group

Min. Wage with Not-Yet-But-Eventually-Treated Comp. Group

# have to do a little hack to get this to work
# drop never-treated group
this_data <- subset(data2, G != 0)
# note: this causes us to lose the 2006 group
# as it no longer has a valid comparison group
# and we lose some periods for the 2004 group
# because it only has a valide comparison group up to 2005
attgt_nye <- did::att_gt(yname="lemp",
                         idname="id",
                         gname="G",
                         tname="year",
                         data=this_data,
                         control_group="notyettreated",
                         base_period="universal")
ggdid(aggte(attgt_nye, type="dynamic"))

Min. Wage with Not-Yet-But-Eventually-Treated Comp. Group

Anticipation

Anticipation occurs when treatments start affecting outcomes in periods before the treatment actually occurs. Examples:

• Policies that are announced before they are implemented

• Ashenfelter’s dip—for applications like job training, often people participate because they experience a shock to their labor market prospects

  • This can cause us to overestimate the treatment effect if the effects of the shock quickly dissipate (so outcomes would have recovered even without the treatment)

Min. Wage with Anticipation

# note: this causes us to lose the 2004 group due
# to not enough pre-treatment periods
attgt_ant <- did::att_gt(yname="lemp",
                         idname="id",
                         gname="G",
                         tname="year",
                         data=data2,
                         control_group="nevertreated",
                         base_period="universal",
                         anticipation=1)
ggdid(aggte(attgt_ant, type="dynamic"))

Min. Wage with Anticipation

Alternative Base Periods (in Pre-Treatment Periods)

In all the results above, we have used a universal base period.

• This means that all differences (including pre-treatment periods) are relative to period \((g-1)\)

• (In my view) this is a legacy of implementing DID in a regression framework, where it is not clear if it is possible to make a different choice

Alternative Base Periods

Choosing between different base periods comes down to the type of information that you would like the event study to highlight

The case for a universal base period:

• Good option if you are mainly interested in highlighting long-run trend differences between groups
• Also, heuristically, this can be better in cases where you have more periods

The case for a varying base period:

• Good for highlighting anticipation
• Good in cases where differences between groups “bounce around”
• Pre-treatment estimates are easier to interpret

Min. Wage with Universal Base Period

attgt_uni <- did::att_gt(yname="lemp",
                         idname="id",
                         gname="G",
                         tname="year",
                         data=data2,
                         control_group="nevertreated",
                         base_period="universal")
ggdid(aggte(attgt_uni, type="dynamic"))

Min. Wage with Universal Base Period

Min. Wage with Varying Base Period

attgt_var <- did::att_gt(yname="lemp",
                         idname="id",
                         gname="G",
                         tname="year",
                         data=data2,
                         control_group="nevertreated",
                         base_period="varying")
ggdid(aggte(attgt_var, type="dynamic"))

Min. Wage with Varying Base Period

Pre-Test of Parallel Trends with Universal Base Period

summary(attgt_uni)

Call:
did::att_gt(yname = "lemp", tname = "year", idname = "id", gname = "G", 
    data = data2, control_group = "nevertreated", base_period = "universal")

Reference: Callaway, Brantly and Pedro H.C. Sant'Anna.  "Difference-in-Differences with Multiple Time Periods." Journal of Econometrics, Vol. 225, No. 2, pp. 200-230, 2021. <https://doi.org/10.1016/j.jeconom.2020.12.001>, <https://arxiv.org/abs/1803.09015> 

Group-Time Average Treatment Effects:
 Group Time ATT(g,t) Std. Error [95% Simult.  Conf. Band]  
  2004 2003   0.0000         NA            NA          NA  
  2004 2004  -0.0327     0.0210       -0.0858      0.0205  
  2004 2005  -0.0683     0.0218       -0.1232     -0.0133 *
  2004 2006  -0.1234     0.0219       -0.1786     -0.0681 *
  2004 2007  -0.1311     0.0230       -0.1891     -0.0731 *
  2006 2003  -0.0341     0.0114       -0.0630     -0.0052 *
  2006 2004  -0.0167     0.0082       -0.0375      0.0041  
  2006 2005   0.0000         NA            NA          NA  
  2006 2006  -0.0194     0.0093       -0.0430      0.0042  
  2006 2007  -0.0661     0.0093       -0.0895     -0.0427 *
---
Signif. codes: `*' confidence band does not cover 0

P-value for pre-test of parallel trends assumption:  0.01325
Control Group:  Never Treated,  Anticipation Periods:  0
Estimation Method:  Doubly Robust

Pre-Test of Parallel Trends with Varying Base Period

summary(attgt_var)

Call:
did::att_gt(yname = "lemp", tname = "year", idname = "id", gname = "G", 
    data = data2, control_group = "nevertreated", base_period = "varying")

Reference: Callaway, Brantly and Pedro H.C. Sant'Anna.  "Difference-in-Differences with Multiple Time Periods." Journal of Econometrics, Vol. 225, No. 2, pp. 200-230, 2021. <https://doi.org/10.1016/j.jeconom.2020.12.001>, <https://arxiv.org/abs/1803.09015> 

Group-Time Average Treatment Effects:
 Group Time ATT(g,t) Std. Error [95% Simult.  Conf. Band]  
  2004 2004  -0.0327     0.0223       -0.0882      0.0229  
  2004 2005  -0.0683     0.0222       -0.1235     -0.0130 *
  2004 2006  -0.1234     0.0216       -0.1771     -0.0696 *
  2004 2007  -0.1311     0.0244       -0.1917     -0.0705 *
  2006 2004   0.0174     0.0096       -0.0066      0.0414  
  2006 2005   0.0167     0.0085       -0.0046      0.0380  
  2006 2006  -0.0194     0.0090       -0.0418      0.0030  
  2006 2007  -0.0661     0.0093       -0.0893     -0.0429 *
---
Signif. codes: `*' confidence band does not cover 0

P-value for pre-test of parallel trends assumption:  0.01325
Control Group:  Never Treated,  Anticipation Periods:  0
Estimation Method:  Doubly Robust

Discussion

To me, it looks like the universal base period would be a better choice in our application

• It does not look like there are anticipation effects.

• It does look like there are trend differences between the treated and untreated units.

What about violations of parallel trends?

Parallel trends assumptions don’t automatically hold in applications with repeated observations over time

• The most common “response” is pre-testing…checking if parallel trends holds in pre-treatment periods

DID + pre-tests are a very powerful/useful approach to “validating” the parallel trends assumption. But they are less than a full test of parallel trends.

• Just because parallel trends holds in pre-treatment periods doesn’t mean it holds in post-treatment periods

• Pre-tests can suffer from low power (Roth (2022))

What about our case?

Partial Identification / Sensitivity Analysis

References: Manski and Pepper (2018), Rambachan and Roth (2023), Ban and Kedagni (2022)

Two versions of sensitivity analysis in RR:

1. Violations of parallel trends are “not too different” in post-treatment periods from the violations in pre-treatment periods
   • Allow for violations of parallel trends up to \(\bar{M}\) times as large as were observed in any pre-treatment period.
   • And we’ll vary \(\bar{M}\).
2. Violations of parallel trends evolve smoothly
   • Allow for linear trend up to \(M\) times as large as linear trend difference in pre-treatment periods.
   • We’ll vary \(M\)

Next slides: show these results in minimum wage application for the “on impact” effect of the treatment

What about violations of parallel trends?

What about violations of parallel trends?

Sampling Weights

Sampling weights are common in DID applications

• Can come from survey design (e.g., oversampling certain subpoopulations)

• Can also arise from working with aggregated data (e.g., counties) where we might want to count larger counties more than smaller counties

Sampling Weights

Tentative Heuristic Advice:

• If you have unit-level data (e.g., job displacement example) and there are sampling weights, use them

• If you have aggregate data (e.g., county-level data), but you are interested in effects at level of underlying unit (e.g., person-level), then you should consider using sampling weights

  • Possibly depends on nature of the outcome variable too: employment vs. log(employment) vs. per capita employment

Sampling Weights in Min. Wage Example

# create weights based on population
data2$pop <- exp(data2$lpop)
data2$avg_pop <- BMisc::get_Yibar(data2, "id", "pop")
attgt_sw <- did::att_gt(yname="lemp",
                        idname="id",
                        gname="G",
                        tname="year",
                        data=data2,
                        control_group="nevertreated",
                        base_period="universal",
                        weightsname="avg_pop")
ggdid(aggte(attgt_sw, type="dynamic"))

Sampling Weights in Min. Wage Example

Replace log(Employment) with Employment Rate

data2$emp_rate <- exp(data2$lemp)/data2$pop
attgt_er <- did::att_gt(yname="emp_rate",
                        idname="id",
                        gname="G",
                        tname="year",
                        data=data2,
                        control_group="nevertreated",
                        base_period="universal",
                        weightsname="avg_pop")
ggdid(aggte(attgt_er, type="dynamic"))

Replace log(Employment) with Employment Rate

Introduction

There are a number of complications that can arise for inference in DID settings:

1. Serial correlation

2. Fixed population inference

3. Small number of treated units

4. Clustered treatment assignment

5. Multiple hypothesis testing

Examples

Different issues related to inference arise given different types of data, all of which are common in DID applications.

Serial Correlation

Probably the most common inference issue in DID settings is serial correlation.

Sample consists of:

\[\{Y_{i1}, Y_{i2}, \ldots, Y_{iT}, D_{i1}, D_{i2}, \ldots D_{iT} \}_{i=1}^n\]

which are iid across units, drawn from a “super-population”, and where the number of units in each group is “large”. This is a common sampling scheme in panel data applications (e.g., job displacement).

Fixed Population Inference

The previous discussion has emphasized traditional sampling uncertainty arising from drawing a sample from an underlying super-population.

In many DID applications, we observe the entire population of interest (e.g., all 50 states).

• Does this means that we “know” the \(ATT\)?…probably not.

Fixed Population Inference

One possibility is to condition on (i.e., treat as fixed/non-random) \((G_i, \theta_t, \eta_i)\) while treating as random \(e_{it}\).

• Intuition: repeated sampling thought experiment where we redraw \(e_{it} \sim (\mu_i, \sigma^2_i)\) for each unit \(i\) and time period \(t\) while unit fixed effects, time fixed effects, and treatment status are held fixed.

• Adjusted target parameter: mean of finite sample \(ATT\) across repeated samples \[ATT(g,t) := \frac{1}{n_g} \sum_{G_i=g} \E[Y_{t} - Y_{t}(0) | G_i=g]\]

• Adjusted parallel trends: mean of finite sample parallel trends across repeated samples \[\frac{1}{n_g} \sum_{G_i=g} \E[\Delta Y_{t}(0) | G_i=g] - \frac{1}{n_\infty} \sum_{G_i=\infty} \E[\Delta Y_{t}(0) | U_i=1]\]

\(\implies\) somewhat different interpretation, but use same \(\widehat{ATT}\) and constructing unconditional standard errors (i.e., same as before) can be conservative in this case. See Borusyak, Jaravel, and Spiess (2024), for example.

Few Treated Clusters with Aggregate Data

What if the number of treated units is small (e.g., 1 treated unit)?

In this case, if there is one treated unit, we can often come up with an unbiased (though not consistent) estimator of the \(ATT\).

For inference, there are a number of ideas, often involving some kind of permutation test

• See, for example, T. G. Conley and Taber (2011), Ferman (2019), Hagemann (2019), Roth and Sant’Anna (2023) for examples

• These approaches often require auxiliary assumptions such as restrictions on heteroskedasiticity or differences in cluster sizes

Clustered Treatment Assignment

Many DID applications have clustered treatment assignment—all units within the same cluster (e.g., a state) have the same treatment status/timing.

The most common approach in this setting is to cluster at the level of the treatment.

• When we additionally cluster at the unit level (to allow for serial correlation), this is often referred to as two-way clustering. Most software implementations readily allow for this.

To rationalize conducting inference in this way typically requires a large number of both treated and untreated clusters.

• This rules out certain leading applications such as the Card and Krueger (1994) minimum wage study, where the only clusters are New Jersey and Pennsylvania.

Clustered Treatment Assignment

Can we conduct inference in cases where there are only two clusters (but we observe underlying data)?

• Yes, but it may require additional assumptions.

• Let’s start with the setting where there are exactly two clusters. Then (for simplicity just focusing on untreated potential outcomes), \[\begin{align*} Y_{ij,t}(0) &= \theta_t + \eta_i + e_{ij,t} \\ &= \theta_t + \eta_i + \underbrace{\nu_{j,t} + \epsilon_{ij,t}} \end{align*}\] where \(\nu_{j,t}\) is a cluster-specific time-varying error term and \(\epsilon_{ij,t}\) are idiosyncratic, time-varying unobservables (possibly serially correlated but independent across units).

• \(\nu_{j,t}\) is often viewed as a common cluster-specific shock that affects all units in the cluster (e.g., some other state-level policy).

• For inference with two clusters, we need \(\nu_{j,t}=0\).

  • Without this condition, parallel trends would be violated.
  • If holds, then we can cluster at the unit-level rather than the cluster-level \(\implies\) possible to conduct inference.

Comparing Different Strategies

Suppose that we are studying a policy that is implemented in a few states.

Option 1: Include a large number of states, cluster at state-level

• For clustering: We may only need the weaker condition that \(\E[\nu_{j,t} | G_i] = 0\).

• For identification: less robust to small violations of parallel trends.

Option 2: Only include a few similar states, cluster at unit-level

• For clustering: We need the stronger condition that \(\nu_{j,t}=0\).

• For identification: possibly more robust to small violations of parallel trends.

In my view, there is a fairly strong case for (in many applications) using tighter comparisons with a smaller number of clusters while arguing that \(\nu_{j,t}=0\) by a combination of

• researcher legwork

• pre-testing

Further Reading on Clustering

• T. Conley, Gonçalves, and Hansen (2018) - on clustering in general

• Roth et al. (2023) - on clustering in the context of DID

Multiple Hypothesis Testing

If you report multiple hypothesis tests and/or confidence intervals—e.g., an event study—it’s a good idea to make an adjustment for multiple hypothesis testing.

sup-t confidence band—construct confidence intervals of the form \[\Big[\widehat{ATT}^{es}(e) \pm \hat{c}_{1-\alpha/2} \textrm{s.e.}\big(\widehat{ATT}^{es}(e)\big) \Big]\] but instead of choosing the critical value as a quantile of the normal distribution, choose a (slightly) larger critical value that accounts for the fact that you are testing multiple hypotheses.

• Typically, an appropriate choice for the critical value to ensure the correct (uniform) coverage requires resampling methods.

• The reason for this is that the appropriate critical value depends on the joint distribution of \(\sqrt{n}(\widehat{ATT}^{es}(e) - ATT^{es}(e))\) across all \(e\), and these are generally not independent of each other.

• That said, this is not too difficult in practice, and is a default option in the did package.

Min. Wage w/o uniform confidence bands

# note: these are the varying base period results
# that were "just barely" not statistically significant
# in pre-treatment periods
attgt_poi <- did::att_gt(yname="lemp",
                         idname="id",
                         gname="G",
                         tname="year",
                         data=data2,
                         control_group="nevertreated",
                         base_period="varying",
                         cband=FALSE)
ggdid(aggte(attgt_poi,type="dynamic",cband=FALSE))

Min. Wage w/o uniform confidence bands

Min. Wage w/ uniform confidence bands

# critical value for uniform confidence band
# computed using resamplin
attgt$c

     95% 
2.532749