Policy Evaluation during a Pandemic

.title[
# Policy Evaluation during a Pandemic
]
.author[
### Brantly Callaway1 and Tong Li2 1University of Georgia, 2Vanderbilt University 
]
.date[
### November 20, 2022 Southern Economics Association Conference
]

---

# Introduction

`$$\newcommand{\E}{\mathbb{E}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\var}{\mathrm{var}}
\newcommand{\cov}{\mathrm{cov}}
\newcommand{\Var}{\mathrm{var}}
\newcommand{\Cov}{\mathrm{cov}}
\newcommand{\Corr}{\mathrm{corr}}
\newcommand{\corr}{\mathrm{corr}}
\newcommand{\L}{\mathrm{L}}
\renewcommand{\P}{\mathrm{P}}
\newcommand{\independent}{{\perp\!\!\!\perp}}$$`

border-top: 80px solid #BA0C2F;

.inverse {
  background-color: #BA0C2F;
}

.alert {
    font-weight:bold; 
    color: #BA0C2F;
}

.alert-blue {
    font-weight: bold;
    color: blue;
}

.remark-slide-content {
    font-size: 23px;
    padding: 1em 4em 1em 4em;
}

.highlight-red {
 background-color:red;
 padding:0.1em 0.2em;
}

.assumption-box {
    background-color: rgba(222,222,222,.5);
    font-size: x-large;
    padding: 10px; 
    border: 10px solid lightgray; 
    margin: 10px;
}

.assumption-title {
 font-size: x-large;
 font-weight: bold;
 display: block;
 margin: 10px;
 text-decoration: underline;
 color: #BA0C2F;
}
</style>

- In this paper, we consider the compatibility of different reduced form policy evaluation strategies with leading pandemic models

- Tons of recent empirical work evaluating the effects of Covid-related policies

- Shelter-in-place orders, school closings, mask mandates, contact tracing, testing policies, etc.

- Recent summary of the literature: Haber et al. (2021)

- At first glance, this seems in the wheelhouse of economists

- Large amount of relevant data
    
    - Policies enacted at different points in time across different locations
    
--
    
    - `$\implies$` Difference-in-differences
    
---

# Introduction

In this paper, we are going to take a second look

1. Hard to rationalize DID at all

- Not generally compatible with leading pandemic models
    
    - Pre-testing does not work well (especially early in the pandemic)
    
--

2. We suggest using a particular version of unconfoundedness that involves conditioning on important pre-treatment pandemic related variables

- We'll largely ignore a number of other potential challenges: e.g., measurement error in cases, multiple policies, spillovers, non-random/limited Covid-19 testing
    
--

3. We also propose an approach to study effects of Covid-related policies on economic outcomes: e.g., unemployment, travel, etc.

---

# Outline

1. Stochastic SIRD Models

2. Policy Effects on Covid-19 Cases

3. Application: Shelter-in-Place Orders

---

# Stochastic SIRD Models

---

# Stochastic SIRD Model

SIRD models are the workhorse model for the spread of an epidemic

- Kermack and McKendrick (1927), Allen (2008), and Allen (2017)

- In economics: Oka, Wei, and Zhu (2020), Fernandez-Villaverde and Jones (2020), Ellison (2020), Acemoglu, Chernozhukov, Werning, and Whinston (2020), others

In a particular location, all individuals are in one of four states:

S - Susceptible

I - Infected

R - Recovered

`$\delta$` - Dead

---

# Stochastic SIRD Model

Additional Notation:

- `$N_l$` - number of individuals in location `$l$`

- `$C_{lt}$` - cumulative cases in location `$l$` by time period `$t$`

- `$\mathcal{F}_{lt} = (S_{lt}, I_{lt}, R_{lt}, \delta_{lt})$` - "state" of the pandemic in location `$l$` in time period `$t$`

Key idea: Markov transition equations for active cases, recoveries, deaths, number susceptible, and cumulative cases

Additional Parameters:

- `$\lambda$` - recovery rate

- `$\gamma$` - death rate

- `$\beta$` - infection rate

---

# Stochastic SIRD Model

Active Cases:

`$$\E[I_{lt} | \mathcal{F}_{lt-1}] = (1-\lambda-\gamma) I_{lt-1} + \beta \frac{I_{lt-1}}{N_l} S_{lt-1}$$`

Recoveries:

`$$\E[R_{lt} | \mathcal{F}_{lt-1}] = R_{lt-1} + \lambda I_{lt-1}$$`

Deaths:

`$$\E[\delta_{lt} | \mathcal{F}_{lt-1}] = \delta_{lt-1} + \gamma I_{lt-1}$$`

---

# Stochastic SIRD Model

Susceptible:

`$$\E[S_{lt}| \mathcal{F}_{lt-1}] = S_{lt-1} - \beta \frac{I_{lt-1}}{N_l} S_{lt-1}$$`

Cumulative Cases:

`$$\E[C_{lt}| \mathcal{F}_{lt-1}] = C_{lt-1} + \beta \frac{I_{lt-1}}{N_l}S_{lt-1}$$`

---

# Simulated SIRD Pandemic

---

# Policy Effects on Covid-19 Cases

---

# Setup

Additional Potential Outcomes Notation:

- `$D_l$` - indicates whether or not location `$l$` participates in the treatment

- Treated potential outcomes: `$S_{lt}(1)$`, `$I_{lt}(1)$`, `$R_{lt}(1)$`, `$\delta_{lt}(1)$`, `$C_{lt}(1)$`

- Untreated potential outcomes: `$S_{lt}(0)$`, `$I_{lt}(0)$`, `$R_{lt}(0)$`, `$\delta_{lt}(0)$`, `$C_{lt}(0)$`

- `$\mathcal{T}$` total time periods and treatment occurs in time period `$t^*$`.

- For simplicity, suppose that treatment timing is the same across all locations
    
--

- Observed outcomes (for example):

- In pre-treatment periods, when `$t < t^*$`: `$C_{lt} = C_{lt}(0)$` for all locations 
 
 - In post-treatment periods, when `$t \geq t^*$`: `$C_{lt} = D_l C_{lt}(1) + (1-D_l) C_{lt}(0)$`

- Long difference notation, for `$t_2 > t_1$`: `$\Delta^{(t_1, t_2)} C_t := C_{t_2} - C_{t_1}$`

---

# Target Parameter

- Target parameter, for `$t \geq t^*$`:

`$$ATT^C_t = \E[C_t(1) - C_t(0) | D=1]$$`

---

# Review of Difference-in-Differences

The main assumption underlying a DID approach is the parallel trends assumption:

## Parallel Trends Assumption

For all `$t=2,\ldots,\mathcal{T}$`,

`$$\E[\Delta C_t(0) | D=1] = \E[\Delta C_t(0) | D=0]$$`

DID estimand given by:

`$$DID^C_t = \E[\Delta^{(t^*-1,t)}C_t | D=1] - \E[\Delta^{(t^*-1,t)} C_t |D=0]$$`

Under parallel trends, it essentially immediately follows that

`$$ATT^C_t = DID^C_t$$`

---

# Difference-in-Differences

DID is very closely related to the following model for untreated potential outcomes:

`$$C_{lt}(0) = \theta_t + \eta_l + v_{lt}$$`

(see, for example, Gardner (2021), Borusyak, Jaravel, and Spiess (2021), Ghanem, Sant'Anna, and Wuthrich (2022)) and where

- `$\theta_t$` - is a time fixed effect

- `$\eta_l$` - is a location (individual) fixed effect (can be distributed differently for the treated group relative to )

- `$v_{lt}$` - idiosyncratic unobservables

Taking difference `$C_{lt}(0)$` over time gets rid of `$\eta_l$` and immediately implies parallel trends holds

---

# Difference-in-Differences

DID is very closely related to the following model for untreated potential outcomes:

`$$C_{lt}(0) = \theta_t + \eta_l + v_{lt}$$`

This is a much different model from the SIRD model we discussed earlier

`$$C_{lt}(0) = C_{lt-1}(0) + \beta \frac{I_{lt-1}(0)}{N_l}S_{lt-1}(0) + v_{lt}$$`

Side-comment: You can re-arrange this to get

`$$\Delta C_{lt}(0) = \beta \frac{I_{lt-1}(0)}{N_l}S_{lt-1}(0) + v_{lt}$$`

but `$S_{lt-1}(0) = N_l - C_{lt-1}(0) \implies$` need to condition on pre-treatment outcome `$\stackrel{\textrm{in my opinion}}{\implies}$` no longer DID because it reduces to unconfoundedness (see Abadie (2005))

---

# Main Results

Under a stochastic SIRD model for untreated potential outcomes:

1. Parallel trends does not generally hold

2. The bias from incorrectly imposing parallel trends is:

- Potentially complicated
  
    - Depends on differences in the distribution of the "state" of the pandemic in period `$t^*-1$` between the treated and untreated group

- Cannot "sign" the bias in general, but suppose that (i) early policy adopters have more pre-policy Covid-19 cases, and (ii) looking at short-term effects of policy `$\implies$` may spuriously estimate *positive* effects of policies
    
--

3. A version of unconfoundedness holds (mainly: conditional on the pre-treatment "state" of the pandemic)

---

# Complications

There are a number of ways you could make things more complicated

Some are compatible with our framework:

- SIRD parameters that vary over time

- SIRD parameters that depend on a location's observed characteristics (e.g., region)

- Anticipation effects

Some are not (immediately) compatible:

- Measurement error in cases

- Spillovers

- TWFE versions of SIRD parameters, such as: `$\beta_{lt} = \beta_l + \beta_t$`

---

# Application: Shelter-in-Place Orders

---

# Application Setup

We study Shelter-in-Place Orders (SIPOs) early in the pandemic, and their efffect on

- Covid-19 cases (from CDC Covid Data Tracker) <-- This one only today

- Travel (from Google Mobility reports)

These have been widely studied in a number of papers

- Consider early part of the pandemic: March 10, 2020 to May 1, 2020

- State level data <-- This one only today

- County-level data

Main emphasis for today: (non-) robustness of DID methods in this context

---

# Various Versions of DID

To start with, we'll show results from various ways to implement DID

- Outcomes in levels, parallel trends

- Outcomes in levels, linear trends

- Outcomes in logs, parallel trends

- Outcomes in logs, linear trends

We'll also use TWFE regression versions of these as well as "treatment effect heterogeneity robust" approaches from Callaway and Sant'Anna (2021) and Gardner (2021).

---

# TWFE, Levels, Parallel Trends

---

# TWFE, Levels, Linear Trends

---

# TWFE, Logs, Parallel Trends

---

# TWFE, Logs, Linear Trends

---

# CS, Levels, Parallel Trends

---

# Gardner, Levels, Linear Trends

---

# CS, Logs, Parallel Trends

---

# Gardner, Logs, Linear Trends

---

# Discussion

(1) DID results are all over the place...not robust to functional form (e.g., Roth and Sant'Anna (2021))

* none of the specifications here compatible with SIRD model (see also Alcott et al. (2020))

(2) Positive effects of SIPOs on Covid-19 cases not reasonable

(3) Its infeasible to decide between approaches based on performance is pre-treatment periods

* None of them are rejected in pre-treatment periods.

---

# Additional Results

Under unconfoundedness: We do not find strong evidence that SIPOs decreased Covid-19 cases

- However 1: What's the counterfactual? - Even among untreated states, there are massive decreases in travel during the time period that we consider

- However 2: Confidence bands are actually pretty wide too (compatible with approximately 30% reduction in cases)

Using county-level data (which has its own advantages and disadvantages), we provide further evidence that:

- DID results are all over the place

- DID can result in relatively large estimates of effects of "placebo policies" in states that never implemented a SIPO

- Unconfoundedness performs better along these dimensions, and tends to result in small or no effect of SIPOs on Covid-19 cases.

---

# Conclusion

We studied policy evaluation strategies for understanding Covid-19 related policies

- Because pandemics are highly nonlinear, the choice of identification strategy can be very important

- In particular, DID seems unlikely to perform well here

- Other identification strategies like unconfoundedness are at least compatible leading pandemic models (and are likely better choices)

- That said, there are still a number of complications and, in my view, evaluating Covid-19 related policies is quite challenging

Five Minute Summary: <a href="https://bcallaway11.github.io/posts/five-minute-pandemic-policy">https://bcallaway11.github.io/posts/five-minute-pandemic-policy</a>

Code: <a href="https://github.com/bcallaway11/ppe">https://github.com/bcallaway11/ppe</a>