Assessing Bias and Precision in State Policy Evaluations

Using state-level opioid overdose mortality data (1999-2016), we evaluated the performance of panel data estimators for capturing time-varying impacts of state-level policies. Most health policy evaluations assume static treatment effects, yet many interventions exhibit dynamic impacts, raising methodological questions about optimal estimation strategies.

We simulated four time-varying treatment scenarios reflecting common policy dynamics (gradual increase, gradual decline, temporary effects, and inconsistent trajectories) and compared seven methods: two-way fixed effects event study, debiased autoregressive model, augmented synthetic control, difference-in-differences with staggered adoption, event study with heterogeneous treatment, two-stage differences-in-differences, and differences-in-differences imputation. Performance was assessed using bias, standard errors, coverage probability, and root mean squared error.

Estimator performance varied substantially across scenarios. Augmented synthetic controls showed lower bias but higher variance when policy effectiveness diminished over time. Difference-in-difference approaches provided reasonable coverage in some scenarios but struggled with non-monotonic effects, while autoregressive methods exhibited lower variability but underestimated uncertainty.

Overall, no single estimator performed best across settings. For epidemiological policy evaluations, particularly time-sensitive interventions like opioid-related policies, researchers should weigh bias-variance tradeoffs and align methodological choices with expected effect trajectories. Careful selection of analytic approaches is critical to avoid misattribution of policy effects and ensure valid conclusions about population health outcomes.