On Measuring and Reducing Selection Bias With a Quasi-Doubly Randomized Preference Trial

Abstract

Randomized experiments provide unbiased estimates of treatment effects, but are costly and time consuming. We demonstrate how a randomized experiment can be leveraged to measure selection bias by conducting a subsequent observational study that is identical in every way except that subjects choose their treatment—a quasi-doubly randomized preference trial (quasi-DRPT). Researchers first strive to think of and measure all possible confounders and then determine how well these confounders as controls can reduce or eliminate selection bias. We use a quasi-DRPT to study the effect of class time on student performance in an undergraduate introductory microeconomics course at a large public university, illustrating its required design elements: experimental and choice arms conducted in the same setting with identical interventions and measurements, and all confounders measured prospectively to treatment assignment or choice. Quasi-DRPTs augment randomized experiments in real-world settings where participants choose their treatments.

Citation: Joyce T, Remler DK, Jaeger DA, Altindag O, O’Connell SD, Crockett S. On Measuring and Reducing Selection Bias With a Quasi‐Doubly Randomized Preference Trial. Journal of Policy Analysis and Management. 2017 Mar;36(2):438-59. https://doi.org/10.1002/pam.21976