Estimands in experiments

Identification with experiments

Limits of identification

Identification without experiments

Agnostic bounds


What can we learn in an experiment?

Testing Theories with Experiment

Paper Topic Theoretical Motivation Intervention Outcome
Kremer and Glennerster (2011) Demand Law of demand price of health goods quantity \(\downarrow\)
Jensen and Miller (2008) Demand Giffen goods coupons for grain by income quantity \(\uparrow\)
Caunedo and Kala (2021) Industrialization Lewis dual sector models Subsidize tractor rental non-farm labor supply \(\uparrow\)
Crépon et al. (2013) Labor Search Matching function job search assistance by city congestion \(\uparrow\)
Breza, Kaur, and Shamdasani (2018) Wage Rigidity Bewley (1999) unequal wages within teams productivity \(\downarrow\)
Balboni et al. (2021) Poverty trap inflection point in returns give people asset (cows) inflection point found

Estimating treatment effects

Interpretation: Random coefficients

Relating to standard linear model

Estimation in Experiments


Fisher permutation test

  1. For \(j=1...J\), draw \(\{X^j_i\}_{i=1}^{n}\) from known assignment mechanism (assumed independent of \((Y_i^1,Y_i^0)_{i=1}^{n}\))
  2. Compute \(\{\widehat{\beta}_1^{j}\}_{j=1}^{J}\) by computing difference in means as if \(\{X^j_i\}_{i=1}^{n}\) had been the true realization of \(j\)
  3. Calculate \(p=\frac{1}{J}\sum_{j=1}^{J} 1\{|\widehat{\beta}|>\widehat{\beta}^j\}\)
  4. If \(p<\alpha\) reject sharp null

Analysis with covariates

Properties of regression adjustment for RCTs

Analysis with other models

Experimental design

What to choose when designing an experiment

How to randomize

Planning and preliminary data gathering phases


Application: Masks



Abaluck, Jason, Laura H Kwong, Ashley Styczynski, Ashraful Haque, Md Alamgir Kabir, Ellen Bates-Jefferys, Emily Crawford, et al. 2021. “The Impact of Community Masking on COVID-19: A Cluster-Randomized Trial in Bangladesh.” J-PAL.
Balboni, Clare A, Oriana Bandiera, Robin Burgess, Maitreesh Ghatak, and Anton Heil. 2021. “Why Do People Stay Poor?” National Bureau of Economic Research.
Bewley, Truman F. 1999. Why Wages Don’t Fall During a Recession. Harvard University Press.
Blair, Graeme, Jasper Cooper, Alexander Coppock, and Macartan Humphreys. 2019. “Declaring and Diagnosing Research Designs.” American Political Science Review 113: 838–59. https://declaredesign.org/paper.pdf.
Breza, Emily, Supreet Kaur, and Yogita Shamdasani. 2018. “The Morale Effects of Pay Inequality.” The Quarterly Journal of Economics 133 (2): 611–63.
Caunedo, Julieta, and Namrata Kala. 2021. “Mechanizing Agriculture.” National Bureau of Economic Research.
Crépon, Bruno, Esther Duflo, Marc Gurgand, Roland Rathelot, and Philippe Zamora. 2013. “Do Labor Market Policies Have Displacement Effects? Evidence from a Clustered Randomized Experiment.” The Quarterly Journal of Economics 128 (2): 531–80.
Ding, Peng. 2017. “A Paradox from Randomization-Based Causal Inference.” Statistical Science, 331–45.
Gupta, Shantanu, Zachary C. Lipton, and David Childers. 2021. “Efficient Online Estimation of Causal Effects by Deciding What to Observe.” NeuRIPS.
Imbens, Guido W, and Donald B Rubin. 2015. Causal Inference in Statistics, Social, and Biomedical Sciences. Cambridge University Press.
Jensen, Robert T, and Nolan H Miller. 2008. “Giffen Behavior and Subsistence Consumption.” American Economic Review 98 (4): 1553–77.
Kasy, Maximilian. 2016. “Why Experimenters Might Not Always Want to Randomize, and What They Could Do Instead.” Political Analysis 24 (3): 324–38.
Kremer, Michael, and Rachel Glennerster. 2011. “Improving Health in Developing Countries: Evidence from Randomized Evaluations.” In Handbook of Health Economics, 2:201–315. Elsevier.
Recht, Ben. 2021. “Arg Min Blog: Effect Size Is Significantly More Important Than Statistical Significance.” Stanford University. http://www.argmin.net/2021/09/13/effect-size/.