Causality again
- Last class, discussed causal models and experiments
- Potential outcomes \(Y^x\) describe how \(Y\) would be if \(X\) changed to \(x\)
- Can always observe \(Y=Y^X\), what happened given observed \(X\)
- With experiment, can learn distribution of \(Y^x\), \(P(Y|do(X=x))\)
- Assigning \(X\) at random implies \(P(Y|do(X=x))=P(Y|X)\)
- Today:
- Inference in experiments
- What to do with non-experimental data
Experimental Data
- Increasingly, substantial amount of economic data from experiments
- Field experiments: assign social program at random
- JPAL runs hundreds of economic experiments all around the world
- Businesses experiment with design, marketing, customer interactions
- Called “A/B tests” by tech companies
- Microsoft, Amazon, Facebook, Google conduct over 10,000 each/year
Problems with nonexperimental data
Example: Job Training and Earnings
- US runs many job training programs for low skill workers
- Goal is to get them back from bad economic situation to find higher-paying work
- Is training effective? Hard to tell \[E[\text{earnings|training}]-E[\text{earnings|no training}]=\] \[E[\text{earnings change|do(training),trained}]+\] \[(E[\text{untrained earnings|trained}]-E[\text{untrained earnings|not trained}])\]
- Want to know effectiveness of program
- First term gives this, at least for participants
- Problem is, second term is probably very negative
- People get job training because they have a bad job, or no job at all
- On average, those who get training are those with lower untrained earnings
Experiments and treatment effects
- Suppose treatment assigned randomly: \((Y_i^0,Y_i^1) \perp X_i\)
- Independence means conditional distributions same as unconditional
- Selection bias now \[E[Y_i^0|X_i=1]-E[Y_i^0|X_i=0]=E[Y_i^0]-E[Y_i^0]=0\]
- ATT now \[E[Y_i^1-Y_i^0|X_i=1]=E[Y_i^1-Y_i^0]=\text{ATE}\]
- By L.L.N. Difference in means consistently estimates ATE for randomized experiments
- In small samples, estimate not exact
- May have drawn sample where unobseved variables differ between treatment and control groups
Random coefficients
- Write potential outcomes model in more familiar form \[Y_i=Y_i^0+(Y_i^1-Y_i^0)X_i\]
- Define \(\beta_{0,i}=Y_i^0\), \(\beta_{1,i}=Y_i^1-Y_i^0\), then \[Y_i=\beta_{0,i}+\beta_{1,i}X_i\]
- Slope is treatment effect, intercept is value if not treated
- Result is a linear model with random coefficients
- Like linear model, but slope terms no longer constant
Relating to standard linear model
- Taking averages, can write as
- \(\beta_{0,i}:=\bar{\beta}_0+e_{0i}\)
- \(\beta_{1,i}:=\bar{\beta}_1+e_{1i}\)
- \(E[e_{0i}]=E[e_{1i}]=0\)
- Random coefficients model becomes \[Y_i=\bar{\beta}_0+\bar{\beta}_1X_i+e_{0i}+X_ie_{1i}\]
- A standard linear model with heteroskedastic errors
- Slope coefficient \(\bar{\beta}_1\) is ATE
- Endogeneity: under nonrandom assignment, residual may be correlated with \(X_i\)
Estimation
- If X assigned randomly, \(X_i\perp e_{0i}\) (no selection bias) and \(X_i\perp e_{1i}\) (treatment effect independent of treatment assignment)
- Model becomes standard linear model satisfying Assumptions (1-4)
- \(\hat{\beta}_1\) OLS estimator same as difference in means
- Heteroskedasticity has meaningful interpretation
- Residual \(e_{0i}+X_ie_{1i}\) has variance which depends on \(X\) so long as \(e_{1i}\neq 0\)
- “Heterogeneous treatment effects”
- OLS with robust standard errors gives valid inference on ATE for experimental data
- Equivalent to two-sample t-test on difference in means with unequal variances
Example: National Supported Work Program Experiment (Code 1)
#Load package containing data set
#see sekhon.berkeley.edu/matching/ for info
#If not installed, run following command
install.packages("Matching",
repos = "http://cran.us.r-project.org",
dependencies=TRUE)
library(Matching)
data(lalonde) #Load data set
library(sandwich) #Robust SEs
library(lmtest) #testing with robust SEs
# Show treatment unrelated to covariates
balancereg<-lm(formula = treat ~ age + educ + black +
hisp + married + nodegr + re74 + re75 +
u74 + u75, data = lalonde)
Example: National Supported Work Program Experiment (Code 2)
#Build robust standard errors
balancereg.vcov<-vcovHC(balancereg, type="HC0")
#Conduct Wald test that coefs jointly 0
#using asymptotic Chi-squared distribution
jointtest<-waldtest(balancereg,
vcov=vcovHC(balancereg,type="HC0"), test = "Chisq")
Example: National Supported Work Program Experiment
- National Supported Work Program (analyzed in Lalonde 1986, Dehejia & Wahba 1998)
- Take group of low earnings workers, randomly assign to on the job training (\(X=1\)) or no intervention (\(X=0\))
- Compare post program earnings for treated and untreated workers
- Can see it is uncorrelated with observable characteristics by regressing tratment on pre-treatment covariates
- A Wald test that all coefficients 0 tests a null which is implied by independence
jointtest<-waldtest(balancereg,
vcov=vcovHC(balancereg,type="HC0"), test = "Chisq")
Results
## Wald test
##
## Model 1: treat ~ age + educ + black + hisp + married + nodegr + re74 +
## re75 + u74 + u75
## Model 2: treat ~ 1
## Res.Df Df Chisq Pr(>Chisq)
## 1 434
## 2 444 -10 21.564 0.01749 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
- Non-rejection (at 1% level) reassuring
- Note that point of experiment is that treatment independent of all characteristics, observed and unobserved
- These tests can’t be used to verify experiment assumption
Estimating treatment effect (Code)
#Run simple regression of outcome (Real earnings in 78)
# on treatment in program
experiment<-lm(formula = re78 ~ treat, data = lalonde)
#Construct Robust SEs, test significance of causal effect
experiment.vcov<-vcovHC(experiment, type="HC0")
experiment.output<-coeftest(experiment,
df=Inf,vcov. = experiment.vcov)
library(stargazer) #Display table of results
stargazer(experiment.output,type="html",style="all",
header=FALSE,omit.table.layout="ldn",
title="Job Training and 1978 Real Wages")
Estimating treatment effect
- Setting is experimental, with binary treatment
- Run OLS regression of earnings on training, with robust SEs
- Coefficient has interpretation as Average Treatment Effect
Job Training and 1978 Real Wages
|
|
|
treat
|
1,794.343***
|
|
|
(669.316)
|
|
|
t = 2.681
|
|
|
p = 0.008
|
|
Constant
|
4,554.802***
|
|
|
(339.438)
|
|
|
t = 13.419
|
|
|
p = 0.000
|
|
|
|
|
Interpretation
- Strong and significant coefficient on treatment
- Treated earn $1800 more annually on average
- Treatment effect \(\bar{\beta}_1\) large relative to SE
- Unlikely difference due to sampling error
- Also large relative to intercept
- \(\bar{\beta}_0\): average earnings of workers conditional on no training
- Experimental design suggests this difference is causal
- If you took more workers from same population and put them in same program, they would also earn about $1800/year more on average
Observational Data and Natural Experiments
- For many questions, no experiment run
- Too costly or unethical, or just not run
- Instead have observational data from observed economic process
- Key to experimental setup is random assignment
- Someone or something set \(X\) independently of other determinants of \(Y\)
- Doesn’t have to be conscious
- Could be random natural event or capricious policymaker
- If this happen have natural experiment
- Can again interpret effect causally
- Rare, but useful to exploit when it does happen
Control
- In practice, rare in observational data that relevant variable affecting \(Y\) randomly assigned
- Instead, have many \(X_{ij}\) determined by a variety of mechanisms with causal links between them
- Today: convenient special cases where regression suffices
- Next class: some more general theory
Controlling for covariates
- New setup
- \(Y\) is outcome of interest
- \(X\) is treatment
- \(W\) are pre-treatment covariates
- \(W\) may cause both X and Y, but are not caused by them
- If we observe all the \(W\) that determine assignment of treatment and might themselves affect \(Y\), can still learn about effects of \(X\)
- Instead of random assignment \(Y^x\perp X\), we can weaken to conditionally random assignment \[Y^x\perp X|W\]
Example: Consumer Credit
- Effect of obtaining mortgage loan \(X\) on consumer spending \(Y\)
- People who can get credit would likely have very different spending habits than those who can’t, even in world where they didn’t get loan
- “Selection Bias”: \(E[Y^0|X=1]-E[Y^0|X=0]\neq 0\)
- But, if we have electronic loan application data, info on form, \(W\), contains all characteristics that determine treatment \(X\)
- Any variation in loan approval comes from bank decisions, made without knowledge of other individual characteristics and so independent of them
- Result: \(Y^x\perp X|W\)
- Conditional on loan data, there should be no relationship between consumer characteristics and what spending would be if given or not given loan
- Given \(W\), approval is “effectively” randomly assigned
Identifying Treatment Effects \(P(Y|do(x))\)
- Assume \(Y^x\perp X|W\), and go through same identification argument as in experiment case \[P(Y|X=x,W=w)=P(Y^X|X=x,W=w)\] \[=P(Y^x|X=x,W=w)\] \[=P(Y^x|W=w)=P(Y|do(X),observe(W=w))\]
- By causal consistency, conditioning, and conditional independence
- Result: recover distribution of conditional outcome Y, in situations where \(W\) is observed
- To get unconditional effect \(P(Y^x)\), integrate over w \[=\int P(Y^x|W=w) P(w)dw=\int P(Y|X=x,W=w)P(w)dw\]
Estimating Treatment Effects
- Result says we can recover causal effects if we know \(P(Y|X,W)\)
- We have tool for estimating conditional expectations
- Impose linear model on P(Y|X,W) \[E[Y|X,W]=X\beta_1+W^\prime\beta_2\]
- If c.e.f. correctly specified, \(\beta_1\) is causal effect of \(X\) on \(Y\), given \(W\) observed
- Need to regress \(Y\) on \(X\) AND \(W\)
- Get unbiased estimate of causal effect
- In this linear additive case, treatment effect is slope \(\beta_1\)
- In this case, omitting \(W\), or some component of it, from regression would cause bias in estimate of \(\beta_1\)
- This is called “confounding” or “omitted variable bias”
Omitted variables again
- Unbiased estimation requires correct specification of conditional expectation function
- Suggests including appropriate functional forms and all possible elements of \(W\) important here
- We now have causal interpretation of omitted variables formula
- Suppose \(E[Y|X,W]=\beta_0+\beta_1X+\beta_2W\)
- Regression of \(Y\) on \(X\) alone gives \(\tilde{\beta}_0+\tilde{\beta}_1X\)
- Regression of \(W\) on \(X\) gives \(\delta_0+\delta_1X\)
- Omitted variables formula says \(\tilde{\beta}_1=\beta_1+\beta_2*\delta_1\)
- Omitted variables bias occurs when both
- confounder correlated with outcome (\(\beta_2\neq 0\))
- confounder correlated with treatment
- Error precisely corresponds to selection bias: causal effect of treatment is to take unit with same characteristics and change status, but if units who get or don’t get treatment differ in some way that affects the outcome, comparison reflects both the treatment effect and the difference in the units selected
Covariates in binary treatment framework
- Random assignment \((Y_i^0,Y_i^1) \perp X_i\) relaxed to \[(Y_i^0,Y_i^1) \perp X_i | W_i\]
- Outcome is \(Y_i=Y^0_i+(Y^1_i-Y^0_i)X_i\), with expectation \[E[Y_i|X_i,W_i]=E[Y^0_{i}|W_i]+E[Y^1_i-Y^0_i|W_i]X_i\]
- Assume a linear CEF for \(Y^0_i\): \(Y^0_{i}=\bar{\beta}_0+\gamma W +e_{0i}\) with \(E[e_{0i}|W]=0\)
- If treatment effect independent of controls, have \[Y^1_i-Y^0_i=\bar{\beta}_1+e_{1i}\]
- Result is random coefficients linear regression, with ATE=\(\bar{\beta}_1\) \[Y_i=\bar{\beta}_0+\bar{\beta}_1X_i+\gamma W_i+e_{0i}+X_ie_{1i}\]
- Under these conditions, OLS with robust SEs recovers average treatment effect
Heterogeneous treatment effects
- In general, causal effect may vary with \(W\)
- Loan increases spending more for consumers with some characteristics than others
- In structural representation, this means \(W_i\) correlated with \(e_{1i}\)
- When \(W\) added to regression, it is now correlated with residual
- Assume linear CEF \[Y^1_i-Y^0_i=\bar{\beta}_1+\delta W_i+e_{1i}\] structural formula now \[Y_i=\bar{\beta}_0+\bar{\beta}_1X_i+\gamma W_i+\delta W_i\times X_i+e_{0i}+X_ie_{1i}\]
- Use linear regression again, with interaction terms (and robust SEs)
- Average treatment effect is \(\bar{\beta}_1+\delta E[W_i]\)
Lessons
- Causality defined in terms of model of what things could be like
- Experiments can recover causal effects on average
- Regression with appropriate controls can recover causal effects
- Regression with missing controls causes omitted variable bias
Next class
- More on causality
- Structural models
- When regression does and doesn’t work