-JA ]]>

It’s worthwhile to note that you can end up with significant biases when you have in mismeasurement of the independent variable as well as the dependent variable.

]]>In a completely randomized experiment with a binary outcome, if you want to adjust for covariates to improve precision, you can use either logit (with an average marginal effect calculation) or OLS to consistently estimate the average treatment effect, even if your model’s “wrong”. Probit doesn’t enjoy this robustness property.

The first-order conditions for OLS and the logit MLE imply a nice property: if you compute an “untreated” predicted probability for each person, using her actual covariate values but setting the treatment dummy to 0, then the average “untreated” prediction in the control group equals the raw control mean. In large enough samples, this will be very similar to the average “untreated” prediction in the full sample (since the distribution of covariates in the control group will resemble the distribution in the full sample). The latter is a regression-adjusted control mean. So we have an adjusted control mean that enjoys the same consistency properties as the raw control mean.

Similarly, we can compute a “treated” predicted probability for each person, and the resulting adjusted treatment group mean enjoys the same consistency properties as the raw treatment group mean. So the difference between the adjusted treatment and control group means is consistent for ATE. None of this depends on the model being correct.

The probit MLE first-order conditions don’t imply the same nice property.

David Freedman gave a rigorous proof for logit in “Randomization does not justify logistic regression”. (The negative message is that you can’t just use predictions at the mean covariate values, and the coefficient on treatment doesn’t estimate anything meaningful if the model’s wrong. But diehard MHE fans already know that.)

Freedman also briefly discussed probits:

“On the other hand, with the probit, the plug-in estimators are unlikely to be consistent, since the analogs of the likelihood equations (16–18) below involve weighted averages rather than simple averages. In simulation studies … Numerical calculations also confirm inconsistency of the plug-in estimators [average marginal effect estimates from the probit], although the asymptotic bias is small.”

A couple other references:

D. Firth and K. Bennett (1998). “Robust models in probability sampling.” JRSSB 60: 3-21.

J. Wooldridge (2007). “Inverse probability weighted estimation for general missing data problems.” J. Econometrics 141: 1281-1301. (See section 6.2.)

]]>Hansen doesn’t mention Canada but he does report Monte Carlos for a scenario with N=10 in the cross-sectional dimension.

A caution, as we noted in MHE 8.2.3 on fewer than 42 clusters: Alas, the bottom line here is not entirely clear.

Also, we don’t know too much about the finite sample behavior of two-way clustering procedures. It’s not clear you wouldn’t be better off just to fatten up the cluster in one dimension and ignore the second. I’m sure someone is working on that (or at least they should be!)

JA

]]>