Your data
Paste data or load the example.
Result
Estimates
| Effect | Method | Estimate | SE | 95% CI | p |
|---|
Balance: standardised mean differences
before after weighting after matching; dashed lines at ±0.1
Overlap: propensity scores by arm
treated (above) control (below)
Covariate balance
| Covariate | Treated mean | Control mean | SMD before | SMD weighted | SMD matched | Variance ratio weighted |
|---|
Propensity model (logistic regression)
| Term | Coefficient | SE |
|---|
Sample
| Quantity | Treated | Control |
|---|
Method and citation
The propensity score e(x) is the probability of treatment given the covariates (Rosenbaum & Rubin 1983). The tool estimates it by a logistic regression of the treatment on the covariates you list (maximum likelihood by Newton-Raphson). If treatment depends only on those covariates (no unmeasured confounding) and every unit had some chance of either arm (overlap), comparing units with the same score compares like with like.
Inverse probability weighting gives each treated unit weight 1/e and each control 1/(1−e) for the ATE; for the ATT treated units weigh 1 and controls e/(1−e); for the ATC treated units weigh (1−e)/e and controls 1. Each arm's weights are normalised to sum to one. Regression adjustment fits a linear regression of the outcome on the covariates in each arm and averages the two predictions over the target group. The doubly robust AIPW estimator (Robins, Rotnitzky & Zhao 1994) adds the inverse-probability-weighted residuals to those predictions, so it is consistent if either model is right. Standard errors for all three stack the logistic model's, the outcome models' and the means' estimating equations and use the sandwich variance (Lunceford & Davidian 2004), so they include the noise of the estimated score.
Matching pairs each treated unit (ATT) or control (ATC) with the nearest unit of the other arm on the logit of the score, within a caliper of 0.2 standard deviations of the logit (Austin 2011). With replacement, the default, a unit can be used more than once and the standard error is Abadie & Imbens's (2006); without replacement it is greedy, hardest-to-match first, with the paired standard error. Balance is the standardised mean difference of each covariate, the difference in means over the pooled standard deviation before adjustment, with |SMD| > 0.1 flagged (Austin 2009). Trimming drops units with scores outside [0.1, 0.9] (Crump, Hotz, Imbens & Mitnik 2009) or a range you choose, then refits everything.
Verified by tests in the source against statsmodels 0.15.0's TreatmentEffect (run only as an external program to record reference numbers): on three covariate specifications of the example data, the IPW and regression adjustment estimates for the ATE, ATT and ATC and the AIPW estimate of the ATE, with their potential-outcome means and standard errors, agree within 1e-6, and the logistic coefficients within 1e-12. On 1,000 simulated samples of 2,000 from the example's design, the 95% intervals covered the true effect in 92.5% (IPW), 94.3% (regression adjustment) and 94.0% (AIPW) of samples for the ATE, and 93.4%, 94.6% and 95.2% (matching with replacement) for the ATT; greedy matching without replacement was biased by −0.53 and covered only 54.0%, which is why it is not the default.
Limits
- No unmeasured confounding. Every estimate here assumes the covariates include every common cause of treatment and outcome. Balance on measured covariates says nothing about unmeasured ones; a sensitivity analysis is the next step for a serious claim.
- Pre-treatment covariates only. A covariate the treatment itself changed (a mediator) biases the estimate; leave it out.
- Main-effects models. The propensity and outcome models are linear in the columns you give. Add squares or interactions as new columns if balance is poor.
- Matching's standard error ignores the estimated score (Abadie & Imbens 2016 show this can matter), and matching is 1:1 with no bias correction.
- Binary treatment, one time point. No continuous doses, no time-varying treatments, no clustered standard errors.
BibTeX
@article{rosenbaum1983central,
author = {Rosenbaum, Paul R. and Rubin, Donald B.},
title = {The Central Role of the Propensity Score in Observational Studies for Causal Effects},
journal = {Biometrika},
volume = {70}, number = {1}, pages = {41--55}, year = {1983},
doi = {10.1093/biomet/70.1.41}
}
@article{lunceford2004stratification,
author = {Lunceford, Jared K. and Davidian, Marie},
title = {Stratification and Weighting via the Propensity Score in Estimation of Causal Treatment Effects: A Comparative Study},
journal = {Statistics in Medicine},
volume = {23}, number = {19}, pages = {2937--2960}, year = {2004},
doi = {10.1002/sim.1903}
}
@article{robins1994estimation,
author = {Robins, James M. and Rotnitzky, Andrea and Zhao, Lue Ping},
title = {Estimation of Regression Coefficients When Some Regressors Are Not Always Observed},
journal = {Journal of the American Statistical Association},
volume = {89}, number = {427}, pages = {846--866}, year = {1994},
doi = {10.1080/01621459.1994.10476818}
}
@article{abadie2006large,
author = {Abadie, Alberto and Imbens, Guido W.},
title = {Large Sample Properties of Matching Estimators for Average Treatment Effects},
journal = {Econometrica},
volume = {74}, number = {1}, pages = {235--267}, year = {2006},
doi = {10.1111/j.1468-0262.2006.00655.x}
}
@article{austin2011optimal,
author = {Austin, Peter C.},
title = {Optimal Caliper Widths for Propensity-Score Matching When Estimating Differences in Means and Differences in Proportions in Observational Studies},
journal = {Pharmaceutical Statistics},
volume = {10}, number = {2}, pages = {150--161}, year = {2011},
doi = {10.1002/pst.433}
}
@article{austin2009balance,
author = {Austin, Peter C.},
title = {Balance Diagnostics for Comparing the Distribution of Baseline Covariates Between Treatment Groups in Propensity-Score Matched Samples},
journal = {Statistics in Medicine},
volume = {28}, number = {25}, pages = {3083--3107}, year = {2009},
doi = {10.1002/sim.3697}
}
@article{crump2009dealing,
author = {Crump, Richard K. and Hotz, V. Joseph and Imbens, Guido W. and Mitnik, Oscar A.},
title = {Dealing with Limited Overlap in Estimation of Average Treatment Effects},
journal = {Biometrika},
volume = {96}, number = {1}, pages = {187--199}, year = {2009},
doi = {10.1093/biomet/asn055}
}
Example data: 2,000 people simulated by eval/simulate.py, not a real study. Younger people, people with less schooling, the recently unemployed and low earners were more likely to join, and the same things drive later earnings, so the raw difference in means is −3.6 thousand dollars while the true sample average effect is +1.33 (+1.64 for those who joined). The effect is larger for the unemployed and for low earners, which is why the ATT exceeds the ATE.
Questions
What does a propensity score analysis answer?
What a treatment did when it was not randomly assigned: a training programme people chose to join, a drug some doctors prescribed, a feature some customers switched on. Treated and untreated units differ in ways that also drive the outcome, so the raw difference in means mixes the effect with those differences. The propensity score is each unit's estimated chance of being treated given its measured covariates; weighting or matching on it compares like with like on those covariates.
What is the difference between the ATE, the ATT and the ATC?
The ATE (average treatment effect) is the effect averaged over everyone in the data. The ATT is the effect for the units that were actually treated, which is the usual question for a programme people opted into. The ATC is what the treatment would have done for the units that were not treated. When the effect differs by person and the treated differ from the controls, the three differ, and the example shows that.
Which estimator should I report?
When they roughly agree, any of them, and the agreement is itself evidence. Inverse probability weighting relies on the propensity model, regression adjustment on the outcome model, and the doubly robust AIPW estimator is consistent if either one is right, so it is the usual headline for the ATE. Matching with replacement is easy to explain and makes few modelling assumptions about the outcome. If they disagree, the answer depends on modelling choices, and you should say so.
How do I know the propensity model is good enough?
By balance, not by how well it predicts treatment. After weighting or matching, each covariate's standardised mean difference between the arms should be small, under 0.1 by the usual rule of thumb (Austin 2009). The page shows the difference before and after for every covariate. If one stays imbalanced, add a square or an interaction of it to the data as a new column and run again.
What does poor overlap mean?
Some units have an estimated propensity score near 0 or 1: they look like nobody in the other arm. Their weights are huge and the effect for them is extrapolated. The histogram shows where the two arms' scores sit. Trimming to scores between 0.1 and 0.9 (Crump, Hotz, Imbens and Mitnik 2009) drops those units, which makes the estimate steadier but changes the population it describes.
Does it match statsmodels or Stata?
On three test specifications of the example data, the inverse probability weighting, regression adjustment and AIPW estimates and standard errors for the ATE, ATT and ATC agree with statsmodels 0.15.0's TreatmentEffect within 1e-6, and the logistic coefficients within 1e-12. statsmodels documents that its standard errors match Stata's teffects. Matching is checked by a simulation study instead.
Is my data uploaded anywhere?
No. The file is read and analysed by JavaScript in this browser tab. There are no accounts, cookies or analytics vendors.