PROPENSITY SCORES · WEIGHTING · MATCHING · DOUBLY ROBUST · RUNS IN YOUR BROWSER · NOTHING UPLOADED

What did the treatment do, once you account for who chose it?

People who join a training programme, patients a doctor puts on a drug, customers who switch on a feature: they differ from the rest in ways that also drive the outcome, so the raw difference in means mixes the effect with who signed up. Upload one row per unit with a 0/1 treatment, the outcome and the things measured before treatment that drove both.

You get four estimators side by side: inverse probability weighting, regression adjustment, the doubly robust AIPW estimate and nearest-neighbour matching, for the effect on everyone (ATE), on the treated (ATT) and on the controls (ATC), with standard errors that count the estimated score. And the checks a reviewer asks for: a covariate balance table and love plot, an overlap histogram, effective sample sizes and optional trimming.

Nothing you enter leaves this browser tab. Statistical output from data you supply, not advice. Not legal, tax, or financial advice, and not medical or policy advice.

Your data

Paste data or load the example.

Method and citation

The propensity score e(x) is the probability of treatment given the covariates (Rosenbaum & Rubin 1983). The tool estimates it by a logistic regression of the treatment on the covariates you list (maximum likelihood by Newton-Raphson). If treatment depends only on those covariates (no unmeasured confounding) and every unit had some chance of either arm (overlap), comparing units with the same score compares like with like.

Inverse probability weighting gives each treated unit weight 1/e and each control 1/(1−e) for the ATE; for the ATT treated units weigh 1 and controls e/(1−e); for the ATC treated units weigh (1−e)/e and controls 1. Each arm's weights are normalised to sum to one. Regression adjustment fits a linear regression of the outcome on the covariates in each arm and averages the two predictions over the target group. The doubly robust AIPW estimator (Robins, Rotnitzky & Zhao 1994) adds the inverse-probability-weighted residuals to those predictions, so it is consistent if either model is right. Standard errors for all three stack the logistic model's, the outcome models' and the means' estimating equations and use the sandwich variance (Lunceford & Davidian 2004), so they include the noise of the estimated score.

Matching pairs each treated unit (ATT) or control (ATC) with the nearest unit of the other arm on the logit of the score, within a caliper of 0.2 standard deviations of the logit (Austin 2011). With replacement, the default, a unit can be used more than once and the standard error is Abadie & Imbens's (2006); without replacement it is greedy, hardest-to-match first, with the paired standard error. Balance is the standardised mean difference of each covariate, the difference in means over the pooled standard deviation before adjustment, with |SMD| > 0.1 flagged (Austin 2009). Trimming drops units with scores outside [0.1, 0.9] (Crump, Hotz, Imbens & Mitnik 2009) or a range you choose, then refits everything.

Verified by tests in the source against statsmodels 0.15.0's TreatmentEffect (run only as an external program to record reference numbers): on three covariate specifications of the example data, the IPW and regression adjustment estimates for the ATE, ATT and ATC and the AIPW estimate of the ATE, with their potential-outcome means and standard errors, agree within 1e-6, and the logistic coefficients within 1e-12. On 1,000 simulated samples of 2,000 from the example's design, the 95% intervals covered the true effect in 92.5% (IPW), 94.3% (regression adjustment) and 94.0% (AIPW) of samples for the ATE, and 93.4%, 94.6% and 95.2% (matching with replacement) for the ATT; greedy matching without replacement was biased by −0.53 and covered only 54.0%, which is why it is not the default.

Limits

BibTeX

@article{rosenbaum1983central,
  author  = {Rosenbaum, Paul R. and Rubin, Donald B.},
  title   = {The Central Role of the Propensity Score in Observational Studies for Causal Effects},
  journal = {Biometrika},
  volume  = {70}, number = {1}, pages = {41--55}, year = {1983},
  doi     = {10.1093/biomet/70.1.41}
}
@article{lunceford2004stratification,
  author  = {Lunceford, Jared K. and Davidian, Marie},
  title   = {Stratification and Weighting via the Propensity Score in Estimation of Causal Treatment Effects: A Comparative Study},
  journal = {Statistics in Medicine},
  volume  = {23}, number = {19}, pages = {2937--2960}, year = {2004},
  doi     = {10.1002/sim.1903}
}
@article{robins1994estimation,
  author  = {Robins, James M. and Rotnitzky, Andrea and Zhao, Lue Ping},
  title   = {Estimation of Regression Coefficients When Some Regressors Are Not Always Observed},
  journal = {Journal of the American Statistical Association},
  volume  = {89}, number = {427}, pages = {846--866}, year = {1994},
  doi     = {10.1080/01621459.1994.10476818}
}
@article{abadie2006large,
  author  = {Abadie, Alberto and Imbens, Guido W.},
  title   = {Large Sample Properties of Matching Estimators for Average Treatment Effects},
  journal = {Econometrica},
  volume  = {74}, number = {1}, pages = {235--267}, year = {2006},
  doi     = {10.1111/j.1468-0262.2006.00655.x}
}
@article{austin2011optimal,
  author  = {Austin, Peter C.},
  title   = {Optimal Caliper Widths for Propensity-Score Matching When Estimating Differences in Means and Differences in Proportions in Observational Studies},
  journal = {Pharmaceutical Statistics},
  volume  = {10}, number = {2}, pages = {150--161}, year = {2011},
  doi     = {10.1002/pst.433}
}
@article{austin2009balance,
  author  = {Austin, Peter C.},
  title   = {Balance Diagnostics for Comparing the Distribution of Baseline Covariates Between Treatment Groups in Propensity-Score Matched Samples},
  journal = {Statistics in Medicine},
  volume  = {28}, number = {25}, pages = {3083--3107}, year = {2009},
  doi     = {10.1002/sim.3697}
}
@article{crump2009dealing,
  author  = {Crump, Richard K. and Hotz, V. Joseph and Imbens, Guido W. and Mitnik, Oscar A.},
  title   = {Dealing with Limited Overlap in Estimation of Average Treatment Effects},
  journal = {Biometrika},
  volume  = {96}, number = {1}, pages = {187--199}, year = {2009},
  doi     = {10.1093/biomet/asn055}
}

Example data: 2,000 people simulated by eval/simulate.py, not a real study. Younger people, people with less schooling, the recently unemployed and low earners were more likely to join, and the same things drive later earnings, so the raw difference in means is −3.6 thousand dollars while the true sample average effect is +1.33 (+1.64 for those who joined). The effect is larger for the unemployed and for low earners, which is why the ATT exceeds the ATE.

Questions

What does a propensity score analysis answer?

What a treatment did when it was not randomly assigned: a training programme people chose to join, a drug some doctors prescribed, a feature some customers switched on. Treated and untreated units differ in ways that also drive the outcome, so the raw difference in means mixes the effect with those differences. The propensity score is each unit's estimated chance of being treated given its measured covariates; weighting or matching on it compares like with like on those covariates.

What is the difference between the ATE, the ATT and the ATC?

The ATE (average treatment effect) is the effect averaged over everyone in the data. The ATT is the effect for the units that were actually treated, which is the usual question for a programme people opted into. The ATC is what the treatment would have done for the units that were not treated. When the effect differs by person and the treated differ from the controls, the three differ, and the example shows that.

Which estimator should I report?

When they roughly agree, any of them, and the agreement is itself evidence. Inverse probability weighting relies on the propensity model, regression adjustment on the outcome model, and the doubly robust AIPW estimator is consistent if either one is right, so it is the usual headline for the ATE. Matching with replacement is easy to explain and makes few modelling assumptions about the outcome. If they disagree, the answer depends on modelling choices, and you should say so.

How do I know the propensity model is good enough?

By balance, not by how well it predicts treatment. After weighting or matching, each covariate's standardised mean difference between the arms should be small, under 0.1 by the usual rule of thumb (Austin 2009). The page shows the difference before and after for every covariate. If one stays imbalanced, add a square or an interaction of it to the data as a new column and run again.

What does poor overlap mean?

Some units have an estimated propensity score near 0 or 1: they look like nobody in the other arm. Their weights are huge and the effect for them is extrapolated. The histogram shows where the two arms' scores sit. Trimming to scores between 0.1 and 0.9 (Crump, Hotz, Imbens and Mitnik 2009) drops those units, which makes the estimate steadier but changes the population it describes.

Does it match statsmodels or Stata?

On three test specifications of the example data, the inverse probability weighting, regression adjustment and AIPW estimates and standard errors for the ATE, ATT and ATC agree with statsmodels 0.15.0's TreatmentEffect within 1e-6, and the logistic coefficients within 1e-12. statsmodels documents that its standard errors match Stata's teffects. Matching is checked by a simulation study instead.

Is my data uploaded anywhere?

No. The file is read and analysed by JavaScript in this browser tab. There are no accounts, cookies or analytics vendors.