As‑If Random: What We Mean, What We Don’t

Econometrics keeps coming back to a simple constraint: we only ever see one path. We do not get to watch the same people both take a programme and not take it, nor the same firm both adopt a technology and refrain from adopting it. In experiments we sidestep this by letting chance decide who gets treatment. In observational work we do not have that luxury, so we try to recreate the consequences of random assignment. That attempt is what people mean when they say “as‑if random”.

“As‑if random” is not a compliment about how neat the dataset looks like. It is a claim about a mechanism. The claim says: once we condition on a set of genuinely pre‑treatment features, the remaining differences in who received treatment behave as if they were generated by chance, for the purpose of the comparison we want to make. If that claim is credible, we can read a treated‑versus‑untreated difference as a causal effect subject to the usual caveats about who is in the data and what comparison we are making.

This is the same theme that ran through the recent posts on selection bias and on the meaning of 95% confidence intervals. In the selection‑bias article I argued that who gets into your dataset matters more than how many observations you have. If chance did not choose them, your statistics do not mean what you think they mean. In the confidence‑interval post I argued that “95%” is a promise about a procedure under a mechanism, not a probability about a specific interval. The “as‑if” assumption simply tries to recover a mechanism that is missing in observational work, so that the procedure you use (a regression, a weighted difference) can inherit a familiar interpretation.

Let us start from an example. A city launches a training programme. Applicants decide whether to take it; there is no lottery. If we compare wages after the programme, trainees and non‑trainees will typically differ in ways that also move wages such motivation, experience, job networks, firm quality. A raw difference in means mixes the programme with all of that. The “as‑if” move is to say: once I condition on the right pre‑treatment information such as prior earnings, age, sector, tenure, education, local unemployment, the remaining variation in who trained and who did not is essentially random with respect to the unmeasured influences on wages. Under that claim the treated‑vs‑untreated contrast behaves like a fair assignment for the purposes of this comparison.

Note what is being asserted. It is not that the two groups look similar on a few tables. It is not that the regression has a long list of controls. It is not that the estimate survived three robustness checks. It is a statement about how treatment varies after you have used the information that, in your setting, plausibly drives both treatment and outcome. The target is the leftover link between treatment and the unmeasured remainder of the outcome. “As‑if” says that link has been severed.

Why could such a claim be credible? Because in many settings the forces that sort people into treatment are, to a large extent, visible and time‑ordered. Eligibility rules, application thresholds, travel distance, prior performance, age bands, school cohorts, sector and occupation etc. often sit upstream of the decision to take up a programme and also predict outcomes. If you can measure the relevant ones carefully and if treated and untreated overlap on those characteristics, removing their influence can leave behind a residue that is practically speaking close to chance.

Two phrases matter here. The first is pre‑treatment. If a variable is actually an effect of the treatment (or moves in response to the treatment), adjusting for it can wash away part of the very effect you look for. The second is overlap. If all high‑tenure, high‑skill workers take the programme and no one else does, your linear line has to invent counterfactuals it never saw; no amount of adjustment manufactures a fair comparison where none exists. “As‑if” is a promise that lives in the region where like can be compared with like.

Because the phrase is mis‑used so often, it is worth being blunt about what it does not mean. It does not mean “the groups look balanced”. Tables can hide differences that matter; more importantly, balance on observables says nothing about the unobservables that worried you in the first place. It does not mean “many controls”. A long list is not a causal model. If the list includes post‑treatment variables (bad controls) or colliders, you can create bias from nowhere. It does not mean “robust across specifications”. Stability is good, but you can be consistently wrong. And it certainly does not mean “big data”. Larger convenience samples make you precisely wrong when the mechanism is wrong.

“As‑if” is a mechanism claim: after conditioning on a defensible set of pre‑treatment features, the part of treatment that remains unpredictable behaves like luck with respect to the unmeasured drivers of the outcome. If you cannot tell that story credibly in your setting, avoid the phrase.

In applied work one tries to make the “as‑if” claim believable using tools that aim at the same target. Matching and stratification compare people with very similar pre‑treatment profiles and then average those local contrasts. Inverse‑probability weighting re‑weights the sample to align the distribution of pre‑treatment characteristics across groups, so that the treated and untreated populations you contrast look like each other on the drivers you can see. Regression adjustment models the outcome as a function of treatment and pre‑treatment variables, interpreting the treatment coefficient as the remaining difference once those variables are accounted for. Combinations of these approaches—so‑called “doubly robust” estimators—hedge against getting one of the ingredients wrong.

Each of these methods is trying to remove the predictable part of treatment selection, in the hope that what remains is close enough to chance. None of them creates chance where none operated. They are only as good as the information you bring to them, the design discipline you apply (pre‑treatment only, genuine overlap), and the story that makes them plausible in your context.

“As‑if random” is not a ritual that works the same way everywhere; it is a model‑guided choice. Even without equations, you are committing yourself to a view of the world: which variables sit upstream of treatment and outcome; which do not; how relationships bend or interact; where overlap is real and where it is absent. If you force linearity where the world is curved, the structure you leave out slips into the “unexplained” part of the outcome, and if that structure is related to treatment you recreate the very link you were trying to break. If you adjust for consequences of treatment, you dilute the effect by design. If you extrapolate into regions with no overlap, you are inventing comparisons you never observed.

The safest habit is design first, model second. Start by writing down, in prose, why treatment decisions in your setting can be made to look like chance once you control for specific, pre‑treatment features. Check that those features really are pre‑treatment and that treated and untreated units overlap on them. Only then ask a model to do the tidying (flexible forms, interactions, weighting) without asking it to perform miracles.

This part completes the triangle with the two earlier themes. Selection bias taught us that convenience corrodes meaning: if chance didn’t choose them, your statistics don’t mean what you think they mean. The 95% post taught us that coverage belongs to a procedure under a mechanism. “As‑if random” is the bridge that lets observational work inherit those interpretations: give me a believable mechanism for the comparison—constructed out of pre‑treatment information and overlap—and I will treat the remaining variation as luck for the purpose of this contrast.