The Joke of Simpson's Paradox (rude)

2026-03-18

Contents
  • What Even Is It, Homie?
  • Trend Reversal After Aggregation
  • Correlation Reversal for Subgroups
  • Non-Random Assignment as Confounder - A Case Study
  • Simulations

What Even Is It, Homie?

Here are some things that can happen:

Trend Reversal After Aggregation

Correlation Reversal for Subgroups

Non-Random Assignment as Confounder - A Case Study

An example that shows this in action is the classic kidney stone study (Charig et al., 1986). Working for the NHS, the authors were interested in comparing efficacy and cost of two different treatments for kidney stones - Treatment A and Treatment B, the details aren’t important.

%%{init: {'flowchart': {'curve': 'stepBefore'}}}%%
flowchart TD
    classDef default fill:#ffffff,stroke:#000000,color:#000000

    Data["700 Patients<br/>NON-RANDOMLY Assigned to Treatment A or B"]
    Data --> Split

    Split{"Analyze by<br/>stone size?"}
    Split --> Small
    Split --> Large
    Split --> All

    subgraph DISAGGREGATED
    Small["Small Stones<br/>357 patients"]
    Large["Large Stones<br/>343 patients"]

    Small --> SA["Treatment A<br/>81 / 87 = 93%"]
    Small --> SB["Treatment B<br/>234 / 270 = 87%"]
    SA & SB --> SW["A wins"]

    Large --> LA["Treatment A<br/>192 / 263 = 73%"]
    Large --> LB["Treatment B<br/>55 / 80 = 69%"]
    LA & LB --> LW["A wins"]
    end

    subgraph AGGREGATED
    All["All Patients<br/>700 patients"]
    All --> AA["Treatment A<br/>273 / 350 = 78%"]
    All --> AB["Treatment B<br/>289 / 350 = 83%"]
    AA & AB --> AW["B wins"]
    end

    style DISAGGREGATED fill:#ffffff,stroke:#000000
    style AGGREGATED fill:#ffffff,stroke:#000000
    style SW fill:#000000,color:#ffffff
    style LW fill:#000000,color:#ffffff
    style AW fill:#000000,color:#ffffff

As we can see, there is something strange going on - when we split by stone size on the “disaggregated” side, Treatment A seems superior for both splits. However, in the overall population it appears that Treatment B is superior.

The problem is that the size of the stone is a “confounder”. Treatment A was given much more frequently to those with large kidney stones than to those with small kidney stones - but the baseline recovery rate for those with large kidney stones is just generally lower! This allows for the statistics for Treatment A, when aggregated, to look worse.

The diagram of influence looks something like this:

flowchart LR
    classDef default fill:#ffffff,stroke:#000000,color:#000000

    C["Stone Size"]
    T["Treatment<br/>Assignment"]
    O["Recovery Rate"]

    C --> L1["biases assignment"]
    L1 --> T
    C --> L2["affects baseline"]
    L2 --> O
    T --> L3["what we care about"]
    L3 --> O

    style L1 fill:none,stroke:none,color:#666666
    style L2 fill:none,stroke:none,color:#666666
    style L3 fill:none,stroke:none,color:#666666

Simulations