How not to miss what’s missing: Survivorship bias in decision making

Survivorship bias is the human tendency of concentrating on the sample that survived (made it past some selection process) and overlooking those that did not . If failures are not considered, then naturally you pay more attention to success.

Not only do you fail to recognise that what is missing might have been important, but also you fail to recognise there is any missing information at all.

• We tend to only consider information that’s presented to us and ignore absent information that may be very relevant.
• Focusing on one side of the equation while neglecting the other can distort thinking and decision making process.
• This bias frequently arises in many contexts; once you’re familiar with it, you’ll be primed to notice it wherever it’s hiding.

Like health and longevity advice. We look to old people on guidance for living a long life when we should really examine those who died early to learn what to avoid.

104-year-old woman’s secret: 3 Dr. Peppers a day:

“This stuff is good.. it’s got sugar in it and 2 doctors have told me if I drink it I will die, but they died first”

In the video the great grandmother of 13, attributes some part of her longevity to Dr. Pepper

An Example from World War II

The most famous example of application of survivorship bias was During World War II,

The statistician Abraham Wald took survivorship bias into his calculations when considering how to minimize bomber losses to enemy fire.

The problem: You don’t want your planes to get shot down by enemy fighters, so you armour them. But armour makes the plane heavier, and heavier planes are less manoeuvrable and consume more fuel.

Armouring the planes too much is a problem; armouring the planes too little is a problem. The problem was to add optimal armour.

The sample: When American planes came back from engagements over Europe, they were covered in bullet holes. But the damage wasn’t uniformly distributed across the aircraft.

There were more bullet holes in the fuselage, not so many in the engines. There was an opportunity for efficiency;

You could get the same protection with less armour if you armour the plane on the areas with the greatest need, where the planes are getting hit the most.

This was the initial thinking of the air force officers.

The missing sample:

However Wald questioned some of the assumptions of the officers: where are the missing holes? The ones that would have been all over the engine casing. If the damage had been spread equally all over the plane?

The missing bullet holes were on the missing planes. The reason planes were coming back with fewer hits to the engine is that planes that got hit in the engine were not coming back.

Whereas the large number of planes returning to base with many bullet holes to the wings and fuselage, this was pretty strong evidence that hits to the fuselage can be tolerated.

Thus Wald advised to armour the places where there were no bullet holes like the engine.

Survivorship bias in business

If you’re calculating annual recurring revenue (ARR) based on your current customer base alone, you may be poised for a harsh reality when your actual revenue turns out to be lower.

While focusing on current customers is key to business planning, survivorship bias encourages us to study the customer churn rate of the same time last year and factor that into our ARR calculations.

This gives us a more accurate picture of what the company can expect to bring in while incorporating churn and avoids disappointments or unexpected cash flow shortages down the line.

But most of us are regularly fooled by the survivorship bias. Consider the plethora of business books that feature the most successful companies and entrepreneurs, but you can never find a book featuring all the failures

David Cowan of Bessemer Venture Partners told Scientific American, “For every wealthy start-up founder, there are 100 other entrepreneurs who end up with only a cluttered garage.”

How to avoid survivorship bias ?

If you don’t want to be misled by survivorship bias you need to look at the whole picture.

To avoid survivorship bias we need to question “What assumptions are we making? And are they justified? Are we looking at the complete data ? Is the sample a good representation of the population ?

Once you’re familiar with it, you’ll be primed to notice it wherever it’s hiding