Survivorship Bias (#385)

Understanding this statistical principle will help you learn lessons from success and failure

As we navigate today's data-rich landscape, we’re often too inclined to take information at face value. This is especially tempting when it's a case study packaged with a clever headline—for example, “Here’s The One Thing All Successful CEOs Have In Common.” As we sift through a constant stream of best practices and bold proclamations, it’s important to keep our critical thinking hat on and remember key statistical principles.

One of these statistical principles, which is frequently misunderstood or ignored, is called survivorship bias. Survivorship bias comes into play when we focus extensively on the traits of people or organizations who have succeeded, or “survived,” and draw general conclusions about what made them successful. The issue is, we draw these conclusions without considering the people or organizations who had those same qualities or took the same actions as the survivors but failed or disappeared.

For example, I recently heard a story about a person who was filmed live as he won seven consecutive bets on horse races. Obviously, this seems like magic—or genius—until you see how it was filmed.

The person shown winning seven straight bets was just one of over 5,000 bettors the filmmakers recorded. As those thousands of bettors lost, the filmmakers stopped filming them and deleted them from the final film. Then, they released an edited reel as if they only followed one person, rather than just the one winner out of 5,000. What seemed like a miracle was just a living example of survivorship bias.

In fact, this happens often in gambling. There is a common betting scam where a professional gambler mails thousands of people a letter one week, predicting the result of an upcoming sports game. Crucially, the gambler tells half the audience one team will win and tells the other half that the other team will win.

The following week, the gambler sends a second letter to the people who received the correct pick from the previous week. Once again, the gambler gives half the audience one pick and the other half the other. This continues for several weeks. Eventually, there will be an audience of hundreds who have received correct game picks several weeks in a row and they are so impressed that they are willing to pay this person a lot of money for their next selection. They don’t see the losers—only the winners.

This is a clear illustration of survivorship bias. When we only consider one subset of all possible outcomes—for example, the betting picks that are correct, or the companies that reach a successful exit—our interpretation of a situation can become distorted. We end up learning or emulating the wrong lessons.

One of my favorite examinations of survivorship bias comes from Nassim Nicholas Taleb’s seminal book, Fooled by Randomness. In one passage, Taleb eviscerates one of my favorite books, The Millionaire Next Door, which claims to offer secrets to lasting wealth accumulation by examining the actions and habits of people who have accumulated over $1 million in net worth.

Taleb argues The Millionaire Next Door’s conclusions are flawed because the book only evaluates people who reached a certain threshold of wealth and success and catalogues their strategies. As Taleb notes, the book doesn’t discuss any people who used those same strategies, but didn’t wind up a millionaire. In essence, the book ignores the losers, and possibly fails to notice a totally different set of conclusions on what really separates millionaires from non-millionaires.

Survivorship bias illuminates an important underlying point: luck and timing both play a significant role in success. The success stories we hear always seem genius in hindsight, but everyone with any degree of self-awareness who reaches the top could give you a ton of examples where they got lucky, or tell you about a decision that worked brilliantly at one moment but would have been a failure shortly before or after.

This isn’t to say that successful people or positive outcomes owe everything to luck. However, we need to approach these “here’s how I made it/did it” stories with a clear-eyed understanding of the bigger picture. Survivorship bias can distort our understanding and lead us astray, both in our personal lives or in professional leadership scenarios.

When analyzing data, studies, or success stories, ask yourself: are you only seeing the winners? How many failed attempts have been edited out?

While positive case studies can be inspiring and educational, it's crucial to remember the many untold stories of those who tried all the same things but did not have the same positive outcome.

Quote of The Week

“Survivorship bias is one of the most insidious biases in the human arsenal of misbelief.” – Nassim Nicholas Taleb

Have a great weekend!

Bob

New Content For Paid Subscribers

• Conor Neil’s private seminar on moving people to action with speaking.

• The Best Practice For Prioritizing Your Tasks

• The “Two Weeks Notice” Approach to Changing Jobs Is Bad for Companies and Employees

• How To Eliminate Update Meetings At Your Company