Noise summary

by Daniel Kahneman, Olivier Sibony and Cass R. Sunstein · 2021 · 8 key ideas · 20-minute read

Besides bias, human judgment suffers from noise, unwanted variability between and within judges, and it can be measured and reduced with simple 'decision hygiene'.

Why read it

For anyone who makes or relies on professional judgments: hiring, lending, medicine, law, insurance or performance reviews. It reveals an invisible source of error and practical ways to shrink it.

Noise in brief

When professionals judge the same case, they should reach similar conclusions. The authors show that they often don't. Judges give very different sentences for the same crime; insurance underwriters quote very different premiums for the same risk; doctors diagnose the same patient differently. This unwanted variability is noise, and it's far larger than most organisations imagine.

Noise is different from bias. Bias is a systematic error in one direction; noise is random scatter. Both contribute equally to overall error, but noise is less visible, because you only see it when you compare many judgments of the same thing. That's why organisations rarely measure it.

The authors break noise into types: level noise (some judges are consistently harsher), pattern noise (judges react differently to particular cases) and occasion noise (the same judge varies with mood, time and circumstance). They explain the psychology behind these, including our overconfidence in our own judgment.

A major theme is that simple rules and models often outperform human judgment, not because they're brilliant but because they're consistent. Where human judgment is still needed, the authors propose 'decision hygiene': techniques such as independent assessments, structured evaluations and aggregation that reduce noise without needing to identify its causes.

Finally, they address objections. Reducing noise can be costly, can feel dehumanising to the people being judged and to the professionals doing the judging, and rules can be gamed. The authors argue these costs are often worth paying, but that each organisation must strike its own balance between firm rules and human discretion.

The 8 key ideas

1. Bias and noise are different errors

Bias leans. Noise scatters.

2. Noise counts as much as bias

Error = bias² + noise². Don't ignore half the problem.

3. Run a noise audit

You can't fix noise you've never measured.

4. Level, pattern and occasion noise

Different judges, different days, different verdicts.

5. Ask the crowd, including the one inside you

Independent first, then average.

6. Simple rules often beat experts

Consistency beats brilliance more often than we'd like.

7. Practise decision hygiene

Wash your judgments before you serve them.

8. Weigh the costs of reducing noise

Noise has costs; so do rules. Choose consciously.

Key takeaways

• Measure noise by having several people judge the same case independently.
• Collect independent judgments before discussion, then aggregate.
• Make two separate estimates and average them when deciding alone.
• Break complex evaluations into separate criteria scored independently.
• Use simple rules or scorecards where consistency matters.
• Weigh the costs of rigid rules against the costs of noise.

Our take

Noise shines a light on an under-appreciated source of error, and its practical tools, noise audits, independent judgments and structured evaluations, are valuable in hiring, medicine, law and beyond. Its weaknesses: it's long and repetitive, and some critics argue the concept of noise is old wine in a new bottle, closely related to well-established ideas like reliability. Several vivid examples of occasion noise, such as judges influenced by sports results, come from single studies that may not be robust. The book also says less about when variability is healthy. Still, it makes a convincing case that organisations should measure how consistently they judge.

About Daniel Kahneman, Olivier Sibony and Cass R. Sunstein

Daniel Kahneman was a Nobel-winning psychologist and author of Thinking, Fast and Slow. Olivier Sibony is a professor at HEC Paris and a former McKinsey partner; Cass Sunstein is a Harvard Law School professor and co-author of Nudge.

If you liked this

• Thinking, Fast and Slow by Daniel Kahneman
• Nudge by Richard H. Thaler and Cass R. Sunstein
• Predictably Irrational by Dan Ariely
• The Happiness Hypothesis by Jonathan Haidt
• Flow by Mihaly Csikszentmihalyi
• Emotional Intelligence by Daniel Goleman