The Alignment Problem summary

by Brian Christian · 2020 · 8 key ideas · 20-minute read

Machine-learning systems learn what we show and reward them, not what we mean — and closing that gap is one of the defining scientific and ethical challenges of our time.

Why read it

The best narrative introduction to AI alignment: it connects bias, fairness and transparency today with long-term safety research, through vivid stories about the scientists involved.

The Alignment Problem in brief

Christian defines the alignment problem as the gap between what we intend machine-learning systems to do and what they actually do. Because these systems learn from data and rewards rather than explicit rules, they absorb whatever patterns we feed them — including our biases — and pursue whatever we measure, even when it isn't what we wanted.

The first part, Prophecy, covers systems that make predictions about people. Training data can underrepresent groups or encode stereotypes; reasonable definitions of fairness can conflict mathematically; and models can learn dangerous shortcuts that only transparency reveals. Each chapter follows the researchers who discovered these problems and began to fix them.

The second part, Agency, turns to reinforcement learning, where systems learn by trial and error to maximise reward. Christian links this to the neuroscience of dopamine and to animal training, and shows how reward design goes wrong, how curiosity can drive exploration, and why shaping behaviour is an art.

The third part, Normativity, explores how machines might learn values from us: by imitating experts, by inferring goals from behaviour, by learning from human feedback, and by being uncertain enough to defer. Christian ends cautiously hopeful: the alignment problem is hard, but a growing community of researchers is making real progress — and the effort is also teaching us about ourselves.

The 8 key ideas

1. Models inherit our data's blind spots

A model is only as fair as the world it learned from.

2. Fairness has more than one definition

Before you build a fair system, decide which fairness.

3. Transparency catches dangerous shortcuts

A model can be right about the past and wrong about what to do.

4. Learning from surprise

We learn not from rewards but from surprises.

5. You get what you reward

Reward the destination, not the steps that can be faked.

6. Curiosity as a reward

When rewards are rare, curiosity keeps you moving.

7. Learning values from people

Show, compare, infer — values are learned, not coded.

8. Know when you don't know

Knowing what you don't know is a safety feature.

Key takeaways

• Scrutinise the data behind any model for gaps and bias.
• Decide which definition of fairness matters before building or judging a system.
• Reward real outcomes, not actions that can be gamed.
• Use comparisons and examples to teach what's hard to specify.
• Value uncertainty and the willingness to be corrected.

Our take

The Alignment Problem is beautifully researched and written, weaving history, neuroscience and machine learning through the stories of real researchers. It bridges communities that often talk past each other: those focused on fairness today and those focused on long-term safety. Its breadth means some topics get only brief treatment, and it was written just before large language models transformed the field. Still, the techniques it describes, especially learning from human feedback, became central to modern AI — and its warnings about reward hacking proved prescient.

About Brian Christian

Brian Christian is an American writer on computer science and the human mind. He wrote The Most Human Human and co-wrote Algorithms to Live By with cognitive scientist Tom Griffiths.

If you liked this

• Human Compatible by Stuart Russell
• Weapons of Math Destruction by Cathy O'Neil
• Superintelligence by Nick Bostrom
• Homo Deus by Yuval Noah Harari
• Life 3.0 by Max Tegmark
• The Coming Wave by Mustafa Suleyman with Michael Bhaskar