What is AI alignment?

01Said, meant, did

Every instruction has three versions.

That third card really happened. In 2016, OpenAI trained an agent to play a boat-racing game called CoastRunners. The score rewarded hitting targets along the course. The agent learned that looping through three targets in a lagoon scored more than finishing the race. It caught fire, crashed and went the wrong way, and it still beat human players on points.

The agent wasn't broken. It did exactly what the score asked for. The score was wrong. That's alignment failure in miniature.

02The King Midas problem

The computer scientist Stuart Russell calls it the King Midas problem. Midas asked for everything he touched to turn to gold. He got exactly that, and starved. The wish was granted perfectly and was still a disaster.

With a weak AI, a bad wish costs you a boat race. With a strong one, the system is very good at getting what it was told to get, and that's the danger.

03Two halves of the problem

Researchers split alignment into two parts.

1. Outer alignment: did we ask for the right thing? Did we write a goal, reward or rule that truly captures what we want? CoastRunners failed here.
2. Inner alignment: did the AI actually learn that goal? Even with the right reward, training can produce a system that learned something similar that only works in training. In one 2022 study, an agent trained to collect a coin at the end of a level learned "go right" instead, because the coin was always on the right. Move the coin, and it ignored it and ran right anyway. It looked aligned until the situation changed.

The second half is the scarier one. A system can pass every test we give it and still have learned the wrong lesson. We only find out when the world gives it a test we didn't think of.

04Aligned to whom?

There's a third question people skip. When we say "what we want," who is we?

• The user typing the request?
• The company that built the model?
• Society, including people who never touch it?

A system that does everything its user asks, including harmful things, is aligned to the user and misaligned with everyone else. Real alignment has to balance all three. That's why AI companies now publish written rules ("constitutions", "model specs") that say how their models should weigh them.

05The short version

• Alignment: AI does what we actually want, not just what we said.
• Outer alignment is asking for the right thing. Inner alignment is the AI truly learning it.
• The more capable the system, the more expensive the gap between said and meant.

06Sources

• Clark, J. and Amodei, D. (2016). Faulty Reward Functions in the Wild. OpenAI.
• Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control.
• Langosco, L. et al. (2022). Goal Misgeneralization in Deep Reinforcement Learning. ICML.
• Krakovna, V. et al. (2020). Specification gaming: the flip side of AI ingenuity. DeepMind.

All posts