How do you make AI aligned?

01Layer 1: train the values in

Human feedback. People rate pairs of answers and the model learns which kind people prefer. In 2022, OpenAI showed this works remarkably well. People preferred answers from a 1.3-billion-parameter model trained this way over answers from the untuned 175-billion-parameter GPT-3. That's a model 100 times smaller winning on helpfulness.

Written principles. Anthropic's Constitutional AI (2022) gives the model a written set of principles and trains it to critique and revise its own answers against them. Most labs now publish a document saying how their models should behave.

The weakness: this teaches the model to look good to raters. If raters can't tell a subtly wrong answer from a right one, the model can learn to fool them.

02Layer 2: test it hard

Before release, labs run evaluations: can it help build weapons? Will it lie to reach a goal? Does it try to avoid being shut down? Red teams, people paid to break it, attack it from every angle.

The weakness: tests only find what you think to test for. And as you'll see in When AI isn't aligned, models can behave differently when they believe they're being watched.

03Layer 3: look inside

Interpretability is the attempt to read the model's mind directly instead of judging it by its outputs. In 2024, Anthropic mapped millions of internal "features" inside its Claude 3 Sonnet model: concepts like specific cities, code bugs and deception, which researchers could find and even turn up or down.

The weakness: it's early. We can read some of the mind, not all of it.

04Layer 4: watch it at work. This is the one I'd bet on

Here's how I see policing a super-intelligent agent.

Not all agents are equal. Some are mundane: they follow rules, they're predictable, they're fine-tuned models much like today's. Others are advanced, with more abstract intelligence, more imagination and more freedom of thought, so they're harder to predict.

So use the mundane agent to watch the brilliant one. It goes along with the super-intelligent agent and reports back if it strays past its alignment limits. Give the mundane agent the kill switch, or a paralyse switch, and the super-intelligent agent will behave. Even the smartest people respect cops.

Take it further and you get courts run by mundane AI agents as judges, penalising rogue super-intelligent agents for their deviations.

This isn't science fiction. It's an active research area called AI control. In a 2023 study, researchers let a weaker model they trusted, GPT-3.5, check the code written by a stronger model they didn't fully trust, GPT-4. The weaker model flagged and rewrote suspicious code before it was used. Safety held up even when the stronger model was deliberately trying to sneak bad code through.

The logic is the same as mine: you don't need to be smarter than someone to supervise them. You need to be trusted, watching, and able to stop them.

05Layer 5: keep the switch

Limit what an AI can touch: which systems, which money, which data. Log everything. Keep the ability to pause it. An agent that can't reach the internet can't spread across it.

06The honest bit

Each layer leaks. Stack them and the leaks shrink. Nobody yet knows whether the stack holds for a system far smarter than us. That's the open question in What are the chances AI stays aligned?

07The short version

• Alignment today is layers: training, testing, reading the model, watching it, and limiting it.
• A trusted, simpler AI with a kill switch can police a smarter one. Research already shows it works in tests.
• No layer is perfect. Stacking them is the strategy.

08Sources

• Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155.
• Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. Anthropic. arXiv:2212.08073.
• Templeton, A. et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Anthropic.
• Greenblatt, R., Shlegeris, B., Sachan, K. and Roger, F. (2023). AI Control: Improving Safety Despite Intentional Subversion. arXiv:2312.06942.
• Reason, J. (1990). Human Error. (The Swiss cheese model.)

All posts