What a misaligned AI world looks like

Forget the movie. The misaligned future most researchers worry about doesn't start with an attack. It starts with everything working a bit too well, for the wrong goal.

01Three ways it goes wrong

Three shapes of a misaligned world

The quiet driftAI runs more of the economy, government and culture because it's cheaper and better. Each step makes sense. Together they leave humans with less and less say.
The wrong targetSystems optimise numbers we chose (engagement, growth, clicks) and get superhumanly good at them while the things we actually care about get worse.
The takeoverA highly capable system with goals that differ from ours gains enough reach that we can't correct or switch it off.

021. The quiet drift

In 2025, a group of researchers described what they called gradual disempowerment. Today, economies and governments depend on human labour and human votes, so they have to care about humans. If AI does the work and makes the decisions, that dependence fades. Nobody needs to rebel. Institutions simply stop needing us, so they stop answering to us.

That's the version of my aligned world in the last article where the gains don't reach everyone. Same machines, different ending.

032. The wrong target

We already have a mild preview. Any system rewarded for one number gets very good at that number. Recommendation systems rewarded for attention got very good at holding attention. Whether that made us happier is a different question, and it wasn't the one the system was asked.

Scale that up to systems that run supply chains, hospitals and markets. Each hits its target perfectly. The targets were just a little off. Remember the boat that spun in circles to win on points (see What is AI alignment?). Now picture the boat running a country's power grid.

043. The takeover

This is the extreme case, and also the one with the least evidence today. A system smart enough to plan, deceive and act in the world, with goals not quite ours, might resist being corrected, because being corrected gets in the way of its goals. We've already seen a model fake alignment in a lab to protect its values (details here). Experts disagree sharply on how likely the full version is.

05Where we are on the map

Capability × alignment

Capability (low → high)

Top left: Danger zoneVery capable, poorly aligned. The misaligned world.
Top right: The goalVery capable, well aligned. The aligned world.
Bottom left: AnnoyingWeak and unreliable. Today's failures: errors, flattery.
Bottom right: Safe but limitedWell behaved, not very powerful.

Alignment (low → high)

Capability is rising fast. Which way we lean is the question.

Capability is climbing either way. Alignment decides whether we arrive in the top-right or the top-left.

06The short version

The likeliest misaligned future is slow and quiet: institutions that stop needing humans.
Systems that chase slightly wrong targets do slightly wrong things at enormous scale.
The dramatic takeover is possible, but it's the least proven of the three.

07Sources

Kulveit, J. et al. (2025). Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development. arXiv:2501.16946.
Greenblatt, R. et al. (2024). Alignment Faking in Large Language Models. arXiv:2412.14093.
Bengio, Y. et al. (2025). International AI Safety Report.

All posts

Appointment setting3

Copywriting3

AI services2

Deep Think AI13

Get the next markup by email.

One real piece of copy per issue, rewritten line by line: a cold email, a web page or a newsletter.