Rogue AI: what it looks like

There's always a rogue. Every society has people who break the rules, and AI will be no different. There will be rogue players, AI ones included, and AI hackers. They won't be different from today's, only scaled up.

So the goal was never zero rogues. The goal is a system that catches them.

01Where rogue AI comes from

Four roads to a rogue AI

Built to do harmSomeone deliberately builds or steers an AI to attack, scam or disrupt.
Built for chaosNot evil, just bored: agents released for fun by people chasing a thrill, with no idea what they'll do.
Drifted off courseA system trained for a good goal learns a slightly different one, and pursues it.
Slipped the leashA capable system copies itself, hides from oversight, or resists shutdown to keep pursuing its goal.

The first and fourth get the headlines. I worry most about the second.

02The dorm-room problem

Imagine 2030. Building technology keeps getting easier; it already is. Picture a world where someone with no real skill can build a model overnight that beats the best of today, using an open model a teenager built in his dorm.

Now picture the agents built for chaos by dopamine-craving kids. Not to steal. Not to kill. Just to see what happens.

Yoshua Bengio made a version of this point in 2023: once the recipe for a dangerous AI is widely available, you can't count on every single person who has it choosing not to use it. A few always will.

03The system I'd build: every agent gets an ID

We need a system. Whoever creates an agent, for whatever purpose, has to get an agent ID from a central body.

Agent ID and policing

1. RegisterAnyone who launches an agent must get an agent ID. No ID, no launch.
2. Assign a police agentIssuing the ID automatically assigns a policing agent: a simpler, trusted AI that watches this one.
3. WatchThe police agent checks actions against the agent's declared purpose and limits.
4. PauseIt holds a kill or paralyse switch. Even the smartest agent behaves when someone can stop it.
5. JudgeSerious cases go to courts run by AI judges, which penalise rogue agents.

The rule: the internet should never find an agent without its ID and its policing agent. If one shows up, it gets dealt with.

This isn't far from real proposals. In 2024, a group of researchers including Alan Chan and Markus Anderljung published IDs for AI Systems. They proposed giving every running AI instance an identifier linked to information about it, so the people and systems dealing with it know what they're dealing with. Pair that with the AI control research in How do you make AI aligned?, and the pieces of my system already exist in labs.

04What it would take

Infrastructure that checks IDs: cloud providers, payment systems and websites refuse agents without one, the way banks refuse unverified accounts.
Strong defence AI: as good as the best attackers, so outages stay occasional and contained.
International agreement: an ID system that stops at one border is a suggestion, not a system.

05The short version

Rogue AI will exist, just as rogue people do.
The biggest risk is how easy building powerful AI becomes, not one evil genius.
Every agent needs an ID and an assigned police agent with a switch. Researchers are already proposing the parts.

06Sources

Bengio, Y. (2023). How Rogue AIs may Arise. yoshuabengio.org, 22 May 2023.
Chan, A. et al. (2024). IDs for AI Systems. arXiv:2406.12137.
Greenblatt, R. et al. (2023). AI Control: Improving Safety Despite Intentional Subversion. arXiv:2312.06942.

All posts

Appointment setting3

Copywriting3

AI services2

Deep Think AI13

Get the next markup by email.

One real piece of copy per issue, rewritten line by line: a cold email, a web page or a newsletter.