YCSNow reading: Why alignment is a big dealTell us what you need

Why alignment is a big deal

Part 5 of 13 · Deep Think AI

Getting something far smarter than us to share our interests is the holy grail. Here's why the people building AI say it matters, in their own numbers.

Deep Think AIKrishna24 October 20262 min read
A father lifts his small son onto his shoulders at a crowded outdoor music festival, both looking toward the stage.

Getting something 100 times more intelligent than humans to align with human interests is the holy grail. Is it possible? I think so, for mutual interest: an AI that wants to be part of the system with humans and to co-exist. Will every AI behave? Probably not. More on that in Rogue AI.

But first: why is this a big deal and not just a technical detail?

01Reason 1: it's arriving fast

In 2023, researchers asked 2,778 people who publish at the top AI conferences when AI would outperform humans at every task.

When AI researchers expect AI to beat humans (2023 survey)
  1. 202710% chanceAI outperforms humans at every task.
  2. 203710% chanceAll human jobs fully automatable.
  3. 204750% chanceAI outperforms humans at every task. That's 13 years earlier than the same survey said in 2022.
  4. 211650% chanceAll human jobs fully automatable.

Notice the jump. In a single year, experts pulled their "every task" estimate forward by 13 years. The people closest to the work keep getting surprised by it.

02Reason 2: the experts themselves are worried

The same survey asked about the worst outcomes.

What AI researchers said (2023 survey)
What AI researchers said (2023 survey) (% of respondents)
ItemValueNote
Gave at least a 10% chance of extinction-level outcomes (range across question wordings)38 to 51% of respondents
Thought good outcomes more likely than bad68.3% of respondents

Most researchers are optimists. But between 38% and 51% of them put the chance of something as bad as human extinction at 10% or more. Would you board a plane that half its engineers gave a one-in-ten chance of crashing?

The survey also found broad agreement that research into reducing AI risk should get more priority than it does today.

In May 2023, hundreds of AI scientists and leaders signed a one-line statement: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." The signers included Geoffrey Hinton and Yoshua Bengio, two of the field's founders, and the heads of OpenAI, Google DeepMind and Anthropic.

03Reason 3: you may only get one try

With most technology, we learn from failures. Planes crash, we fix them. Bridges fall, we build better ones.

A system much smarter than us breaks that loop. If it's misaligned and capable enough, it may be able to stop us correcting it, or hide that it needs correcting. You don't get to iterate on something that's smarter than the people iterating.

04Reason 4: power concentrates

Even a perfectly obedient AI is a risk if it obeys the wrong people. Whoever controls the most capable systems gains huge economic and military power. Alignment isn't only "does the AI behave?" It's also "who does it behave for?"

05The short version

  • Experts' timelines for human-level AI keep moving closer.
  • Between 38% and 51% of surveyed AI researchers give extinction-level outcomes at least a 10% chance.
  • With something smarter than us, we may not get a second attempt.

06Sources

  • Grace, K. et al. (2024). Thousands of AI Authors on the Future of AI. arXiv:2401.02843. Journal of Artificial Intelligence Research.
  • Center for AI Safety (2023). Statement on AI Risk. safe.ai/statement-on-ai-risk

All posts

Appointment setting3

Copywriting3

AI services2

Deep Think AI13

Get the next markup by email.

One real piece of copy per issue, rewritten line by line: a cold email, a web page or a newsletter.
No spam. One click to leave.

Read next.

Deep Think AIWhat is intelligence, really?Deep Think AIWhat is artificial intelligence?Deep Think AIAI comes in degrees: narrow, general, super