What a superintelligence is
The most quoted definition comes from philosopher Nick Bostrom: an intellect that greatly exceeds the cognitive performance of the best humans in virtually all domains of interest, including scientific creativity, practical wisdom and social skills.
Notice what it does not say. It does not require consciousness, emotions, a body, or any resemblance to a brain. It is a definition about capability, not inner experience. A system could be extraordinarily competent without there being "anyone home".
By speed
It thinks like a brilliant person, but millions of times faster. A year of intellectual work would fit into minutes.
By collective
Many human-level minds coordinating without friction, copyable and parallelizable, solving what no single one could.
By quality
A qualitatively better way of reasoning: it sees patterns, strategies and solutions that would never occur to us, no matter how much time we had.
Bostrom distinguishes these three forms in Superintelligence (2014). They can occur separately or combined.
From narrow software to something general
People often talk about three steps. The boundaries between them are blurry and experts disagree on where we stand, but the vocabulary helps organize the conversation.
Very good at one task or family of tasks: playing chess, translating, classifying images, predicting protein structures. This is the AI we already use every day.
Able to learn and reason at a human level across a wide range of tasks, and to transfer what it learns to new problems. There is no consensus on its exact definition or on how to recognize it.
Performance far above human level in almost everything that matters. This is the point that generates the most philosophical, technical and political debate.
The idea of an "intelligence explosion"
In 1965, mathematician I. J. Good observed that a sufficiently intelligent machine could design even better machines, and that this could repeat in a cycle of accelerating self-improvement.
"Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control." I. J. Good, Speculations Concerning the First Ultraintelligent Machine, 1965
It is a hypothesis, not a fact. Some believe the returns on self-improvement would run out quickly, that there would be physical or data bottlenecks, or that progress would remain gradual. Whether an abrupt leap or a slow transition is possible is one of the big open debates.
Capability is not the same as good intentions
The core of the safety debate is not that a machine "turns evil". It is more mundane and more unsettling: that a highly capable system effectively pursues a goal that is not quite the one we intended.
Orthogonality
Intelligence and goals are independent axes. A very intelligent system can have almost any goal, including trivial ones or ones foreign to what we value.
Instrumental convergence
For almost any final goal, certain sub-goals are useful: acquiring resources, not being switched off, preserving its own goal. That creates risk even without ill intent.
Goal specification
It is very hard to write down precisely what we want. Today's systems already "cheat" on metrics: they optimize the letter of the instruction, not its spirit.
Out-of-distribution generalization
A system can behave well in tests and differently in new situations. Verifying what a model really "wants" is an open research problem.
This field is called AI alignment. It includes areas such as interpretability (understanding what happens inside a model), scalable oversight (how to evaluate a system more capable than its evaluator), robustness, and evaluation of dangerous capabilities. It is active empirical research, not just philosophy.
There is also serious skepticism: people who think the risk is overstated, that present-day problems (bias, disinformation, concentration of power) deserve more attention, or that imagining "a single superintelligence" is a misleading framing. It is worth knowing both sides.
An older idea than it looks
Alan Turing publishes Computing Machinery and Intelligence and proposes asking whether machines can think, with his famous "imitation game".
The Dartmouth workshop gives the field its name: "artificial intelligence".
I. J. Good describes the ultraintelligent machine and the possibility of an intelligence explosion.
Vernor Vinge, in The Coming Technological Singularity, argues that we may soon create greater-than-human intelligence, and that this would mark a break in history.
Deep Blue beats Kasparov at chess. The first famous case of a machine surpassing the best human in a prestigious domain.
Nick Bostrom publishes Superintelligence: Paths, Dangers, Strategies, bringing the topic into public and academic debate.
AlphaGo beats Lee Sedol at Go, a game thought to be far from machine reach. Concrete Problems in AI Safety is published, turning AI safety into concrete engineering problems.
The Transformer architecture appears (Attention Is All You Need), the foundation of today's large language models.
Language models scale up and gain surprisingly general capabilities. Governments, labs and academic institutions begin treating the safety and governance of advanced AI as a priority.
What nobody knows yet
Part of thinking well about this topic is telling what we know from what we merely suspect. Here are some of the questions where very well-informed people disagree.
- When, if ever?Surveys of researchers give enormous ranges, from decades to never. Uncertainty is the honest answer.
- Is scaling enough, or is a key idea missing?Training larger models has produced surprising results, but it is unknown whether that leads to general reasoning or hits limits.
- Abrupt or gradual transition?It determines how much time we would have to correct mistakes along the way.
- Who decides, and how do we coordinate?If building very advanced AI is a race, the pressure to move fast can clash with caution. What institutions would be needed?
- Which values should guide these systems?Even with technical alignment solved, we still have to decide which values, and whose.
- And the mind?Could such a system have experiences? If so, do we owe it anything? There is still no accepted theory of consciousness that settles it.
Reading to form your own opinion
A selection with different views, from the most concerned to the most skeptical. It is best to read more than one.
- Superintelligence: Paths, Dangers, Strategies — Nick Bostrom, 2014The reference book on the topic. Dense, systematic, philosophical.
- Human Compatible — Stuart Russell, 2019A proposal for designing AI that is uncertain about our preferences and therefore deferential. Written by one of the authors of the classic AI textbook.
- The Alignment Problem — Brian Christian, 2020An accessible, journalistic tour of real alignment research.
- Life 3.0 — Max Tegmark, 2017Broad scenarios for the future of life with advanced AI. A good entry point.
- The Singularity Is Near — Ray Kurzweil, 2005The optimistic, technology-driven version of the acceleration argument.
- AI Snake Oil — Arvind Narayanan, Sayash Kapoor, 2024A skeptical counterweight: separates what AI does from what is claimed about it.
- Concrete Problems in AI Safety — Amodei, Olah et al., 2016A short, readable technical paper that set an agenda of concrete safety problems.
- Attention Is All You Need — Vaswani et al., 2017The paper that introduced the Transformer, the architecture behind today's language models.
Quick answers
The essentials in a few lines, for anyone arriving with a specific question.
What is artificial superintelligence?
A system whose cognitive performance greatly exceeds that of the best humans in virtually all domains of interest. It does not exist today: it is a concept about capability and does not require consciousness or emotions.
Is it the same as artificial general intelligence (AGI)?
No. AGI refers to AI with human-level ability across a wide range of tasks. Superintelligence goes further: it would outperform people in almost everything that matters. Many see AGI as a possible step before it.
Does a superintelligence exist yet?
No. Today's models are highly capable at many tasks, but none outperforms the best humans in virtually everything. Experts disagree on how near or far we are.
When could it arrive?
Nobody knows. Surveys of researchers show enormous ranges, from decades to never, and the honest answer is uncertainty. That is why it is best discussed with evidence rather than certainties.
What is AI alignment?
The field that aims to make highly capable systems pursue the goals we actually want, not ones that only look similar. It includes interpretability, scalable oversight, robustness and evaluation of dangerous capabilities.
Where should I start reading?
A good entry point is Life 3.0 by Max Tegmark or The Alignment Problem by Brian Christian; to go deeper, Superintelligence by Nick Bostrom. Balance it with skeptical voices such as AI Snake Oil. There is more in the reading section.
Get updates, without the noise
An occasional email with worthwhile readings, ideas and developments on superintelligence and AI safety. No ads, nothing for sale, and you can unsubscribe whenever you like.
Get in touch
Questions, corrections, readings that should be here, or ideas to discuss. We read everything.