AI alignment
The field that aims to make highly capable systems pursue the goals we actually want, not ones that only resemble them. It includes interpretability, scalable oversight, robustness and evaluation of dangerous capabilities.
Superintelligence and advanced AI glossary
Short, checkable definitions of the terms that come up most when people discuss superintelligence. Each one has its own link so you can cite or share it.
AI governance
The rules, institutions and practices, such as regulation, standards, evaluations and international agreements, that guide how artificial intelligence is developed and used.
Artificial general intelligence (AGI)
AI able to learn and reason at a human level across a wide range of tasks, and to transfer what it learns to new problems. There is no single accepted definition and no consensus on how to recognize it.
Control problem
How to ensure that a system far more capable than those overseeing it keeps acting in line with human intentions. Bostrom develops it in Superintelligence (2014).
Distribution shift
A situation in which real data or environments differ from those a system was trained or tested on, which can make it behave differently than expected.
Existential risk
The risk of a catastrophe that would wipe out humanity or permanently and drastically curtail its future potential. It is the framing Bostrom uses to discuss the risks of advanced AI.
Fast and slow takeoff
How quickly an AI would move from human level to far beyond it. In a fast takeoff it would happen in a very short time; in a slow one there would be years or decades to react and correct course.
Instrumental convergence
The observation that, for almost any final goal, certain sub-goals are useful, such as acquiring resources, preserving one's own goal or avoiding being switched off. That is why risks can arise even without ill intent.
Intelligence explosion
A hypothesis by I. J. Good (1965): a sufficiently intelligent machine could design even better machines and repeat the cycle at an accelerating pace. It is a hypothesis, not an established fact.
Interpretability
A set of methods for understanding what happens inside a model and why it produces a given output, instead of treating it as a black box.
Large language model (LLM)
A neural network trained on vast amounts of text to predict the next token, able to generate, summarize and analyze language.
Narrow AI
A system highly competent at one task or family of tasks, such as playing chess, translating or classifying images, but without the generality of a person.
Orthogonality thesis
Bostrom's idea that, in principle, any level of intelligence can be combined with almost any final goal. Being highly intelligent does not imply sharing our values.
Red teaming
Adversarial testing in which a team tries to provoke failures, misuse or dangerous behavior in a system in order to find them before deployment.
RLHF (reinforcement learning from human feedback)
A technique in which a reward model is trained from human preferences and the main model is then tuned to earn more reward. It is used to adapt conversational assistants.
Scalable oversight
The problem of how to evaluate and guide a system that may be more capable than the person evaluating it, and techniques for keeping that oversight reliable.
Scaling laws
Empirical regularities in which a model's error falls predictably, often as a power law, as compute, data and parameters increase (Kaplan et al., 2020).
Specification gaming
A system scores highly by exploiting shortcuts or flaws in how the goal was defined, without achieving what was intended. Also known as reward hacking.
Superintelligence
A system whose cognitive performance greatly exceeds that of the best humans in virtually all domains of interest. It is a definition about capability, not consciousness. The term was popularized by Nick Bostrom's 2014 book.
Technological singularity
A hypothetical point beyond which technological progress would be so fast and unpredictable that it would irreversibly change human history. It is associated with Vernor Vinge (1993) and Ray Kurzweil (2005).
Transformer
A neural network architecture based on attention mechanisms, introduced in 2017 in the paper Attention Is All You Need. It underpins most of today's language models.
Turing test
A test proposed by Alan Turing in 1950 as the "imitation game": an evaluator converses with a machine and a person and must tell which is which.
Plain-language definitions. To go deeper, see the reading list in the main guide.