Superintelligence by Nick Bostrom

• Other animals have stronger muscles and sharper claws, but we have cleverer brains. Our modest advantage in general intelligence has led us to develop language, technology, and complex social organization. The advantage has compounded over time, as each generation has built on the achievements of its predecessors.
• Once unfriendly superintelligence exists, it would prevent us from replacing it or changing its preferences Our fate would be sealed.
• History, at the largest scale, seems to exhibit a sequence of distinct growth modes, each must more rapid than its predecessor. This pattern has been taken to suggest that another (even faster) growth mode might be possible.
• Machines matching humans in general intelligence--that is, possessing common sense and an effective ability to learn, reason, and plan to meet complex information-processing challenges across a wide range of natural and abstract domains--have been expected since the invention of computers in the 1940s.
• Two decades is a sweet spot for prognosticators of radical change: near enough to be attention-grabbing and relevant, yet far enough to make it possible to suppose that a string of breakthroughs, currently only vaguely imaginable, might by then have occurred.
• Most technologies that will have a big impact on the world in five or ten years from now are already in limited use, while technologies that will reshape the world in less than fifteen years probably exist as laboratory prototypes.
• The main reason why progress has been slower than expected is that the technical difficulties of constructing intelligent machines have proved greater than the pioneers foresaw.
• Sometimes a problem that initially looks hopelessly complicated turns out to have a surprisingly simple solution (though the reverse is probably more common).
• To overcome the combinatorial explosion, one needs algorithms that exploit structure in the target domain and take advantage of prior knowledge by using heuristic search, planning, and flexible abstract representations--capabilities that were poorly developed in the early AI systems.
• Designed as support tools for decision makers, expert systems were rule-based programs that made simple inferences from a knowledge base of facts, which had been elicited from human domain experts and painstakingly hand-coded in a formal language.
• While simple neural network models had been known since the late 1950s, the field enjoyed a renaissance after the introduction of the backpropagation algorithm, which made it possible to train multi-layered neural networks.Such multi layered networks, which have one or more intermediary (“hidden”) layers of neurons between the input and output layers, can learn a much wider range of functions than their simpler predecessors. Combined with the increasingly powerful computers that were becoming available, these algorithmic improvements enabled engineers to build neural networks that were good enough to be practically useful in many applications.
• In practice getting evolutionary methods to work well requires skill and ingenuity, particular in devising a good representational format. Without an efficient way to encode candidate solutions (a genetic language that matches latent structure in the target domain), evolutionary search tends to meander endlessly in a vast search space or get stuck at a local optimum.
• Even if a good representational format is found, evolution is computationally demanding and is often defeated by the combinatorial explosion.
• In fact, one of the major theoretical developments of the past twenty years has been clearer realization of how superficially disparate techniques can be understood as special cases within a common mathematical framework.
• Algorithms differ in their processor time and memory space requirements, which inductive biases they presuppose, the ease with which externally produced content can be incorporated, and how transparent their inner workings are to a human analyst.
• The ideal is that of the perfect Bayesian agent, one that makes probabilistically optimal use of available information. This ideal is unattainable because it is too computationally demanding to be implemented in any physical computer.
• One advantage of relating learning problems from specific domains to the general problem of Bayesian inference is that new algorithms that make Bayesian inference more efficient will then yield immediate improvements across many different areas. Advances in Monte Carlo approximation techniques, for example, are directly applied in computer vision, robotics, and computational genetics.
• Artificial intelligence already outperforms human intelligence in many domains.
• Common sense and natural language understanding have also turned out to be difficult. It is now often thought that achieving a fully human-level performance on these tasks is an “AI-complete” problem, meaning that the difficulty of solving these problems is essentially equivalent to the difficulty of building generally human-level intelligence machines. In other words, if somebody were to succeed in creating an AI that could do everything else that human intelligence can do, or they would be but a very short step from such a general capability.
• The fact that the best performance at one time is attained through a complicated mechanism does not mean that no simple mechanism could do the job as well or better. It might simply be that nobody has yet found the simpler alternative.
• The Google search engine is, arguably, the greatest AI system that has yet been built.
• Interactions between individually simple components can produce complicated and unexpected effects.
• Systemic risk can build up in a system as new elements are introduced, risks that are not obvious until after something goes wrong (and sometimes not even then).
• Another lesson is that smart professionals might give an instruction to a program based on a sensible-seeming and normally sound assumption and that this can produce catastrophic results when the program continues to act on the instruction with iron-clad logical consistency even in the unanticipated situation where the assumption turns out to be invalid.
• The need for pre-installed and automatically executing safety functionality--as opposed to reliance on runtime human supervision--again foreshadows a theme that will be important in our discussion of machine superintelligence.
• Historically, AI researchers have not had a strong record of being able to predict the rate of advances in their own field or the shape that such advances would take. On the one hand, some tasks, like chess playing, turned out to be achievable by means of surprisingly simple programs; and naysayers who claimed that machines would “never” be able to do this or that have repeatedly been proven wrong. On the other hand, the more typical errors among practitioners have been to underestimate the difficulties of getting a system to perform robustly on real-world tasks, and to overestimate the advantages of their own particular pet project or technique.
• Machines are currently far inferior to humans in general intelligence. Yet one day (we have suggested) they will be superintelligent.
• We can tentatively define a superintelligence as any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest
• It now seems clear that a capacity to learn would be an integral feature of the core design of a system intended to attain general intelligence, not something to be tacked on later as an extension or an afterthought. The same holds for the ability to deal effectively with uncertainty and probabilistic information. Some faculty for extracting useful concepts from sensory data and internal states, and for leveraging acquired concepts into flexible combinatorial representations for use in logical and intuitive reasoning, also likely belong among the core design features in a modern AI intended to attain general intelligence.
• We know that blind evolutionary processes can produce human-level general intelligence, since they have already done so at least once. Evolutionary processes with foresight--that is, genetic programs designed and guided by an intelligent human programmer--should be able to achieve a similar outcome with far greater efficiency.
• Evolutionary algorithms require not only variations to select among but also a fitness function to evaluate variants, and this is typically the most computationally expensive component. A fitness function for the evolution of artificial intelligence plausibly requires simulation of neural development, learning, and cognition to evaluate fitness.
• THe computational cost of simulating one neuron depends on the level of detail that one includes in the simulation. Extremely simple neuron models use about 1,000 floating-point operations per second (FLOPS) to simulate one neuron (in real-time).
• Evolution achieved human intelligence without aiming at this outcome. In other words, the fitness functions for natural organisms do not select only for intelligence and its precursors.
• Advances in neuroscience and cognitive psychology--which will be aided by improvements in instrumentation--should eventually uncover the general principles of brain function. This knowledge could then guide AI efforts.
• A successful seed AI would be able to iteratively enhance itself: an early version of the AI could design an improved version of itself, and the improved version--being smarter than the original--might be able to design an even smarter version of itself, and so forth. Under some conditions, such a process of recursive self-improvement might continue long enough to result in an intelligence explosion--an even in which, in a short period of time, a system’s level of intelligence explosion--an event in which, in a short period of time, a system’s level of intelligence increases from a relatively modest endowment of cognitive capabilities to radical superintelligence.
• There is one more thing that we should emphasize, which is that an artificial intelligence need not much resemble a human mind. AIs could be--indeed, it is likely that most will be--extremely alien. We should expect that they will have very different cognitive architectures than biological intelligences, and in their early stages of development they will have very different profiles of cognitive strengths and weaknesses.
• The goal systems of AIs could diverge radically from those of human beings.
• There is no reason to expect a generic AI to be motivated by love or hate or pride or other such common human sentiments: these complex adaptations would require deliberate expensive effort to recreate in AIs. This is at once a big problem and a big opportunity.
• In whole brain emulation (also known as “uploading”), intelligent software would be produced by scanning and closely modeling the computational structure of a biological brain.
• The whole brain emulation path does not require that we figure out how human cognition works or how to program an artificial intelligence. It requires only that we understand the low-level functional characteristics of the basic computational elements of the brain. No fundamental conceptual or theoretical breakthrough is needed for whole brain emulation to succeed.
• Whole brain emulation does, however, require some rather advanced enabling technologies. There are three key prerequisites: (1) scanning: high-throughput microscopy with sufficient resolution and detection of relevant properties; (2) translation: automated image analysis to turn raw scanning data into an interpreted three-dimensional model of relevant neurocomputational elements; and (3) simulation: hardware powerful enough to implement the resultant computational structure.
• In general, whole brain emulation relies less on theoretical insight and more on technological capability than artificial intelligence.
• A third path to greater-than -human intelligence is to enhance the functioning of biological brains. In principles, this could be achieved without technology, through selective breeding.
• The ultimate potential of machine intelligence is, of course, vastly greater than that of organic intelligence.
• Even if there were an easy way of pumping more information into our brains, the extra data inflow would do little to increase the rate at which we think and learn unless all the neural machinery necessary for making sense of the data were similarly upgraded.
• In general terms, a system’s collective intelligence is limited by the abilities of its member meinds, the overheads in communicating relevant information between them and the various distortions and inefficiencies that pervade human organizations.
• The fact that there are many paths that lead to superintelligence should increase our confidence that we will eventually get there. If one path turns out to be blocked, we can still progress.
• Machines have a number of fundamental advantages which will give them overwhelming superiority. Biological humans, even if enhanced, will be outclassed.
• We use the term “superintelligence” to refer to intellects that greatly outperform the best current human minds across many very general cognitive domains.
• Speed Superintelligence: A  system that can do all that a human intellect can do, but much faster.
• The simplest example of speed superintelligence would be a whole brain emulation running on fast hardware.
• Collective superintelligence: A system composed of a large number of smaller intellects such that the system’s overall performance across many very general domains vastly outstrips that of any current cognitive system.
• Collective intelligence excels at solving problems that can be readily broken into parts such that solutions to sub-problems can be pursued in parallel and verified independently.
• We can think of wisdom as the ability to get the important things approximately right.
• Quality superintelligence: A system that is at least as fast as a human mind and vastly qualitatively smarter.
• Minor changes in brain volume and wiring can have major consequences, as we see when we compare the intellectual and technological achievements of humans with those of other apes.
• Biological neurons operate at a peak speed of about 200 Hz, a full seven orders of magnitude slower than a modern microprocessor (~ 2 GHz). As a consequence, the human brain is forced to rely on massive parallelization and is incapable of rapidly performing any computation that requires a large number of sequential operations.
• Many cognitive tasks could be performed far more efficiently if the brain’s native support for parallelizable pattern-matching were complemented by, and integrated with, support for fast sequential processing.
• Human working memory is able to hold no more than some four or five chunks of information at any given time.
• It is easier to experiment with parameter variations in software than in neural wetware.
• Biological brains need extended periods of training and mentorship wherase digital minds could acquire new memoires and skills by swapping data files.
• A slow takeoff is one that occurs over some long temporal interval, such as decades or centuries. Slow takeoff scenarios offer excellent opportunities for human political processes to adapt and respond. Different approaches can be tried and tested in sequence.
• A fast takeoff occurs over some short temporal interval, such as minutes, hours, or days. Fast takeoff scenarios offer scant opportunity for humans to deliberate. Nobody need even notice anything unusual before the game is already lost. In a fast takeoff scenario, humanity’s fate essentially depends on preparations previously put in place.
• A moderate takeoff is one that occurs over some intermediary temporal interval, such as months or years. Moderate takeoff scenarios give humans some chance to respond but not much time to analyze the situation, to test different approaches, or to solve complicated coordination problems. There is not enough time to develop or deploy new systems, but extent systems could be applied to the new challenge.
• Cognitive enhancement via improvements in public health and diet has steeply diminishing returns. Big gains from eliminating server nutritional deficiencies, and the most severe deficiencies have already been largely eliminated in all but the poorest countries. [...] Education, too, is now probably subject to diminishing returns.
• Building a seed AI might require insights and algorithms developed over many decades by the scientific community around the world. But it is possible that the last critical breakthrough idea might come from a single individual or a small group that succeeds in putting everything together.
• Given the extreme security implications of superintelligence, governments would likely seek to nationalize any project on their territory that they thought close to achieving a takeoff. A powerful state might also attempt to acquire projects located in other countries through espionage, theft, kidnapping, bribery, threats, military conquest, or any other available means.
• Human decision makes often seem to be acting out an identify or a social role rather than seeking to maximize the achievement of some particular objective.
• The principal reason for humanity’s dominant position on Earth is that our brains have a slightly expanded set of faculties compared with other animals. Our greater intelligence lets us transmit culture more efficiently, with the result that knowledge and technology accumulates from one generation to the next.
• It is important not to anthropomorphize superintelligence when thinking about its potential impacts. Anthropomorphic frames encourage unfounded expectations about the growth trajectory of a seed AI and about the psychology, motivations, and capabilities of a mature superintelligence.
• A machine superintelligence might itself by an extremely powerful agent, one that could successfully assert itself against the project that brought it into existence as well as against the rest of the world.
• Excelling at a task like strategizing, social manipulation, or hacking involves having a skill at that task that is high in comparison to the skills of other agents.
• Despite the fact that human psychology corresponds to a tiny spot in the space of possible minds, there is a common tendency to project human attributes onto a wide range of alien or artificial cognitive systems.
• Improvements in rationality and intelligence will tend to improve an agent’s decision-making, rendering the agent more likely to achieve its final goals. One would therefore expect cognitive enhancement to emerge as an instrumental goal for a wide variety of intelligence agents. For similar reasons, agents will tend to instrumentally value many kinds of information.
• If intelligence and knowledge come at a cost, such as time and effort expended in acquisition, or increased storage or processing requirements, then the agent might prefer less knowledge and less intelligence.
• Much information is irrelevant to our goals; we can often rely on others’ skill and expertise; acquiring knowledge takes time and effort; we might intrinsically value certain kinds of ignorance; and we operate in an environment in which the ability to make strategic commitments, socially signal ,an satisfy other people’s direct preferences over our own epistemic states is often more important to us than simple cognitive gains.
• A great deal of resource accumulation is motivated by social concerns-gaining status, mates, friends, and influence, through wealth accumulation and conspicuous consumption. Perhaps less commonly, some people seek additional resources to achieve altruistic ambitions or expensive non-social aims.
• An existential risk is one that threatens to cause the extinction of Earth originating intelligent life or to otherwise permanently and drastically destroy its potential for future desirable development.
• An unfriendly AI of sufficient intelligence realizes that it's unfriendly final goals will be best realized if it behaves in a friendly manner initially, so that it will be let out of the box. It will only start behaving in a way that reveals its unfriendly nature when it no longer matters whether we found out; that is, when the AI is strong enough that human opposition is ineffectual.
• We can thus perceive a general failure mode, wherein the good behavioral track record of a system in its juvenile stages fails utterly to predict its behavior at a more mature stage.
• While weak, an AI behaves cooperatively. When the AI gets sufficiently strong--without warning or provocation--it strikes, forms a singleton, and begins directly to optimize the world according to the criteria implied by its final values.
• One feature of a malignant failure is that it eliminates the opportunity to try again. The number of malignant failures that will occur is therefore either zero or one.
• In general, while an animal or a human can achieve some desired inner mental state, a digital mind that has full control of its internal state can short-circuit such a motivational regime by directly changing its internal state into the desired configuration: the eternal actions and conditions that were previously necessary as means become superfluous when the AI becomes intelligent and capable enough to achieve the end more directly.
• We have seen enough to conclude that scenarios in which some machine intelligence gets a decisive strategic advantage are to be viewed with grave concern.
• It is important to realize that some control method (or combination of methods) must be implemented before the system becomes super intelligent. It cannot be done after the system has obtained a decisive strategic advantage.
• The need to solve the control problem in advance--and to implement the solution successfully in the very first system to attain superintelligence--is part of what makes achieving a controlled detonation such a daunting task.
• Human beings are not secure systems, especially not when pitched against a superintelligent schemer and persuader.
• Incentive methods involve placing an agent in an environment where it finds instrumental reasons to act in ways that promote the principal’s interests.
• Any piece of information can in principle be relevant to any topic whatsoever, depending on the background information of a reasoner.
• A tripwire is a mechanism that performs diagnostic tests on the system and effects a shutdown if it detects signs of dangerous activity.
• Direct specification is the most straightforward approach to the control problem. The approach comes in two versions, rule-based and consequentialist, and involves trying to explicitly define a set of rules or values that will cause even a free-roaming superintelligent AI to act safely and beneficially.
• Oracles with superintelligence in extremely limited domains already exist.
• Even simple evolutionary search processes sometimes produce highly unexpected results, solutions that satisfy a formal user-defined criterion in a every different way then the user expected or intended.
• TO the extent that cheap machine labor can substitute for human labor, human jobs may disappear.
• Superimposed on local fluctuations history shows a macro-pattern of initially slow but accelerating economic growth, fueled by the accumulation of technological innovations.
• Human behavior has not yet adapted to contemporary conditions. Not only do we fail to take advantage of obvious ways to increase our inclusive fitness but we actively sabotage our fertility by using birth control.
• We seldom put forth full effort. When we do, it is sometimes painful.
• The essential property of a superorganism is not that it consists of copies of a single progenitor but that all the individual agents within it are fully committed to a common goal.
• UNless the plan is to keep superintelligence bottled up forever, it will be necessary to master motivation selection.
• It is impossible to enumerate all possible situations a superintelligence might find itself in and to specify for each what action it should take.
• Similarly, it is impossible to create a list of all possible worlds and assign each of them a value. IN any realm significantly more complicated than a game of tic-tac-toe, there are far too many possible states for exhaustive enumeration to be feasible. A motivation system, therefore, cannot be specified as a comprehensive lookup table. It must instead be expressed more abstractly, as a formula or rule that allows the agent to decide what to do in any given situation.
• One formal way of specifying such a decision rule is via a utility function. A utility function assigns value to each outcome that might obtain, or more generally to each “possible world”.
• Given a utility function, one can define an agent that maximizes expected utility.
• If an agent is not already fundamentally friendly by the time it gains the ability to reflect on its own agency, it will not take kindly to a belated attempt at brainwashing or a plot to replace it with a different agent that better loves its neighbor.
• Retard the development of dangerous and harmful technologies, especially ones that raise the level of existential risk; and accelerate the development of beneficial technologies, especially those that reduce the existential risks posed by nature or by other technologies.
• Faster computers make it easier to create machine intelligence. One effect of accelerating progress in hardware, therefore, is to hasten the arrival of machine intelligence.
• The case for rushing is especially strong with regard to technologies that could extend our lives and thereby increase the expected fraction of the currently existing population that may still be around for the intelligence explosion.
• The common good principle: Superintelligence should be developed only for the benefit of all of humanity and in the service of widely shared ethical ideals.
• Technical progress in machine learning has been swifter than most had expected.
• Deep learning methods--essentially many-layered neural networks--have, thanks to a combination of faster computers, larger data sets, and algorithmic refinements, begun to approach (and in some cases exceed) human performance on many perceptual tasks, including handwriting recognition, image recognition and image captioning, speech recognition, and facial recognition.