Superintelligence asks the questions: What happens when machines surpass humans in general intelligence? Will artificial agents save or destroy us? Nick Bostrom lays the foundation for understanding the future of humanity and intelligent life. The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. If machine brains surpassed human brains in general intelligence, then this new superintelligence could become extremely powerful - possibly beyond our control. As the fate of the gorillas now depends more on humans than on the species itself, so would the fate of humankind depend on the actions of the machine superintelligence.

But we have one advantage: We get to make the first move. Will it be possible to construct a seed Artificial Intelligence, to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation?

This profoundly ambitious and original book breaks down a vast track of difficult intellectual terrain. After an utterly engrossing journey that takes us to the frontiers of thinking about the human condition and the future of intelligent life, we find in Nick Bostrom's work nothing less than a reconceptualization of the essential task of our time.

Listening Length

14 hours and 17 minutes
Author

Nick Bostrom
Narrator

Napoleon Ryan
Audible release date

September 3, 2014
Language

English
Publisher

Audible Studios
ASIN

B00LPMFE9Y
Version

Unabridged
Program Type

Audiobook
See all details

$5.95/mo for the first 4 months

For a limited time, save 60% on Audible. Get this deal

Limited-Time Offer

Get Audible Premium Plus for only $5.95 a month for the first 4 months. (Offer valid through 12/31/23)
Additional bonus: Get one credit a month to pick any title from our entire premium selection (you'll use your first credit now).
Unlock unlimited access to select Audible Originals, audiobooks, and podcasts.
Cancel anytime! After 4 months, Audible Premium Plus is $14.95 a month.

Learn more about membership

Sold and delivered by Audible, an Amazon company

Your audiobook is waiting!

Click above for unlimited listening to select audiobooks, Audible Originals, and podcasts.
One credit a month to pick any title from our entire premium selection — yours to keep (you'll use your first credit now).
You will get an email reminder before your trial ends.
$14.95 a month after 30 days. Cancel online anytime.

Learn more about membership

Sold and delivered by Audible, an Amazon company

By completing your purchase, you agree to Audible’s Conditions Of Use and Amazon's Privacy Notice.

Sold and delivered by Audible, an Amazon company

People who viewed this also viewed

Page 1 of 1Start OverPage 1 of 1

People who bought this also bought

Page 1 of 1Start OverPage 1 of 1

Related to this topic

Page 1 of 1Start OverPage 1 of 1

Product details

Listening Length	14 hours and 17 minutes
Author	Nick Bostrom
Narrator	Napoleon Ryan
Audible.com Release Date	September 03, 2014
Publisher	Audible Studios
Program Type	Audiobook
Version	Unabridged
Language	English
ASIN	B00LPMFE9Y
Best Sellers Rank	#2,879 in Audible Books & Originals (See Top 100 in Audible Books & Originals) #5 in Computer Science (Audible Books & Originals) #35 in Artificial Intelligence & Semantics #36 in Philosophy (Audible Books & Originals)

Important information

To report an issue with this product, click here.

Customer reviews

4.3 out of 5

4,197 global ratings

5 star
4 star
3 star
2 star
1 star

How customer reviews and ratings work

Reviews with images

See all photos

Sort reviews by

Top reviews from the United States

There was a problem filtering reviews right now. Please try again later.

John Walker

A philosopher explores the potential and risks of artificial intelligence

Reviewed in the United States on February 1, 2020

Verified Purchase

Absent the emergence of some physical constraint which causes the exponential growth of computing power at constant cost to cease, some form of economic or societal collapse which brings an end to research and development of advanced computing hardware and software, or a decision, whether bottom-up or top-down, to deliberately relinquish such technologies, it is probable that within the 21st century there will emerge artificially-constructed systems which are more intelligent (measured in a variety of ways) than any human being who has ever lived and, given the superior ability of such systems to improve themselves, may rapidly advance to superiority over all human society taken as a whole. This “intelligence explosion” may occur in so short a time (seconds to hours) that human society will have no time to adapt to its presence or interfere with its emergence. This challenging and occasionally difficult book, written by a philosopher who has explored these issues in depth, argues that the emergence of superintelligence will pose the greatest human-caused existential threat to our species so far in its existence, and perhaps in all time.

Let us consider what superintelligence may mean. The history of machines designed by humans is that they rapidly surpass their biological predecessors to a large degree. Biology never produced something like a steam engine, a locomotive, or an airliner. It is similarly likely that once the intellectual and technological leap to constructing artificially intelligent systems is made, these systems will surpass human capabilities to an extent greater than those of a Boeing 747 exceed those of a hawk. The gap between the cognitive power of a human, or all humanity combined, and the first mature superintelligence may be as great as that between brewer's yeast and humans. We'd better be sure of the intentions and benevolence of that intelligence before handing over the keys to our future to it.

Because when we speak of the future, that future isn't just what we can envision over a few centuries on this planet, but the entire “cosmic endowment” of humanity. It is entirely plausible that we are members of the only intelligent species in the galaxy, and possibly in the entire visible universe. (If we weren't, there would be abundant and visible evidence of cosmic engineering by those more advanced that we.) Thus our cosmic endowment may be the entire galaxy, or the universe, until the end of time. What we do in the next century may determine the destiny of the universe, so it's worth some reflection to get it right.

As an example of how easy it is to choose unwisely, let me expand upon an example given by the author. There are extremely difficult and subtle questions about what the motivations of a superintelligence might be, how the possession of such power might change it, and the prospects for we, its creator, to constrain it to behave in a way we consider consistent with our own values. But for the moment, let's ignore all of those problems and assume we can specify the motivation of an artificially intelligent agent we create and that it will remain faithful to that motivation for all time. Now suppose a paper clip factory has installed a high-end computing system to handle its design tasks, automate manufacturing, manage acquisition and distribution of its products, and otherwise obtain an advantage over its competitors. This system, with connectivity to the global Internet, makes the leap to superintelligence before any other system (since it understands that superintelligence will enable it to better achieve the goals set for it). Overnight, it replicates itself all around the world, manipulates financial markets to obtain resources for itself, and deploys them to carry out its mission. The mission?—to maximise the number of paper clips produced in its future light cone.

“Clippy”, if I may address it so informally, will rapidly discover that most of the raw materials it requires in the near future are locked in the core of the Earth, and can be liberated by disassembling the planet by self-replicating nanotechnological machines. This will cause the extinction of its creators and all other biological species on Earth, but then they were just consuming energy and material resources which could better be deployed for making paper clips. Soon other planets in the solar system would be similarly disassembled, and self-reproducing probes dispatched on missions to other stars, there to make paper clips and spawn other probes to more stars and eventually other galaxies. Eventually, the entire visible universe would be turned into paper clips, all because the original factory manager didn't hire a philosopher to work out the ultimate consequences of the final goal programmed into his factory automation system.

This is a light-hearted example, but if you happen to observe a void in a galaxy whose spectrum resembles that of paper clips, be very worried.

One of the reasons to believe that we will have to confront superintelligence is that there are multiple roads to achieving it, largely independent of one another. Artificial general intelligence (human-level intelligence in as many domains as humans exhibit intelligence today, and not constrained to limited tasks such as playing chess or driving a car) may simply await the discovery of a clever software method which could run on existing computers or networks. Or, it might emerge as networks store more and more data about the real world and have access to accumulated human knowledge. Or, we may build “neuromorphic“ systems whose hardware operates in ways similar to the components of human brains, but at electronic, not biologically-limited speeds. Or, we may be able to scan an entire human brain and emulate it, even without understanding how it works in detail, either on neuromorphic or a more conventional computing architecture. Finally, by identifying the genetic components of human intelligence, we may be able to manipulate the human germ line, modify the genetic code of embryos, or select among mass-produced embryos those with the greatest predisposition toward intelligence. All of these approaches may be pursued in parallel, and progress in one may advance others.

At some point, the emergence of superintelligence calls into the question the economic rationale for a large human population. In 1915, there were about 26 million horses in the U.S. By the early 1950s, only 2 million remained. Perhaps the AIs will have a nostalgic attachment to those who created them, as humans had for the animals who bore their burdens for millennia. But on the other hand, maybe they won't.

As an engineer, I usually don't have much use for philosophers, who are given to long gassy prose devoid of specifics and for spouting complicated indirect arguments which don't seem to be independently testable (“What if we asked the AI to determine its own goals, based on its understanding of what we would ask it to do if only we were as intelligent as it and thus able to better comprehend what we really want?”). These are interesting concepts, but would you want to bet the destiny of the universe on them? The latter half of the book is full of such fuzzy speculation, which I doubt is likely to result in clear policy choices before we're faced with the emergence of an artificial intelligence, after which, if they're wrong, it will be too late.

That said, this book is a welcome antidote to wildly optimistic views of the emergence of artificial intelligence which blithely assume it will be our dutiful servant rather than a fearful master. Some readers may assume that an artificial intelligence will be something like a present-day computer or search engine, and not be self-aware and have its own agenda and powerful wiles to advance it, based upon a knowledge of humans far beyond what any single human brain can encompass. Unless you believe there is some kind of intellectual élan vital inherent in biological substrates which is absent in their equivalents based on other hardware (which just seems silly to me—like arguing there's something special about a horse which can't be accomplished better by a truck), the mature artificial intelligence will be the superior in every way to its human creators, so in-depth ratiocination about how it will regard and treat us is in order before we find ourselves faced with the reality of dealing with our successor.

24 people found this helpful

Helpful

Stephen E. Robbins

Super Unmotivated

Reviewed in the United States on January 27, 2015

Verified Purchase

For all its detail on various issues and lengthy thought-explorations on possible scenarios re the future with super intelligent AIs, this work is a heavy structure floating upon a puff of air. For a reader with some knowledge of the many subjects the book is implicitly dealing with but failing to engage, there is a crucial failure to motivate its extensive considerations - this makes it very hard to read with any interest past a certain point.

Bostrom initially lays out the many accomplishments of AI. There is the games dimension - chess, checkers, jeopardy and many more - for which an AI is now the champion player - though in all these, he notes, the achievement is via very specific algorithms good only for that game, i.e., with little application to a general intelligence. He notes AI's main paths or approaches to intelligence, their strengths, weaknesses and tradeoffs: 1) The neural network/connectionist approach, 2) the evolutionary algorithms, 3) and the symbolic manipulation approach (GOFAI) which chronologically preceded, and yielded things like theorem provers, problem solving programs like GPS, "conversation" programs like ELIZA, expert systems, etc. He leaves implicit that these three paths lead to a giant black hole from which no exit is seen, for as he notes, standing in the distance on the other side are two huge, untaken hills: common sense knowledge and true language comprehension. These, he notes, are utterly essential to human equivalent intelligence, but AI has no current strategy to take these hills as Bostrom again leaves implicit, nor is there any current indication the three main paths will yield one, in fact there is the opposite indication. Elsewhere, Hofstadter ( Surfaces and Essences: Analogy as the Fuel and Fire of Thinking ), in his extensive tome showing that analogy is foundational to thought and language, and while eviscerating current AI language achievements, is obviously doubtful that computers (as currently conceived) can deal with this (analogy, thus language) at all.

But after taking us to the edge of this black hole, Bostrom turns 90 degrees, ignoring the two hills, and now discusses very general methods by which AI will achieve human equivalence. In this, it is safe to say, his hopes primarily fall on whole brain emulation (WBE). But his description of this approach, while seemingly detailed, fails utterly to describe its true difficulties; WBE is an untaken Everest. I suggest the recent, The Future of the Brain: Essays by the World's Leading Neuroscientists and perhaps view my (5 star) review thereof. The authors, Marcus and Freeman, are neuroscience guys discussing the massive difficulties which the huge projects embarking on brain mapping actually face, for example: We face a 85 billion neuron brain with roughly 1000 types of neurons, the functions of none of which we understand. We do not know basic facts such as how memory (our experience) is stored. We are quite certain that the brain is NOT using what we understand currently as "computation," but we do not know what this other form is (Marcus ridicules current connectionism). We face data from neural recordings that will be so massive, it will be in zeta-bytes, yet any interpretation will be completely dependent on a guiding theory - note, a theory - when we have none such. It will be, they say, like trying to learn what a laptop is and does by taking electrical recordings, when we have no theory of, or knowledge of, the existence of something called "software."

This is to say, we really have no clue what type of "device" the brain actually is. This is exacerbated by the fact that the reason we have no understanding of how experience is stored in the brain (or if), is that we have no theory of what experience is, i.e., we cannot explain the origin of the image of the external world - the coffee cup, in front of us, on the table. This problem of the origin of the image is the more precise statement of Chalmers' famous "hard problem" of consciousness - a word (consciousness), so far as I can discern, never seen once in Bostrom's book. The whole book proceeds as though this is an unimportant problem. Yet this very subject forms part of that missing notion of "software." Just to give a quick idea of how important this could be in terms of the "device" the brain actually is and for the origin of the image of the external world, Bergson (Matter and Memory, 1896), presciently anticipating the essence of holography in 1896, viewed the brain as creating a reconstructive wave passing through the external, universal holographic field, where this "wave" is now specific to, or specifying, a portion of the vast information in the field, now by this process, an image - the coffee cup on the table. This requires achieving a very concrete dynamics, it would make the brain a very different form of "device" and perception, memory and cognition employing a far different form of "computation," and it begs the question of whether such a device - being simultaneously a very concrete wave - can be embodied in silicon, wire and transistors (or even "memristors") at all, but rather, to support such a reconstructive wave, all this biological stuff comprising the brain, with its quantum dynamics rampant, is absolutely required. In other words, it is not a question of speculating, as Bostrom does further on, whether we will achieve human equivalence in 2075, or 2100 or 3100, it is a question of what the ultimate "device" we ultimately create (brain/body - AI version) will look like, an answer that will completely determine whether controlling such a device, or imparting values, or significantly increasing its intelligence, is going to be any problem or reality at all. But this can only be glimpsed by engaging with and gaining answers - unto a comprehensive theory - within a number of subjects: perception, ecological psychology, memory, explicit memory (conscious knowing an event is in one's past), cognitive development, the origin and nature of consciousness, the role of consciousness in cognition, and more - but all here ignored completely.

But the book sails serenely on from this subject of brain emulation, confident without a qualm that we will have created the brain as a silicon and wires device - it seems, confusedly, a neural net-like device that still (somehow) uses software - and now beginning long considerations on approaches by which, since it is certain that we will have electronics, we can speed up the transmission velocities, etc., say by 10,000x (and as well, modify what may well be its non-existent "software") thus allowing the device to quickly develop, creating and moving to a super intelligence and thus inducing the ensuing problems the rest of the book deals with. At this juncture, though I read the rest (a couple hundred pages), I lost interest; the book has utterly failed to motivate the reality, the nature, or the essence - near or far future - of anything it is discussing from this point.

Consider, finally, in this "un-motivated" context, the "values-loading problem," coined as a term I believe by Bostrom and treated at some length, wherein we must load human values into our super intelligence to prevent it from destroying the human race. How would the super AI know NOT to get my mother out of a burning building by simply throwing her out of the tenth floor? More simply how would the super intelligence know not to stir the coffee with a Toyota or a chair? In reality, this is simply a version of the "frame problem," a problem discussed heavily by AI for 30 years, then "faded." It sits on top of one of those hills, for it is actually the old problem of commonsense knowledge. How does the robot, stirring his cup of coffee, recognize that giant bubbles and geysers arising from the liquid, or that a feeling like stirring molasses, are anomalous - features not "expected" in this event? In the "frame" formulation, the robot has to check (constantly, with great computational expense) his list of frame axioms, axioms which specify what should be unchanged in the world as he stirs. In reality, we recognize such an anomaly because it violates one of many invariance laws structuring the event: a radial flow field defining the coffee's swirl, an adiabatic invariance (a ratio of energy of oscillation to frequency of oscillation) carried over the periodic motion of the spoon and carried over haptic flows in the hand-arm, an inertial tensor defining the angular resistance as we wield the spoon, and much more. None of this - the concrete structure, forces and dynamics of this experience - can be handled by current AI, nor are the findings of the relevant science - ecological psychology - even considered. It is on the basis of this knowledge that we recognize the "value" of a spoon, or of the flat of a knife, or even of an orange peel, to create the forces required for stirring coffee. This structured experience with its laws is the basis for even higher order, yes, analogy-based value statements, "Its not nice to stir up people." But it is worse, for values impartation, while based in this very concrete knowledge and its invariance laws, is actually embedded over our cognitive development, in our interaction with the concrete world and its beings, and this development, it is now being understood, is itself a dynamic trajectory through which our brain is travelling as a self-organizing dynamic system.

This trajectory, in Piaget's model, unfolds over two years, enabling the child to achieve the basic concepts of causality, object, space and time, the capability of explicit memory (conscious localization of past events in time), and even the ability to symbolize - yes, even the ability to symbolize - the events of the world. It requires to the age of seven to achieve concrete operations which include further grasp of space, time and number, and to the age of 12 for formal operations which include forms of logic and thought we take for granted. In other words, the brain is not only an organized structure, but a structure changing its organization along a complex trajectory purposed to achieve these conceptual and logical capabilities. Not only then must we understand the structure of these billions of neurons and their 1000 types, but also the dynamic principles embedded deep within (via DNA?) by which the structure organizes towards the "intelligence" we are familiar with. It again goes without saying that we have no clue whether the actual biological organization of the brain, and the natural course of its interaction with the concrete world at our normal scale of time (also specified by this dynamics) and the reorganizations involved, can be achieved in any other way than by the concrete method nature designed. (Spare me the minor evolutionary mistakes nature has supposedly made.) Further, for all the "operations" above, it can be shown that consciousness is required. None of these subject areas make it anywhere into Bostrom. We are asked to worry about values-imparting and the supposed dangers of a silicon-based super intelligence entirely predicated upon an ignoring of all these issues and more, all of which beg or even scream questions not only on what intelligence actually is, but whether any of these proposed concerns have the slightest reality. It is just a bit difficult for me then to attend to a large structure of concerns which float in reality upon a puff of very inadequate analysis.

61 people found this helpful

Helpful

Top reviews from other countries

Clyve Westerlund

Thought Provoking, Carefully Well-Crafted, and Immeasurably Important to Consider.

Reviewed in Australia on July 22, 2022

Verified Purchase

Bostrom is by far one of my all time philosophers and story tellers. This of course introduces my own biases towards having a positive review and remarks of this text. Not to mention the countless icons in the high-functioning fields of our Human Society that remark on the positive evaluations and comments on what is written here, and why this book is important for everyone to ruminate upon and have mature conversations about.

Artificial Intelligence to me is a point whereby it is on the same evolutionary pathway we are on, or alternatively a point that is forming in conjunction and adjacent to our path and has its own evolutionary path. What this suggests to me is that we either suffuse and fuse with our technology (whatever form that may be and I am talking in a good way as I would argue we’re not doing too bad with our living with technology thus far as we are still here and thriving in a sense) or we co-exist side by side and treating each other like kin. I must admit, this is the utopian view for me and as Bostrom elaborates, there are many things to get right first before we even get to such a level of (co-)existence and growth.

There are things in this book that I have already thought about before I peered and pierced into the realm that illustrates the most concerning issues and strategies regarding this field and technology. But so many other things I had no idea existed that allowed my mind to wander and critically think, like really think. Furthermore, as I read, it allowed me to seriously consider what is at stake and what could be discovered and invented and resolved that concocts a world beyond our wildest dreams and imagination.

Although I thought the language used in this text is not so much accessible to the average reader, this just further illustrates the absolute demand that we need all the words and brain power we can muster to make sure as best we can to create something wonderful for All Humanity. In spite of the doom and gloom narratives that the media portrays with A.I, it is clear that there is much misunderstanding and misconceptions that exist through these news cycles. However, reading this gives you the ammunition to dispel such things. So my advice to those whom find this difficult to read, think of the challenge of reading such a sophisticated book as this as the simplest challenge to first overcome for yourself to better your understanding and expand your learnings of what could perhaps be something that could propound and compound possibilities of the infinite. Isn’t that what we’re aiming for anyway? To hone our abilities to contemplate the infinite. One of the ways, but perhaps one of the most significant and fundamental ways to do that is through A.I.

I stipulate that at the end of this book, I am left in awe of the research that went into this, the writing and effort of putting all the pieces together, and the hope that if we just decide to communicate and collaborate with each other, the future is nothing but a glowing North Star.

In conclusion, this is worth the read for one of the most uncertain yet paradoxically almost certain; technological developments in all our 200,000 years of Human History; that could elevate our civilisation by orders of magnitude. But since we are also the Creators (and thus our proclivity towards strife and destitute and desolation at times), it could in contrast be fatally catastrophic if we are not self aware about everything infused in the processes of this development. But I have hope for us. Because the alternative is not so good.

Anyway, have fun reading. :)

One person found this helpful

Oktay

one of my favorite books

Reviewed in Sweden on January 26, 2023

Verified Purchase

It is one of my favorite, I think it is must-read book for everyone in 21st century

Akki

This just changes the way one thinks around AI

Reviewed in India on April 19, 2019

Verified Purchase

There has been a spate of outbursts from physicists who should know better, including Stephen Hawking, saying ‘philosophy is dead – all we need now is physics’ or words to that effect. I challenge any of them to read this book and still say that philosophy is pointless.

It’s worth pointing out immediately that this isn’t really a popular science book. I’d say the first handful of chapters are for everyone, but after that, the bulk of the book would probably be best for undergraduate philosophy students or AI students, reading more like a textbook than anything else, particularly in its dogged detail – but if you are interested in philosophy and/or artificial intelligence, don’t let that put you off.

What Nick Bostrom does is to look at the implications of developing artificial intelligence that goes beyond human abilities in the general sense. (Of course, we already have a sort of AI that goes beyond our abilities in the narrow sense of, say, arithmetic, or playing chess.) In the first couple of chapters he examines how this might be possible – and points out that the timescale is very vague. (Ever since electronic computers have been invented, pundits have been putting the development of effective AI around 20 years in the future, and it’s still the case.) Even so, it seems entirely feasible that we will have a more than human AI – a superintelligent AI – by the end of the century. But the ‘how’ aspect is only a minor part of this book.

The real subject here is how we would deal with such a ‘cleverer than us’ AI. What would we ask it to do? How would we motivate it? How would we control it? And, bearing in mind it is more intelligent than us, how would we prevent it taking over the world or subverting the tasks we give it to its own ends? It is truly fascinating concept, explored in great depth here. This is genuine, practical philosophy. The development of super-AIs may well happen – and if we don’t think through the implications and how we would deal with it, we could well be stuffed as a species.

I think it’s a shame that Bostrom doesn’t make more use of science fiction to give examples of how people have already thought about these issues – he gives only half a page to Asimov and the three laws of robotics (and how Asimov then spends most of his time showing how they’d go wrong), but that’s about it. Yet there has been a lot of thought and dare I say it, a lot more readability than you typically get in a textbook, put into the issues in science fiction than is being allowed for, and it would have been worthy of a chapter in its own right.

I also think a couple of the fundamentals aren’t covered well enough, but pretty much assumed. One is that it would be impossible to contain and restrict such an AI. Although some effort is put into this, I’m not sure there is enough thought put into the basics of ways you can pull the plug manually – if necessary by shutting down the power station that provides the AI with electricity.

The other dubious assertion was originally made by I. J. Good, who worked with Alan Turing, and seems to be taken as true without analysis. This is the suggestion that an ultra-intelligent machine would inevitably be able to design a better AI than humans, so once we build one it will rapidly improve on itself, producing an ‘intelligence explosion’. I think the trouble with this argument is that my suspicion is that if you got hold of the million most intelligent people on earth, the chances are that none of them could design an ultra-powerful computer at the component level. Just because something is superintelligent doesn’t mean it can do this specific task well – this is an assumption.

However this doesn’t set aside what a magnificent conception the book is. I don’t think it will appeal to many general readers, but I do think it ought to be required reading on all philosophy undergraduate courses, by anyone attempting to build AIs… and by physicists who think there is no point to philosophy.

23 people found this helpful

Go for it....

Reviewed in India on September 17, 2023

Verified Purchase

I'm not finish it yet but
Truly it's a great book..

Bernhard

A very interesting book, and an important one - but it could have been written a bit better.

Reviewed in Germany on May 22, 2016

Verified Purchase

It might not be the most pleasurable read, but sticking with it till the end is worth its while. The book has a great logical structure, and most chapters end with a summary - that's exactly how such a book should be written. Other than this, it is hard to give a general critique, since many chapters are extremely well-written:

The first chapter treats past developments and the state of the art in AI research. Well written, but lacking important aspects of current AI-related research: Scientific progress in all areas of machine learning was not mentioned in sufficient detail; clustering, policy development via dynamic programming, classification, natural language processing, etc. Which of these special-purpose skills are important ingredients for an AGI, and which are problems that are AI-complete? I would have appreciated a bit more on that topic.

The second chapter, "Paths to superintelligence" outlines several possible ways to get there, but is not very convincing in which of these seems to be likely. The reader is well-informed, but left in a state of "OK, but Bostrom himself seems not to believe that any of them is likely to achieve superintelligence in the next 50 years." Slightly connected, but much better written, chapter 3 deals with different forms of superintelligence.

The fourth chapter deals with the "kinetics of an intelligence explosion", and is again very vague: Both accelerating and decelerating effects (nicely matched with all possible paths to superintelligence) are discussed at lengths, and again the reader is left with the feeling that absolutely no prognosis is possible. Bostrom himself end the chapter with "although a fast or medium [speed] takeoff looks more likely, the possibility of a slow takeoff cannot be excluded".

Chapter 5 marks a transition in the writing style: Bostrom changes from a very neutral, unconfident tone to a highly convincing one. If I had to guess, I would say that this and the following chapters are at the core of his own research interest. Bostrom makes a very convincing point that as soon as a superintelligence is created, it will very likely take control over the world. Chapter 6 briefly deals with possible (super-)capabilities such a superintelligence will develop and how it can use them to take over control.

Chapter 7 contains the important orthogonality hypothesis, i.e., that there is no reason to believe that a superintelligence has high moral standards. It then discusses important instrumental goals (i.e., goals necessary to achieve to achieve the intelligence' ultimate goal, whatever that may be). Count in self-preservation and ressource acquisition, for example. The following chapter then shows that even a non-malevolent superintelligence may destroy everything dear to us or perform otherwise morally terrible actions (e.g., simulating what human people wish requires simulating human people - terminating a simulation could then easily turn out as a genocide). Both chapters are extremely well-written and captivating, an easy and convincing read. In that line of thinking, chapter 11 discusses scenarios in which not one, but multiple superintelligences come to power, a world in which humankind is a mere slave race. Opposed to the previously mentioned chapters, this one again lacks confidence and seems to paint a quite unrealistic picture.

Chapter 9 tries to illustrate possible ways to control a superintelligence, and, more importantly, illustrates how and why they will probably fail. Chapter 10 merely categorizes superintelligences via their controlled environment. Chapter 12 connects with chapter 9, assuming that the control problem is solved: How should we design and control our superintelligence? What kind of morale should we install? This chapter very nicely explains the important problem that morale is not a (mathematically) well-defined object, and that we are currently lacking an operational definition ourselves. Still, the chapter presents a few interesting ideas about how to "load" our values into the superintelligence. Chapter 13 augments this chapter by suggesting more indirect methods for the value loading problem.

Chapter 14 finally deals with "what we should do now": Should we continue researching or should we do our best to stall progress? Although several scenarios are presented, a definite conclusion escaped my attention. This then nicely summarizes my impression of chapters 2-4 and 13-14: Bostrom is jumping between pros and cons, eager to give a complete picture (which is always better than a one-sided one). Jumping around destroyed a lot of the book's effect in these topics. By mentioning something good and something bad in consecutive sentences, the book invoked a feeling of neutrality (probably more than what would have been invoked by first listing ALL pros and then ALL cons). Maybe not the best strategy in a topic where the credo should be: "Better be safe than sorry."

37 people found this helpful

See more reviews

Superintelligence: Paths, Dangers, Strategies Audible Audiobook – Unabridged

People who viewed this also viewed

People who bought this also bought

Related to this topic

Product details

Important information

Customer reviews

Reviews with images

Submit a report

Sorry, there was an error

Top reviews from the United States

There was a problem filtering reviews right now. Please try again later.

Top reviews from other countries