Rationality: From AI to Zombies Kindle Edition

"There is a spoon." <-- Best line of the book.

WHAT TO EXPECT
This "book" is a compilation of 333 blog posts that Eliezer Yudkowsky wrote for the blog Less Wrong, with a few transitional explanations inserted. Eliezer Yudkowsky draws on knowledge about cognitive biases, the math of human decision-making, probability theory, and other fields to present a coherent picture of how humans think and some of the common mistakes that we should try to avoid. (For more details on that, see below.)

TONE: Yudkowsky's tone is eager, sometimes derogatory, challenging. I list this here because it is both a con and a pro. If you disagree with something he says, it's harder to listen when he doesn't seem to be respecting your position. But the style is also more engaging and fun to read than if he was drily presenting facts and leaving it up to you how to feel about it.

NOTE TO RELIGIOUS READERS
While the author does not set out to explicitly disprove religious beliefs, he does sometimes use examples of religious thinking (from a wide variety of faiths) to illustrate irrational thinking. He also draws on examples from the history of science and from politics, so religion is not just getting the short end of the stick. If you're wondering whether this book will offend your beliefs, here is a sample—if you can tolerate this, you should be fine with the rest of the book:

"Nobel laureate Robert Aumann, of Aumann's Agreement Theorem, is an Orthodox Jew... This should scare you down to the marrow of your bones. It means you can be a world-class scientist and conversant with Bayesian mathematics and still fail to reject a belief whose absurdity a fresh-eyed ten-year-old could see. It shows the invincible defensive position which a belief can create for itself, if it has long festered in your mind."

He uses this to give the atheist reader a feel for how much effort it would take them to question or destroy their own most cherished beliefs.

CONS

* Crazy LONG. (It's a good thing the TOC numbered the individual blog posts, because if I had tried to count them myself, I'd have given up after 108 or so. I can't tell you how long it took me to read it, because I'm a sleep-deprived mom and my memory of personal events extends to roughly yesterday. It took my smartphone two minutes just to scan the document for a word.)

* JUMPS AROUND A LOT instead of building the matter from the ground up. JUST KEEP READING; don't follow all the links to later points in the book, just read it through in the order given. It will all fit together, in the end.

* HIGHLY TECHNICAL REFERENCES are thrown in. Light cones, minimum message length, configuration space, logarithmic utility functions, Python scripts, and Occam priors may all be mentioned without explicit explanation. JUST KEEP READING. The bulk of the language is easily accessible, and the ideas still come across. Although I could follow most of those, so if you're less interested in science/math/programming than I am, maybe it will affect you more.

* NOT PRACTICAL ENOUGH. The author rightly identifies that his work is primarily aimed at what we should know rather than how to apply what we know in real life. (Although the examples given do serve that purpose, at least somewhat.) I sincerely hope he tackles that front with the same skill that he tackled the first.

Mr. Yudkowsky, please take note: these ideas need a proper book. One single book to encode the ideas you've got so far, written to simply, engagingly, and briefly-as-possible introduce the subject to the general public.

* Oh, and a minor quibble. The Quantum Mechanics sequence in the e-book is incomplete. Since I have an ongoing interest in understanding QM better, I went back and read through the whole sequence online. If you just want to know about QM's application to rationality, and aren't trying to grasp the theory overall, the sections in the book are fine.

PROS

* HIGHLY ENGAGING STYLE. Despite throwing in technical language regularly, the author's style is not dry, formal, or stiff; it's quite easy and fun to read.

* GREAT HUMOR. I love when he ends a blog post by displaying (in an ironic manner) the very cognitive error he was talking about.

* FICTIONAL INTERLUDES. Masters of Bayesian fu, zombies that turn the world normal, and more fictional interludes help demonstrate his points. Since I first started looking into his sequences after reading his Harry Potter and the Methods of Rationality fanfic, I was pleased to see more of his skills along those lines.

* SERIOUS ABOUT APPLYING HIS OWN IDEAS. As in, he's really thought his ideas through and tried to apply his knowledge of biases to his own thinking. He's not just on a superiority trip; he uses examples from his past to show problematic thinking and openly discusses how his biases might be affecting his conclusions and what he's done to try to avoid that.

* The ideas. OH MY GOSH THE IDEAS. In no particular order:

- grounding words and theories in expected differences of experience, so that your words mean something and aren't just floating terms disconnected from anything real. [This was my favorite because it resolved a dilemma I'd been having]

- how something can feel like an explanation without concretely being one

- why it matters if you play the lottery

- the reasons political arguments so frequently refuse to admit any downside to proposed policies

- the way we fail to properly take into account new evidence, avoid our belief's real weak points, don't look hard enough for counter-evidence, and look harder for evidence against something if we don't want to believe it than if we do

- the way thinking people like to talk about reasons to believe something that aren't our true reasons for believing it

- how our thinking is influenced by fictional accounts

- the importance of talking about problems before trying to come up with solutions

- and then once you start proposing solutions, don't stop at the first likely-sounding one, but consider as many as possible (at least if the issue is important enough to warrant the time)

- experiments that show the effect of unbounded and bounded scales in human estimates, and the application of this to jury-awarded penalties

- what conformity and dissent really look like, the experiments related to them, and what the role of each should be

- how hard it is to say oops, you made a mistake, and change your mind—and how reminding yourself that you're already living with reality can help make that easier

- evolution as a stupid optimization process that does not necessarily result in the kind of things that humans (especially nice humans) desire ... an example being that researchers tried to make insects self-control their own reproduction by eliminating excessive numbers, only to end up with insects that cannibalized their neighbors nests instead of controlling their own breeding

- what to do when arguments devolve into arguing about definitions

- why it feels like we need an answer to the question "does a tree falling in the forest make a sound if no one hears it?", even if we agree on what actually happens in the forest

- the mind projection fallacy and why sci-fi aliens try to kidnap beautiful women [I've found multiple applications for this in the last couple months, everything from the philosophy of aesthetics to online discussions of homosexuality to my husband's view of life in general... and pretty sure I caught myself doing it a few times, too]

- the danger of knowing about cognitive biases, because then they become more fuel with which you destroy others' arguments but don't become tools that you apply to your own thinking

- taking joy in the "merely real" and seeing the poetry of things as they are, without needing extra mysticism added on [This part occasionally rang hollow to me... not because I thought he was wrong about enjoying reality as it is or the beauty in science and curiosity, but because I felt he didn't quite "get" poets and the different way their brains work - their different goals, as it were.]

- reductionism and how to understand consciousness in a world "merely" made of fundamental particles; he discusses the Descartian "ghost in the machine", supernaturalism, and zombie consciousness theory along the way

- a sort of overview of what quantum physics is doing, focusing on quantum collapse vs. multi-worlds ... and why we should believe the latter

- the model of Traditional Science, where you can believe any theory you want as long as you can come up with a prediction and then test it ... vs. the model of Bayes, where you use every scrap of evidence at hand, worked into the hard math of Bayes' probability theorem if possible, to drag yourself as close as possible to the truth

- why we would want to be good people even if reductionism is true and there is no "ultimate" morality being beamed at us from outside the universe

- experiments that show humans don't pay attention to absolute numbers (scope insensitivity), and how this should be changed when thinking about charitable giving

- what steps rationalist atheists might take that could match and surpass the community-building effects of religion in general and the charity-giving of the Catholic Church in particular

I was already familiar with a lot of this material from LessWrong. However, it is very nice to have it organized in a reasonably coherent linear way; among other things, it makes keeping track of my progress much easier.

Yudkowsky is very good at explaining complicated ideas in an approachable way. While I haven't finished the book yet (it's quite long), if I had to sum up what I've absorbed so far, I would say: "The only way to understand the way the world *really is* is to consider the probabilities of the various possible ways the world *could be*, based on the available evidence. This isn't an esoteric concept to be used only by white-coated scientists in labs - we all do it, all the time. And we do it wrong, because our brains are built to help us survive, not necessarily to provide us with accurate understanding. But we can learn to do it better, and this is how."

Most (all?) of magisterial is available on Eliezer's website, but I was enjoying it so much that I bought the book to support him. This ebook is MASSIVE, unbelievably huge, and provided me with weeks of engaged reading. So many topics are covered that it would be silly to try and summarize. Eliezer has a great writing style and manages to draw out the simple lessons from even the most complex logical or ethical problem. His work changed my life, it really did. It made me a more thorough philosopher and a critic of my own thoughts. I'd like to think that it made me a better person. I cannot recommend this book enough.

This is a huge project. I am nibbling at the book, chapter by chapter. Fortunately, it is written to be read in exactly this fashion.

Nonetheless, I can say after having read only a quarter of it that it is an excellent book. I have little to add to the many, many positive reviews.

If you've studied a bit of symbolic logic and boolean algebra, then you might think know all you need to know about rational thinking. You couldn't be more wrong. Astonishing advances in decision theory, psychology, and neuroscience have revealed subtle, but important, flaws in the way we all go about trying to make sense of the world. "Rationality: From AI to Zombies" is a collection of short essays by the author, explaining what it really means to think rationally and how to make the best decisions, given limited information. Far from being dry and academic, the author discusses everything from cryogenics to artificial intelligence with humor and eloquence. Most of the essays are non-technical. Some of them get into the mathematics of decision theory, which make them quite challenging, but you can safely skip over those parts if you don't have the necessary background. I will warn you... although the individual essays are very short, the book as a whole is very, very (even unbelievably) long -- I'm still not even half way through it. Even so, I highly recommend it.

Humans are pretty terrible at thinking, and we're especially bad at judging just how bad we are at it.

This book presents a huge list of failure modes and cognitive biases - like optical illusions, only for critical thinking.

The wonderful and horrible part is that, just as with optical illusions, seeing the trick doesn't make you immune to it. You can understand precisely why and how you get it wrong, and watch yourself continue to make the same mistakes. It's glorious.

There is nothing to say about this book that will do it justice. Let this be just another piece of anecdotal evidence you consider, and hopefully the multitude of similar reviews will be enough to convince you.

I read this as a recommendation of Sebastian Marshall, an online strategy blogger. This is a summarized version of the Sequences at LessWrong.com. It makes for great reading, but some of this is read better on the website. I find that I am using more of the ideas on Bayesian thought in my daily life. I think that anyone who thinks themselves rational should read this text and then re-evaluate.