The Signal and the Noise: Why So Many Predictions Fail-But Some Don't 1st Edition

This book was first published in 2012, at a time when Big Data (or if you prefer, big data) was only beginning to receive the attention it deserves as a better way to use analytics within and beyond the business world. One key point is that big data should also be right data and in sufficient quantity. I recently re-read the book, in its paperbound edition. Thde quality and value of its insights have held up remarkably well.

In the years that followed publication of the first edition, as Nate Silver notes in the new Preface, the perception that statisticians are soothsayers was proven to be an exaggeration, at best, and a dangerous assumption, at worst. This new edition "makes some recommendations but they are philosophical as much as technical. Once we're getting the big stuff right -- coming to a better [i.e. more accurate and more reliable] understanding of probability and uncertainty; learning to recognize our biases; appreciating the value of diversity, incentives, and experimentation -- we'll have the luxury of worrying about the finer points of technique."

In the Introduction to the First Edition, Silver observes, "If there is one thing that defines Americans -- one thing that makes us exceptional -- it is our belief in Cassius' idea that we are in control of our own fates." In t his instance, Silver refers to a passage in Shakespeare's play, Julius Caesar, when Cassius observes:

"Men at some time are masters of their fates.
The fault, dear Brutus, is not in our stars,
But in ourselves, that we are underlings."
(Act 1, Scene 2, Lines 146-148)

Cassius' assertion has serious implications and significant consequences. It is directly relevant to a theory named after Reverend Thomas Bayes (1701–1761), who first provided an equation that allows new evidence to update beliefs in his An Essay towards solving a Problem in the Doctrine of Chances (1763). Silver: "Bayes's theorem is nominally a mathematical formula. But it is really much more than that. It implies that we must think differently about our ideas [predictions, for example] -- and how to test them. We must become more comfortable with probability and uncertainty. We must think more carefully about the assumptions and beliefs that we bring to a problem."

Silver cites another passage in Julius Caesar when Cicero warns Caesar: "Men may construe things, after their fashion / Clean from the purpose of things themselves." According to Silver, man perceives information selectively, subjectively, "and without much self-regard for the distortions this causes. We think we want information when we want knowledge." I take "want" to have a double meaning: lack and desire. Silver goes on to suggest, "the signal is the truth. The noise is what distracts us from the truth. This is a book about the signal and the noise...We may focus on those signals that advance our preferred theory about the world, or might imply a more optimistic outcome. Or we may simply focus on the ones that fit with bureaucratic protocol, like the doctrine that sabotage rather than an air attack was the more likely threat to Pearl Harbor."

In their review of the book for The New Yorker (January 25, 2013), Gary Marcus and Ernest Davis observe: "Switching to a Bayesian method of evaluating statistics will not fix the underlying problems; cleaning up science requires changes to the way in which scientific research is done and evaluated, not just a new formula." That is, we need to think about how we think so that we can make better decisions.

In Thinking, Fast and Slow, Daniel Kahneman explains how an easy question ("How coherent is the narrative of a given situation?") is often substituted for a more difficult one ("How probable is it?"). And this, according to Kahneman, is the source of many of the biases that infect our thinking. Kahneman and Tversky's System 1 jumps to an intuitive conclusion based on a “heuristic” — an easy but imperfect way of answering hard questions — and System 2 lazily endorses this heuristic answer without bothering to scrutinize whether it is logical). And this, according to Kahneman, is the source of many of the biases that infect our thinking. System 1 jumps to an intuitive conclusion based on a “heuristic” — an easy but imperfect way of answering hard questions — and System 2 lazily endorses this heuristic answer without bothering to scrutinize whether it is logical.

When an unprecedented disaster occurs, some people may feel at least some doubt that they are in control of their fate. Nate Silver offers this reminder: "But our bias is to think we are better at prediction than we really are. The first twelve months of the new millennium have been rough, with one unpredicted disaster after another. May we arise from the ashes of these beaten but not bowed, a little more modest about our forecasting abilities, and a little less likely to repeat our mistakes."

A Jewish proverb suggests that man plans and then God laughs. The same could be said of man's predictions.

This is the best general-readership book on applied statistics that I've read. Short review: if you're interested in science, economics, or prediction: read it. It's full of interesting cases, builds intuition, and is a readable example of Bayesian thinking.

Longer review: I'm an applied business researcher and that means my job is to deliver quality forecasts: to make them, persuade people of them, and live by the results they bring. Silver's new book offers a wealth of insight for many different audiences. It will help you to develop intuition for the kinds of predictions that are possible, that are not so possible, where they may go wrong, and how to avoid some common pitfalls.

The core concept is this: prediction is a vital part of science, of business, of politics, of pretty much everything we do. But we're not very good at it, and fall prey to cognitive biases and other systemic problems such as information overload that make things worse. However, we are simultaneously learning more about how such things occur and that knowledge can be used to make predictions better -- and to improve our models in science, politics, business, medicine, and so many other areas.

The book presents real-world experience and critical reflection on what happens to research in social contexts. Data-driven models with inadequate theory can lead to terrible inferences. For example, on p. 162: "What happens in systems with noisy data and underdeveloped theory - like earthquake prediction and parts of economic and political science - is a two-step process. First, people start to mistake the noise for a signal. Second, this noise pollutes journals, blogs, and news accounts with false alarms, undermining good science and setting back our ability to understand how the system really works." This is the kind of insight that every good practitioner acquires through hard-won battles, and continues to wrestle every day both in doing work and in communicating it to others.

It is both readable and technically accurate: it presents just enough model details yet avoids being formula-heavy. Statisticians will be able to reproduce models similar to the ones he discusses, but general readers will not be left out: the material is clear and applicable. Scholars of all stripes will appreciate the copious notes and citations, 56 pages of notes and another 20 pages of index, which detail the many sources. It is also important to note that this is perhaps the best general readership book from a Bayesian perspective -- a viewpoint that is overdue for readable exposition.

The models cover a diversity of areas from baseball to politics, from earthquakes to finance, from climate science to chess. Of course this makes the book fascinating to generalists, geeks, and breadth thinkers, but perhaps more importantly, I think it serves well to develop reusable intuition across domains. And, for those of us who practice such things professionally, to bring stories and examples that we can tell and use to illustrate concepts with the people we inform.

There are three audiences who might not appreciate the book as much. First are students looking for a how-to book. Silver provides a lot of pointers and examples, but does not get into nuts and bolts details or supply foundational technical instruction. That requires coursework in research methods and and statistics. Second, his approach to doing multiple models and interpreting them humbly will not satisfy those who promote a naive, gee-whiz, "look how great these new methods are" approach to research. But then, that's not a problem; it's a good thing. The third non-fitting audience will be experts who desire depth in one of the book's many topic areas; it's not a technical treatise for them and I can confidently predict grumbling in some quarters. Overall, those three audiences are small, which happily leaves the rest of us to enjoy the book.

What would make it better? As a pro, I'd like a little more depth (of course). It emphasizes games a little too much for my taste. And a clearer prescriptive framework could be nice (but also could be a problem for reasons he illustrates). But those are minor points; it hits its target better than any other such book I know.

Conclusion: if you're interested in scientific or statistical forecasting, either as a professional or layperson, or if you simply enjoy general science books, get it. Cheers!