Customer Reviews

Expert Political Judgment: How Good Is It? How Can We Know?

5 star:		(14)
4 star:		(2)
3 star:		(0)
2 star:		(1)
1 star:		(2)

 

 

Tetlock shows conclusively two key points: First, the best experts in making political estimates and forecasts are no more accurate than fairly simple mathematical models of their estimative processes. This is yet another confirmation of what Robyn Dawes termed "the robust beauty of simple linear models." The inability of human experts to out-perform models based on their expertise has been demonstrated in over one hundred fields of expertise over fifty years of research; one of the most robust findings in social science. Political experts are no exception.
Secondly, Tetlock demonstrates that experts who know something about a number of related topics (foxes) predict better than experts who know a great deal about one thing (hedgehogs). Generalist knowledge adds to accuracy.
Tetlock's survey of this research is clear, crisp, and compelling. His work has direct application to world affairs. For example he is presenting his findings to a conference of Intelligence Community leaders next week (Jan 2007) at the invitation of the Director of National Intelligence.
"Expert Political Judgment" is recommended to anyone who depends on political experts, which is pretty much all of us. Tetlock helps the non-experts to know more about what the experts know, how they know it, and how much good it does them in making predictions.

This book is a rather dry description of good research into the forecasting abilities of people who are regarded as political experts. It is unusually fair and unbiased.
His most important finding about what distinguishes the worst from the not-so-bad is that those on the hedgehog end of Isaiah Berlin's spectrum (who derive predictions from a single grand vision) are wrong more often than those near the fox end (who use many different ideas). He convinced me that that finding is approximately right, but leaves me with questions.
Does the correlation persist at the fox end of the spectrum, or do the most fox-like subjects show some diminished accuracy?
How do we reconcile his evidence that humans with more complex thinking do better than simplistic humans, but simple autoregressive models beat all humans? That seems to suggest there's something imperfect in using the hedgehog-fox spectrum. Maybe a better spectrum would use evidence on how much data influences their worldviews?
Another interesting finding is that optimists tend to be more accurate than pessimists. I'd like to know how broad a set of domains this applies to. It certainly doesn't apply to predicting software shipment dates. Does it apply mainly to domains where experts depend on media attention?
To what extent can different ways of selecting experts change the results? Tetlock probably chose subjects that resemble those who most people regard as experts, but there must be ways of selecting experts which produce better forecasts. It seems unlikely they can match <a href="http://hanson.gmu.edu/ideafutures.html">prediction markets</a>, but there are situations where we probably can't avoid relying on experts.
He doesn't document his results as thoroughly as I would like (even though he's thorough enough to be tedious in places):
I can't find his definition of extremists. Is it those who predict the most change from the status quo? Or the farthest from the average forecast?
His description of how he measured the hedgehog-fox spectrum has a good deal of quantitative evidence, but not quite enough for me check where I would be on that spectrum.
How does he produce a numerical timeseries for his autoregressive models? It's not hard to guess for inflation, but for the end of apartheid I'm rather uncertain.
Here's one quote that says a lot about his results:

Beyond a stark minimum, subject matter expertise in world politics translates less into forecasting accuracy than it does into overconfidence

As both a consultant and an investment manager I have spent a lot of years studying decision theory. One limitation in a lot of the work I encountered was its heavy reliance on lab studies using students. You were never quite sure if the conclusions applied in the "real world." This outstanding book puts that concern to rest. It is by far the richest body of evidence I have encountered on decision making in real world situations. Anybody interested in decision making and decision theory will profit from reading it.

Judging the possibility of future outcomes depends less on what a person knows (i.e. expertise), and more on how a person thinks. Best able to judge the future are those who see the world in fluid terms, embrace complexity and nuance, empathize with all sides of an argument, and seek out different opinions. Least able are those who see the world in black-and-white terms, tend toward dogma, hold to an idealistic view, and shut out others' opinions. Education and being up-to-date on current events were also required, but flexible thinking was the biggest variable. The book is extremely quantitative and deeply researched, but it is needlessly verbose in many places. Politics is the focus domain, but the principles extrapolate to other fields. Interestingly, the single most accurate predictor of the future was the autoregressive distributed lag function that the author used as a control, thus unintentionally showing the value of quantitative analytics.

"The fox knows many things but the hedgehog knows one big thing." - Isaiah Berlin, The Hedgehog and the Fox.

Tetlock uses the Fox-Hedgehog thinking style framework from Berlin's classic essay to distinguish between top-down ideological Hedgehogs and the far more circumspect and integrative Foxes. After a long term study of predictive ability in the political sphere - Tetlock clearly shows the superiority of Foxes over Hedgehogs (more vulnerable to cognitive biases).

However, worryingly for experts/pundits/consultants everywhere, the study also reveals the inability of humans to outperform statistical and base rate extrapolation algorithms in the predictive arena in any complex process with stochastic elements (politics/finance/economics...)

Read the New Yorker review at http://www.newyorker.com/critics/books/articles/051205crbo_books1#top

This is an important book for it gives us an insight in how to evaluate the thousands of experts who are continually bombarding us with their predictions. Tetlock chooses the difficult and murky area of political judgment, and on it centers his analysis, though his basic conclusions relate to forecasting in other areas, such as business and finance. Roughly he takes Isaiah Berlin's distinction between the hedgegog who would know one big thing, and the fox, who knows many little things as basis of his analysis. As he sees it the Hedgehogs who base themselves on big theories are invariably wrong, while the foxes who tend be more open to the actual play of reality, have a far better record.
As he understands it the Hedgehogs go overboard in making Boom or Bust predictions. He provides empircal studies data to show how they are most often wrong, even more wrong by the way when they are predicting Disaster. Those who qualify their predictions the uses of 'perhaps' and 'however' and 'nonetheless' and 'possibly' have a far better chance of getting it right.
The irony of this however is that it is precisely the Hedgehogs who are rewarded, and receive greatest Media attention. They are never punished for being wrong, for few seem to follow and check on the accuracy of the prediction. The more accurate qualified assessments are given, on the other hand, more scanty space and attention. After all when we are uncertain about the future who wants to hear a prediction which itself says it is uncertain.
This is a very instructive book, although I wish at times its systems of classification were a bit less awkward.
On the whole though this is a highly recommended and important work, which can be of real help to most of us in understanding how to separate the ' wheat ' from the 'bull'.

Why isn't this book on the front page of every newspaper everywhere? The author makes a cogent argument that informed amateurs are as good as "experts" in seriously important fields from investing to politics. He provides experimental evidence that this is the case. This is the hypothesis that a bunch of farmers and merchants tested when they sat down and organized the United States of America. This is the hypothesis that every smart investor proves when he meets or beats hedge funds spending millions on experts and computer power. A very empowering book. Well-written, well-argued, well-referenced. I dare you to read this one and not give it to a friend.

If you want to find out what makes a forecaster a real expert or a lucky guesser, this book explains the complicated set of necessary talents. Author Philip E. Tetlock is a researcher and political psychologist. He tracks a wide academic path into psychological investigations about predicting the future - in business, politics or other arenas - and the implications of its results. He finds some surprises, especially in the study of objectivity and how people think. He explains psychological experiments on forecasting, and uses them as a trail through tangles of complex research. As you climb, enjoy the occasional clearings where some great ideas (such as Amos Twersky's "Support Theory") come to light. We find Tetlock's insights worth the journey. Despite its sometimes dense thickets, this book is necessary for people who want to understand the role of self-described "expert" prognosticators. If you wonder why the predictions of political, media and sports forecasters often are not worth heeding, Tetlock shows you how to distill the best from the rest. We recommend this book to journalists, political scientists and managers or executives who rely on "expert" opinions or futuristic scenarios.

...can't afford to do without this book. It is scary to think that many people will be writing PhDs in the Social Sciences, and then be called upon to make or influence policy without being familiar with this book's central arguments and evidence.