The Wisdom of Crowds Paperback – August 16, 2005

Surowiecki's The Wisdom of Crowds documents and analyzes an extremely important phenomenon. When people guess at a question to which nearly no one knows the answer but most people can make a sensible guess (e.g., what proportion of the world's airports are in the USA; how many marbles can fit into a box that is a meter on each side) the average of a large group is nearly always more accurate than the guess of any member of that group. Moreover, the more people involved, the more accurate the average is.
This phenomenon was first discovered by Francis Galton, Charles Darwin's first cousin. Throughout his life, Galton was obsessed with measurement and categorization. His study of fingerprints led to their use by the police to identify criminals. His study of twins revealed that biological heredity determines intelligence and temperament. He also worked out the coefficient of correlation, which is a basis of modern statistics. In 1906, when Galton was eighty-five and still as inquisitive as ever, he visited a country fair. One of the events was a contest to try to guess what the weight of an ox, which was on display, would be after it had been killed and dressed. In order to enter the contest, a person had to pay sixpence and write his guess, along with his name and address, on a piece of paper. After the contest was over, Galton borrowed the papers with the guesses. There were 787 papers in total. To his amazement, the average guess was only one pound less that the actual weight (1,198 pounds). That was closer than any individual guess. Yet, some of the participants in the contest were butchers and cattle raisers, who would be expected to do much better than the average.
This phenomenon also applies to predictions of future events. That is why it is nearly impossible to make money consistently by betting on sporting events - because the odds are determined by the average of all bets, which is extremely accurate. The most striking illustration of this phenomenon is the otherwise inexplicable fact that, with very few exceptions, professional stock investors (managers of mutual funds, pension funds, etc.) do worse than the stock market indexes. Professional investors are highly intelligent people, who devote all their time to analyzing stocks and stock market trends; they have specially developed soft-ware and teams of assistants. But their analyses are not as accurate as the average analysis of all investors.
However, for the wisdom of crowds to work, two conditions must be met (pages 10-11, 36-7). One is that the opinions of the individuals involved must be formed and expressed independently of the others. When decisions are made by groups meeting together, the group is often swayed by a consensus that seems to have formed or by one or two of its member who seem to have more expertise, or who express their opinions forcefully. Also, the individuals involved must have some knowledge of what is involved. For example, the average prediction of Indonesian peasants as to which team will win the National Football League championship would be worthless.
In line with the second condition, the wisdom of crowds does not apply to areas of technical expertise. The average guess of a crowd as to the weight of an ox when it has been slaughtered and dressed is more accurate than the estimate of any butcher in the crowd. But the butchers in the crowd are more adept than any non-butchers at carving the ox, storing its meat, and preparing it for sale.
Surowiecki offers a general explanation (pages 10-11) for why the crowds are wiser than any of their individual members and specific reasons (pages 34, 44-50) for why professional stock investors do worse than the stock indexes. I found the latter explanations more convincing than the former.
Surowiecki also discusses (pages 236, 245-6) the obvious objection that the average prices of stocks, houses, and commodities often rise and fall sharply, without regard to the value of the assets involved. Such price swings do not occur with ordinary goods and services, such as televisions or haircuts; nor does a rise in price of ordinary goods and services induce more people to buy them. Average predictions of the results of sporting events or of elections are also immune from irrational price bubbles. In these cases, the accuracy of the predictions is decided unequivocally at a specific time in the near future. That keeps the crowd tethered to reality. But when the prices of stocks, houses, and commodities rise, that means that they can be resold at a higher price; and there is no point at which the bet is definitely over and the bettors have undeniably been proved right or wrong.

Suroweiki engrossed me from the beginning. Though this book appears to be a collection of anecdotes about how crowds often outthink the experts, it struck me as a blueprint for how decision-makers should harness the power of people. Thus it is a treatise on smart business and marketing, good government, and sound organization management.

As a U.S. Army veteran, the author propelled me to thoughts on how the military could use its people's collective wisdom, something on which I have written extensively:Nine Weeks: a teacher's education in Army Basic Training

Among the most relevant claims from the book is this cogent bit of logic:

"To state the obvious, unless people know what the truth is, it's unlikely they'll make the right decisions. This means being honest about performance. It means being honest about what's not happening. It means being honest about expectations. Unfortunately, there's little evidence that this kind of sharing takes place....One of the things that gets in the way of the exchange of real information is the deep-rooted hostility on the part of bosses to opposition from subordinates. This is the real cost of a top-down approach to decision making: it confers the illusion of perfectability upon the decision makers and encourages everyone else simply to play along. What makes this especially damaging is that people in an organization already have a natural inclination to avoid conflict and potential trouble. It's remarkable, in fact, that in an autocratic organization good information ever surfaces.

It's a book that anyone who has been around people should read.

Surowiecki's book focuses on the strengths and weaknesses of collective decision-making. The text covers a large number of issues rather superficially, and it will not satisfy those looking to dive into and master the underlying economic theories. But it makes an important point and can be read quickly.

His thesis is that groups of people can aggregate information to produce surprisingly accurate decisions. The book starts by recounting a famous experiment, in which a 19th century Scottish scientist studied the results of a typical contest where county fair-goers guessed the weight of an ox. He found that, while many guesses were wide of the mark, the mean of the guesses was spot on -- and closer to the ox's actual weight than any individual guess. Apparently this phenomenon holds true for other such contests (how many jellybeans in a jar or how much a pumpkin weighs) -- and this ought to make one stop and think about the power of aggregating information from groups of ordinary people.

Surowiecki relates the power of group thinking to other situations, ranging from the efficacy of market prices to the eery accuracy of the Iowa Electronic Markets (which forecast election results among other things).

He contrasts these examples with the dangers of "group think," the consensual and wrongheaded approach to decision-making that plagues governments and corporations, the frequently poor quality of "expert" forecasts, and the volatility of security prices (he explains the theory of "information cascades" in a clear and easy-to-follow manner).

The moral of the story: "Diversity and independence are important because the best collective decisions are the product of disagreeemant and contest, not consensus or compromise." It strikes me that this bit of wisdom could be usefully applied in government, business, investing, and many other walks of life; hence the value of the book, even if you skim it quickly.