Customer Reviews

The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty

5 star:		(22)
4 star:		(8)
3 star:		(7)
2 star:		(4)
1 star:		(4)

 

 

So, let me start by saying that I think this book had a ton of potential. I couldn't put it down after reading the first several chapters. The author is witty, clearly very intelligent and has thought a lot about his subject matter. He clearly describes the problem(s) through the use of clever analogies. My favorite is his comparison of using averages to that of a drunk walking down the highway. On average, the drunk's path is along the center line - but on average, he is dead. But, the book in my opinion has some serious flaws.

The essence of the book is that you should not use an average number in your predictions / forecasts / etc. Rather, you should use a distribution (e.g., simulations) instead. More specifically, he recommends the use of "Probability Management" which is his brainchild - basically boiling down to sharing probability distributions that are "certified" by designated specialists in the organization. This ensures that your work takes interrelationships into account, whereas separate simulations might miss the boat.

The main problem with the book, in my opinion, is that he talks far too much about the problem and far too little about the solution. He spends chapter after chapter talking about where the problem exists (or previously did exist), but doesn't give much in the way of details for the alternative. Even when the alternative is discussed, it is rarely in enough detail to glean any real kind of information about the solution to the problem other than a cursory overview.

The book also includes a lot of superfluous chapters that don't seem to fit with the book. For example, chapter 35 is about World War II statisticians (one in particular) that used a clever trick to figure out how many German tanks were created based on those captured by the US Army. While the chapter was interesting, I'm not exactly sure why it was in the book - maybe leading up to the next chapter regarding the war on terror? In the end, it seemed to me that there was too much fluff.

I would have given the book a lower rating, but the author was too funny and interesting to go that far. Also, I think it depends on the type of person you are - I'm a doer, a self-taught programmer and I like solving problems and implementing solutions. A different personality may have enjoyed the book differently and not minded the things that I did.

Sam Savage has written a book that reveals what is behind a simple flaw in so many management decisions. This book leads the manager who is used to accounting-style, point-estimate thinking to the world of thinking with probabilities. His writing style is light (sometimes even funny) but the content is meaty.

Savage criticizes what he calls "steam era" concepts of statistics which most stats courses seem stuck in and introduces decision making under uncertainty in a way that is much more welcoming than most books on this topic. I suspect that if more people had a professor like Sam Savage as their first mentor on statistics, there would be far fewer people with bad memories of that course.

His approach is all about avoiding intimidating terminology and getting hung up on esoteric concepts. In particular, he explains the concepts of Monte Carlo simulations in a way that might just get the reader excited about the power of the tool. He is not only an expert in MC simulations himself (he has developed many new innovations in the method) but is also an expert in how to explain it to a wide audience.

This is a book written for laymen with enough interesting insights to engage even the most scholarly professional.

As other reviewers have noted, this is an exceptional account of what you need to know about statistics, without any of the boring or intimidating stuff people like to layer on. No combinatorics, no measure theory, no calculus. It's clear and entertaining, with helpful links to simulations and animations on the web. The illustrations are amusing and useful, the overall production quality is significantly better than similar books. The writing is skillful and lively.

Experts (and I consider myself one) will learn some new things and, more important, learn effective ways to explain things they already know. Novices will learn what they need, and sharpen their thinking skills. People in between will unlearn a lot of nonsense, and replace it with good stuff, and get the confidence to ignore self-proclaimed experts with dense jargon and impenetrable formulas. Key concepts are reduced to easy-to-remember "mindles." There are examples from most areas of finance, including some quite advanced, and business; with a few from other fields. What more could you want?

Well, one more thing, but it's impossible. The author is the son of Jimmie Savage, and I consider his The Foundations of Statistics one of the great accomplishments of human thought. It was his intellectual precision and genius, and those of a few other people, that allows statistics to be made simple. Before that work, people were impossibly confused about the basics. It would have been nice to see that acknowledged instead of ridiculed.

However, I realize so many people are traumatized and intimidated by statistics that it takes a little iconoclasm to motivate them. But why did it have to be from Jimmy Savage's son? Why not someone whose parents were killed by an overmathematical analysis that overruled common sense?

The sections on finance, my specialty, are quite deep. His explanations of options, portfolio management and risk are excellent. It reads at the level of Kiplinger's Personal Finance magazine, but it makes the points of more intimidating authors such as Benoit Mandelbrot, Nassim Taleb and Kent Osband. His accounts of business management issues a bit more superficial, but still excellent.

I do have a few specific gripes, that will bother no one but nerds. He uses "Monte Carlo simulation" to mean simulation of a random event. This is a near-universal error. Monte Carlo means creating randomness that doesn't exist to get a deterministic result. It matters because anyone can simulate a random event, and it's an obvious idea. Monte Carlo can wonderful, unexpected answers to seemingly intractable questions, but it requires a lot of precise mathematics to do correctly.

Another gripe is he defines "Value-at-Risk" (VaR) as just a percentile. There is much more to VaR. For example, he estimates the distribution of return on movie investments by resampling from 28 past movies, which includes one blockbuster. Someone familar with VaR would realize that one observation is not enough to reliably estimate either the probability or potential size of blockbusters; and those factors are very important to the result. So she would resample among the other 27 movies, and call that the distribution inside the 95% VaR limit. To estimate what might happen outside the VaR limit, she would look at a much longer record of movies to get enough observations of blockbusters. This is necessarily a judgmental process (it's called "stress testing") because she's bringing in less comparable data. The end result is a range of profits where normal conditions apply and you can make reliable probability forecasts; you optimize in this range; and a much larger range where you have only qualitative guesses about probabilities; you create plans to maximize probability of survival in these ranges.

I cannot think of a person who will not benefit significantly from this book. It won't make people forget his father (thankfully) but it's one of the few books worthy to be on the same shelf.

Unusual style, intermediate between a serious attempt to explain a simple yet important statistical idea in plain words and graphics, and the frothy "management secrets" style that permeates the Business section of an airport bookstore, the latter reflected in breezy writing, a layout of short chapters containing short sections, and the admirably inexpensive price. The idea can be said in 5 words -- "don't put averages into spreadsheets" -- where averages refer to expectations about the future. A typical illustration is what Operations Research calls the Newsboy Problem, which supposes you buy the daily paper wholesale, sell retail, but are stuck with any unsold copies. If you sold exactly 100 copies every day then you would simply order 100 copies; if your sales vary but average 100 a day, then the best number to order is (typically) not exactly 100, because of asymmetry between profit on sold copies and opportunity cost of lost sales if you run out. In such simple settings the idea is rather obvious, but now imagine some business spreadsheet, with actual figures from last year which combine in some complicated way to get last year's actual profit. It's natural to try to estimate next year's profit by entering estimates of each line entry and having the spreadsheet recalculate; but in any complicated setting there are always asymmetries between upside and downside and one does not get the correct "average profit" so easily. And one is interested not only in averages but also in risks of large losses.

This is no mystery to statisticians, just one of 100 workaday concepts not particularly emphasized in a typical college statistics course -- though as the book implies, worthy of more emphasis in a business statistics course. Scattered through the book are the components of a potentially great 30-page chapter illuminating this idea via graphical ideas of representing distributions as histograms, and describing contexts where asymmetries are important. For instance: bonuses for meeting a target, inventory and supply-demand, options, a project takes as long as its longest component, and interesting analogs with portfolio theory.

It would have been valuable to extend these 30 pages to either (i) in depth case studies, or (ii) a wide-ranging book explaining 10 different statistics ideas in 10 chapters. But instead, this book becomes a mashup of historical and personal anecdotes, superficial descriptions of case studies, some of the "popular science" style topics in probability and statistics covered in many other books, and witticisms about traditional mathematical statistics as "Steam Era statistics". As the author implies, it would be useful if Excel and open source analogs had an option for entering distributions instead of numbers. Alas, this book turns into a plug for the author's Probability Management software, and case studies turn into one paragraph testimonials.

I should begin by saying that I am a professional mathematician and also a professional money manager, so I know a lot about uses and abuses of statistics. The author starts of with a triviality (one number is not enough to describe the world), and then goes off the deep end: he says that people should think in terms of distributions, but without understanding the underlying mathematics -- this is a recipe for disaster, as the recent financial crisis has (yet again) demonstrated: people used probability distributions to model the pricing of certain assets, conveniently forgetting that this model had a certain range of validity. Those of us who know probability theory also know that the Central Limit Theorem is called that because it only describes the CENTER of the probability distribution (of a sum of identical independent random variables -- never even mind that in real world nothing is identitical or independent), and there is a whole'nother area of probability to model the tails ("large deviation theory"). Actually deriving these distributions is also highly nontrivial, and they generally should not be trusted (because to make deriving them possible at all, one has to make simplifying assumptions, which may or may not be appropriate). The author also suggests Monte Carlo simulation as the silver bullet -- I have never heard anything so stupid (yes, it has its place, but you REALLY need to know what you are doing).

In any case, the author's prose shouts "management consultant", and the book is a collection of marketing materials for him and colleagues (many of whom appear to be in the business of selling Excel add-ons). If you want to be able to reason about the world, think, and learn some mathematics (not for the facts as much as to learn how to think). The author dismisses mathematical reasoning as "steamship statistics", or some such, but this is disingenuous: anybody who believes this is doomed to depending on shysters, er, management consultants for the rest of their days.

A huge disappointment.

I found this to be an easy, delightful read. I think the book is best directed to business professionals or decision-makers with a firm-grasp of MBA-like coursework who need to rethink some aspect of their decision-making. For me, that means some rethinking how I make sales forecasts & demographic projections ... and perhaps more importantly, how I communicate those forecasts to broader audiences of decision-makers. That is no easy task, and the book gave me some direction on how to help my clients.

However, I have two major problems with the book (or, perhaps more correctly, this approach). My first issue. Monte Carlo simulations are not within the grasp of most business decision-makers -- conceptually, time-wise, tool-wise. That means the simulations are left up to specialists, who frequently are removed from the business or are far too young to make seasoned judgments. That formula is just as bad (if not more so) as forecasting point estimates. My second issue. These simulations depend upon specifying distributions, which are frequently assumptions themselves. Frankly, I would rather prep a range of different forecasts with point estimates that my clients & I understand & grasp, than prep a forecast based on distributional assumptions that are WAGs (wild a** guesses). The distributions are one of the black-boxes in this approach, and one of inputs most likely to be misunderstood. When you don't know the variability and its distributional form, it's GIGO. Forecasting points you know, understand & can communicate is better than forecasting distributions that are unknown, guessed, or too uncertain.

The approach is not the holy grail, and it can be dangerous in the wrong hands. But this book can inform & enlighten. I should be a must-read for any quantitatively oriented MBA. Put it on your reading list.

4*.

In The Flaw of Averages, Sam exposes the epic failure of using single number value averages in models. His cure is an invention called probability distribution strings, where thousands of numbers populate a single cell, to create visual, interactive simulations, which can inform business decisions with unprecedented clarity.

This book is a fun read, and written in everyday language, so even if you're not a quant, you will still benefit from Sam's genius.

This book is difficult to pigeonhole, but I would describe it as an insightful, somewhat irreverent examination of risk and risk analysis from a "right-brain" perspective, rather than a statistics book for non-statisticians. (The irreverent part stems from the author's disregard for what he refers to as "Steam Era Statistics.") Rather than starting with statistical theory, equations, proofs, and Greek-alphabet soup, the author uses clever and engaging examples to clarify the principles.

If there is an "organizing principle" for the book, I believe the author sums it up in this statement: "My position is that decisions are made using the seat of the intellect at one extreme and the seat of the pants at the other and that the best decisions are those upon which both extremes agree."

Speaking from the standpoint of one who has a personal and professional interest in making abstruse mathematical and statistical concepts understandable to "right-brain thinkers," I am happy to report that this book lived up to my expectations.

A thoughtful look into how business management and how to overcome the field's bias in its decision making process of discounting risk. Sam Savage constructs what amounts to a resource textbook for the masses . It is written intuitively, but rather clumsily. Savage, despite trying hard to frame himself as a common man, cannot hide the fact that he has been in academia for too long and unable to mask his inability to transcribe in `commonspeak.' Savage's inability to express complex ideas in a clear concise manner led to a patchwork of watered down examples broken into non-flowing parts that lacked much order or improved even slightly the explanation power resulted in me feeling lost throughout the text. The book is a textbook-lite which fails miserably. He should have just went all in and made a probability management textbook.

The book's strongest point is its use of the internet. Rather than footnoting or appendixing particular points or thoughts, he utilizes interactive online pages on his website. The books true value comes from the connecting readers with several powerful probability management programs that are also free, such as solver and Analytica. I have downloaded solver since and find its potential uses rather numerous, though to be honest I was exposed to solver in a limited context during graduate school. Additionally, the book recommends reading of several books that I intend in the future to read, such as Peter Bernstein's `Capital Ideas' and Roger Lowenstein's `When Genius Failed.'

The book spent a considerable length of time explaining the difficulties that arise when managers use one single number, usually an average, but sometime can be a median, within their budget forecasts and decision making process. This is problematic, Savage argues, as, in the example of the drunk who on average is walking within the center of the road but spends most of his time walking towards incoming traffic or away from outgoing traffic, this does not really provide an accurate view of the true uncertain trend . Instead managers should use simulations that account directly for uncertainty using statistical principles, minus the esoteric language and terms, and chart and graphs that clearly display the range of possibilities of outcomes given a decision. I agree with this assessment.

Savage does not fail to include an important caveat. That probability and its universe of uncertainty and risk analyzing techniques should not preclude genuine understanding of risk, rather than as a tool to post-hoc protection of liability if things go sour. The most recent economic crisis and resulting recession is a derivative of not modern portfolio theory failing but rather the people who ignored the very basis of modern theory of accounting for risk and not as merely representing some abstract and uninterruptable number. Other great bits of knowledge include "The computer should be used to chronicle your thoughts, rather than form them" and "the best models are those you no longer need because they have changed the way you think."

The week used reading this book was well spent, and I feel I've walked away with an added decision making skill-set that I did not have before, nor would have gotten until perhaps years of first hand trial and error. Though I don't feel I have really learned anything new, what I currently know has been reframed, which I believe was how Savage desired anyone who read his book to leave. In this regard , Savage's writing style was a success. Again, there is just feeling that there could have been much clearer and concise treatment of the topic of probability management than Savage's attempt in this textbook-lite.

"Just give me the number, willya?" Ugh. I've had the question asked many times in my life working in or alongside financial reporting in a company. They need a number for the budget, even though that number will certainly be wrong, leading to numerous explanations for why we are mistracking the budget.

The truth is though there will be one result for the question you ask prospectively, hypothetical answers will often systematically mis-estimate the result when average inputs are fed into models.

As I have experienced in the insurance industry many times, good companies accept feedback from claim experience into new product pricing, and consider the potential downside risks. Bad companies rely on industry tables (averages), and assume that downside deviations are just random.

Would we manage companies better if we shared data on how uncertain our estimates are? Certainly, but the whole company would have to be geared toward understanding how to deal with risk and uncertainty.

Often there is option-like behavior in companies, where if sales are low, expenses will be cut back to a baseline level, but if they are high, expenses will run. Average expense numbers rarely express the likely result.

Most people/companies assume that things will be stable. If we look at projections of investment results, stability is the norm. But our world is unstable. There are booms and busts; there are wars. Plans lose their validity when the real world appears.

"The Flaw of Averages" is a popular book on statistics. It points out many ways in which statistics are abused. This book will make you a skeptical and reasoned consumer of statistics.

Quibble: My main difficultly with the book, is that much as the author tries to simplify the concept of complex simulations, is that in the ninth part of the book, he seems to overly encourage use of his own software.

Aside from that, the average reader will learn many ways that statistics such as averages can deceive. As Benjamin Disraeli, once said, "There are three kinds of lies: lies, damned lies, and statistics." This book will help you avoid the last sort of lie. I recommend this book.