The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives (Economics, Cognition, and Society) [Paperback]

Stephen T. Ziliak (Author), Deirdre N. McCloskey (Author)

Book Description

Publication Date: February 19, 2008 | ISBN-10: 0472050079 | ISBN-13: 978-0472050079

“McCloskey and Ziliak have been pushing this very elementary, very correct, very important argument through several articles over several years and for reasons I cannot fathom it is still resisted. If it takes a book to get it across, I hope this book will do it. It ought to.”

—Thomas Schelling, Distinguished University Professor, School of Public Policy, University of Maryland, and 2005 Nobel Prize Laureate in Economics

 

“With humor, insight, piercing logic and a nod to history, Ziliak and McCloskey show how economists—and other scientists—suffer from a mass delusion about statistical analysis. The quest for statistical significance that pervades science today is a deeply flawed substitute for thoughtful analysis. . . . Yet few participants in the scientific bureaucracy have been willing to admit what Ziliak and McCloskey make clear: the emperor has no clothes.”

—Kenneth Rothman, Professor of Epidemiology, Boston University School of Health

 

The Cult of Statistical Significance shows, field by field, how “statistical significance,” a technique that dominates many sciences, has been a huge mistake. The authors find that researchers in a broad spectrum of fields, from agronomy to zoology, employ “testing” that doesn’t test and “estimating” that doesn’t estimate. The facts will startle the outside reader: how could a group of brilliant scientists wander so far from scientific magnitudes? This study will encourage scientists who want to know how to get the statistical sciences back on track and fulfill their quantitative promise. The book shows for the first time how wide the disaster is, and how bad for science, and it traces the problem to its historical, sociological, and philosophical roots.

 

Stephen T. Ziliak is the author or editor of many articles and two books. He currently lives in Chicago, where he is Professor of Economics at Roosevelt University. Deirdre N. McCloskey, Distinguished Professor of Economics, History, English, and Communication at the University of Illinois at Chicago, is the author of twenty books and three hundred scholarly articles. She has held Guggenheim and National Humanities Fellowships. She is best known for How to Be Human* Though an Economist (University of Michigan Press, 2000) and her most recent book, The Bourgeois Virtues: Ethics for an Age of Commerce (2006).

Editorial Reviews

Review

"[Steve Ziliak and Deirdre McCloskey] explain to us why the misunderstanding of statistical significance has lead to bad government policy making and how one particularly famous brewery employed the technique to improve the pints we enjoy today."
—Tim Harford, BBC

(Tim Harford BBC 20090123)

"Despite appearing to be a book of limited appeal - it is after all a book th (NA London Book Review 20081223)

"Persuading professionals that their procedures are wrong is a long and lonely task. McCloskey, joined later by Ziliak, has been conducting such a crusade against the misuse of significance testing for over 25 years. This book presents their argument, gives lots of examples of the adverse consequences of misuse, and provides some history of the controversy, which dates from the origins of mathematical statistics."
—Ron P. Smith, Journal of Economic Issues

(Ron P. Smith Journal of Economic Issues 20090101)

"The Cult of Statistical Significance has virtues that extend beyond its core message. It is clearly written and should be accessible to those who have neither formal training in statistics nor a desire to secure any. It is full of examples that illustrate why it is the strength of relationships and not their statistical significance that mainly matters."
—Richard Lempert, Law and Social Inquiry

(Richard Lempert Law and Social Inquiry 20090101)

"A clear trade-off: how much confidence [in a result] is "enough" depends on the costs of further research and the benefits of extra precision. Ziliak and his co-author Deirdre McCloskey argue in The Cult of Statistical Significance that most academic disciplines have forgotten this trade-off . . . A sharp line for statistical significance makes no sense, and it has a cost."
—Tim Harford, The Financial Times

(Tim Harford Financial Times 20090207)

"If not Fisherian significance, what should be the Holy Grail of statistics? (Theodore Porter Science 20090605)

"The book is a model of scholarship, transparent in its method, wide-reaching in its disciplinary expertise, and highly literate, including occasional haiku poems and humor such as, 'If the variable doesn't fit/you may not have to acquit.' The authors convincingly argue that environmental quality, jobs, and even lives are at stake."
—M. H. Maier, Glendale Community College, Choice

(M. H. Maier Choice 20091021)

"What is important is a shift of emphasis away from a dichotomous world of true and false towards a recognition of "oomph". This is what the presented book tries to achieve. It is also fun to read, rich with historical information and an excellent reminder of what empirical work of any sort is all about."
—Walter Kramer, Stat Papers

(W. Kramer Stat Papers ) --This text refers to the Hardcover edition.

Product Details

• Paperback: 352 pages
• Publisher: University of Michigan Press (February 19, 2008)
• Language: English
• ISBN-10: 0472050079
• ISBN-13: 978-0472050079
• Product Dimensions: 6 x 0.9 x 9 inches
• Shipping Weight: 15.2 ounces (View shipping rates and policies)
• Average Customer Review: 3.5 out of 5 stars  See all reviews (19 customer reviews)
• Amazon Best Sellers Rank: #216,539 in Books (See Top 100 in Books)
  • #72 in Books > Business & Investing > Economics > Econometrics

Most Helpful Customer Reviews

88 of 90 people found the following review helpful

4.0 out of 5 stars Important work on misuse of statistics by academics May 29, 2008

By David J. Aldous

Format:Hardcover|Amazon Verified Purchase

Tests of statistical significance are a particular tool which is appropriate in particular situations, basically to prevent you from jumping to conclusions based on too little data. Because this topic lends itself to definite rules which can be mechanically implemented, it has been prominently featured in introductory statistics courses and textbooks for 80 years. But according to the principle "if all you have is a hammer, then everything starts to look like a nail", it has become a ritual requirement for academic papers in fields such as economics, psychology and medicine to include tests of significance. As the book argues at length, this is a misplaced focus; instead of asking "can we be sure beyond reasonable doubt that the size of a certain effect is not zero" one should think about "how can we estimate the size of the effect and its real world significance". A nice touch is the authors' use of the word oomph for "size of effect".

Misplaced emphasis on tests of significance is indeed arguably one of the greatest "wrong turns" in twentieth century science. This point is widely accepted amongst academics who use statistics, but perversely the innate conservatism of authors and academic journals causes them to continue a bad tradition. All this makes a great topic for a book, which in the hands of an inspired author like Steven Jay Gould might have become highly influential. The book under review is perfectly correct in its central logical points, and I hope it does succeed in having influence, but to my taste it's handicapped by several stylistic features.

(1) The overall combative style rapidly becomes grating.

(2) A little history -- how did this state of affairs arise? -- is reasonable, but this book has too much, with a curious emphasis on the personalities of the individuals involved, which is just distracting in a book about errors in statistical logic.

(3) The authors don't seem to have thought carefully about their target audience. For a nonspecialist audience, a lighter How to Lie With Statistics style would surely work better. For an academic audience, a more focused [logical point/example of misuse/what authors should have done] format would surely be more effective.

(4) Their analysis of the number of papers making logical errors (e.g. confusing statistical significance with real-world importance) is wonderfully convincing that this problem hasn't yet gone away. But on the point "is this just an academic game being played badly, or does it have harmful real world consequences" they assert the latter but merely give scattered examples, which are not completely convincing. If people fudge data in the traditional paradigm then surely they would fudge data in any alternate paradigm; if one researcher concludes an important real effect is "statistically insignificant" just because they didn't collect enough data, then won't another researcher be able to collect more data and thereby get the credit for proving it important? Ironically, they demonstrate the harmful real world effect is of the cult is non-zero but not how large it is ......

116 of 135 people found the following review helpful

2.0 out of 5 stars Mean-spirited and Misguided June 29, 2008

By Peter Kwok

Format:Paperback

I attended a seminar by McCloskey when she announced she was working on this then-upcoming book. So I knew beforehand that its style would be more like a victim-tells-all revenge than a fun-seeking discovery typical of most popular science books. The first half of the book (up to Chapter 13) did turn out to be bitter. However, at least that part was largely based on facts, such as a comprehensive count of academic papers failing to meet certain standards. The second half of the book was devoted to the biographies of key persons who led to the rise of what the authors called the "cult of statistical significance". The book lost any pretense of integrity at that point, and just started slinging muds. Gosset was portrayed as a good-natured figure who worked hard like a bee; and Fisher, a mad scientist who stole the labor of others and would attack people by any means to defend his status. At one point the authors didn't even bother to call Fisher by his name, and just referred to him as the Wasp. They also dragged Fisher's mother into the ordeal by making suggestions that she was responsible for turning Fisher into a cold-hearted person that they claimed.

I was not only disgusted by this kind of tabloid sensationalism, but was also disappointed by how little useful information I got out of this long-awaited book. The authors "irrationalized" the popularization of statistical significance by framing it as the work of a cult. To further illegitimatize the use of statistical significance, they argued that it is wrong to rely on it to evaluate scientific hypotheses because (1) what we really want is how likely for a hypothesis to be true given the data, not the other way around; and (2) there are other clues just as, if not more, important, especially the effect size. These could have been reasonable positions if they did not make statistical significance a scapegoat for being a "fallacy" just because it is defined on the likelihoods of observing data given the hypotheses. As the way it is defined, statistical significance provides a measure of precision. That's all. Just because it doesn't answer all the questions of scientific interest doesn't mean it provides no useful information and certainly doesn't automatically make it a fallacy. Furthermore, many hypothesis tests used in academic researches are based on likelihood "ratios" rather than just the conditionals. At least there would be NO fallacy for the believers of the Likelihood Principle. It is quite regrettable that they fail to elaborate on such crucial information to make other people look stupid, whether it was their intention or not. As for the second point, I agree that researchers should have paid more attention to other factors, such as statistical power and sample size, IN ADDITION TO statistical significance. But I think it is misguided to hail any ban on reporting statistical significance as a heroic act of revolt as the authors did in the book. One can report all the effect sizes he wants. But it all means nothing if his inferences are what they appear to be mostly due to "bad luck" in sampling the wrong subjects.

If my views above are on the right track, then this book would serve the research community no good by martyrizing Gosset and demonizing Fisher. There has been no cult all along. If we are justified in believing that some vested interests overemphasized statistical significance to divert our attention away from the more important issues, then we should encourage people (authors and readers alike) to focus on those more important issues instead of treating statistical significance as if it were irrelevant. For a more serious and more informative discussion on this topics, I would recommend Chow's Statistical Significance: Rationale, Validity and Utility (Introducing Statistical Methods) . His first chapter explains the key issues in 12 pages with more varieties of arguments and more intellectually stimulating details than what Ziliak and McClosky attempted in 251 pages.

I give 3 stars for this book's good intent but average quality, and, on top of that, took 1 star off for its mean-spirited rhetorics.

39 of 44 people found the following review helpful

2.0 out of 5 stars Disappointing December 9, 2008

By Sergei Soares

Format:Paperback|Amazon Verified Purchase

I know and admire Deirdre McCloskey's work and I am an empirical economist who has to work every day with t and F tests and p-values. So I was quite excited when I read that this particular author had co-authored a book on this particular subject.

Unfortunately, I was quite disappointed. I was expecting either a narrative of errors made in the name of statistical significance or an in-depth analysis of what tests really mean. The authors do neither.

In the first half of the book, they superficially narrate the problems with the Vioxx clinical trials, but tell few other stories of how the standard error "costs jobs, justice and lives." A narrative along the lines of "Normal Accidents", by Charles Perrow, which documents an extensive list of accidents to tell of the perils of complexity, would have made for much better reading. After reading the book, I am none the wiser as to why or how the jobs, justice and lives were lost to statistical significance.

Alternatively, the book could have explained in terms clear to those who do not work every day with tests what is meant by significance and power of a test and what these terms really mean. I have never seen an explanation of these terms that is really clear and sticks in your mind. Unfortunately this was not the case either. The authors mention that statistical significance is more complex than just p-values, affirm that most economists not understand why, and leave it at that. They confuse more than explain.

As a final problem, the book takes a good versus evil attitude that is nowhere good science. Gosset is good and Fischer is bad. Please.

In conclusion, while I agree with the author's main thesis, their book argues it very poorly, very lengthily, and very tediously.