Customer Reviews

Wrong: Why experts* keep failing us--and how to know when not to trust them *Scientists, finance wizards, doctors, relationship gurus, celebrity CEOs, ... consultants, health officials and more

5 star:		(17)
4 star:		(9)
3 star:		(4)
2 star:		(3)
1 star:		(3)

 

 

Freedman is a good writer and this is a good read. He deftly covers the many ways, in numerous fields, that experts can mess us up by passing on "wrong" info and advice. He even tackles the question "is this book wrong" - a smart touch, given the subject.

So why three stars? For all that's good here, the book left me a bit empty for reasons I'll mention shortly. Not that I expected Freedman to have "the answer." He rightly warns against experts claiming to know it all. Unfortunately, his last chapter on how to tell questionable from trustworthy advice, is pretty generic. I wondered why a book that's so good at describing the problem couldn't muster a few more creative solutions.

I also wish Freedman had weaved in what experts do "right." The contrast would have better highlighted their true failures and helped readers tell when to trust expert advice and when to question it. But we mostly hear about how experts fail on complex problems that no one has solved, in areas like education and medicine. We don't learn how they do on regular ones. For example, Freedman ignores service experts (e.g., plumbers, mechanics, electricians) and most sports experts. There's one example of how coaches missed quarterback Tom Brady's talent - but QB is a position for which success is notoriously hard to predict.

The result: Freedman suggests we can't trust most experts most of the time. That's a bit misleading. The time to be especially wary is when experts weigh in on topics where the answers are unknown. By contrast, studies show experts tend to do much better than non-experts on problems that have known solutions. That makes sense. Plumbers have seen thousands of drain clogs and know how to treat. Mechanics have a wealth of experience diagnosing common car problems. And ditto for many of the common problems a family doctor encounters. Does that mean we should always believe them? No. Just that all else being equal, experts are likely to do much better than average on problems that are well understood. Freedman has to know this but doesn't say it clearly, so readers may easily come away thinking otherwise.

Of course, unscrupulous auto-mechanics and repairmen can still mislead us on purpose, just like unethical doctors and researchers can. Freedman addresses ethics in several places but tends to lump together experts who give us wrong info on purpose with those who do it by accident or ineptitude. The bad advice may look similar in these cases, but ethics breaches spring from different problems and point to different solutions. It would have made for a more compelling conclusion if Freedman had pushed harder on possible solutions here and in related areas. Maybe he wanted to do more but just ran out of time before publication. Overall, a good effort but could have been "gooder."

If your job requires you to make decisions based on advice or research studies, then you should read Wrong by David Freedman. The book takes a look at the state of studies and the unsettling observation that a surprising minority of studies is inaccurate, flawed or just plain wrong. The book looks at several types of studies with a concentration on the medical studies we hear so much about and so often hear conflicting advice. For example, red wine prevents heart disease when another study shows no relationship.

The book is Freedman's investigation and exploration of the reasons behind the why these studies are wrong. The book takes the reader on a systematic investigation of the forces that lead to the publication of inaccurate studies from the need to simplify study finding, the bias of publishing only positive findings, to the social pressures that suppress whistleblowers. Freedman paints a comprehensive picture of the weakeness of the scientific research, including research conducted by Nobel Laureates.

Freedman also takes a look at business research and business books which suffer from these same weaknesses and biases. He points out the structural weakness of the two major basis of business books - that today's `winners' offer immutable lessons for everyone else, or that companies need radical new approaches to address new issues. That discussion, in Chapter 6, should be required reading for every business guru and person offering advice. Readers should also go back to Clayton Christensen's HBR article Why Hard-Nosed Executives Should Care About Management Theory that was published in September 2003 to round out their understanding of business research.

Freedman provides practical advice on characteristics of different types of advice. Below is a summary of the statements to give you an idea of how comprehensive the book is, each point is discussed in detail in the last chapter and this cements the value of the book for a researcher or those who make decisions based on research.

Characteristics of less trustworthy advice include:
- Advice that is simplistic, universal, and definitive
- Advice that is supported by only a single study, or many small or less careful ones, or animal studies.
- Advice that is groundbreaking
- Advice that is pushed by people or organizations that stands to benefit from its acceptance.
- Advice that is geared to preventing future occurrences of a prominent recent failure or crisis.

Characteristics of expert advice we should ignore:

- Advice that is mildly resonant
- Advice that is provocative
- Advice that gets a lot of positive attention
- Advice that other experts embrace
- Advice that appears in a prestigious journal
- Advice that is supported by a big, rigorous study
- Advice backed by experts that boast impressive credentials

Characteristics of more trustworthy expert advice

- Advice that does not trip other alarms
- Advice that is a negative finding
- Advice that is heavy on qualifying statements
- Advice that is candid about reputational evidence
- Advice tat provide some context for research
- Advice that provides perspective
- Advice that includes candid, blunt comments

There are to many points here to repeat in this review, but this is the most valuable part of the book and one of the reasons I am recommending the book - particularly for those who engage in research or are making decisions based on research they either commission or review.

STRENGTHS

Wrong offers a comprehensive view of different types of studies from medical, business, public policy, etc. This gives the book broad appeal and the careful reader insight into the overall modern research process.

The discussion on the wisdom of crowds (Chapter 4 - the Idiocy of Crowds) is perhaps one of the better responses to the current craze of social media. This book is definitely worth the read.

The book offers numerous examples, predominantly of medical studies, which the reader can remember hearing about. The South Korean scientist cloning human stem cells, the debate about the connection between red wine and health are among the flawed science and study Mr. Freedman uses to support his analysis.

The book is blunt and direct and the author calls out people by name and points out the weakeness of their past work.

The author gives you direct references to other skeptical scientists and people who have the job of verifying studies and assuring the quality of research. This has led me to their work, something I never would have found on my own.

CHALLENGES

This book is not for everyone and while its premise, that many scientific studies are based on poor research, biased, or just plan wrong will have popular appear, the book is not own written for the mainstream. People who do research or make decisions based on research or give advice will get the most out of this book. That is something I do and I found the book very powerful.

The book occasionally drifts into areas and alleys that seem to be more for the author's benefit than the readers. The middle chapters particularly go into detail on things that I did not find particularly interesting. This required me to work my way through chapters five, seven and eight which I found a bit heavy and indirect.

The book has a strong bias and purpose in pointing out the weaknesses and outright failures of research and the research process. The readers have to remind themselves that this is a book about an issue rather than research on the research process. If you know the bias, then you can get a lot out of this book.

This is the way David Freedman describes his purpose in this book." This book is about why expertise goes wrong and how we may be able to do a better job of seeking out more trustworthy expert advice. To that end, we're going to look at how experts -- including scientists, business gurus, and our other highly trusted sources of wisdom -- fall prey to a range of measurement errors, how they come to have deep biases that lead them into gamesmanship and even outright dishonesty, and how interactions among them tend to worsen rather than correct for these problems. We're also going to examine the ways in which the media sort through the flow of dubious expert pronouncements and further distort them, as well as how we ourselves are drawn to the worst of this shoddy output, and how we end up being even more misled on the Internet. Finally, we'll try to extract from everything we've discovered a set of rough guidelines that can help to separate the most suspect expert advice from the stuff that has a better chance of holding up."
Freedman certainly has a great deal of evidence to draw from. Consider the 2008 economic collapse and how few predicted it. Consider the confusions in regard to medical treatments such as women's taking 'hormones'at menopause, or men undergoing PSE test as indicator for prostate cancer. Consider the contradictory advice given all the time by various experts on almost every subject.
There is no doubt that there is a lot of wrong prediction and prescription out there.
But what fields are specially prone to it? And where are experts most reliable?
To his credit Freedman saying that he may be wrong at the end of the book gives eleven rules for testing what any given expert says.
There is a great deal one can learn from this book. I would only say that I believe in making his case the author tends to neglect the true other side, and that is the way 'experts' are in some areas more right than wrong. I have a personal physician who is not a genius or distinguished. He is an ordinary family physician. But in tens of incidences of various illnesses through the years he has been almost always right. He has used his expert knowledge to help me many many times. He is almost always right. I suspect most of us have in various areas 'experts' we do trust. ( I would make an exception in this day and age of financial advisors. My experience with them is almost the opposite of that with my personal physicians. They have been , thanks also to my own stupidity, of no great help.) But of course I am only talking about one little case my own.
To get back to the book. I believe it is an important one. And there is a great deal to be learned from it. I highly recommend it.

The author of this book completely misses the point and perpetuates a misunderstanding that he (and journalists in general) live by. While science certainly does have its problems, the systemic problem that he points out falls almost wholly on the shoulders of journalists and reporters rather than the experts he derides.

Science doesn't claim to be right; it is a process by which incorrect answers can be discarded. Science has progresses as much as it has, and provided what it has to us, not because it is necessarily right, but because, unlike other common ways of thinking, it allows us to move on from conclusions that are clearly wrong. The scientific method was designed with precisely this understanding: that most answers that we will come up with will be wrong. Studies, those things that journalists love to cite and paraphrase, are not intended to be THE FINAL ANSWER--they are working answers, or a way of working toward an answer. Most scientists not only understand this on an abstract level, but are viscerally aware of it; it shapes their thinking (or at least it should) about every published paper they read. Scientists understand that all conclusions published in all papers are tentative, and they understand that a study is only a model or simulation of the real world and may therefore not present an outcome or conclusion that accurately represents the real world. Yes, some excited moron enamored with his own research may exaggerate the strength and meaning of his conclusions. Especially when prodded by a journalist to do so. (Although far more often the exaggeration is injected by the journalist.)But this is not how science as a whole works. Half or more of Freedman's book, then, is wasted arguing a point that is elementary and obvious to anyone with a basic understanding of the scientific method, and that he, apparently, is shamefully unaware of.

He didn't focus on the real cause of "expert failure" in the realm of science, because he IS that problem: namely, that journalists create most of these public misconceptions of fact by incorrectly representing the work of scientists. It's not only that they often misrepresent the conclusions of a particular study or what those conclusions mean. It's not only that they remove the vitally important language of tentativity that characterizes scientific writing, changing their write-up to display a tone of certitude that the original never contained simply to increase their work's wow factor for the public. It is also, more deeply and more importantly, that they entirely and consistently misrepresent what it means for a conclusion to be supported by data and published at all. If he wrote this book to help the public understand how to interpret scientific results, he failed miserably because he never made this most vital point. I suspect this failure was because 1) He didn't really understand it himself, and 2) Journalism IS the art of spin; it would have undermined the attractiveness of his book to be reasonable.

Further, although Freedman occasionally distinguished between different types of expertise, he generally conflated some fundamentally different types of information suppliers under the term "expert." These types of fact suppliers includes scientists, advertisers, journalists, self-promoting gurus, and idealogues. Each of these types of information suppliers is so fundamentally different that treating them all as "experts" serves to muddle rather than improve a reader's ability to assess information.

Advertisers, idealogues, and self-promoters have the objective of financial gain and therefore cannot generally be trusted. Journalists need to shape facts into news or stories, which are a form of information repackaged as entertainment that is inherently unreliable. These information suppliers need to be assessed and understood differently.

An expert is someone who dedicates a substantial portion of his or her life to learning about and understanding a particular topic. Not a few months, or even a few years. The better part of a working lifetime, or at least a decade or preferably decades. Another huge problem with expertise is that, in order to make a stopry more convincing or interesting, journalists or other people in the entertainment industry will elevate a non-expert, usually another journalist, to the status of "expert" when such a label is totally inappropriate. As cases in point, on public radio I recently heard a journalist who wrote a book on fruit presented as an "expert" but he didn't know what a quince was when a caller called; another supposedly expert journalist who wrote a book on honey didn't even know what sugar honey was composed of (glucose) or have any idea on how it was physiologically produced.

Many of the author's points are important, thoughtful, and well-made, but this content could have been thoroughly addressed in a medium to long magazine article; in no way did it warrant a book. Instead he went on and on with irrelevant or poorly presented examples that demonstrated nothing in particular. For example, he cites, as an example of confusing and contradictory expert opinion, the theories that asthma is caused by sterile environments in childhood and environmental pollution. These ideas are in no way contradictory or exclusive; the only confusion is that introduced by his simplification of the topic for the sake of producing catchy sentences that speciously sound perceptive.

The information that Freedman gives us to help us assess expert advice is almost useless and totally, and astoundingly, misses most of the most important points. If anyone cares, here is what they are:

1) Look for real experts. Advertisers, idealogues, and journalists are not real experts and should not be trusted to provide information. Journalists are better than the first two, but if at all possible a primary source or true expert should be consulted. Freedman confuses these categories; for example, he seems to think of economists as scientists when they are not. Economists are idealogues who theorize within an artificial mathematical world that they create to support their ideas; economics as practiced is not constrained by nor does it adhere to the scientific method. By conflating idealogues, advertisers, and such with real expertise, Freedman is able to display the inherent shortcomings of one group and let it by juxtaposition and association discolor our view of the other. This is a disservice to our understanding of the issue he purports to address.

2) In reading any published or unpublished study, assess the methods used to produce data, the strength of the data, and perhaps most importantly, the logical connection between the data and the author's conclusion. I know this is work, but it is the only way. And any intelligent person who reads a few studies will quickly begin to see that the conclusions are often illogical, the data sparse or weak, or the methods ridiculous. Even journalists almost nnever do this, and I find it telling that Freedman never once in the book dissects a published study, or directly critiques one. This tells me that he has only researched this topic skin-deep, as beautiful and illustrative examples of poor studies are remarkably easy to find.

3) Realize that published conclusions supported by data (ie, studies) are not facts; they are just part of the process of science.

So, in short, I like the author's point, but dislike the way he made it and the misunderstandings he obfuscated it with, and the way he failed to clarify it. But he did what journalists do: they sell words, not information.

Author David Freedman takes on the experts, and how they mislead us, if we let them. This original and common-sense critique of expert opinion traces how it gains credibility within a particular field, often spilling over to a much larger audience. It scrutinizes the many ways advice may take a wrong turn, even when it comes from one of the greats. The book is well researched with examples from medicine, science, business, consumer products and other facets of everyday ife where experts may weigh in accurately or not. With his many substantial contributions as a journalist covering these topics, Freedman has a wealth of case studies and anecdotes to draw upon. He is well equipped to take on the big ones, which he does with ease (Yes, even Albert Einstein was sometimes wrong). This is an excellent read, with much good humor, historical perspective, honest analysis and whimsy. An appendix describes how the book itself may be wrong. Freedman leaves us with a tip list of what to look for in deciding who to trust and who to ignore. I highly recommend this book - it will make you ask the hard questions before believing what passes as expert advice these days.

Do you remember the study from a few years back that claimed that "workers distracted by e-mail and phone calls suffer a fall in IQ more than twice that found in marijuana smokers?" It turns out that the study was the product of corporate sponsorship (Hewlett-Packard); involved an extremely small test pool (eight students) who were subjected to continuous noise and flashing lights while taking a standard IQ test. The author of the study, Dr. Glenn Wilson, later admitted that, "it didn't prove much of anything . . . ." Yet none of this qualifying information was included in the sensational stories that were published at the time. According to author David H. Freedman, in Wrong, this is just one of too many to count examples of misleading or flat out wrong expert information.

Wrong is a provocative book that challenges conventional beliefs about the value of scientific studies (including random controlled trials) and the expert opinions. Some of the most explosive statements come from noted medical researcher John Ioannidis who opines, "most medical treatment simply isn't backed up by good, quantitative evidence." Moreover, Ioannidis claims that these problems are not limited to medicine, but rather opines that "the facts suggest that for many, if not the vast majority, of fields, the majority of published studies are likely to be wrong . . . . [Probably] the vast majority."

Why do experts get it wrong so often? Freedman convincingly argues that several forces are at play including: flawed evidence and faulty assumptions; "fudging" of results or plain fraud by scientists seeking to advance their professional careers; publication preference for positive and sensational findings; audience preference for experts who provide certain and simplistic advice; and internet shenanigans such as for hire services that bump up the paying client's study in Google search results. In Wrong Freedman explores these and other pitfalls in thorough detail.

Wrong is a book that repeatedly had me scratching my head in amazement, but I think I am the wiser for having read it. In addition, I will no longer blindly accept the latest study results or expert de jour opinion. However, I don't claim that I "won't be fooled again." Rather I think I will be a bit more skeptical or at least more cautious in the future.

Wrong is a revealing book that challenges many sacred cows, but ultimately succeeds in reminding the reader that caveat emptor reigns even when the advice is served up by an expert.



Publisher: Little, Brown and Company; 1 edition (June 10, 2010), 304 pages.
Advance review copy provided courtesy of the publisher.

All of us have noticed that some expert, after a scientific study, will make a claim about a food, a common practice or an alcoholic beverage. The claim may be about the harmful effects or the benefits of the subject of the study. Then, some time later, another expert will cite a study contradicting the first expert's claim. Who do you believe and why? In this book, Mr. Freedman tries to help you find the answer to that question. I found this book to be both educational and entertaining. The reader will discover that there are fraudulent researchers, but also that even the best intentioned research can lead to false conclusions. Mr. Freedman gives many examples of both. But even more importantly, he tells us how the research can go wrong and how we use certain clues to spot faulty studies, false claims and bogus "experts". The author never claims that his advice will always work, but it is certainly better than guessing at the truth.

I highly recommend this book.

Like a lot of people, I listen hard to what my doctor says, swallow whole the advice of a lawyer when I seek it, etc... Freedman's book is a wakeup call for me and the many people who tend to equate narrow expertise with reliable, and especially, knowledgeable advice. Oil industry engineers are only the latest experts to give us a false sense of security; who's next? This book offers great insights into the industry of expertise marketing.

Book review by Richard L. Weaver II

The author is "a science and business journalist." Outside of appendices, notes, and index, the text is 230 pages long, and there are 11 pages of notes.

There are a number of reasons I liked this book. First, it is very well-written. Second, it covers areas (finance, doctors, relationship gurus, celebrity CEOs, consultants, health, and more) with which I have some interest (although no expertise).

The third reason I liked this book is that it offers great evidence, interesting facts, and fascinating statistics and insights I would probably never gather elsewhere.

The fourth reason I liked this book is that it produces skeptics. Whether you accept Freedman's ideas or not, he certainly opens your eyes and makes you question -- something we all should be doing all of the time. (If nothing else, it is what colleges and universities should be good at promoting.)

I enjoyed Freedman's examination of the various safeguards that we have to try to root out and address fraud. As I was preparing this review of his book (on January 6, 2011), the British Medical Journal (BMJ) just pronounced a Lancet study by lead scientist Andrew Wakefield, M.D., that connected the Mumps, Measles, and Rubella (MMR) vaccine to autism as an "elaborate fraud." An ironic juxtaposition, to say the least.

On page 120 Freedman writes, "Thank goodness for peer review, the 350-year-old research-journal tradition of sending candidate articles out to knowledgeable researchers for vetting and comments." -- even though he admits that peer review provides only a minimal assurance of quality, and that "lousy research can slip past peer review into journals" (p. 121). Having been part of a number of peer-review teams during my tenure as a university professor, I have to agree with Freedman; however, I also agree that there is really little other way to prevent lousy research from getting published. The Internet makes "publication" an easy process.

What Freedman does is open the whole area of fraud and "the fraudulent police" to further discussion. His chapter conclusions (see page 124) are right on target about scientists.

Freedman's comment about determining whether or not Internet information is accurate was well taken: " . . . we're back to that problem of whether most people in the public are equipped to track down high-quality information on the Internet, as opposed to ending up with advice that may look convincing but is in fact junk" (p. 201). Of course, with respect to students (and the public, too), this isn't a new concern, it is simply a much bigger concern with the glut of information at our fingertips.

Every student should be required to read Chapter 9, "Eleven Simple Never-Fail Rules for Not Being Misled by Experts" (pp. 203-230). Even though the rules are generic, they are important and well explained here. His "Typical Characteristics of Less Trustworthy Expert Advice," "Characteristics of Expert Advice We Should Ignore," and "Some Characteristics of More Trustworthy Expert Advice" is priceless, essential, practical, and incredibly relevant in today's information-saturated world.

Freedman's book is about why expertise goes wrong and how to identify what is most likely suspect. Included among those labeled 'experts' are pop gurus, celebrity advice givers, stock brokers, and auto mechanics, but not engineers, architects, or politicians. (Freedman sees politicians' errors as due to relying on bad expertise.)

Recent examples of experts being proven wrong include changes to CPR advice, the 2007-08 market crash, early pro-football careers of Tom Brady and Kurt Warner (both were ignored), discrepancies between "Consumer Reports" and J.D. Power summaries, differing criteria and results for school evaluation, differing diet advice, exposes of poor crime labs, medical misdiagnosis rates (about 1/6), and the controversy surrounding the diabetes drug Avendia.

Us of 'indirect measurements' (eg. tumor shrinkage, instead of average years of survival) are a common problem, especially in medicine. Freedman also cites 21 recent asthma studies - 487 different measures of 'improvement' were used. Functional MRI scans measure blood flow through varying regions of the brain, and conclude the existence of close links among activities with similar brain maps; others contend this logic is nonsense. Gene links are regularly 'found,' but on average turn out to be correct less than 1% of the time; some contend that an overfocus on genes also brings reduced attention to controllable factors such as diet.

Measurement technique sometimes prove problematic. Freedman cites an example where ocean water temperature differed according to where the water was tested vs. the location of the research ship. Other major problem sources include non-random sample populations, and extrapolating from animal tests (only about 10% of animal test results are similar to those of humans, according to one study). Adding average data to fill in for study dropouts can skew results - generally those that drop out differ from those staying in the study. An even bigger problem is that studies finding no-effect or a negative effect are not usually published, and thus are not available to counter published studies that find a favorable effect. Then there are the drug companies that 'prove' their new wonder-drug is better than competitors' by manipulating the compared dosages - small for the 'old' drug, much more appropriate levels for their own.

Studies that start out looking for whatever differences show up between groups are almost guaranteed to find something - the problem is that whatever is found is almost always random variation. That's why physicians advise taking the average of three blood pressure measurements instead of just one. Confounding variables are another major problem - eg. comparing pupil performance in high-spending vs. low-spending schools is almost always confounded (distorted) by the tendency for higher socioeconomic areas to spend more on schools, thus intermingling the two factors.

Freedman does not believe that groups are more likely to make correct decisions than individuals, citing evidence that groups are frequently dominated not by people more likely to be right, but by those who are more persuasive, manipulative, and persistent. Similarly, pack mentality is another problem - Freedman cites "Candid Camera" for evidence - test subjects entering an elevator with jokesters facing the rear often followed suit. And then there's our recent housing bubble - a massive illustration of pack mentality.

Sometimes studies are distorted by deliberate fraud - researchers pursuing tenure or grants.

We often associate expertise with one having published a book. That's not such a good idea, says Freedman, given today's large market for books and the ease of publication via vanity press houses. Increasing media pressures (more channels, the Internet) for material acerbate the rush to faux experts and fads.

Remember team-teaching, 'new math,' whole-word reading instruction, esteem-boosting, multiculturalism, experiential learning, peer tutoring, English immersion, ESL, tracking, learning centers, and pupil-centered classes? They're just a few examples of innumerable education fads that have come and gone, and sometimes come again. You'd think we'd learn, but I'll bet we jump on at least three new education fads during the remainder of this year. Management has its fads too - MBO, TQM, 'high-grading,' teams, even day-time naps.

How can one distinguish the good from the bad? Freedman suggests advice that is simplistic, universal, and definitive, supported by a single study, and touted by those who stand to benefit are the least likely to be reliable. Unfortunately, some of these characteristics are also what makes an 'authority' most convincing.

As for me, I'm sticking with Paul the Octopus - his World Cup record of picking winners proves he's an expert.