This book is difficult to rate. It is well-written and well-argued, with a lot of useful and interesting material. But it also has fatal gaps. My recommendation is to read the book, but only in combination with three related books: Trust in Numbers, The Politics of Large Numbers and Exploratory Data Analysis. On its own, this book is likely to mislead more than inform.
The central contention of the author is that randomized, double-blind, controlled experiments are the "gold standard" to establish validity. He goes as far as to claim that controlled experiments are necessary for scientific progress, which is patently absurd. But my criticism is not that I disagree, it's that the book fails to mention the reason people do these kinds of experiments.
For example, suppose we want to know if a drug helps lower back pain. We might select 100 healthy patients with severe, persistent lower back pain that has no discernible physical cause. We give half of them 100 mg of the drug daily for a week, the other half get an identical-looking pill with no active ingredients. Neither doctor nor patient knows who is getting the drug. We then ask the patients whether their back improved or not.
Real experiments will be more complicated. We may stratify the sample, test different dosages and regimes, record other data (certainly we will at least record reported side effects) and so forth. But the idea is still to throw away almost all the data, only adding back the stuff we think is important, and then to torture the remaining data to force it to give a definitive answer to a question that may not have one, or that the data may not know.
In contrast, you might treat the 100 patients one at a time, giving the treatment you thought was best, and recording everything about them before, during and after the trial. Blinding, control and randomization are still important. You will enter a treatment, but the computer will sometimes randomly change it, sometimes telling you, sometimes not, sometimes telling the patient, sometimes not. This is the only way to disentangle the effect of your diagnosis from the actual drug administered, and the effect of the drug from what you think and what the patient thinks. But we use the three techniques for the opposite reason as the standard protocol: to increase the dimensionality of our data rather than to reduce it.
The results may be analyzed using exploratory data analysis, letting the data talk to you rather than forcing them into pre-selected simple statistical tests. Or you could use a search algorithm, like the genetic algorithms the author likes. You start will thousands of hypotheses and recombine them randomly, selecting for the ones that explain the data best and are simplest. At the end you will have a weighted population of explanations, whose aggregated predictions give you a reasonable range of answers to what you really want to know, should I take this drug for my back pain? This is continuous learning, not a strict separation between experiment and application. There are no clear, simple answers, just increasingly precise and accurate subjective probability ranges.
A randomized, double-blind, controlled experiment is used when the goal is to get a group of people to agree on a simple action rather than to learn the truth. It is an organizational tool, not a scientific one. The author quotes Francis Bacon for indirect support of the idea, but Bacon was writing at a time when science was under attack by scholasticism and religious dogmatism. The book skips about 350 years to get to Ronald Fisher and randomized field trials. For most of this period, science had no need to prove itself to outsiders, there was rapid progress without significant public opposition. Only when scientists started demanding control over public policy and requesting huge government subsidies was there a need for the kind of experiments the author praises.
The people who came up with these experimental designs had lots of other bad ideas, such as non-expert witnesses, like judges or generals, for experiments. Their movement was popular with fascists. The fact that Nazis liked the ideas doesn't mean they were bad ideas, but it should give some pause. The author mentions Ronald Fisher's dismissal of the claims that smoking is bad for health (but not the money he was paid to argue for the case); he omits Fisher's support for institutionalized racism and eugenics. He also omits John Tukey's famous attack on the Kinsey Report that did so much to bring rationality to public discussion of sexuality, the claim that a random sample of 3 was better than Kinsey's detailed interviews with 300 non-randomly selected people (I recommended a book Tukey wrote much later in life above, when I knew him in the 1970s his opinions had nearly reversed from the 1950s).
The reason this book is five-stars despite the essential omissions, is the core of the argument is based on the author's extensive personal experience. He is really arguing for experiments, skepticism, trial-and-error, limited expectations, controls, blinding, randomization, quantification, careful record-keeping and common sense; he just somehow got it tangled up with a bad theory. This is valuable material you won't find other places, certainly not in as clear and lively form. His recommendations form the same mess of incisive sense mixed with tangled reversals of sense.
If you think for yourself, you will learn a lot from this book. If you don't, you should stay away.
Download the free Kindle app and start reading Kindle books instantly on your smartphone, tablet, or computer - no Kindle device required.
Read instantly on your browser with Kindle for Web.
Using your mobile phone camera - scan the code below and download the Kindle app.
Book recommendations, author interviews, editors' picks, and more. Read it now.
Frequently purchased items with fast delivery
Page 1 of 1 Start over
About the author
Top reviews from the United States
There was a problem filtering reviews. Please reload the page.
- Reviewed in the United States on March 24, 2013
