Customer Reviews

Scientific Reasoning: The Bayesian Approach

5 star:		(3)
4 star:		(1)
3 star:		(2)
2 star:		(2)
1 star:		(0)

 

 

This book is a little-known treasure in the philosophy of science that deserves a spot alongside the better known works of Popper, Kuhn, Lakatos, and Feyerabend, and is more practical than most of those. Herein lies the clearest, simplest, and most persuasive discussion I've ever seen on the limits of Karl Popper's view of science, along with a sound introduction to the Bayesian probability theory requiring no more than high school algebra and a little persistence.

Much of this book will strike students of classical probability theory and philosophy of science as very counter-intuitive at first, but it is so well argued and so clear that I think most readers will begin to warm up to the Bayesian view at least to some degree by the time they finish the book.

The book starts out introducing one version of the traditional "problem of induction": 'how can we be certain of a rule inferred from finite individual observations ?' We then quickly discover why the usual solutions offered don't quite work in actual theory construction in practice. Mainly, the usual solutions (generally based on the disconfirmation of hypotheses) don't address the way _auxilliary_ hypotheses help theories escape refutation, and how webs of evidence of different kinds often converge to help confirm theories.

It has been generally accepted by modern philosophers of science that useful scientific theories go well beyond the experimental data. Hence they can technically not be "proven" in a logical sense, only considered increasingly more likely as their testable predictions are validated.

The Bayesian view is not based so much on a negative attitude toward objective confirmation of theories, as on the observation that classical methods which are the guardians of total objectivity, in fact violate that ideal constantly and in arbitrary ways. The most objective methods, such as those of Fisher and Neyman and Pearson are credibly claimed to rely on personal judgement of likelihood at key points, rather than being the objective logical consequences generally assumed of them.

The Bayesian view starts off acknowledging that subjective assessment of likelihood is an important part of theory selection and construction, and makes it part of the philosophy of science. The central point is that we have degrees of belief in theories, and that these degrees of belief adhere to probability calculus.

The power of scientific reasoning then results not from some elusive objective logic of discovery but because our innate inference abilities lead observation of evidence to beliefs that follow probability calculus, and hence our sense of increasing credibility tends to reflect greater likelihood of a theory making accurate predictions. Although our inferences are not consistently Bayesian by any means, our own intuitions about what represents *correct* inductive reasoning _are_ Bayesian in nature. So when we take pains to correct our inferences based on our own standards of tenability, our subjective assessments lead us to increasingly better theories.

Our beliefs can be measured as probabilities, and probabilities can be used to confirm theories. Among other things, the Bayesian view uniquely predicts, in contrast to the classical view of Popper and statistician Fisher, that novel observations should have and do have special importance in theory construction. The authors not only introduce probability calculus in simple algebraic terms and discuss its application to philosophy of science, but they also devote considerable time to exploring specific weaknesses of alternate views, and considerable time persuasively addressing the strongest criticisms of the Bayesian approach, such as that it is "too subjective." But the Bayesian philosophy of science is actually built on a powerful theory of inference and is itself "unimpeachably objective" because of its strict rules of consistency, even though its subject matter is subjective degrees of belief.

If you've ever wondered exactly what the Bayesian approach to probability is, and what it is supposed to offer science, or you've ever been dissatisfied with the traditional answers to the problem of induction, this book will be your welcome friend for a number of evenings. It combines mathematical elegance and deftness with simple philosophical wisdom and deals convincingly with the controversial nature of its claims.

This book contains lots of useful information for the budding Baysian. Excellent discussions on many topics. However, I have to give this only 3 stars, because on a cardinal point, the authors give very bad advice: they give the impression that Komogorov complexity-based methods are ill motivated. In fact, Kolmogorov complexity is one of the most fruitful new developments in Baysianism, and I have personally used it many times in industrial settings to solve otherwise intractible problems.

However, on most points the book is very useful. I recommend buying the first edition over the second, because the second edition doesn't really add that much useful info over the first. I also recommend buying in addition to this book Ming Li and Paul Vianyi's book on Kolmogorov complexity, for a comprehensive intro to a whole wonderland of Baysianism which Howson & Urbach have overlooked.

Now in its third edition, Scientific Reasoning: The Bayesian Approach, is a basic introduction to the philosophy that scientific reasoning is, and should be, conducted in accordance with the axioms of probability. Called the Bayesian view, after a theorem first proven by Thomas Bayes in the late eighteenth century, has recently gained increased standing as a valuable methodology for examining scientific evidence. Scientific Reasoning explains the elements of probability calculus that are relevant to Bayesian methods and argues that probability calculus should be understood as a form of logic. Accessibly written, even to readers who understand only the basics of probability or calculus, Scientific Reasoning is a solid explanation of how Bayesian theory offers a unified and highly satisfactory accounting of scientific procedure, and is an absolute "must-read" for all scientists and students of science.

Colin Howson continues the importance of the London School of Economics in international philosophy of science with this learned overview of the Bayesian theory of scientific confirmation- that probability can be used to reasonably justify scientific theories. Reconfirming such advantages as the value of novel evidence, uniquely recognised in the Bayesian approach, and answering such criticisms as the problem of old evidence, this is the definitive work on the philosphically popular Bayesian probabalistic theory of scientific confirmation.

Howson and Urbach are totally committed to the subjectivist,Bayesian approach to probability founded by Ramsey,De Finetti,and Savage.This approach is based on the misbelief that all probability estimates are precise,exact ,unique single numbers.The decision maker can initially hold any subjective belief he wants to entertain as long as he is willing to incorporate additional evidence over time using Bayesian updating.This means that the decision maker uses the mathematical laws of the probability calculus (the addition and multiplication rules)to update his subjective probabilities so that they are consistent over time with the mathematical laws of the probablity calculus.His beliefs will be coherent and not subject to having a Dutch book made against him.Whatever his initial a priori ,subjective estimates of the probabilities were,the updated versions will start to converge to the correct a posteriori probabilities.The allegiance to the use of the mathematical laws of the probability calculus guarantee a decision maker that over time his assessments will be consistent.

The problems with this approach are based on the unspecified assumptions that there is always sufficient evidence to specify a sample space or unique probability distribution.This requires that there be NO uncertainty,vagueness,unclearness,ambiguity,or conflict in the evidence used to specify the probability distribution.Keynes specifies this by saying that the weight of the evidence is complete or equal to 1 on the unit interval [0,1].Ellsberg would specify this by requiring that rho =1,where rho is specified on the unit interval[0,1].Urbach and Howson simply assume that there will always be sufficient information that is clear in the present or future so that they can specify their distribution.Nowhere in this book do they show or discuss the difficulty in showing that w=1 or rho=1.Urbach and Howson just assume it.

The second problem is the assumption that all probability estimates are precise exact ,single numbers. Keynes founded his logical theory of probability on the interval estimate approach of George Boole.Theodore Hailperin has shown since 1965 that all of the Boole-Keynes problems using lower-upper probabilities (Howson and Urbach are completely ignorant of the fact that Keynes and Boole,and not Good,Smith,Dempster,etc.,are the founders of the interval,upper-lower probabilities approach to probability )can be translated into linear programming problems.This completely destroys the claims that (a) all a priori logical probabilities must be specified using the principle of indifference and (b) that the logical probability approach is not operational.Urbach and Howson need to completely rewrite their discussions of logical probabiity and upper-lower probability to take into account what it was that Keynes actually did and not what Frank Ramsey and Bruno De Finetti think that he did.The discussions in the book about Logical probability contain so many errors of commission and omission that I recommend that a reader skip them entirely.It turns out that the Bayesian ,Subjectivist approach of Howson and Urbach is a special case that arises when (a) rho and /or w =1 on the interval between 0 and 1 and (b) the lower probability estimates are equal to the upper probability estimates.Recently, Kadane ,Schervish,and Siedenfeld have called into question the entire logical framework of the Bayesian,Subjectivist approach to probability in their " Rethinking the Foundations of Statistics "(Cambridge University Press,1999).They have completely thrown in the towel and admitted implicitly that Keynes was right and Ramsey,De Finetti,and Savage were incorrect.The book is composed of 16 essays,all of which have been previously published in academic journals and/or other books.Throughout the book,the authors concede that there are many holes and deficiencies in the logical foundations of the subjectivist approach.The biggest hole is vastly understated by KSS: " In fact,it seems reasonable to deny that there are consequences in practical decisions.Thus,our position is that,lacking consequences,expected utility theory must treat probability distributions as extraneous(italicized)..."(KSS,1999,p.195).Of course,"...lacking consequences...", means that the outcomes are state independent.On pp.157-160,KSS had already demonstrated the near impossibility in the real world of being able to specify outcomes that would make their utilities state independent.Given that the subjective probabilities are completely extraneous,there is no longer any way in which the a priori beliefs of a decision maker can be represented by a unique probability distribution on purely decision theoretic grounds alone.This means that unique,definite,precise,numerical ,single,exact,hard,sharp point estimates of subjective probabilities do not exist.This result goes to the heart of the entire edifice erected by Ramsey,De Finetti,and Savage,in particular.Savage argued that,based on a careful elicitation of subjective preferences based on betting quotients,a unique probability measure(distribution)can be defined to represent the agent's preference relation.Nowhere is it stated by KSS that this position( of Savage) goes to the heart of the dispute between Keynes and Ramsey about the inherent indeterminateness of many probability estimates.The comparative -interval estimate-approach to estimating probabilities,presented by Keynes for the first time in the A Treatise on Probability(TP;1921,pp.160-163,pp.186-194),is fully operational since all of the problems Keynes presented as examples in the TP in chapters 15 and 17, that used the difficult Boolean approach, can instead use the substantially easier integer-mixed integer linear programming approach of Theodore Hailperin in order to obtain solutions.It appears that Ramsey's approach is a very special case of Keynes's approach that is applicable only when a single,unique probability distribution can be specified a priori. KSS's technical results,first published in 1990 in the Journal of the American Statistical Association(JASA), totally undermines the logical ,decision theoretic foundations of the subjectivist approach to estimating probabilities.
Howson and Urbach have overlooked the numerous logical deficiencies in the current Bayesian approach.Keynes has been right all along.Howson and Urbach's book needs to be completely rewritten .Nevertheless, I recommend that the book be bought.The reader ,however,needs to realize that the theory presented in this book by Urbach and Howson is a special theory,not a general one.

Science is probability. Science is about degrees of belief: should I or should I not believe such-and-such a hypothesis? Such degrees of belief are probabilities (i.e., satisfy the probability axioms). This follows from the assumption that our degrees of belief determine a coherent assignment of fair odds, and the theorem that odds are coherent (i.e., cannot be Dutch-booked) if and only if they satisfy the probability axioms.

Beliefs are updated in accordance with Bayes' theorem. This does actually not follow from the above: one could in principle go from one coherent set of beliefs to another in an erratic manner without being susceptible to Dutch books (despite what some people say, "diachronic" Dutch books do obviously not indicate inconsistency or irrationality). But it follows with a reasonable additional assumption: namely, that a scientists should not change his P(h|e)'s as the evidence e comes in, i.e., the new P'(h|e)'s should be equal to the old P(h|e)'s. This is reasonable because when forming P(h|e) we have already imagined that e were true; that its truth is not factual rather than imagined should not make a difference. With this assumption we see that P'(h)=P'(h|e)=P(h|e), which is precisely the conditionalisation rule.

Bayes' theorem P(h|e)=P(e|h)P(h)/P(e) reflects scientific practice. To what extent e supports h obviously depends on the extent to which h implies e, which is reflected in the term P(e|h), on a scale from entailment P(e|h)=1 to refutation P(e|h)=0. The term P(h) reflects the fact that some hypotheses are a priori more reasonable (simple, non-ad hoc, etc.) than others. The term P(e) assures that surprising evidence carries more weight.

The alleged further explanatory success of Bayesianism is unimpressive. No one should be impressed, for example, that the theory can be made to conform with various aspects of science (e.g., "solve" the Duhem problem) by just-so assignments of probabilities. Our authors also make a big deal out of the rather plain fact their approach is successful in purely probabilistic settings (e.g., statistical tests); but these are not impressive arguments for Bayesianism, as they are merely instances of probability theory conforming with itself.

Objections:

Problem of old evidence. Known e has P(e)=1, so by Bayes' theorem it can never support any hypothesis. The proposed solution is that conditionalisation should take place with respect to background knowledge "minus e." But whatever this is supposed to mean our authors cannot say, except in the trivial case where e is independent of everything else in the background knowledge.

Bayesian conditionalisation does not allow prediction any privilege over accommodation. Our authors' reply is that such a difference can be accounted for by appeal to priors or background knowledge. Sure enough, since Bayesianism does not regulate these factors in any way, they can of course be used to account for anything. The irony in this being an obvious case of ad hoc accommodation should be clear.

Paradox of evidence reinforcing priors. Before I flip a coin for the first time I do not know if it is biased; I assign priors 1/2. Now suppose I flip it 1000 times and it comes out precisely that way. Bayesianism makes the posterior probability the same as the prior without reflecting the fact that the probability is now more securely established. This problem can be solved by allowing probabilities to be distributions rather than numbers.

Subjectivism. Subjective Bayesianism allows arbitrary priors, so it is a "confirmation theory" that relies on opinion as much as fact. The reply is confused. On the one hand there is the analogy with logic suggesting that it is not the premises but the rules of inference that are the proper subject matter, which is fine but obviously dodges the question (i.e., what are we justified in believing?). On the other hand we read that: "The prescription of the same 'objective' prior probability for everybody in the same knowledge state is a prescription for stagnation and eventual catastrophe, as is the suppression of dissent quite generally." This makes no sense. Why celebrate dissent in the priors but suppress it in conditionalisation? Indeed, it is claimed on the same page that "experience is allowed to dominate prior beliefs ...; disagreement is not eradicated at once, but its effect is usually falls off quickly." Bayesianism suppresses dissent, in other words. Or so our authors claim. But this "usual" washing out of the priors is a convenient fiction: the word "usually" here is in effect code fore "when one plugs in the kind of numbers that gives the desired results."

"A curiously outdated work, which might have served a useful purpose 60 years earlier. Mostly a rehash of all the false starts of philosophers in the past, while offering no new insight into them and ignoring the modern developments by scientists, engineers, and economists which have made them obsolete. What little positive Bayesian material there is, represents a level of understanding that Harold Jeffreys had surpassed 50 years earlier, minus the mathematics needed to apply it. They persist in the pre-Jeffreys notation which fails to indicate the prior information in a probability symbol, take no note of nuisance parameters, and solve no problems."

Howson, C. & Urbach, P. (1991), "Bayesian Reasoning in Science", Nature, 350, 371-374.
"An advertisement for the previous work, with the same shortcomings. Since they expound Bayesian principles as they existed 60 years earlier, it is appropriate that Anthony Edwards responded (Nature, 352, 386{387) with the standard counter-arguments given by his teacher, R. A. Fisher, 60 years earlier. But to those actively engaged in actually using Bayesian methods in the real problems of science today, this exchange seems like arguing over two different systems of epicycles."

[1] Both authors are philosophers, not mathematicians.

[2] If you are interested in the philosophy of Bayesianism, Probability Theory : The Logic of Science by E. T. Jaynes is definitely better.

[3] The knowledge required for reading this book is almost nothing, so it is useful to complete beginners of probability theory.

[4] Good Bayesian guys are always good philosophers and skilled at traditional theories. So what? Read the books written by good Bayesian mathematicians.

[5] The title is awesome, but the content is not commensurate.