Customer Reviews

Probability Theory: The Logic of Science

5 star:		(17)
4 star:		(4)
3 star:		(0)
2 star:		(0)
1 star:		(0)

 

 

Reading this book is an exhilarating intellectual adventure. I found that it shed light on many mysteries and answered questions that had long troubled me. It contains the clearest exposition of the fundamentals of probability theory that I have ever encountered, and its chatty style is a pleasure to read. Jaynes the teacher collaborates fully with Jaynes the scientist in this book, and at times you feel as if the author is standing before you at the blackboard, chalk in hand, giving you a private lesson. Jaynes's advice on avoiding errors in the application of probability theory -- reinforced in many examples throughout the book -- is by itself well worth the price of the book.

If you deal at all with probability theory, statistics, data analysis, pattern recognition, automated diagnosis -- in short, any form of reasoning from inconclusive or uncertain information -- you need to read this book. It will give you new perspectives on these problems.

The downside to the book is that Jaynes died before he had a chance to finish it, and the editor, although capable and qualified to fill in the missing pieces, was understandably unwilling to inject himself into Jaynes's book. One result is that the quality of exposition suffers in some of the later chapters; furthermore, the author is not in a position to issue errata to correct various minor errors. Volunteer efforts are underway to remedy these problems -- those who buy the book may want to visit the "Unofficial Errata and Commentary" website for it, or check out the etjaynesstudy mailing list at Yahoo groups.

This book has been on the web in unfinished form for a number of years and has shaped my scientific thinking more than any other book. I believe it constitutes one of the most important scientific texts of the last hundred years. It convincingly shows that "statistics", "statistical inference", "Bayesian inference", "probability theory", "maximum entropy methods" , and "statistical mechanics" are all parts of a large coherent theory that is the unique consistent extension of logic to propositions that have degrees of plausibility attached to them. This is already a theoretical accomplishment of epic proportions. But in addition, the book shows how one actually solves real world problems within this frame work, and in doing so shows what a vastly wider array of problems is addressable within this frame work than in any of the forementioned particular fields.
If you work in any field where on needs to "reason with incomplete information" this book is invaluable.

As others have already mentioned, Jaynes never finished this book. The editor decided to "fill in" the missing parts by putting excercises that, when finished by the reader, provide what (so the editor guesses) Jaynes left out. I find this solution a bit disappointing. The excercises don't take away the impression that holes are left in the text. It would have been better if the editor had written the missing parts and then printed those in different font so as to indicate that these parts were not written by Jaynes. Better still would have been if the editor had invited researchers that are intimately familiar with Jaynes' work and the topic of each of the missing pieces to submit text for the missing pieces. The editor could then have chosen from these to provide a "best guess" for what Jaynes might have written.

Finally, there is the issue of Jaynes' writing style. This is of course largely a matter of taste. I personally like his writing style very much because it is clear, and not as stifly formal as most science texts. However, some readers may find his style too belligerent and polemic.

From a few common sense requirements, the books starts by deriving basic results such as the product and sum rules, for probabilities defined not in terms of frequencies, but as degrees of plausibility. This was an eye-opener for me, having imbibed the common attitude that such probabilities are 'subjective' and, implicitly, lacking rigor and utility.

Jaynes' knowledge of the history and philosophy of statistics is far deeper than that of most statisticians (including myself). His trenchant style gives the book a narrative drive and cover-to-cover readability that, in my experience, is unique in the field. One such strand is the continual battle between his respect for RA Fisher's abilities, and his exasperation at how wrongheadedly he feels they were channelled. And he doesn't hesitate to take on philosophical heavyweights such as Hume in defending the possibility - - in fact, the necessity - - of inductive inference. However, this style also produces some more bitter fruit, such as the way the author repeatedly likens himself to historical victims of religious persecution.

The book weakens when it turns to applications. Regression with errors in both variables is said to be 'the most common problem of inference faced by experimental scientists' who have 'searched the statistical literature in vain for help on this'. Good points. So why don't the author and editor give us at least a reference for just one of the 'correct solutions' which 'adapt effortlessly' to scientists' needs? And Jaynes' argument that the null hypothesis procedure 'saws off its own limb' would also rule out mathematical proof by reductio ad absurdum.

When estimating periodicities, we're told that 'the eyeball is a more reliable indicator of an effect than an orthodox equal-tails test'. So why not show us the data of the example used, to let us use our eyes? In fact, there's only one graph of empirical data in all the book's 600+ pages.

Several convincing arguments are presented for the use of the Jeffreys (reciprocal) prior for scale parameters, including scale independence. However, just when I was ready to go and use it, there's a warning against the use of improper priors except as 'as a well-defined limit of a sequence of proper priors'. A few pages later a uniform prior is used for the mean of a Gaussian, with no such justification as a limit, which makes it far from clear what exactly is being recommended.

I could give a lot more space to the book's many other insights, and several other annoyances. Instead, I'll finish now by recommending it to anyone interested in the foundations and practice of statistical analysis.

I read this book before it was published; I downloaded it from a WU website. It has been of immense use to me in my career, it is a very practical book. Other reviews that say Dr. Jaynes' ideas are at odds with traditional measure theoretic probability are mistaken. Dr. Jaynes is a true Baysian. A Baysian is one who believes that probabilities do not model serendipity in nature, but do model subjective certainty. The Bayesian concept of probability is epistomological, i.e. the uncertainty is in our minds, not in objective reality. Traditional probability takes the reverse view: probabilities model unpredictable events, they are a model of objective reality like any science, i.e. probabilities are ontological. The trick is to realize the two are not mutually exclusive! There can be true ontological randomness in nature, and our minds can have uncertainty from incomplete knowledge as well. Probability theory as a branch of mathematics makes no claim what it models. The beauty is that probabiltity distributions integrate the two seamlessly. Thus, it is perfectly valid to put a distribution on an unknown parameter, epistomologically unknown, and derive that distribution from an experiment with, presumably, ontological randomness. Dr. Jaynes' book is well worth reading for the many case studies he presents. His background as a physicist is key to understanding some of the esoteric philisophical points.

First off, I can in good conscience only recommend this book to experts who already have a deep understanding of both Bayesian and frequentist probability theory. The most useful function of this book is to illuminate puzzling features of probability theory that niggle at the minds of experts. If you don't already understand the subject at a fairly deep level, Jaynes will only leave you confused. (I could not imagine the torment of someone trying to learn probability and statistics for the first time from this book!)

Expect little in the way of examples or practical solutions here. Jaynes is concerned more with fundamentals and philosophy. Phil Gregory's textbook, although overly fond of Mathematica, is a better intro to practical applications. What examples there are tend to be highly idealized, with a high amount of tedious calculation.

Jaynes died with his book in an unfinished state. What he needed was an editor, but what he got instead was a hagiographer. Rather than inject himself into Jaynes' work, the editor instead has left all of the flaws, incomplete explanations, and many out-and-out mistakes in place. This was a bad mistake. Too many important points are left as exercises to the reader.

Jaynes himself is highly infuriating on a number of points. He repeatedly argues for a Haldane prior as a non-informative prior for a binomial distribution, but doesn't come to grips with the fact that this improper prior gives absurd results in some limits, whereas the more commonly used and more robust Jeffreys prior is ignored. Jeffreys priors themselves are scarcely mentioned in most places, while discussion of how to apply KL information measures to construct non-informative priors is completely missing. Jaynes' commentary on the state of quantum mechanics will strike most physicists as misguided as at best.

I find it ironic that I have mostly negative things to say about a book that I rank at 4 out of 5. The trouble is that this could have been the greatest single book ever written on the subject if it only had better editing, fewer polemics, and a more practical bent. I find myself mourning for what this book could have been. What it actually is, however, is a great probability text from a Bayesian perspective. It contains many gems, but you have to wade through a lot to find them.

Outline

This book develops probability theory from first principles as an extension of deductive logic. In deductive logic, propositions can have only three possible truth values: true, false, and irremediable uncertainty. Therefore, the goal of the book is to describe a consistent extended logic that assigns real numbers to the plausibility of propositions. The requirements for such a system are derived from five simple desiderata, which serve as the postulates of this theory - and it turns out that *any* such system is equivalent to probability theory, to within a monotonic transformation.

Probability theory is then developed through applications to problems which grow more and more complex. The author demonstrates its use in direct sampling problems and so-called inverse problems, aka Bayesian probability. He derives procedures for multiple hypothesis testing, parameter estimation, and significance testing, and shows that although there are close connections between probability and frequency of occurrence in a large number of trials, no probability is *simply* a frequency.

Following this, the author presents solutions to the problem of assigning prior probabilities, and develops decision theory as an adjunct to probability theory. The author then compares and contrasts mainstream or "orthodox" statistical theory with probability theory as extended logic, and (perhaps unsurprisingly) finds severe deficiencies in the orthodox methods. The final chapters concern even more advanced applications.

Math Requirements

Readers should be well versed in simple calculus and multivariate calculus; some familiarity with convolution integrals and finite combinatorics is also an asset, but not essential. In isolated places, the author uses or refers to the calculus of variations and the theory of function spaces (in this case Hilbert spaces); but lack of familiarity with these branches of mathematics will not seriously hamper the reader.

Critical Review

This book represents a major step forward in the understanding of what probability theory is and how to use it. In particular, a lack of solutions to the problem of prior probabilities is the main reason that for the past 100 years, mainstream probability theory was taught as a theory of frequencies instead of as an extenstion of logic; therefore, having solutions to the problem of assigning priors in a textbook is a great step forward in the development of probability theory.

The book is a pleasure to read, with a text-to-equation ratio that is uncharacteristically high for a textbook of probability theory. That is not to say that the equations are simplistic; on the contrary, solutions to quite challenging problems are presented. In addition, the author's polemics against orthodox theory are quite entertaining (and convincing); he wields an acerbic pen when describing the efforts of those who actively reject probability theory as extended logic.

One negative feature of the book is its incompleteness: the author passed away before finishing the book, so occasionally large chunks of planned text are missing. The editor has cleverly mitigated this flaw by inserting "Editor's Problem Boxes", which challenge the reader to fill in the missing text. Still, as one reads the book, one gets the vaguely disquieting feeling that the author wanted to include much, much more information, but didn't have the chance.

To "pure" mathematicians, probability theory is measure theory in spaces of measure 1. To the extent to which you remain a "pure" mathematician, this book will be incomprehensible to you.

To frequentist statisticians, probability theory is the study of relative frequencies or of proportions of a population; those are "probabilities".

To Bayesian statisticians, probability theory is the study of degrees of belief. Bayesians may assign probability 1/2 to the proposition that there was life on Mars a billion years ago; frequentists will not do that because they cannot say that there was life on Mars a billion years ago in precisely half of all cases -- there are no such "cases".

To _subjective_ Bayesians, probability theory is about subjective degrees of belief. A subjective degree of belief is merely how sure you happen to be.

"Noninformative" _objective_ Bayesians assign "noninformative" probability distributions when they deal with uncertain propositions or uncertain quantities, and replace them with "informative" distributions only when they update them because of "data". "Data", in this sense, consists of the outcomes of random experiments.

"Informative" _objective_ Bayesians -- a rare species -- ask what degree of belief in an uncertain proposition is logically necessitated by whatever information one has, and they don't necessarily require that information to consist of outcomes of random experiments.

Jaynes is an "informative" objective Bayesian. This book is his defense of that position and his account of how it is to be used.

"Pure" mathematicians will not find that this book resembles that branch of "pure" mathematics that they call probability theory.

Jaynes rails against those he disagrees with at great length. Often he is right. But often he simply misunderstands them. For example, writing in the 1990s, he said that pure mathematicians reject the use of Dirac's delta function and its derivatives, and related topics. That is nonsense; the delta function has long been considered highly respectable, and required material in the graduate curriculum. Unfortunately Jaynes's misunderstandings may cause some others to misunderstand him when he is right. Statisticians are more informed than "pure" mathematicians and will disagree with Jaynes for better reasons. _Some_ statisticians will agree with him.

Jaynes has many flaws, made all the more annoying by the fact that we need to overlook them in order to understand him. His message is important.

Jaynes' work on probability has inspired many students and academics over the years. Jaynes advocates probability as a degree of belief. In the first two chapters he refers to the two axioms of Robert Cox relating probability to plausible reasoning and comments on "subjective" vs. "objective" reasoning. He moves on to discuss many aspects of classical statistics such as hyptohesis testing and parameter estimation from a Bayesian view. He also presents his famous work on prior probabilities and builds on Shannon's entropy definition to present the maximum entropy principle. A lot of people have been waiting for this book to be published.It should be on the bookshelf of every person who deals with probability.

Beginning with three simple reasoning desiderata, Jaynes derives two theorems from which the whole of probability theory as the logic of plausible reasoning is inferred. Deductive, Aristotlean logic, it turns out, is but a theoretical by-product affirming logical certainty. If you're drowning in a flood of epistomelogical doubt concerning the foundations of probability theory, specifically, or inductive reasoning in general, and wish for a cogent, literate account of both, this book is for you. Examples abound, including an analysis of Bertrand's paradox. The text is lively and engaging throughout. Jaynes has poured over 30 years of thought into this work, a masterpiece of clear thinking that interprets probability as a measure of our imperfect information about the world. This book is for truth seekers, mathematicians, scientists, and open-minded philosophers of science and mathematics ready to sort through the facts rather that make a-priori pronouncements about reality. The author's warnings regarding the mind projection fallacy and the paradoxes associated with infinite point sets alone is worth the cost of this extroadinary work.

I've never seen another book like this. Jaynes definitely has an agenda, but he justifies his viewpoint through an amazingly deep tour of probability theory. Not every viewpoint he expresses is convincing (such as his view that quantum theory is inherently probabilistic only because physicists are lazy), but he always raises deep and interesting questions while teaching the ideas. If you can read a book and accept some but not all of its viewpoint, then this is the book on probability for you.