Customer Reviews

Pattern Classification (2nd Edition)

5 star:		(13)
4 star:		(7)
3 star:		(5)
2 star:		(2)
1 star:		(5)

 

 

The 1973 edition of Pattern Classification by Richard Duda and Peter Hart is one of the most cited books in the fields of image processing, machine vision, and classification. It contains perhaps the clearest, most comprehensible descriptions of statistical inference ever written. Though intended for the image processing audience, it is general in its approach, and is broader in coverage than other contemporary books like the redoubtable Van Trees (1969). The section on Bayesian Learning anticipates the EM algorithm which appeared a few years later (Dempster, et al. 1977) and their description of Parzen windows for density estimation is more often cited than Parzen's own papers.

The appearance of the 2000 2nd edition led this writer to wonder if D&H could repeat with an offering as good as their first. In particular, would D&H have kept up with the considerable growth in methodology in the 1990s? Well, they have! With the addition of David Stork as third author, the second addition re-presents the basic theory, illustrated with some beautiful and complex figures, and knits it neatly with an exposition of neural networks, stochastic methods for posterior determination, nonmetric classification (tree search and string parsing), and clustering. Chapter 9 is a particularly interesting review of the recent machine learning research making the point that, absent knowledge of a problem's specific domain, no one classifier is better that any other. This chapter also reviews solutions to the problem of training on too-small samples including the Jackknife and bootstrap methods, and newer bagging and boosting algorithms popular in data mining applications. Each chapter is well-designed, with a summary, many exercises (including computer exercises), and references to the literature (typically 50-100) including many recent references.

This book is designed for an upper-level undergraduate/graduate audience. It doesn't assume a knowledge of statistics, but requires some familiarity with methods from calculus, real analysis, and linear algebra.

The first edition was a particularly important element in this writer's education; the second edition is certain to find a similar place in the working and intellectual lives of many new readers.

This book is a revised edition of Duda and Hart's classic text on Pattern Classification which was originally published in 1973. In fact, the 1973 edition of the book played a pivotal role in introducing me (and countless researchers of my generation) to the field of pattern classification. Needless to say, I was looking forward to the release of the revised edition. Unfortunately, I was extremely disappointed with the new edition. I had expected much more from the masters: Duda and Hart!

My reasons for disappointment with this book are as follows:

Given the 27 years that have elapsed since the publication of the first edition of the book, and the immense progress that has taken place in pattern recognition, machine learning, computational learning theory, grammar inference, statistical inference, algorithmic information theory, and related areas, the revisions and additions in the 2000 edition are essentially of a patchwork nature. In my opinion, they do not reflect the current understanding of the topic of pattern classification.

A disproportionate number of pages are devoted to topics like density estimation despite the fact that it has been well established in recent years, through the work of Vapnik and others, that when working with limited data, trying to solve the problem of pattern classification through density estimation (which turns out to be, in a well-defined sense of the term, a much harder problem than pattern classification) is rather futile. When modern techniques for learning pattern classifiers from limited data sets (e.g., support vector classifiers) are touched on in the book, the treatment is disappointingly superficial and in some cases, misleading.

There is virtually no discussion of problems of learning from large high dimensional data sets, incremental refinement of classifiers, learning from sequential data, distributed algorithms, etc. The treatment of non-numeric pattern recognition techniques (e.g., automata, languages, etc.) is extremely superficial. There is almost no discussion of essential aspects such as preprocessing and feature extraction techniques for dealing with variable length, semistructured, or unstructured patterns.

There is very little contact made with a large body of pattern classification algorithms, results, and approaches developed by the machine learning community, some exceptions.

There is little discussion of the extremely important topic of computational complexity and data requirements of learning algorithms.

On the positive side, the discussion of most topics that were originally covered in the 1973 edition has been further refined and in many cases, made more accessible through the addition of illustrative examples and diagrams. Topics such as Bayesian networks receive an intutive and accessible treatment. It was good to see a treatment of techniques for combining classifiers (although it is placed misleadingly in a chapter titled "Algorithm-Independent Machine Learning" which has an organization that is reminescent of a "kitchen sink"). The exercises at the end of each chapter seems useful.

Perhaps it is too difficult for any individual or a small group of individuals to write a textbook that reflects the state of the art in pattern recognition. Perhaps my expectations of Duda and Hart (based largely on the extraordinary job that did on the 1973 edition of their book) were too high to have a reasonable chance of being met by the 2000 edition. Perhaps I have come to expect more out of graduate level textbooks after having worked as a researcher and an educator in this field for over a decade at a major university.

In short, the book fell significantly short of my expectation.

The 1973 book by Duda and Hart was a classic. It surveyed the literature on pattern classification and scene analysis and provided the practitioner with wonderful insight and exposition of the subject. In the intervening 28 years the field has exploded and there has been an enormous increase in technical approaches and applications.
With this in mind the authors and their new coauthor David Stork go about the task of providing a revision. True to the goals of the original the authors undertake to describe pattern recognition under a variety of topics and with several available methods to cover each topic. Important new areas are covered and old but now deemed less significant are dropped. Advances in statistical computing and computing in general also dictate the topics. So although the authors are the same and the title is almost the same (note that scene analysis is dropped from the title) it is more like an entirely new book on the subject rthan a revision of the old. For a revision, I would expect to see mostly the same chapters with the same titles and only a few new chapters along with expansion of old chapters.

Although I view this as a new book, that is not necessarily bad. In fact it may be viewed as a strength of the book. It maintains the style and clarity of the original that we all loved but represents the state-of-the-art in pattern recognition at the beginning of the 21st Century.

The original had some very nice pictures. I liked some of them so much that I used them with permission in the section on classification error rate estimation in my bootstrap book. This edition goes much further with beautiful graphics including many nice three-dimensional color pictures like the one on the cover page.

The standard classical material is covered in the first five chapters with new material included (e.g. the EM algorithm and hidden markov models in Chapter 3). Chapter 6 covers multilayer neural networks (a totally new area). Nonmetric methods including decision trees and the CART methodology are covered in Chapter 8. Each chapter has a large number of relevant references and many homework exercises and computer exercises.

Chapter 9 is "Algorithm-Independent Machine Learning" and it includes the wonderful "No Free Lunch" theorem (Theorem 9.1), a discussion of the minimum desciption length principle, overfitting issues and Occam's razor, bias - variance tradeoffs,resampling method for estimation and classifier evaluation, and ideas about combining classifiers.

Chapter 10 is on unsurpervised learning and clustering. In addition to the traditional techniques covered in the first edition the authors include the many advances in mixture models.

I was particularly interested in that part of Chapter 9. There is good coverage of the topics and they provide a number of good references. However, I was a bit disappointed with the cursory treatment of bootstrap estimation of classification accuracy (section 9.6.3 on pages 485 - 486). I particularly disagree with the simplistic statement "In practice, the high computational complexity of bootstrap estimation of classifier accuracy is rarely worth possible improvements in that estimate (Section 9.5.1)". On the other hand, the book is one of the first to cover the newer and also promising resampling approaches called "Bagging" and "Boosting" that these authors seem to favor.

Davison and Hinkley's bootstrap text is mentioned for its practical applications and guidance for bootstrapping. The authors overlook Shao and Tu which offers more in the way of guidance. Also my book provides some guidance for error rate estimation but is overlooked.

My book also illustrate the limitations of the bootstrap. Phil Good's book provides guidance and is mentioned by the authors. But his book is very superficial and overgeneralized with respect to guiding practitioners. For these reasons I held back my enthusiasm and only gave this text four stars.

Before this book was published, I considered "Pattern Recognition", by Theordoridis to be the best text for learning pattern recognition and classification. Although Theordoridis' book has some difficulties (not enough concrete exercises, ommission of structural methods, and not enough material on Bayesian Networks and HMMs), it seemed significantly better than previous texts. However, not only does Duda, Hart, and Stork's book succeed in those areas where the former fails, but it also has other strengths that the former book does not have: better illustrations, boxed formulas and algorithms, and highlighted defintions. Although somewhat superficial, these improvements mark the fact that pattern recognition is now considered a mainstream subject, and thus requires a mainstream text that keeps the integrity and rigor of the subject matter, while simultaneously making it more accessible to the average engineer. The new champ, however, does not come without it's own shortcomings. For example, I believe the last 3 chapters of Theodoridis' book should be read by anyone who wants a deeper understanding of clustering techniques for unsupervised learning. Moreover, this book fails to acknowledge the brilliant work done in computational learning by Vapnik and Chervonenkis, which reveals the authors' bias towards practice over theory. I believe it deserves more than passing mention in the historical notes section of unsupervised learning.

[1] This book is good as an introduction to Pattern Recognition, at undergraduate level (compared with the level of Fukunaga's -Introduction to Statistical Pattern Recognition-).
[2] The references may be helpful to those who are interested in kernel methods, SVM, etc for detailed discussions.
[3] Compared with Bishop's or Ripley's book on pattern recognition, this book is not a bible.
[4] As a textbook for undergraduate students, I will mark it 5 stars; as a reference book for researchers, it is worthy 3 stars at most.

I have now used this book 3 times for a class. While the 1st edition did a nice job of covering the material in its time, the additions to in the 2nd addition are a disaster. What the book has going for it is that it at least lists the necessary material for such a course in the table of contents. However, all the additional material is poorly explained at best. The problem sets are too few and the ones that are included are generally weak.

I have tried to use this book, but after constant student complaints and my own difficulty with the text, I have finally concluded that the problem lies with the text and not with the users.

I think an indicator of problems was the large number of errors in the first printing; large here is an understatement. Even in later additions, the 4th, the size of the errata is huge. I think this is indicative of the authors' attention to detail and seriousness in preparation. I have found similar errors and ambiguities in the associate Computer Manual.

The bottom line is that this book has seen its final appearance in our curriculum. I would use any other text, even an older one.

There is simply not enough room or time to point out all the problems with this text. Do yourself a favor if considering this text for a class. Don't bother.

I used the first edition of this book in a class on pattern recognition back in 1998. That old first edition did a great job of explaining the different aspects of pattern recognition as they were generally taught when the first edition came out in 1969. However, over the next 30 years the field expanded enough that a second edition was required. I purchased it, expecting an expanded version that went over the details as well as the first edition, and boy was I wrong. This second edition just glosses over the details of modern pattern classification techniques and doesn't show sufficient examples or even motivation for you to "get it". It's almost like the entire book is an introduction. I'm accustomed to the first chapter of a technical book being an overview that doesn't tell you much, but not the entire book. The only thing the second edition has to offer are slicker illustrations. My advice is find a copy of the first edition. It is very well put together. If you need additional material on subjects the first edition doesn't cover well, then go find more modern books specifically on those subjects. You may spend more money but at least you'll learn something.

I found this book quite useful as an augmentive text to Elements of Statistical Learning used in a grad engineering level data mining course. This book is written more at an engineering level, and I found it to bridge well between advanced texts such as Elements of Statistical Learning and more general audience books that really are lacking. Duda and Hart do a good job at explaining the concepts, however some techniques only recieve a cursory overview while other topics are rather elaborated upon, however this may have been done by the authors experience of which techniques are commonly employed in practice. The excercises at the end of the chapters include a lot of hands on programming and computer-based assignments which I found useful, and a MATLAB workbook associated with this is also offered, however I have not read this book. Nonetheless I have implemented some of the concepts in this book using Matlab and it definately does help to cement the idea, even if this is just serves as an intellectual excercise and isn't intended to be used for anything else. With a little bit of digging through the help or using a book such as Ripley and Venerable's Modern Applied Statistics with S, most if not all of the techniques can be explored using the R statistical software.

I am not sure how this book gets consistently high marks. I am using this text for a graduate level course. While it does a decent job covering most of the topics, it has some glaring flaws.

For one the Homework Problems it provides are not really representative of what you're learning in the text. Almost all of the problems revolve around proofs, as opposed to using the concepts in practice. You can seemingly have a good grasp on the material, yet spend hours trying to solve each of the problems they provide for that particular section. My entire class has complained, and even my professor has admitted that even he isn't sure sometimes how they expect you to solve some of the problems.

Secondly, there are very few example problems demonstrated in the text, so the reader doesn't really get to see the concepts in action so to speak.

Also, there is a typo or error on almost every other page, sometimes even on important formulas.

Overall, I'd have to think there are better books out there. If this truly is "the best there is" as some reviewers claim, God help the field of Pattern Recognition.

The book is the opposite of what I have expected. Unlike to the first volume by Duda and Hart, this book does not go to the roots on many subjects, the review is not comprehensive, but poor and biased, most important keywork, including that which has been greatly reviewed in the first edition, is not there, crucial links between methods and key concepts are not fully understood. References are incomplete and ignore the true key work of so many authors - what a shame !

The first edition by Duda and Hart is one of the best books ever written on the subject. It makes me feel very much sad to compare the two books. The second edition should have never been written.