Statistical Learning Theory [Hardcover]

Vladimir N. Vapnik (Author)

Book Description

ISBN-10: 0471030031 | ISBN-13: 978-0471030034 | Publication Date: September 16, 1998 | Edition: 1

A comprehensive look at learning and generalization theory. The statistical theory of learning and generalization concerns the problem of choosing desired functions on the basis of empirical data. Highly applicable to a variety of computer science and robotics fields, this book offers lucid coverage of the theory as a whole. Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.

Editorial Reviews

From the Publisher

This book is devoted to the statistical theory of learning and generalization, that is, the problem of choosing the desired function on the basis of empirical data. The author will present the whole picture of learning and generalization theory. Learning theory has applications in many fields, such as psychology, education and computer science.

Product Details

• Hardcover: 736 pages
• Publisher: Wiley-Interscience; 1 edition (September 16, 1998)
• Language: English
• ISBN-10: 0471030031
• ISBN-13: 978-0471030034
• Product Dimensions: 9.6 x 6.3 x 1.6 inches
• Shipping Weight: 2.6 pounds (View shipping rates and policies)
• Average Customer Review: 5.0 out of 5 stars  See all reviews (4 customer reviews)
• Amazon Best Sellers Rank: #205,801 in Books (See Top 100 in Books)
  • #16 in Books > Computers & Technology > Computer Science > Artificial Intelligence > Computer Mathematics
  • #33 in Books > Computers & Technology > Computer Science > Artificial Intelligence > Theory of Computing
  • #34 in Books > Computers & Technology > Computer Science > Artificial Intelligence > Machine Learning

More About the Author

Discover books, learn about writers, read author blogs, and more.

Customer Reviews

Most Helpful Customer Reviews

27 of 28 people found the following review helpful:

5.0 out of 5 stars statistical learning based on the VC class, January 23, 2008

By 

Michael R. Chernick "statman31147" (Holland PA) - See all my reviews
(REAL NAME)   

This review is from: Statistical Learning Theory (Hardcover)

Vapnik and Chernovenkis extended the Glivenko-Cantelli Theorem in their work on classification and statistical learning. Vapnik in recent texts has described a form of nonparametric statistical inference based on approximating functions and the Vapnik-Chernovenkis dimension.
In an earlier book published by Springer-Verlag he develops the basics of the theory. However to keep the mathematical level excessible to computer scientists and engineers he avoided the mathematical proofs needed for mathematical rigor. This text is an advanced text that provides the rigorous development. Although the preface and chapter 0 give the reader a idea of what is to come the rest of the text is difficult reading.

The theory has been quite successful at attacking the pattern recognition/ classification problem and provides a basis for understanding support vector machines. However Vapnik sees a much broader application to statistical inference in general when the classical parametric approach fails.

If you have a strong background in probability theory you should be able to wade through the book and get something out of it. If not I recommend reading section 7.9 of "The Elements of Statistical Learning" by Hastie, Tibshirani and Friedman. That will give you an easily understandable view of the VC dimension. Also sections 12.2 and 12.3 of their text will give you some appreciation for support vector machines and the error rate bounds obtainable for them based on the VC dimension.

40 of 44 people found the following review helpful:

5.0 out of 5 stars An excellent overview, July 21, 2004

By 

Dr. Lee D. Carlson (Baltimore, Maryland USA) - See all my reviews
(VINE VOICE)    (HALL OF FAME REVIEWER)    (REAL NAME)   

Amazon Verified Purchase

This review is from: Statistical Learning Theory (Hardcover)

The field of statistical learning theory has not only seen considerable advances in the last fifteen years, it has also found many applications, some of these appearing in commercial packages. It is now classified as a subfield of artificial intelligence, and as such gives an alternative, and frequently more general viewpoint on such topics as pattern recognition, regression estimation, and signal processing. The author of this book is one of the originators of statistical learning theory, and has written a book that will give the mathematically sophisticated reader a rigorous account of the subject. Most of the main results are proven in detail, but the author does find time to include insightful discussion on the origins and intuition behind the concepts involved in statistical learning theory.

Along with a brief introduction, the book consists of three parts, the first being an overview of the statistical theory of learning, the second giving the details of the now widely used support vector machines, and the last one (the most sophisticated mathematically) giving the statistical foundations of learning theory. In writing the book, the author wants to put forward a new approach to dependency estimation problems having their origin in learning theory, and being able to deal with the ?curse of dimensionality?. The origins of the subject lie in the pattern recognition problem and the Glivenko-Cantelli problem in statistics. Both of these problems were discovered to be essentially the same, and the author?s task is to use their similarities to construct a general theory of statistical inference and (inductive) learning. Indeed, a new induction principle, called ?structural risk minimization? (SRM) is paradigmatic in the book, along with the now ubiquitous VC dimension, the latter of which originates in the author?s early research. Both the SRM and the VC dimension illustrate the tension between the need for high accuracy and the need for the minimization of error in data sets.

The learning problem, as the author sees it, is the problem of selecting the correct dependence on the basis of empirical data. Two approaches to this problem are discussed, the first using a ?risk functional?, and the second involving the estimation of stochastic dependencies and the consequent solution of integral solutions. Both of these approaches are modeled in terms of a general model of learning from examples, which consists of a data generator, a supervisor, and a learning machine. The learning machine can either imitate the supervisor or identify how the supervisor operates. These two methods are different, the author says, in that the first one searches for the best prediction based on the data, while the second one attempts to approximate the operator representing the supervisor. Both approaches are studied in the book, with the first one being the easier of the two, while the second involving the solution of ill-posed problems. The author views the learning process in terms of choosing the right function from a given function collection.

Both perceptrons and their generalizations, neural networks, are briefly discussed in the book, along with the back-propagation method. The author gives reasons why he does not think neural networks are well-controlled learning machines, such as the existence of local minima, the slow convergence of the gradient method, and the choice of scaling factors. These problems serve as motivation for the introduction of support vector machines, which are introduced as optimal separating hyperplanes. Support vector machines take input vectors into a high-dimensional feature space via a nonlinear mapping, and an optimal separating hyperplane is then constructed in this feature space.

Similar to the need for neural networks to generalize well, separating hyperplanes must do the same, and due to the large dimensionality of the feature space, a hyperplane that separates the training data may not generalize well. In addition, the large dimensionality of the feature space makes the construction of the hyperplane computationally demanding. The author shows that optimal hyperplanes, found using various mathematical techniques such as quadratic optimization, do generalize well. Also, as the author points out, the explicit form of the feature space need not be known, since only the inner products between the ?support vectors? and the vectors of the feature space need to be calculated. The calculation of the inner product is done with the insight gained from Mercer?s theorem, which gives the existence of a kernel function such that there exists a feature space where this function generates the inner product. This inner product in feature space allows the construction of a decision function that is nonlinear in the input space but that is equivalent to a linear function in the feature space. Different choices of the kernel function give different types of learning machines. The author discusses three examples of support vector machines for pattern recognition: polynomial, radial basis function, and two-layer neural network support vector machines. An entire chapter is spent on the problem of digit recognition using support vector machines.

16 of 23 people found the following review helpful:

5.0 out of 5 stars Rich & Valuable, July 24, 2001

By 

khaled (Jordan) - See all my reviews

This review is from: Statistical Learning Theory (Hardcover)

This book aims at rigorours and deep treatment of statistical learning and is divided into three parts :

(I)THEORY OF LEARNING AND GENERALIZATION;

(II)SUPPORT VECTOR ESTIMATION OF FUNCTIONS;

(III)STATISTICAL FOUNDATION OF LEARNING THEORY'

For anyone intending to dive into this topic intriguing readers shull find their task rather not simple when exploring this mathematical exposition.This is because of the mature nature behind the basic theory .In order to gain most of the benefit ,interested and even involved researchers are urged and should assume all the requirements for a vast and solid mathematical background.

I Think the book constitutes a respectful and organized 'exhibition' that you will not find in any other place. Althought there are excellent books discussing SVMs and Machine-Learning/ Intelligence,eventually all emenate from the theory.Regarding the book rating it is was not rated upon how much you retrieve as concepts, but how well the propositions offer a precious appreciation of the substantial theory.In otherwords, this book is not the place for a first time learning, but it is serves as a bridge between interrelated elements of such incredibly growing area.

For the book: "The Nature of Statistical learning Theory" also by Vapnik you can find a review by Vladimir Cherkassky in The IEEE TRANSACTIONS ON NEURAL NETWORKS VOL. 8, NO. 6, NOVEMBER 1997 .