An Introduction to Kolmogorov Complexity and Its Applications (Texts in Computer Science) (Hardcover)

~ (Author), Paul Vitanyi (Author) "Suppose we want to describe a given object by a finite binary string..." (more)
Key Phrases: unit integrable, recursive semimeasure, prefix machine, Lecture Notes, System Sci, Soviet Math (more...)

Editorial Reviews

Review

"This is a great book. The other extant literature on Kolmogorov complexity is scattered, in need of translation, or otherwise hard to access. This book puts at all in one place in a readable, enjoyable style. In this second edition, the authors have added many new results that have been proven since the first edition was published in 1993." -- ACM's Computing Reviews

...provides a stimulating and welcome presentation that theory which deals with the quantity of information in individual objects...this book should be considered a must read for researchers and practitioners interested in maintaining an awareness of theories important to the advancement in finance. -- Randall. B. Caldwell, Journal of Computational Intelligence, FRANCE

Product Description

Written by two experts in the field, this is the only comprehensive and unified treatment of the central ideas and their applications of Kolmogorov complexity. The book presents a thorough treatment of the subject with a wide range of illustrative applications. Such applications include the randomness of finite objects or infinite sequences, Martin-Loef tests for randomness, information theory, computational learning theory, the complexity of algorithms, and the thermodynamics of computing. In this new edition the authors have added new material on circuit theory, distributed algorithms, data compression, and other topics.

Product Details

• Hardcover: 637 pages
• Publisher: Springer; 2nd edition (February 27, 1997)
• Language: English
• ISBN-10: 0387948686
• ISBN-13: 978-0387948683
• Product Dimensions: 10.2 x 7 x 1.6 inches
• Shipping Weight: 2.9 pounds (View shipping rates and policies)
• Average Customer Review: 5.0 out of 5 stars  See all reviews (8 customer reviews)
• Amazon.com Sales Rank: #975,112 in Books (See Bestsellers in Books)

Customer Reviews

Most Helpful Customer Reviews

34 of 34 people found the following review helpful:

5.0 out of 5 stars Biggest return for the biggest investment, May 7, 2005

By	Randall Helzerman (campbell, ca) - See all my reviews (REAL NAME)

This was the second-hardest book I ever read. Honestly, it took me years and years to get through it. I even had to buy a 2nd copy, because I kept getting frustrated and throwing the first copy across the room until it was destroyed. So yes, this book requires a substantial effort to read.

But the payback!! I've gotten more return on investment from this book than from any other book I've ever read. If you dilligently read and master this book, you will be able to analyze and solve problems your collegues just can't.

The basic idea behind Kolmogorov complexity is straighforward: a good measure of the complexity of an object is the length of the shortest computer program which will construct that object. From this basic idea an amazing variety of insights and powerful techniques have been developed, and this book is quite comprehensive in cataloging and explaining them.

For computer scientists and working programmers, probably the most useful result of Kolmogorov complexity would be the "Incompressibility Method", which is a powerful technique for the analysis of the runtime of algorithms. Typically, it is relatively easy to figure out what the best case or the worst case runtime of an algorithm is. Until now, it was hard to calculate the average runtime of an algorithm, because it usually involved a tricky counting problem, to enumerate all possible runs of the the algorithm and summing over them. The incompressibility method eliminates the need for doing these complicated enumerations, by letting you perform the analysis on a single run of the algorithm which is guarunteed to be representative of the average runtime of the algorithm. If you program for a living like I do, this will give you an edge, because if you can accurately predict that the worst-case runtimes almost never happen, you can usually simplify and streamline your programs by optimizing it for the average case. If your competitors are wasting time optimizing for a worst case which almost never happens--at the expense of _not_ optimizing for the average case, you win bigtime.

For philosophers of science and AI/knowledge representation folks, the most useful results of Kolmogorov complexity are probably the contributions of Kolmogorov complexity to Baysianism. To be a Baysian is to follow a two step process: (STEP 1) for every possible sentence, assign to it a number between 0 and 1 which represents how certain you are that that sentence is true. This initial assignment should be a probability distribution over all possible sentences. It should be a "good" probability distrubution, but of course it won't be perfect, since you don't know everything. (STEP 2) when confronted with new evidence, e.g. an observation, update your current "good" degrees of belief by using Bayes' law, to yield a new "better" set of degrees of belief.

The Baysians always had a good story for Step 2--just use Bayes law. But until now, they were mostly hand-waving on Step 1--what would constitude a "good" initial probability distribution? There were many proposals (e.g. maximum entropy) but all proposals had benefits and drawbacks. What Kolmogorov complexity provides is the so-called "universal" distribution, which is guarunteed to be a "good" initial distirbution. This book devotes much time to explaining and exploring this, and shows how previous techniques, like maximum entropy, minimum description length, etc all can be seen as computable approximations to the (unfortunately uncomputable) universal distribution. This really gives a nice framework for evalutating and formulating good prior distributions.

After remarking on how hard this book was to read, I should emphasize that this is not due to bad writing on the part of the authors! Indeed, after throwing the book across the room, I was always drawn back by Li & Vitanyi's most engaging writing style to pick the book back up, dust it off, and have another go at it. If it were not for their wonderul ability to expain a very complicated subject matter, I never would have gotten through it.

An unsung hero of this book is Peter Gacs, who wrote a set of lecture notes which really could be considered to be an Urtext for this book. If you tackle this book, I highly recommend that you also get ahold of these notes, because it is sometimes very useful, when trying to puzzle out a difficult argument, to get another description/explaination of it from a different point of view. These notes are available on the web, just google for "Lecture note on descriptional complexity and randomness" by Peter Gacs.

If you're up to the challange, then buy this book, dilligently read it, swear at it--then swear by it.

12 of 12 people found the following review helpful:

5.0 out of 5 stars THE book on Kolmogorov Complexity, October 13, 1998

By	Lance Fortnow ([e-mail]) (Chicago) - See all my reviews

When is an object "random"? Kolmogorov (and others) argue that one could measure randomness by the shortest description, i.e. computer program, that generates it.

This simple idea leads to a beautiful mathematical theory and a powerful tool as one can show that random objects have several interesting properties.

Li and Vitanyi have written this wonderful monograph on the area covering the depth of theory and applications not seen anywhere else. They give a clear and complete descriptions of many of the important concepts in the book. I have used this book twice in teaching graduate courses on the topic.

This book is a must have for anyone interested in a serious mathematical treatment of Kolmogorov complexity.

19 of 21 people found the following review helpful:

5.0 out of 5 stars The only one of its kind...., September 22, 2001

By	Dr. Lee D. Carlson (Baltimore, Maryland USA) - See all my reviews (TOP 100 REVIEWER)    (REAL NAME)

The theory of Kolmogorov complexity attempts to define randomness in terms of the complexity of the program used to compute it. The authors give an excellent overview of this theory, and even discuss some of its philosophical ramifications, but they are always careful to distinguish between mathematical rigor and philosophical speculation. And, interestingly, the authors choose to discuss information theory in physics and the somewhat radical idea of reversible computation. The theory of Kolmogorov complexity is slowly making its way into applications, these being coding theory and computational intelligence, and network performance optimization, and this book serves as a fine reference for those readers interested in these applications. Some of the main points of the book I found interesting include: 1. A very condensed but effective discussion of Turing machines and effective computability. 2. The historical motivation for defining randomness and its defintiion using Kolmogorov complexity. 3. The discussion of coding theory and its relation to information theory. The Shannon-Fano code is discussed, along with prefix codes, Kraft's inequality, the noiseless coding theorem, and universal codes for infinite source word sets. 4. The treatment of algorithmic complexity. The authors stress that the information content of an object must be intrinsic and independent of the means of description. 5. The discussion of the explicit universal randomness test. 6. The discussion (in an exercise) of whether a probabilistic machine can perform a task that is impossible on a deterministic machine. 7. The notion of incompressibility of strings. 8. The discussion of randomness in the Diophantine equations; it is shown that the set of indices of the Diophantine equations with infinitely many different solutions is not recursively enumerable; with the initial segment of length n in the characteristic sequence having Kolmogorov complexity n. 9. The discussion on algorithmic probability, especially the test for randomness by martingales. 10. The Solomonoff theory of prediction and its ability to solve the problem of induction. 11. The treatment of Pac-learning and the resultant formalization of Occam's razor. 12. The discussion of compact routing; the optimal space to represent routing schemes in communication networks on the average for all static networks. 13. Computational complexity and its connection to resource-bounded complexity. 14. The notion of logical depth, i.e. the time required by a universal computer to compute the object from its compressed original description. 15. The connection between algorithmic complexity and Shannon's entropy. 16. The discussion on reversible computation, i.e. logically reversible computers that do not dissipate heat. 17. The treatment of information distance, i.e. for two strings, the minimal quantity of information sufficient to translate from one to the other.