Elements of Information Theory (Hardcover)

~ Thomas M. Cover (Author), (Author) "I WENT to work when I was just out of grammar school..." (more)
Key Phrases: rate distortion code, degraded broadcast channel, proportional gambling, Occam's Razor, Kullback Leibler, Mona Lisa (more...)

Editorial Reviews

Review

"As expected, the quality of exposition continues to be a high point of the book. Clear explanations, nice graphical illustrations, and illuminating mathematical derivations make the book particularly useful as a textbook on information theory." (Journal of the American Statistical Association, March 2008)

"This book is recommended reading, both as a textbook and as a reference." (Computing Reviews.com, December 28, 2006) --This text refers to an alternate Hardcover edition.

Product Description

Up-to-date introduction to the field of information theory and its applications to communication theory, statistics, computer science, probability theory and the theory of investment. DLC: 1. Information theory.

See all Editorial Reviews

Product Details

• Hardcover: 542 pages
• Publisher: Wiley-Interscience; 99th edition (August 12, 1991)
• Language: English
• ISBN-10: 0471062596
• ISBN-13: 978-0471062592
• Product Dimensions: 9.6 x 6.4 x 1.3 inches
• Shipping Weight: 2.1 pounds
• Average Customer Review: 4.2 out of 5 stars  See all reviews (25 customer reviews)
• Amazon.com Sales Rank: #497,175 in Books (See Bestsellers in Books)

  Popular in this category: (What's this?)

  #99 in

  Books > Computers & Internet > Computer Science > Information Theory

Customer Reviews

Most Helpful Customer Reviews

74 of 76 people found the following review helpful:

5.0 out of 5 stars Great intermediate to advanced entry into information theory, October 25, 2000

By	Julius Kusuma (Cambridge, MA, USA) - See all my reviews (REAL NAME)

Thomas Cover is a well-known researcher for both his excellent and sometimes surprising work in information theory, and his reputation as a teacher. The result here is a very well-written and gentle "overview" of information theory that is designed as a comprehensive introduction to the subject.

One thing to note about this book is that it is by design both an introduction and a survey of information theory, as the title suggests. It starts off with the basic concepts of information theory such as entropy and mutual information, and continues on with brief and gentle reviews of different more intermediate topics such as entropy rates in random processes, introduction to coding, and finally with the channel coding theorem, rate-distortion theorem, network information theory, and other more advanced topics.

While I find that his treatment of the intermediate and advanced topics to be excellent, there are a few weak aspects on this book's treatment of the introductory topics here and there. However, with just a little persistence the reader will be well rewarded by Cover's excellent writing. At each topic, the reader is presented with reason, motivation, intuition and example before delving into the rigorous treatment of the subject. Therefore even the most casual reader will be rewarded with good insights into the different topics in information theory.

That all said, I highly recommend this book to anybody armed with elementary probability who is interested in the general area of communication, signal processing and information theory. Readers who are alergic to math are recommended to start with J.R. Pierce's "Introduction to Information Theory" and readers looking for a casual introduction to the fundamental concepts in information theory are recommended to find a copy of A. Renyi's hard to find "A Diary on Information Theory".

20 of 22 people found the following review helpful:

5.0 out of 5 stars Very thorough and readable intro text, July 22, 1999

By	Roger Peng (Baltimore, MD USA) - See all my reviews (REAL NAME)

Cover and Thomas have written an excellent book on information theory. This book is suitable for an introductory type course but the entire book probably cannot be covered at once. All of the explanations and proofs are very understandable, although the section on types can be a tad confusing at times! I would not recommend this book to someone with a casual interest in information theory, rather to someone who wants a more rigorous treatment of the underlying mathematics.

9 of 9 people found the following review helpful:

5.0 out of 5 stars An Excellent Introduction to Information Theory, May 16, 2008

By	A Reader - See all my reviews

This review is from: Elements of Information Theory 2nd Edition (Wiley Series in Telecommunications and Signal Processing) (Hardcover)

I am writing this review in response to some confusion and unfairness I see in other reviews. Cover and Thomas have written a unique and ambitious introduction to a fascinating and complex subject; their book must be judged fairly and not compared to other books that have entirely different goals.

Claude Shannon provided a working definition of "information" in his seminal 1948 paper, A Mathematical Theory of Communication. Shannon's interest in that and subsequent papers was the attainment of reliable communication in noisy channels. The definition of information that Shannon gave was perfectly fitted to this task; indeed, it is easily shown that in the context studied by Shannon, the only meaningful measure of information content that will apply to random variables with known distribution must be (up to a multiplicative constant) of the now-familiar form h(p) = log(1/p).

However, Shannon freely admitted that his definition of information was limited in scope and was never envisioned as being universal. Shannon deliberately avoided the "murkier" aspects of human communication in framing his definitions; problematic themes such as knowledge, semantics, motivations and intentions of the sender and/or receiver, etc., were avoided altogether.

For several decades, Information Theory continued to exist as a subset of the theory of reliable communication. Some classical and highly regarded texts on the subject are Gallager, Ash, Viterbi and Omura, and McEliece. For those whose interest in Information Theory is motivated largely by questions from the field of digital communications, these texts remain unrivalled standards; Gallager, in particular, is so highly regarded by those who learned from it that it is still described as superior to many of its more recent, up-to-date successors.

In recent decades, Information Theory has been applied to problems from across a wide array of academic disciplines. Physicists have been forced to clarify the extent to which information is conserved in order to completely understand black hole dynamics; biologists have found extensive use of Information Theoretic concepts in understanding the human genome; computer scientists have applied Information Theory to complex issues in computational vs. descriptive complexity (the Mandelbrot set, which has been called the most complex set in all of mathematics, is actually extremely simple from the point of view of Kolmogorov complexity); and John von Neumann's brilliant creation, game theory, which has been called "a universal language for the unification of the behavioral sciences," is intimately coupled to Information Theory, perhaps in ways that have not yet been fully appreciated or explored.

Cover and Thomas' book "Elements of Information Theory" is written for the reader who is interested in these eclectic and exciting applications of Information Theory. This book does NOT treat Information Theory as a subset of reliable communication theory; therefore, the book is NOT written as a competitor for Gallager's classic text. Critics who ask
for a more thorough treatment of rate distortion theory or convolutional codes are criticizing the authors for failing to include topics that are not even central to their goals for the text!

A very selective list of some of the more interesting topics that Cover and Thomas study includes: (1) the Asymptotic Equipartition Property and its consequences for data compression; (2) Information Theory and gambling; (3) Kolmogorov complexity and Chaitin's Omega; (4) Information Theory and statistics; and (5) Information Theory and the stock market. Item (4) on this list is only briefly introduced in Cover and Thomas's book, and appropriately so; however, readers who wish to pursue the fascinating subject of Fischer Information further should consider B. Roy Frieden's book Physics from Fisher Information: A Unification. Frieden identifies a principle of "extreme physical information" as a unifying theme across all of physics, deriving such classic equations as the Klein-Gordon equation, Maxwell's equations, and Einstein's field equations for general relativity from this information-theoretic principle.

This last point is quite typical of Cover and Thomas's book. I participated in a faculty seminar on Information Thoery at my university a few years ago, in which we studied Cover and Thomas as our primary source. We were a diverse group, drawn from five different academic disciplines, and we all found that Cover and Thomas repeatedly introduced us to exciting and unexpected applications of Information Theory, always sending us to the journals for further, more in-depth study.

Cover and Thomas' book has become an established favorite in university courses on information theory. In truth, the book has few competitors. Interested readers looking for additional references might also consider David MacKay's book Information Theory, Inference, and Learning Algorithms, which has as a primary goal the use of information theory in the study of Neural Networks and learning algorithms. George Klir's book Uncertainty and Information considers many alternative measures of information/uncertainty, moving far beyond the classical log(1/p) measure of Shannon and the context in which it arose. Jan Kahre's iconoclastic book The Mathematical Theory of Information is an intriguing alternative in which the so-called Law of Diminishing Information is elevated to primary axiomatic status in deriving measures of information content. I alluded to some of the "murkier" issues of human communication earlier; readers who wish to study some of those issues will find Yehoshua Bar-Hillel's book Language and Information a useful source.

In conclusion, I highly recommend Cover and Thomas' book on Information Theory. It is currently unrivalled as a rigorous introduction to applications of Information Theory across the curriculum. As a person who used to work in the general area of signals analysis, I resist all comparisons of Cover and Thomas' book with the classic text of Gallager; the books have vastly different goals and very little overlap.