Elements of Information Theory [Hardcover]

Thomas M. Cover (Author), Joy A. Thomas (Author)

Book Description

ISBN-10: 0471062596 | ISBN-13: 978-0471062592 | Publication Date: August 12, 1991 | Edition: 99th

Following a brief introduction and overview, early chapters cover the basic algebraic relationships of entropy, relative entropy and mutual information, AEP, entropy rates of stochastics processes and data compression, duality of data compression and the growth rate of wealth. Later chapters explore Kolmogorov complexity, channel capacity, differential entropy, the capacity of the fundamental Gaussian channel, the relationship between information theory and statistics, rate distortion and network information theories. The final two chapters examine the stock market and inequalities in information theory. In many cases the authors actually describe the properties of the solutions before the presented problems.

Editorial Reviews

From the Publisher

Following a brief introduction and overview, early chapters cover the basic algebraic relationships of entropy, relative entropy and mutual information, AEP, entropy rates of stochastics processes and data compression, duality of data compression and the growth rate of wealth. Later chapters explore Kolmogorov complexity, channel capacity, differential entropy, the capacity of the fundamental Gaussian channel, the relationship between information theory and statistics, rate distortion and network information theories. The final two chapters examine the stock market and inequalities in information theory. In many cases the authors actually describe the properties of the solutions before the presented problems.

From the Inside Flap

Elements of Information Theory is an up-to-date introduction to the field of information theory and its applications to communication theory, statistics, computer science, probability theory, and the theory of investment. Covering all the essential topics in information theory, this comprehensive work provides an accessible introduction to the field that blends theory and applications. In step-by-step detail, the authors introduce the basic quantities of entropy, relative entropy, and mutual information and show how they arise as natural answers to questions of data compression, channel capacity, rate distortion, hypothesis testing, information flow in networks, and gambling. In addition, Elements of Information Theory investigates a number of ideas never before covered at this level in a textbook, including:
* The relationship of the second law of thermodynamics to Markov chains
* Competitive optimality of Huffman codes
* The duality of data compression and gambling
* Lempel Ziv coding
* Kolmogorov complexity
* Portfolio theory
* Inequalities in information theory and their consequences in mathematics
Complete with numerous illustrations, original problems and historical notes, Elements of Information Theory is a practical reference for communications professionals and statisticians specializing in information theory. It also serves as an excellent introductory text for senior and graduate students taking courses in telecommunications, electrical engineering, statistics, computer science, and economics.

See all Editorial Reviews

Product Details

• Hardcover: 542 pages
• Publisher: Wiley-Interscience; 99th edition (August 12, 1991)
• Language: English
• ISBN-10: 0471062596
• ISBN-13: 978-0471062592
• Product Dimensions: 9.6 x 6.4 x 1.3 inches
• Shipping Weight: 2.1 pounds
• Average Customer Review: 4.4 out of 5 stars  See all reviews (15 customer reviews)
• Amazon Best Sellers Rank: #585,758 in Books (See Top 100 in Books)

Customer Reviews

Most Helpful Customer Reviews

83 of 87 people found the following review helpful:

5.0 out of 5 stars Great intermediate to advanced entry into information theory, October 25, 2000

By 

Julius Kusuma (Cambridge, MA, USA) - See all my reviews
(REAL NAME)   

This review is from: Elements of Information Theory (Hardcover)

Thomas Cover is a well-known researcher for both his excellent and sometimes surprising work in information theory, and his reputation as a teacher. The result here is a very well-written and gentle "overview" of information theory that is designed as a comprehensive introduction to the subject.

One thing to note about this book is that it is by design both an introduction and a survey of information theory, as the title suggests. It starts off with the basic concepts of information theory such as entropy and mutual information, and continues on with brief and gentle reviews of different more intermediate topics such as entropy rates in random processes, introduction to coding, and finally with the channel coding theorem, rate-distortion theorem, network information theory, and other more advanced topics.

While I find that his treatment of the intermediate and advanced topics to be excellent, there are a few weak aspects on this book's treatment of the introductory topics here and there. However, with just a little persistence the reader will be well rewarded by Cover's excellent writing. At each topic, the reader is presented with reason, motivation, intuition and example before delving into the rigorous treatment of the subject. Therefore even the most casual reader will be rewarded with good insights into the different topics in information theory.

That all said, I highly recommend this book to anybody armed with elementary probability who is interested in the general area of communication, signal processing and information theory. Readers who are alergic to math are recommended to start with J.R. Pierce's "Introduction to Information Theory" and readers looking for a casual introduction to the fundamental concepts in information theory are recommended to find a copy of A. Renyi's hard to find "A Diary on Information Theory".

24 of 26 people found the following review helpful:

5.0 out of 5 stars Very thorough and readable intro text, July 22, 1999

By 

Roger Peng (Baltimore, MD USA) - See all my reviews
(REAL NAME)   

This review is from: Elements of Information Theory (Hardcover)

Cover and Thomas have written an excellent book on information theory. This book is suitable for an introductory type course but the entire book probably cannot be covered at once. All of the explanations and proofs are very understandable, although the section on types can be a tad confusing at times! I would not recommend this book to someone with a casual interest in information theory, rather to someone who wants a more rigorous treatment of the underlying mathematics.

9 of 9 people found the following review helpful:

5.0 out of 5 stars The best pick for the beginner., January 7, 2002

By 

ALAC (New York, NY USA) - See all my reviews

This review is from: Elements of Information Theory (Hardcover)

If you want a single book on information theory, you must have this one. Tom Cover's writing style is great. He wrote a clear, concise, and understandable book on a somewhat involved topic. The notation is neat and clean, something difficult to find in Information Theory books. The problems at the end of each chapter are very useful and a great aid to master the material. There is also a summary at end of each chapter with the main definitions and results just studied. The intuition behind Information Theory important quantities such as entropy and mutual information is well explained through several toy examples and Venn diagrams. The channel coding theorem for example is thoroughly discussed. There is also a sketchy and intuitive proof of the capacity of the AWGN channel based on a sphere packing argument which is given before the more rigorous one comes along. This is the standard text on information theory, deservedly so.