Customer Reviews

An Introduction to Information Theory: Symbols, Signals and Noise

5 star:		(16)
4 star:		(7)
3 star:		(2)
2 star:		(0)
1 star:		(0)

 

 

Although old this is still the best book to learn the core ideas of this subject, especially what information "entropy" really means. I read Ash's book, and followed the proofs, but I didn't really grasp the ideas until I read this.

The book is geared towards non-mathematicians, but it is not just a tour. Pierce tackles the main ideas just not all the techniques and special cases.

Perfect for: anyone in science, linguistics, or engineering. Very good for: everyone else.

Claude Shannon died last year, and it's really disgraceful that his name is not a household word in the manner of Einstein and Newton. He really WAS the Isaac Newton of communications theory, and his master's thesis on Boolean logic applied to circuits is probably the most cited ever.

This is the ONLY book of which I am aware which attempts to present Shannon's results to the educated lay reader, and Pierce does a crackerjack job of it. Notwithstanding, this is not a book for the casual reader. The ideas underlying the theory are inherently subtle and mathematical, although there are numerous practical manifestations of them in nature, and in human "information transmission" behavior. On the other hand, this is a work which repays all effort invested in its mastery many times over.

Though first printed in 1961 and revised in 1980 this is the best introduction to information theory there is. Very easy to read and light on math, just as an introduction should be. I expect it will be in print for a very, very long time.

I give this book five stars because it succeeds brilliantly at what it sets out to do - to introduce the field of information theory in an accessible non-mathematical way to the completely uninitiated. Information theory is that branch of mathematics that deals with the information content of messages. The theory addresses two aspects of communication: "How can we define and measure information?" and "What is the maximum information that can be sent through a communications channel?". No other book I know of can explain these concepts of information, bits, entropy, and data encoding without getting bogged down in proofs and mathematics. The book even manages to equate the concept of language with the information it inherently transmits in a conversational and accessible style. The book rounds out its discussion with chapters on information theory from the perspectives of physics, psychology, and art. The only math necessary to understand what's going on in this book is high school algebra and the concept of logarithms. If you are an engineer or engineering student who knows anything about information theory, you probably will not find this book helpful. Instead you would do better to start off with a more advanced book like "An Introduction To Information Theory" by Reza, which introduces concepts from a more mathematical perspective.

Pierce is a very talented educator, and this book demonstrates his ability to take a very deep subject, information and communication theory, and disect it into small, bite-size pieces that don't bite back. i think the best researchers should be able to explain things in the simplest and most intuitive way possible, and Pierce is a clear winner. however, this book is quite old, especially due to the pace of progress in information and coding theory in the past decade. however, this book also gives an excellent overview of the historical development of information theory, which is something that a lot of other books miss.

Pierce's book is an excellent introduction to the subject of information theory. It is not a text on the subject, although it does have some limited mathematical content which is more than the casual reader can handle. The beauty of this book is that unlike most engineers and scientists turned authors, Pierce not only relates much of the history of the subject (from first hand knowledge), but does so with incredible conciseness and clarity. The non-technical approach allows that, and Pierce takes full advantage of his chosen format. It is better to say a non-textual approach really since this isn't a text. Yet, like Feynman, Pierce is able to explain a great amount of the fundamental details of information theory without the rigor of difficult equations and derivations. Any student truly interested in the subject should keep this volume as a companion to their textbook.

The update of this book should have been updated. While it is understandable that at the time of the first print of this book in 1961 the author saw little or no practical use for Shannon's information theory (other than perhaps his channel capacity theorem) it was well known by the second printing in 1980 that it has profound implications in studying biology (and modern technology). For instance in an article published in Nature in 1967, A. L. MacKay showed how the genetic code is highly optimal using Huffman's algorithm. More recently Ardell and Sella (with summaries available on the net) have 'demonstrated that the code's present structure was also shaped by natural selection (though non-Darwinian, see below). In this process, the codons - the triplets of nucleotides that map a particular nucleic acid sequence into proteins - are arranged to minimize the negative effects of genetic error, and to optimize the process of 'readout' of genes during protein synthesis. By permuting all 20 amino acids across all possible codon sets, both groups found that the 'universal' genetic code - the one found in nearly every organism on earth...-falls in the best .0001% of all possible codes and perhaps even better, in terms of its capacity to be an error-correcting code...' By showing modifications are possible in one generation the evidence points away from Crick's thesis of the genetic code being a 'fozen accident' but instead possible Lamarckian beginnings with horizantal gene transfer leading to Carl Woese's early RNA World hypothesis before Darwinian vertical descent begins.

The author also tends to perpetuate the widespread misunderstanding (generally by physicists who tend to contort the meaning away from Shannon's into 'available' states or choices such as with Black Holes) that information is uncertainty; he confuses (readers potentially with) surprise versus information by not taking into account the other half of the necessary equation for information transmission, being noise. He says "The amount of information conveyed by the message increases as the amount of uncertainty as to what message actually will be produced becomes greater." [pg 23] While he clears this up in a later chapter on noise it becomes so technical that it appears most readers of Shannon's theory have been mislead. At this point the scientists (usually physicists who actually work with a different concept of 'available information') typically equate the uncertainty with Kolmogorov complexity and assume that maximum information and complexity is randomess.

For instance consider Philip Nelson's comment in his book Biological Physics that 'random messages carry the most information!' In one footnote of his nearly 600 page book he effectively dismisses all of Nobel Prize winner Shannon's information achievements.

Much of the trouble is with terminology. We think of noise as impure sound. Shannon tried to avoid this problem by introducing the term 'equivocation' but on the other hand this seems to have no intuitive meaning in this context. One really has to go to the math to sort it out. The critical equation to potentially eradicate the confusion does not appear in the book -
R = Hbefore - Hafter
H is an entropy-like formula without Boltzman's constant; however the concepts are very different. (Reportedly Von Neuman told Shannon in the 1940's to call his uncertainty 'entropy, as noone will know what you mean!' Apparently this is still working!) Entropy of the universe apparently increases under the 2nd law of thermodynamics (at least ignoring gravity and extensivity), information begins and ends with life (one needs a recognizer to measure it). A random message in fact carries no information as there is no resolution (reduction) of uncertainty. This is all explained at molecular biologist's Dr. Tom Schneider's website, I know of no other comprehensive source and certainly no book that gets it right. (As yet! 'Hope springs eternal!' A. Pope; 1688 - 1744)

Pierce is an accomplished scientist/engineer, and was influential in the development of information theory/signal processing. This book has some mathematics, but lays a solid qualitative foundation for understanding the material. This book is a classic, good for computer engineers/scientists (as is his book Signals: The Science of Telecommunications). The presentation is accessible, and first hand accounts of important discoveries motivates a real appreciation for Pierce's contributions.

However, the clarity of the presentation tends to obscure just how profound and deep the thinking involved really is. During the first reading, Pierce's insights made the material seem almost obvious. Later I would get doubts that such straightforward approaches could be correct, and then would think about the correctness of his assertions. This is why this is a great book, because it focuses on important stuff, and doesn't shy away from deep topics. This is a great book for those interested in the basis of information theory, on a side note Shannon's original papers are also quite readable.

Without scaring the reader through complex mathematics from page 1, Pierce introduces the subject in an intutive manner, which is the main strength of the Book. Any student interested in studying communication theory without being scared away, should study this book first

This is a good introduction to the concepts of information theory: entropy, stationarity, ergodic sources, efficient coding, error detection, error correction and geometrical modelling. It is aimed at the layman, so there is plenty of explanatory text for each equation presented. If you know basic algebra, probability and logarithms (especially logarithms of base 2), you would get more out this book.

There should have been a more detailed explanation of how signals are transmitted and received; the difficulty of receiving a signal is described in Chapter II and there is a mention of FM transmission in Chapter IX. While the actual transmission and reception of signals may not be part of information theory, including a discussion on the most familiar use of information theory would have made this book more satisfying.

In the second half of the book, Pierce applies information theory to areas other than communications. The chapters relating to physics (X) and art (XIII) are clear, but I found the ones on cybernetics (XI) and psychology (XII) a bit of a strain. For computer-savvy folk, the chapter on cybernetics is very dated but this is no fault of the author since this book was last updated in 1979.

Kam-Hung Soh, 15 May 2005.