Customer Reviews

Ending Spam: Bayesian Content Filtering and the Art of Statistical Language Classification

5 star:		(8)
4 star:		(4)
3 star:		(1)
2 star:		(1)
1 star:		(1)

 

 

The sub-title of this scared me a bit, because it sounds like heavy geek territory. A review of chapter titles raised my eyebrows a too: "Fifth Order Markovian Discrimination" - I visualized page after page of unintelligible mathematical symbols.

That's not the case at all. Actually Markovian Discrimination is a technique I've used in other programming efforts, and the author explains it in simple and entertaining language. There's nothing here that any competent programmer can't grasp.

I'm a little hesitant to call this book entertaining, although it absolutely is. I only hesitate because that might give the impression that it's more fluff than substance, and that's not the case at all. There's a lot of substance here, both in theory and in practical advice. And although the subject is definitely spam, some of the techniques and methods discussed here apply to other programming challenges as well.

The first part of the book is especially enjoyable. It's a history of spam, and I learned things I hadn't known before about spam's early days. It then segues into analysis; in a sense you get desert before the meat and potatoes.

Overall, worth reading, even by non-programmers wanting to understand more about what current anti-spam efforts are all about.

Author Jonathan A. Zdziarski starts this book by giving the reader a history of Spam as well as the historical approaches to fighting Spam. This is followed by a very practical guide for the serious Spam fighter; including details on statistical filtering, tokenization, Markovian discrimination, and Bayesian filtering. Although it is very technical in many respects most readers should be able to comprehend the text if they read carefully. Readers who already understand the basics of filtering and email analysis will find it both easy and educational to read.

The author includes an excellent section on spammer tricks and how they get past fileters as well as what to do about it. This section alone makes the book worth the price. Ending Spam is a highly recommended read for anyone in charge of controlling spam in a corporate environment as well as on their own system.

Ending Spam from Mr. Zdziarski is a well written BASIC and easy to understand INTRODUCTION to get a technical overview of todays spam fighting solutions on the market.

Also it is written on the cover that it is f.e focused towards developers, network admins etc. I would consider the target customer to be IT Managers, or other curious people who want to get an overview.

Thats what it does and it does it very well in my eyes.
The book provides simplified, abstract overviews of some available spam filters solutions.

The book is provided into 3 parts

- An Introduction part to spam filtering (Chapter 1-4)
- A part describing "Fundamentals of Statistical Filtering" (Chapter 5-9)
- an the third part describing "Advanced Concepts of Statistical Filtering" (Chapter 10-14)

Its a bit confusing that Chapter 4 has the same title than Part II. So perhaps Chapter 4 should have been part of "Part II" ?

The Chapters which I found most interesting were:

Chapter 4 "Fundamentals of Statistical Filtering"
Chapter 7 "The Low down dirty Tricks of spammers"
Chapter 9 "Scaling in Large Environments"

I am sure the author could have easily filled the book with Chapter 7 alone. The book is very entertaining and has a nice motivating writing style. You might at times find some rant about the spammers which I have chosen to ignore as it doesnt contain any valuable information or anything which I didnt know already. While I might agree to some of the authors views, I believe that the rant does unfortunately do exactly the opposite in my eyes and does give spammers credit to how they do their work.

I personally was actually looking for a companion book to "The Book of Postfix" to help me further explore new anti spam technology.
I was hoping to find overview charts, being able to compare different solutions,features, (dis)advantages. So in this sense, I was actually looking for workshop style instructions, tuning advice, troubleshooting advice etc.

The authors does explain f.e (Chapter 14) Collaborative Algorithms but he does not go into detail which products support the feature and how to perform the setup. He does provide some weblinks in his book from which the interested reader might further investigate the topic.

From reading the Chapter10 on "Testing Theory" its easier to conclude why the author doesnt go into more detail. If he would have done so, the book could have been easily 2-3 times the size.

I assume, this is partly due to the fact that the anti spam technology /products/market is still fairly young .


Summary:

"Ending Spam" gives a very BASIC INTRODUCTION to the current available Anti spam technology and some chosen products. After you have read the book you have a first vague idea what type of solutions exist. You will actually need other books to intensify the "knowledge" you have gained here.

The fact that the book is written in simple terms makes it easily acessable for a wide market, however if you are a technichian you will perhaps find that the book just doesnt contain enough "meat" for you.

I would still recommend the book for Managers which need to know only the rough details, beginners, or a first time read for newcomers.

Reading this book was fun. I was doing some research on spam and found this book was exactly what I was looking for. This book covers (almost) all aspects of spam, including the history, the current status, the principles of anti-spam systems, statistical algorithms, case studies, etc. This book is a good start point for understanding spams and means to stop them, although it does not contain a lot of in-depth technical details. I was amazed by the author's style, which was quite energetic and entertaining. This book made my research a pleasant experience. I strongly recommend this book for those who are interested to know how spams came and how we fight them.

This is one of my favorite NLP books because it offers an extremely readable introduction to Bayesian statistics in a very applied context. If you don't have a strong background in statistics and/or text classification, this book is a great way to get an intuitive feel for how Bayesian classifiers work. If you're a developer looking to do some coding, what's explained in the book is easy to translate into code. I recommend this book to upper-level undergrads and graduate students in linguistics who take an applied computational linguistic class I teach.

I have been watching the development of dspam for quite a while and was interested in the concept behind the new crop of spam detection engines.

This book gives you a lot of background on the growth of spam and the various type of spam on how each new variant of spam effects different type of spam detection with obvious concentration on statistical analysis and not the tradition aka "dumb" forms of regex matching and etc.

The only gripe I have is that I feel there is too much time spent on the growth/history of spam. It would be better if the entire book is dedicated at the art/science of statisical analysis and fighting spam.

Note that this book does not target the lowest denominator and some the science flew over my head. Well, I did flunk almost all my higher ed physics and stat classes so take this with a spoon of salt.

In an extraordinarily well-researched book, one of the few areas where it fails to deliver on its promise is Zdziarski's disappointingly simplistic definition of spam-filter "accuracy". On page 185, in a surprisingly brief discussion of this key metric, he defines accuracy as (100 - error percentage), where the error rate is the *total* number of misclassifications divided by the number of messages. Unfortunately, this equation gives equal weighting to missed junk mail as to legitimate messages which are mistakenly spam-binned (false positives). Any user of a spam filter will tell you that a false positive error is *far* more significant than a sneaky spam sliding into their inbox, especially if their junk mail is quarantined on a server.

Professional anti-spam researchers and filter developers have long recognized that any single-percentage "accuracy" metric is an apparition having such a high coefficient of bogosity that only a marketeer could love it! When comparing (or improving) spam filters, the most significant accuracy measurements are the true False Positive (FPR) and False Negative rates (FNR), derived over a statistically significant number of messages (e.g. N > 100k). These rates provide us with "sensitivity" and "specificity" percentages, which together clearly indicate the quality (and the underlying aggression setting) of a filter's mail-discrimination logic.

Most modern filtering engines provide for user-configurable aggression settings. A "lenient" setting reduces the risk of false positives, but lets more spam get through. After training their filter, many users opt to increase the aggression level, thereby reducing spam leakage (but risking a higher level of false positives). Statistics from high-volume mail feeds at service providers clearly indicates that false positives are perceived as far more costly, by a factor of 10-100 times, than false negative errors (misses). Any single-figure "accuracy" percentage would be far more useful if the false positives were weighted by ~25 in the calculation.

With many open-source and commercial filters now approaching seriously high levels of filtration accuracy, it's increasingly difficult to compare technologies (or implementations) without a robust definition of "accuracy", plus a reasonably large test corpus. Otherwise, test results will be "way down in the noise", in which case we can't compare filter performance with any degree of statistical confidence.

All that aside, I only wish I'd been able to read a book like this a year ago! A definite buy recommendation for anyone interested in the nuts and bolts of 3rd generation (probabilistic) email filtering technology.

Awesome read. For those who are in the SpamAssassin mindset and are considering DSPAM, this is a definite must!

The problem of spam is of enormous significance. You may spend only a few minutes a day deleting unwanted unsolicited e-mail. Yet multiply this time spent by the number of individuals dealing with the same task and you have unwanted work of an extraordinary magnitude. In Ending Spam Jonathan A. Zdziarski provides a highly readable, while at the same time, technical treatment of the problem of spam. The reader of this lucid work will acquire background in the history, theory and current direction of spam detection technology.

Does the average computer user need to know about Bayesian filtering techniques and external innoculation? The answer is "yes" and here's why: Increasingly, the technologies used to handle spam are implemented by ISPs at the mail server level. That means that ISPs may be making decisions about what e-mail messages are delivered to you. That is, they may very well censor your mail when they suspect that it is spam. If spam filters always detected actual spam, that would be just fine. But as Zdziarski shows, the problem of spam analysis is a difficult and constantly changing problem in computational linguistics. In order to understand the challenge of e-mail flow and delivery, a problem that affects each and every user of e-mail, one must be acquainted with the variety of spam delivery techniques and spam detection techniques available to both ISPs and individual users.

Although this is a technical work, it is highly accessible, and its value goes beyond covering spam. The author writes clearly and with enthusiasm about the intellectual challenge posed by spam analysis. The theoretical and technological issues covered in the book go well beyond the narrow (but still very important) topic of spam itself. Anyone who is interested in how the analysis of the syntactic properties of language can be mined for semantic, i.e. meaningful information, will be interested in this book.

This book provides the history of spam, so we know how it all started, as well as the reasoning and theories behind the current spam technologies, whithout getting bogged down in minutia. I found this book quick and enjoyable to read. Very informative. Highly suggested if you are a sysAdmin (like me) who has or will build a spam filter, or wants to know how they work and why. Good for programmers as well looking for the theories.