Mr. A.J. O'Neill wrote on Friday, November 17, 2006 7:25 PM -0500:

One of the many steps involved in the processing was the removal (or ignoring of) punctuation before searching for search tokens. I draw your attention to the following extract from a Spam Clues report

'beneficiary' 0.844828 0 1 'beneficiary.' 0.844828 0 1

I would argue that there is no difference between these two tokens and that the inclusion of the punctuation adds nothing to the process but in this instance is likely to give the token a lower score than may be appropriate.

This type of specific choice in the tokenizer resulted from testing in a number of people's working environments. It was shown to improve classification empirically. This suggests that the intuition behind your argument, which I originally shared as well, is not correct for the purpose of classifying email as ham/spam at the time this was tested. A lot of the small choices in Spambayes turn out to be the results of empirical testing rather than intuition, and it's surprising (non-intuitive) how often our intuition about our own language is incorrect. If you're looking for a reason to explain the empirical results, one possibility is that it provides differentiation based on grammar, as opposed to just word occurrence. This is something that you normally don't get with a tokenizer that only recognizes words and not sentence structure.

I also used a stop list of words which are so common that they are useless to index or use in search engines or other search indexes. Below are a number of instances of words which I believe are not appropriate tokens to use to differentiate between spam and ham emails.

There is a clash between the philosophy of naive Bayesian classification and rule-based schemes. The idea behind rule-based schemes is that we can tap human beings' pattern recognition ability to create rules that we run in a computer. Since we can recognize spam easily when we see it, we are the best experts to consult when forming a rule set. The problem with this notion is that computers are not currently capable of creating inferences in the same way as people because the system architecture is so different. While people can indeed reliably distinguish spam, often from only a part of the message, they cannot reliably tell you how they made the decision. The aim of naive Bayesian classification is to avoid all the particular problems of trying to construct a useful rule set and instead look at simple statistical properties of language the do not require human-like inference. The underlying model is fundamentally different. A Bayesian classifier is not trying to emulate a speaker of natural language. The approach has strengths as well as weaknesses. One of the strengths is that you don't have to decide what words you think are the best or worst spam indicators. If you tend to favor rule-based approaches, this also looks like a huge weakness. The classifier learns word probabilities by observing your message classifications. To the extent that you are surprised by the spam probabilities of individual words, you would make the classifier worse by manually overriding the training results on a token-by-token basis. This happens far more often than you would think. Words that indicate a spam likeliness equal to a ham likeliness score somewhere near 0.5 and do not contribute to the final score. Another of the strengths is that the word probabilities vary widely among different recipients. It's a strength because there is no such thing as a ham word list that will reliably avoid Bayesian classifiers. That's also a weakness, if you wish to apply Bayesian methods on a server without tracking the word probabilities separately for each mailbox. What this suggests is that it is equally difficult to come up with a list of words that the classifier should ignore that would work for most users. There is a fundamental disagreement in the approaches of Bayesian and rule-based systems. Proponents of rule-based systems believe that people can best identify what clues are most significant, while proponents of Bayesian systems either believe that people cannot reliably identify the most important clues, or even if they can, they don't care to do so. The last condition is important if spam avoidance is simply a utilitarian goal, not a hobby. Personally, I tried rule-based systems first and then experimented with Spambayes. I found that my intuition on word probabilities was indeed wrong a significant proportion of the time and the naive Bayesian approach did about as well as my rule-based system when it was at its peak. The Bayesian approach required much less maintenance and it works well for a wide variety of end-users without requiring insight from them. I still feel there are very useful rules to help detect spam that are complimentary to word frequency. These are things such as whether the message comes from a particular mailing list, whether the sending IP is on a DNS blacklist that I choose or to which one of my mailbox addresses the message is addressed. My own compromise on this is to either put them in the domain MTA, or to write Outlook rules that run before the Bayesian classifier. In terms of overall system architecture, I tend to believe that the rule-based approaches belong in the domain MTA, whenever possible, and should generate rejections during the SMTP session, preferably before DATA. This eliminates most of the spam at the lowest possible system cost and with the largest savings in bandwidth. You can eliminate another significant amount of spam by running rule-based content filters, such as SpamAssassin, in the MTA. This is very expensive, so it is important to run it on as few messages as possible. This generates rejections at the end of DATA, which are still useful for legitimate messages that are improperly classified. For the spam that slips through global rule-base systems, it then makes sense to do computationally intensive and user-specific content filtering like Spambayes in the MUA. The spam load is hopefully reduced enough that the end-user doesn't mind scanning the junk folder for the occasional false positive. -- Seth Goodman