Mailman 3 August 2004 - spambayes-dev

message subject filtering
by John Moriarty Sept. 1, 2004

Sept. 1, 2004

message subject filtering I'm not a programmer. I have asked a similar question before, but recently, mounting spams have made me alter it significantly, and so I hope it worthy of reconsideration. A lot of spam shows: * Ungrammatical and/or irrelevant wording * Random words * Gibberish words * Deliberately weird or obscure punctuations * Since this is true in the header as well as the text body, this potentially reduces the loads on the filter. Random words not seen before seem to allow stuff through more easily. Therefore the presence of these certain features, I don't know for sure if they fall under the definition of tokens, are high probability signals. Is this what's new -error signals computed from the entire (or a substantial subset of) message? * I also note spam outnumbers ham by up to 100 to one, so header filtering seems good at throwing up warnings. And invariably the text body contains the web address of the seller, so a web address of itself is a giveaway. I am fast at identifying spam by the header alone, using the above observations I reckon I spot 90% plus in a blink. However it's still a pain rubbing them out. It seems to me that application of rules based the above would be a more sophisticated way of developing spambayes. I think the analysis of text would then focus in better on the more subtle forms of spam, using the tokens to greater effect. Apologies if any of this is rubbish or goes against the theories. Kind regards, John Moriarty (+353) (0)87 2833 530 www.helimodels.com

4 4

The naive bayes classifier algorithm in spambayes doesn't take in frequency?
by Austine Jane Aug. 31, 2004

Aug. 31, 2004

Hello. I have a question on the naive bayes classifier algorithm used in spambayes. I suppose if word1 appeared in three ham mail, the probability of word1 being in ham mail would be greater than when it appeared in one ham mail: (using spambayes-1.0rc2) >>>c=storage.DBDictClassifier('test.db') >>>def tok(s): return s.split() >>>c.learn(tok('word1'),is_spam=False) >>>c.spamprob(tok('word1')) 0.15517241379310343 >>>c.learn(tok('word1'),False) >>>c.spamprob(tok('word1')) 0.091836734693877542 >>>c.learn(tok('word1'),False) >>>c.spamprob(tok('word1')) 0.065217391304347783 As you see the spam probability declines. So far so good. >>>c.learn(tok('word1'),True) And word1 also appeared in one spam mail, but it appeared in three ham mail before. >>>c.spamprob(tok('word1')) 0.5 Hm... Sounds not very right. >>>c.learn(tok('word1'),False) >>>c.spamprob(tok('word1')) 0.5 Stays still. This doesn't sound intuitive. For example, word1 occurred in 1000 spam email and occured in 1 ham mail. What is the probability of one mail that contains word1 being spam mail? Half and half? Doesn't it take in the number of occurences(it does seem to take in the number of distinct tokens though)? It seems like the concept of the number of occurences and the number of distinct tokens are mixed in spambayes' classifier. Machine Learning by Tom Mitchell(esp. page 183 and http://www-2.cs.cmu.edu/afs/cs/project/theo-11/www/naive-bayes.html see also p.s.) suggests a formula that gives a quite different result from spambayes' classifier. It always takes in the number of occurences, hence more intuitive. Am I missing something big? Thanks in advance, Jane ------------- p.s. The formula in Tom Mitchell's book is: Vocabulary is the set of all distinct words and other tokens occuring in any text document from Examples For each target value v_j in V do * docs_j is the subset of documents from Examples for which the target value is v_j * P(v_j)= | docs_j | / | Examples | * Text_j is a single document created by concatenating all members of docs_j * n is the total number of distinct word positions in Text_j * for each word w_k in Vocabulary * P(w_k|v_j) = ( n_k + 1 )/( n + |vocabulary| ) * n_k is the number of times word w_k occurs in Text_j As you see it uses m-estimate. _________________________________________________________________ Help STOP SPAM with the new MSN 8 and get 2 months FREE* http://join.msn.com/?page=features/junkmail

3 2

problem with "junk suspects"
by Joshua Ronen Aug. 29, 2004

Aug. 29, 2004

I kept the default threshold at 50% and yet no items are ever moved to this folder. What's wrong? Professor Joshua Ronen Stern School of Business New York University 40 West 4th St. Tel: 212-998-4144 Fax: 212-995-4599 E-mail: <mailto:jronen@stern.nyu.edu> jronen(a)stern.nyu.edu Web: www.stern.nyu.edu/~jronen/

1 0

Re: Chung-Kwei algorithm (from BBC news)
by Graham Toal Aug. 26, 2004

Aug. 26, 2004

Firmicus(a)ankabut.net quoted a press release: > 'DNA analysis' spots e-mail spam > > By Jo Twist > > BBC News Online science and technology staff > > > Few would have thought that when Crick and Watson discovered DNA, > it would help in making a tool to fight spam. > > But computational biologists at IBM's TJ Watson Research Center > have devised an anti-spam filter based on the way scientists analyse > genetic sequences. > > Called after Feng Shui character Chung-Kwei, the formula automatically > learns patterns of spam vocabulary and has proved to be 96.5% efficient. > > In tests, the filter only misidentified one message in 6,000 as spam. > > Pillar of protection > > Isidore Rigoutsos and Tien Huynh, at IBM's bioinformatics and pattern > discovery research group, started to develop the formula - or algorithm > - a little over a year ago. > > They named the formula, Chung-Kwei, after a Feng Shui character who is > usually shown carrying a bat, and also holds a sword behind him. > > He is an important figure for those involved in business and who have > expensive goods that need protection. > > Chung-Kwei grew out of another algorithm called Teiresias which the > researchers were using for pattern discovery in computation biology > sequencing, specifically, in protein annotation. > > "To train 88,000 messages takes about 15 minutes on a normal single processor. > If, an hour, later we have more spam we can add to the collection so we keep > on learning more and more" > -- Isidore Rigoutsos, IBM > > The algorithm helped in automatically determining the properties of > a protein, like function and structure, directly from a string. > > "Obviously algorithms that pertain to pattern discovery are applicable > to a vast range of problems," Mr Rigoutsos explained to BBC News Online. > > Instead of looking at strings of protein, Chung-Kwei uses Teiresias to > identify strings of character sequences which appear in spam, but never ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exactly my algorithm > in non-spam mail. > > Their work, said Mr Rigoutsos, was helped by the large volume of spam > which they received at their own workplace. > > "We have lots of e-mails that we know are bona fide spam. If we run a > pattern analysis on those, it can see letters that appear frequently. > > "One of the properties of the algorithm is that it will spot two or ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > more occurrences. It doesn't matter where it is in the message. ^^^^^^^^^^^^^^^^ Exactly what mine does > > "If you do this, effectively you get small collections of letters so > you can think of these as a vocabulary of sorts. If you have lots of > data to work with, your vocabulary will be able to describe the data > in a different form." > > Spam training > > The algorithm can be trained so that it will not be fooled by cunning > replacements of "S" with "$", a common ploy used by spammers to bypass > conventional e-mail filters. > > The Chung-Kwei method builds up its database of known true-spam > patterns and constantly adds new patterns it spots. > > It compares its vocabulary to e-mails which it knows do not contain > spam. So, an incoming message hit with this pattern analysis will be > rejected if it contains a large proportion of the same vocabulary > patterns. > > If a message received had a lot of spam patterns in it, it was scored > highly. Chung-Kwei succeeded in spotting almost 97% of junk mails. > > "We experimented with large collections of e-mail. We have 66,000 > training messages that are all spam and 22,000 training messages that > are all 'white' [non-spam]. > > "To train 88,000 messages takes about 15 minutes on a normal single > processor. If, an hour, later we have more spam, we can add to the > collection so we keep on learning more and more." > > Various anti-spam software use several techniques to spot and kill > junk mail, but IBM believes the Chung-Kwei algorithm to be the only ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > anti-spam tool that uses pattern discovery in this way. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (mine does) > > Some tools look at the route an e-mail has taken and its origins; > others involve identity verification and black and white listing > of accepted and not accepted addresses. > > Others use Bayesian combinations of individual words that statistically ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (ie this doesn't) > make up spam messages. > > The system has to go through some more pilot studies and testing > before it is let loose to protect inboxes. I published what I am fairly sure is the same algorithm, or at least close enough to invalidate a patent on the grounds of prior art, back in February of this year. I also mailed it to a few prominent researchers like Bill Yerazunis, just to get it on the record. (One of those researchers was a spam researcher at IBM incidentally, though I don't believe he was involved in the project quoted above) Here's the links, decide for yourself. I'm not interested in taking credit for this, if it is the same algorithm, just in making sure it is not patented. ------------------ ] Date: Fri, 18 Jun 2004 12:33:02 -0500 ] From: Graham Toal <gtoal(a)gtoal.com> ] To: mertz(a)gnosis.cx ] Subject: spam algorithms ] Message-ID: <40D3274E.mailN3311EYMT(a)gtoal.com> ] User-Agent: nail 10.5 4/27/03 ] MIME-Version: 1.0 ] Content-Type: text/plain; charset=us-ascii ] Content-Transfer-Encoding: 7bit ] ] I was searching the IBM search engine for spam algorithms and ] found your page: ] ] > . Bayesian trigram filters ] > I decided to look into how well a much more starkly limited ] > model space would work for a Bayesian spam filter. ] > Specifically, I decided to use trigrams for my probability ] > model rather than "words". ] ] Have a look at some notes I wrote up earlier this year: ] http://www.gtoal.com/mt/archives/2004_02.html ] ] You may find this approach interesting. It's sort of like your idea but ] taken to the extreme. Subsequent to writing the description in the above ] link, I have actually hacked up some prototype code and the idea does show ] promise. I've not had the resources to make a fully usable spam filter ] out of it, but it's a convincing proof of concept. ] ( http://www.gtoal.com/spam/devel-temp/tokra3.c.html ) ] ] I hope you don't mind me mailing this to you unsolicited, but I want ] a few people working in the field to be aware of it as I think it is ] original and by making it public I can forstall someone reinventing ] it later and claiming a bogus patent on it! ] ] ] Regards ] ] Graham Toal --------------------------- I am BCC'ing this post to Isidore Rigoutsos and Tien Huynh for their opinion as to whether their "pattern discovery" algorithm and my "token recognition" algorithm are the same algorithm (and to ask as to whether IBM is attempting to patent it...) G

1 0

crash from sb_imapfilter.py
by Sjoerd Mullender Aug. 26, 2004

Aug. 26, 2004

I received a couple of junk mails that caused my sb_imapfilter.py to crash. The problem was that the messages contained a space inside the brackets of a Message-ID (even though that's probably illegal). This caused the "id" of the message to contain a space, and this gave difficulty in the IMAP SEARCH command in the Save method. I see two possible solutions: - strip() the message-id before it is assigned to the "id" field, or - properly quote the search command in the Save method. I think it's probably a good idea to do the latter anyway, so here is a patch. I can check this in if there is agreement. (I haven't tested whether this actually works with these messages, but it does work for normal messages.) -- Sjoerd Mullender <sjoerd(a)acm.org> Index: scripts/sb_imapfilter.py =================================================================== RCS file: /cvsroot/spambayes/spambayes/scripts/sb_imapfilter.py,v retrieving revision 1.38 diff -u -r1.38 sb_imapfilter.py --- scripts/sb_imapfilter.py 19 Aug 2004 13:11:45 -0000 1.38 +++ scripts/sb_imapfilter.py 26 Aug 2004 08:21:28 -0000 @@ -557,8 +557,9 @@ # Although we don't use the UID to keep track of messages, we do # have to use it for IMAP operations. self.imap_server.SelectFolder(self.folder.name) - search_string = "(UNDELETED HEADER %s %s)" % \ - (options["Headers", "mailid_header_name"], self.id) + search_string = "(UNDELETED HEADER %s \"%s\")" % \ + (options["Headers", "mailid_header_name"], + self.id.replace('\\',r'\\').replace('"',r'\"')) response = self.imap_server.uid("SEARCH", search_string) data = self.imap_server.check_response("search " + search_string, response)

1 0

Chung-Kwei algorithm (from BBC news)
by Firmicus＠ankabut.net Aug. 25, 2004

Aug. 25, 2004

Hello spambayes developers, Heard of this yet? Regards, F ======== 'DNA analysis' spots e-mail spam By Jo Twist BBC News Online science and technology staff Few would have thought that when Crick and Watson discovered DNA, it would help in making a tool to fight spam. But computational biologists at IBM's TJ Watson Research Center have devised an anti-spam filter based on the way scientists analyse genetic sequences. Called after Feng Shui character Chung-Kwei, the formula automatically learns patterns of spam vocabulary and has proved to be 96.5% efficient. In tests, the filter only misidentified one message in 6,000 as spam. Pillar of protection Isidore Rigoutsos and Tien Huynh, at IBM's bioinformatics and pattern discovery research group, started to develop the formula - or algorithm - a little over a year ago. They named the formula, Chung-Kwei, after a Feng Shui character who is usually shown carrying a bat, and also holds a sword behind him. He is an important figure for those involved in business and who have expensive goods that need protection. Chung-Kwei grew out of another algorithm called Teiresias which the researchers were using for pattern discovery in computation biology sequencing, specifically, in protein annotation. "To train 88,000 messages takes about 15 minutes on a normal single processor. If, an hour, later we have more spam we can add to the collection so we keep on learning more and more" -- Isidore Rigoutsos, IBM The algorithm helped in automatically determining the properties of a protein, like function and structure, directly from a string. "Obviously algorithms that pertain to pattern discovery are applicable to a vast range of problems," Mr Rigoutsos explained to BBC News Online. Instead of looking at strings of protein, Chung-Kwei uses Teiresias to identify strings of character sequences which appear in spam, but never in non-spam mail. Their work, said Mr Rigoutsos, was helped by the large volume of spam which they received at their own workplace. "We have lots of e-mails that we know are bona fide spam. If we run a pattern analysis on those, it can see letters that appear frequently. "One of the properties of the algorithm is that it will spot two or more occurrences. It doesn't matter where it is in the message. "If you do this, effectively you get small collections of letters so you can think of these as a vocabulary of sorts. If you have lots of data to work with, your vocabulary will be able to describe the data in a different form." Spam training The algorithm can be trained so that it will not be fooled by cunning replacements of "S" with "$", a common ploy used by spammers to bypass conventional e-mail filters. The Chung-Kwei method builds up its database of known true-spam patterns and constantly adds new patterns it spots. It compares its vocabulary to e-mails which it knows do not contain spam. So, an incoming message hit with this pattern analysis will be rejected if it contains a large proportion of the same vocabulary patterns. If a message received had a lot of spam patterns in it, it was scored highly. Chung-Kwei succeeded in spotting almost 97% of junk mails. "We experimented with large collections of e-mail. We have 66,000 training messages that are all spam and 22,000 training messages that are all 'white' [non-spam]. "To train 88,000 messages takes about 15 minutes on a normal single processor. If, an hour, later we have more spam, we can add to the collection so we keep on learning more and more." Various anti-spam software use several techniques to spot and kill junk mail, but IBM believes the Chung-Kwei algorithm to be the only anti-spam tool that uses pattern discovery in this way. Some tools look at the route an e-mail has taken and its origins; others involve identity verification and black and white listing of accepted and not accepted addresses. Others use Bayesian combinations of individual words that statistically make up spam messages. The system has to go through some more pilot studies and testing before it is let loose to protect inboxes. The research was originally reported in the New Scientist magazine. Story from BBC NEWS: http://news.bbc.co.uk/go/pr/fr/-/2/hi/technology/3584534.stm Published: 2004/08/25 09:38:12 GMT © BBC MMIV

1 0

email, mala direta, propaganda e-mail, marketing por e-mails listas de divulgação
by Paula O. Grassi Aug. 23, 2004

Aug. 23, 2004

Visite agora: http://www.divulgamail.mx.gs mala direta e-mail, email regiões, e-mails região, mala direta por email, marketing e-mail, regiões, cadastro e-mails, publicidade por email, emails região, divulgar, enviar emails, campanha emails, propaganda emails, email cidade, envio anônimo emails, email estados, divulgar e-mail, programas emails, e-mails por estados, e-mails cidade, cadastro e-mail, mala direta por e-mail, listas emails, e-mail regiões, propaganda email, enviar email anônimo, envio mala direta, estados, campanha, cidade, envio, publicidade e-mails, Visite agora: http://www.divulgamail.mx.gs campanhas e-mail, lista e-mail, programas e-mails, e-mails estado, publicidade emails, marketing digital, cidade, divulgar, lista email, emails estados, propaganda digital e-mails, e-mail por regiões, e-mails por cidades, email cidades, campanha e-mail, e-mail estado, listas email, lista emails, propaganda por e-mails, mala direta email, publicidade, cidades, marketing emails, cidade, email por regiões, envio propaganda, listas e-mails, e-mails regiões, divulgar e-mails, envio mala-direta, e-mail cidades, email estado, e-mails por Visite agora: http://www.divulgamail.mx.gs região, marketing por emails, propaganda, software email em massa, propaganda digital e-mail, programas email, email, mala direta, propaganda e-mail, marketing e-mails, e-mail, mala-direta email, propaganda digital, emails por região, email segmentado, estado, campanhas e-mails, e-mails cidades, e-mails segmentados, email por estado, marketing por email, emails segmentado, divulgação, e-mails estados, cidade, campanha e-mails, software, email segmentados, região, enviar e-mails anônimo, enviar emails anônimo, mala direta emails, marketing email, emails segmentados, programas e-mail, e-mails por cidade, lista e-mails, propaganda, mala direta por e-mails, campanha email, software spam internet, Visite agora: http://www.divulgamail.mx.gs emails estado, publicidade e-mail, e-mail por cidades, enviar e-mail anônimo, software propaganda internet, emails cidade, emails, campanhas emails, mala-direta e-mail, publicidade email, mala direta e-mails, e-mail região, listas, listas segmentadas, marketing, marketing digital por emails, email região, divulgação e-mail, emails por cidade, mala-direta por email, marketing digital por e-mails, listas email, lista segmentada, cidades, cadastro email, divulgue seu produto, mala-direta por e-mails, e-mail por estado, segmentos, email por cidades, propaganda por e-mail, emails cidades, publicidade por emails, envio e-mail, e-mails por estado, mala direta, mala-direta, mala-direta por emails, e-mail segmentado, marketing digital emails, cidades, divulgação e-mails, marketing, e-mail estados, cidades, marketing por e-mail, envio emails, marketing digital email, propaganda Visite agora: http://www.divulgamail.mx.gs por email, envio anônimo email, divulgue sua propaganda, propaganda digital emails, cidade, emails por cidades, e-mails segmentado, propaganda por emails, divulgar email, e-mail cidade, enviar e-mails, e-mails, cadastro emails, e-mail por cidade, envio email, cadastro, lista, envio e-mails, propaganda digital email, publicidade por e-mails, marketing digital, e-mail por região, email por estados, divulgação, emails por estados, segmentados, mala-direta emails, envio publicidade, campanhas, mala direta por emails, e-mail por estados, marketing por e-mails, emails por estado, mala-direta e-mails, marketing digital e-mail, divulgar emails, emails regiões, publicidade, email por região, e-mails por regiões, listas e-mail, divulgação emails, mala-direta por e-mail, enviar e-mail, enviar email, Visite agora: http://www.divulgamail.mx.gs divulgação email, cidades, publicidade por e-mail, enviar, emails por regiões, marketing digital por e-mail, email por cidade, campanhas email, marketing digital por email, marketing digital e-mails, propaganda e-mails, e-mail segmentados, envio anônimo e-mail, software publicidade internet, segmentados, envio anônimo e-mails, lista mala direta, programa email anônimo, mala direta internet, publicidade email, mala direta segmentada, emails segmentados, marketing digital, mala direta email, publicidade, spam, mala direta e-mail, email regiões, e-mails região, mala direta por email, marketing e-mail, regiões, cadastro e-mails, publicidade por email, emails região, divulgar, enviar emails, campanha emails, propaganda emails, email cidade, envio anônimo emails, email estados, divulgar e-mail, programas emails, e-mails por estados, e-mails cidade, cadastro e-mail, mala direta por e-mail, listas emails, e-mail regiões, propaganda email, enviar email anônimo, envio Visite agora: http://www.divulgamail.mx.gs mala direta, estados, campanha, cidade, envio, publicidade e-mails, campanhas e-mail, lista e-mail, programas e-mails, e-mails estado, publicidade emails, marketing digital, cidade, divulgar, lista email, emails estados, propaganda digital e-mails, e-mail por regiões, e-mails por cidades, email cidades, campanha e-mail, e-mail estado, listas email, lista emails, propaganda por e-mails, mala direta email, publicidade, cidades, marketing emails, cidade, email por regiões, envio propaganda, listas e-mails, e-mails regiões, divulgar e-mails, envio mala-direta, e-mail cidades, email estado, e-mails por região, marketing por emails, propaganda, software email em massa, propaganda digital e-mail, programas email, email, mala direta, propaganda e-mail, marketing e-mails, e-mail, mala-direta email, propaganda Visite agora: http://www.divulgamail.mx.gs digital, emails por região, email segmentado, estado, campanhas e-mails, e-mails cidades, e-mails segmentados, email por estado, marketing por email, emails segmentado, divulgação, e-mails estados, cidade, campanha e-mails, software, email segmentados, região, enviar e-mails anônimo, enviar emails anônimo, mala direta emails, marketing email, emails segmentados, programas e-mail, e-mails por cidade, lista e-mails, propaganda, mala direta por e-mails, campanha email, software spam internet, emails Visite agora: http://www.divulgamail.mx.gs estado, publicidade e-mail, e-mail por cidades, enviar e-mail anônimo, software propaganda internet, emails cidade, emails, campanhas emails, mala-direta e-mail, publicidade email, mala direta e-mails, e-mail região, listas, listas segmentadas, marketing, marketing digital por emails, email região, divulgação e-mail, emails por cidade, mala-direta por email, marketing digital por e-mails, listas email, lista segmentada, cidades, cadastro email, divulgue seu produto, mala-direta por e-mails, e-mail por estado, segmentos, email por cidades, propaganda por e-mail, emails cidades, publicidade por emails, envio e-mail, e- Visite agora: http://www.divulgamail.mx.gs mails por estado, mala direta, mala-direta, mala-direta por emails, e-mail segmentado, marketing digital emails, cidades, divulgação e-mails, marketing, e-mail estados, cidades, marketing por e-mail, envio emails, marketing digital email, propaganda por email, envio anônimo email, divulgue sua propaganda, propaganda digital emails, cidade, emails por cidades, e-mails segmentado, propaganda por emails, divulgar email, e-mail cidade, enviar e-mails, e-mails, cadastro emails, e-mail por cidade, envio email, cadastro, lista, envio e-mails, propaganda digital email, publicidade por e-mails, marketing digital, e-mail por região, email por estados, divulgação, emails por estados, segmentados, mala-direta emails, envio publicidade, campanhas, mala direta por emails, e-mail por estados, marketing por e- Visite agora: http://www.divulgamail.mx.gs mails, emails por estado, mala-direta e-mails, marketing digital e-mail, divulgar emails, emails regiões, publicidade, email por região, e-mails por regiões, listas e-mail, divulgação emails, mala-direta por e-mail, enviar e-mail, enviar email, divulgação email, cidades, publicidade por e-mail, enviar, emails por regiões, marketing digital por e-mail, email por cidade, campanhas email, marketing digital por email, marketing digital e-mails, propaganda e-mails, e-mail segmentados, envio anônimo e-mail, software publicidade internet, segmentados, envio anônimo e-mails, lista mala direta, programa email anônimo, mala direta internet, publicidade email, mala direta segmentada, emails segmentados, marketing digital, mala direta email, publicidade, spam

1 0

Olimpics
by John Stovas Aug. 22, 2004

Aug. 22, 2004

1 0

The next generation?
by G. Waleed Kavalec Aug. 19, 2004

Aug. 19, 2004

DNA technique protects against 'evil' emails http://www.newscientist.com/news/print.jsp?id=ns99996292 11:34 19 August 04 Exclusive from New Scientist Print Edition. Subscribe and get 4 free issues. A technique originally designed to analyse DNA sequences is the latest weapon in the war against spam. An algorithm named Chung-Kwei (after a feng-shui talisman that protects the home against evil spirits) can catch nearly 97 per cent of spam. Chung-Kwei is based on the Teiresias algorithm, developed by the bioinformatics research group at IBM's Thomas J Watson Research Center in New York, US. Teiresias was designed to search different DNA and amino acid sequences for recurring patterns, which often indicate genetic structures that have an important role. Instead of chains of characters representing DNA sequences, the research group fed the algorithm 65,000 examples of known spam. Each email was treated as a long, DNA-like chain of characters. Teiresias identified six million recurring patterns in this collection, such as "Viagra". Each pattern represented a common sequence of letters and numbers that had appeared in more than one unsolicited message. The researchers then ran a collection of known non-spam (dubbed "ham") through the same process, and removed the patterns that occurred in both groups. Genuine email Incoming email was given a score based on how many spam patterns it had. A long email that only had a few spammy sentences would get a relatively low score; but one with many patterns spread across the length of the message would score much higher. The Chung-Kwei correctly identified 64,665 of 66,697 test messages as being spam or 96.56 per cent. More importantly, its rate of misidentifying genuine email as spam was just 1 in 6000 messages. Losing a single email in a torrent of spam is a greater failing in a filter than letting the occasional spam email through. Chung-Kwei deals with common spammer strategies to dodge pattern-recognition schemes, such as replacing the s with a $, as in "increa$e your $ex power" using its built-in tolerance for different, but functionally equivalent, DNA sequences. Just as in genetic analysis, Teiresias could be taught that CCC and CCU codons both produce the same amino acid, proline, the anti-spam system can be trained to accept $ and s as identical. IBM intends to include Chung-Kwei in its commercial product, SpamGuru. Justin Mason, who developed SpamAssassin, one of the most popular open-source anti-spam filters, says that Chung-Kwei looks promising. "I think there is still a lot of work to be done. But what is exciting is not the particular algorithm, but the fact that IBM has shown there is the entire field of bioinformatics techniques to explore in the fight against spam." Danny O'Brien, San Jose

1 0

Absence
by Tony Meyer Aug. 18, 2004

Aug. 18, 2004

Hi everyone :) Just in case anyone wonders why I've suddenly gone silent and stopped answering people on spambayes(a)python.org, I'm leaving for a business trip/vacation for 3.5 weeks from tomorrow. No email (well, not enough to handle all this mail), so no answering SpamBayes questions. The positive side is that I'm also not here to check anything in and break the scripts, so there should be no *new* problems <wink>. =Tony Meyer p.s. If anyone wants to, there isn't much to do to get 1.0 out the door. The sf release is all done, the source archives are uploaded, the website is uploaded (but not pushed to the live site). All that needs to be done is build the binary (needs to be done with Outlook 2k), upload it, make the release visible, and send out the announcement emails (remembering Anthony's suggestion to make a big deal of the exhaustive alpha/beta cycle). You might be able to get Mark to build the binary, but I haven't managed to get hold of him recently. p.p.s. 1.0 is no different to 1.0rc2, so there shouldn't be any new bugs then, either.

1 0