Mailman 3 May 2007 - spambayes-dev

SpamBayes developer needed for small paid job
by Robert Savage May 29, 2007

May 29, 2007

Hi, This is a request for paid help from a SpamBayes developer, please. We have source files which are creating a COM object from SpamBayes 1.1a2 or 1.1a3, but the COM isn't working well. This can be because these versions are alpha. We want to create a SpamBayes COM by using the latest stable release, 1.0.4. However, we tried to do it ourselves, and the build was created without errors, but the resulting COM isn't working. So having the COM correctly built with version 1.0.4 is the first task. Once we have the COM, it's possible that the SpamBayes COM still won't function as expected; in which case the Python source file which implements the COM would need to be fixed. This is the second task, and may not be necessary as migrating to version 1.04 may fix the problem. We do not know Python, and the original author is unreachable. Robert

2 1

spambayes crash due to bad image
by Sjoerd Mullender May 25, 2007

May 25, 2007

I got this crash, probably due to a bad image. The code in ImageStripper.py that calls PIL is currently in a except IOError, but should probably be in a bare except to catch any and all errors from PIL. In my case I got an IndexError. I could make this change, if there is consensus and once Sourceforge lets me get at the spambayes repository. 16:02:17+ sb_imapfilter.py -c SpamBayes IMAP Filter Version 1.1a3 (August 2006). Traceback (most recent call last): File "/ufs/sjoerd/bin/x86_64/sb_imapfilter.py", line 1301, in <module> run() File "/ufs/sjoerd/bin/x86_64/sb_imapfilter.py", line 1281, in run imap_filter.Filter() File "/ufs/sjoerd/bin/x86_64/sb_imapfilter.py", line 1080, in Filter self.unsure_folder, self.ham_folder) File "/ufs/sjoerd/bin/x86_64/sb_imapfilter.py", line 955, in Filter evidence=True) File "/ufs/sjoerd/lib/python2.6/site-packages/spambayes/classifier.py", line 196, in chi2_spamprob clues = self._getclues(wordstream) File "/ufs/sjoerd/lib/python2.6/site-packages/spambayes/classifier.py", line 498, in _getclues for word in Set(wordstream): File "/ufs/sjoerd/lib/python2.6/site-packages/spambayes/tokenizer.py", line 1284, in tokenize for tok in self.tokenize_body(msg): File "/ufs/sjoerd/lib/python2.6/site-packages/spambayes/tokenizer.py", line 1643, in tokenize_body text, tokens = crack_images(engine_name, parts) File "/ufs/sjoerd/lib/python2.6/site-packages/spambayes/ImageStripper.py", line 367, in analyze pnmfiles, tokens = PIL_decode_parts(parts) File "/ufs/sjoerd/lib/python2.6/site-packages/spambayes/ImageStripper.py", line 144, in PIL_decode_parts image.load() File "/ufs/sjoerd/lib/python2.6/site-packages/PIL/ImageFile.py", line 189, in load s = read(self.decodermaxblock) File "/ufs/sjoerd/lib/python2.6/site-packages/PIL/PngImagePlugin.py", line 349, in load_read cid, pos, len = self.png.read() File "/ufs/sjoerd/lib/python2.6/site-packages/PIL/PngImagePlugin.py", line 92, in read len = i32(s) File "/ufs/sjoerd/lib/python2.6/site-packages/PIL/PngImagePlugin.py", line 40, in i32 return ord(c[3]) + (ord(c[2])<<8) + (ord(c[1])<<16) + (ord(c[0])<<24) IndexError: string index out of range -- Sjoerd Mullender

2 1

Standalone SpamBayes classifier for websites
by skip＠pobox.com May 16, 2007

May 16, 2007

CC'ing Richard Jones - Roundup guru and Reimar Bauer - MoinMoin guru. Reimar, I don't seem to have Marian Neagul's email handy. Can you forward this to him? I've been trying (rather unsuccessfully) to figure out how to integrate a SpamBayes classifier into Roundup. Basically I know zilch about Roundup's code. You need to score form submissions (the easy part), save them for later retraining and allow misclassified submissions to be reinjected into the website (the hard parts). I had similar problems when I tried to incorporate SpamBayes into MoinMoin. These sites generally treat all submissions as valid. Presuming we have a SpamBayes training database and classifier we can talk to it's a fairly easy task to score a submission and reject it if it looks like spam. Alas, if the submission is scored as spam Roundup and MoinMoin have no convenient way to save the submission yet keep it sequestered so it doesn't turn up on the web. It occurred to me yesterday that the SpamBayes POP3 proxy and IMAP filter solve the storage and classification problems for the specific case where you're talking those two email protocols. The only trick is that they are tied to POP3 and IMAP. Instead of email I need some other way to get a "message" into and out of the classifier/database manager. Given an arbitrary form submission I should be able to convert it to a MIME message (file uploads map to attachments) and hand it off to a standalone SpamBayes server for scoring and storage. If the submission is originally marked as spam (or unsure) but is later deemed okay, I should be able to convert the MIME message back into the necessary bits for resubmission. If the submission is originally marked as ham but is later deemed to be spam the regular Roundup or MoinMoin facility for deleting tickets, pages or attachments would get rid of it. Alas, sb_server.py and sb_imapfilter.py don't seem to share a lot of code (save for using Dibbler to build the web user interface). Is that true? It seems the user interface, classifier bits and storage should be essentially identical. All that should be different between the them is the way you transmit messages to and from external systems: sink ^ | +------------------+ +----------+ | | | | | Core |<------>| Protocol | | Server | | Adapter | | | | | +------------------+ +----------+ ^ ^ | | v source web & msg storage For POP3 the source would be the email client and the sink would be the real POP3 server. For IMAP the source and sink would be the IMAP server. For websites the source and sink would be the web site (Roundup, MoinMoin, etc). The data sent from the protocol adapter to the core server would be MIME messages. The data sent to the protocol adapter would be simply score info (ham, spam, unsure, perhaps raw scores). Any ideas on the shortest route to a core server that provides the user, training and storage interfaces? Start from scratch? Rip the POP3 stuff out of sb_server.py? Rip the IMAP stuff out of sb_imapfilter.py? I'd really hate to reinvent the wheel since we seem to have two wheels already. Once that core server is available, adapting to different environments should be possible by plugging in specific protocol adapters Thx, Skip

2 2

Vista compatibilty
by Marko von Oppen May 10, 2007

May 10, 2007

Under Vista win32traceutil does not work with normal user privileges. I have published a patch on Sourceforge which switches back to the Logfile mechanism from the binary release in such an environment. Marko

1 0

I used SpamBayes for class project - thanks!
by Elizabeth Gifford May 3, 2007

May 3, 2007

Hi SpamBayes developers, I just wanted to let you know that I used SpamBayes for my term project in my Machine Learning class, and to say thanks. I thought there might be some academic interest here, and that I'd share with the list in return for the great code of yours that I used. The overall assignment was to "do something with evolutionary algorithms." The goal of my project was more or less to attack a spam filter and attempt to use an EA to find weaknesses it might have. I set up a blind watchmaker sort of natural selection/ evolution environment using a SpamBayes filter that I had trained on an old training corpus as the fitness function. The program I wrote attempted to evolve a generic formula that worked sort of like "MadLibs" to write spam emails that could get past the filter. I started off with almost zero knowledge of how a spam filter works, and tried to keep it that way while I wrote the algorithm so that it would be as naive as possible. I've since read up on SpamBayes and some of the math behind it, and it looks really interesting. I'm looking forward to taking more time to look through the codebase. Now that the project is completed, I see where I could have trained the filter on different content or parts of the evolution structure that I could have written differently to make this project more successful, but that's research for you! This was a very quick proof of concept project, and if I were to continue with this research there are certainly a lot of things I would do differently and many more complicated techniques that I'd like to try. If anyone is interested in reading about the project in more detail, I've posted the paper I wrote here: http://www.cs.brandeis.edu/~egifford/classes/egiffordTermProject.pdf Thanks again for maintaining this cool project. I hope you'll take my contribution in the intellectual sense that is intended, and not as an attack. :) Liz Gifford

1 0