Mailman 3 September 2006 - spambayes-dev

Ancient spambayes-dev messages popping up, maybe via spamlab.co.uk?
by skip＠pobox.com Sept. 26, 2006

Sept. 26, 2006

I received four or five spambayes-dev messages today that were dated in 2004. Looking at the received headers: Received: from mail.mojam.com [198.49.126.96] by montanaro.dyndns.org with IMAP (fetchmail-6.3.4) for <skip@localhost> (single-drop); Mon, 25 Sep 2006 13:02:47 -0500 (CDT) Received: from fence.pobox.com (fence.pobox.com [208.210.124.76]) by orca.mojam.com (Postfix) with ESMTP id EAABF1170024 for <skip(a)mail.mojam.com>; Mon, 25 Sep 2006 12:01:04 -0600 (MDT) Received: from fence.pobox.com (localhost [127.0.0.1]) by fence.pobox.com (Postfix) with ESMTP id 6A7CB72C for <skip(a)mail.mojam.com>; Mon, 25 Sep 2006 14:01:26 -0400 (EDT) Delivered-To: skip(a)pobox.com Received: from prodrts7.prod.spamlab.co.uk (unknown [213.208.70.139]) by fence.pobox.com (Postfix) with ESMTP id EF14B755 for <skip(a)pobox.com>; Mon, 25 Sep 2006 14:01:21 -0400 (EDT) Received: from prodrts0 (prodrts0.prod.spamlab.co.uk [172.16.139.50]) by prodrts7.prod.spamlab.co.uk (8.13.1/8.13.1) with SMTP id k8PIuu4g012727; Mon, 25 Sep 2006 19:56:56 +0100 List-Archive: <http://mail.python.org/pipermail/spambayes-dev> X-Spam-Status: No, hits=-4.9 required=5.0 tests=AWL, BAYES_00 autolearn=ham version=2.70-cvs List-Post: <mailto:spambayes-dev@python.org> In-Reply-To: Message from "Seth Goodman" <nobody(a)spamcop.net> of "Tue, 13 Jan 2004 17:40:29 CST." <MHEGIFHMACFNNIMMBACACEFAHDAA.nobody(a)spamcop.net> it appears they originated somewhere outside of python.org, perhaps spamlab.co.uk. (cc'ing their postmaster just in case...) Skip

4 5

Considering IronPython ...
by Sean True Sept. 18, 2006

Sept. 18, 2006

We're considering sponsoring a project to port the Spambayes code base to IronPython, and to then port the InBoxer product on to that code base. We would be interested in funding the first part and contributing the port back to the community, and funding the second part, while greedily keeping the results of the second part for ourselves. As you might imagine, I'd like to find someone capable of and interested in doing _both_. Comments, thoughts, or volunteers welcome -- in private or in forum. -- Sean Sean True InBoxer, Inc.

1 0

multi-frame GIFs
by skip＠pobox.com Sept. 10, 2006

Sept. 10, 2006

I checked a change to spambayes/ImageStripper.py that purports to handle multi-frame images. It selects the frame most likely to contain text as the one with the fewest number of pixels in the background color. That's easily circumvented, but I'll wait until the spammers accomplish that feat. I also removed netpbm support. I'll wager a beer that anyone who has netpbm either has or can install PIL, making it the more logical choice. Skip

1 0

Junk E Mail Folder Problem
by John Gagliardi Sept. 8, 2006

Sept. 8, 2006

My Junk E Mail folder has a red circle with a line through it. I can't delete it or move it. I can't even drag it to the Deletec Items folder. What caused this and what can I do to correct the problem?

1 0

Domain expiration
by Rob Hooft Sept. 7, 2006

Sept. 7, 2006

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 It is that time of year again: I will have to renew the spambayes.org domain name before 2006-11-02. Has anything changed in some relevant organization somewhere that the domain name should be transferred, or shall I just pay another year myself? Regards, Rob - -- Rob W.W. Hooft || rob(a)hooft.net || http://www.hooft.net/people/rob/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFE+SZWH7J/Cv8rb3QRApyaAJ9P42fzvAxJULn4Kq0b1iLHqAc9GgCfRpMJ nmQkDBP8jS63CLUdlh3311E= =8x2T -----END PGP SIGNATURE-----

4 4

Re: [spambayes-dev] Choosing which image to OCR
by Robert Mezzone Sept. 6, 2006

Sept. 6, 2006

This probably doesn't help u much but as an fyi: We run Trend ScanMail 7.0 on our Exchange server and until a few months ago it kept our mailboxes spam free. Being a long time user or spambayes, even I was suprised how well their spam filter worked. Then we started seeing a lot of spam, mainly the image based stuff, get past their filter. I called them and they told me they were developing a new scan engine. I haven't checked the logs to see when it was updated but they figuered out a way to dectect this junk because our mailboxes are once again spam free. -----Original Message----- From: spambayes-dev-bounces(a)python.org <spambayes-dev-bounces(a)python.org> To: Mark Hammond <mhammond(a)skippinet.com.au> CC: spambayes-dev(a)python.org <spambayes-dev(a)python.org> Sent: Wed Sep 06 00:05:19 2006 Subject: Re: [spambayes-dev] Choosing which image to OCR >> I find it hard to believe it's in response to what we're doing here. >> I'm sure some other much bigger groups must be doing OCR analysis of >> image-based spam these days. Mark> Apparently SpamAssassin is developing OCR support. I found some Mark> vaguely interesting info from this slashdot discussion: Mark> http://it.slashdot.org/article.pl?sid=06/09/04/1712233 Thanks for the pointer. That led me to gocr and libgocr (http://jocr.sf.net/). That seems to be what the SA folks are using for OCR. At first blush, gocr seems to be about as good (or as bad) as ocrad, though the text it generates appears to be bad in some dimension orthogonal to ocrad's badness. I haven't looked at libgocr yet. Skip _______________________________________________ spambayes-dev mailing list spambayes-dev(a)python.org http://mail.python.org/mailman/listinfo/spambayes-dev

1 0

Integrate SpamBayes into a Win32 application2
by Vilmos Schnedarek Sept. 6, 2006

Sept. 6, 2006

Hi, I want to integrate SpamBayes into a Win32 application and I would like to ask your help for it. I read about something similar on the internet and there the sb_bnfilter has been suggested as a solution. I downloaded the sources from CVS and I tried to compile the C version of the sb_bnfilter but it isn't possible because it has several Unix specific C commands, which doesn't exists under Win32. But before I continue to work with sb_bnfilter I would like to get some more inormation about SpamBayes. Here are my questions, please answer them: 1, first at all what are the main modules of the SpamBayes? In the sources I see several files, but I am sure only a few is related to the main functionality and the rest is UI or for other things. 2, what is the architecture of the app? it is a client-server application, meaning a server daemon is running in the background and one or more clients are connecting to it to request mail scanning services? Or it is a stand-alone application, which is loaded in memory every time when a mail is scanned then is unloaded when it finish? 3, what storage is used to save the training informations and where is it located? 4, what is the minimum required source files to have Spambayes working? Here I mean working without UI and use it in command line, by sb_filter. PS: I apologize posting this message for second time but the first one has been sent by mistake in HTML format. Thanks, Vilmos

2 2

Choosing which image to OCR
by skip＠pobox.com Sept. 6, 2006

Sept. 6, 2006

I took a few minutes to examine a couple (as in exactly two) multi-frame GIF images from stock spams I received in the past couple days. I'd like a cheap test to decide which frame is the best candidate for OCR without OCRing every frame. The computational costs are high enough already. I have two images, bogus-0.gif and bogus-1.gif (both attached to this message). For each one I ran the following loop: >>> img = Image.open("bogus-0.gif") >>> for (i, frame) in enumerate(ImageSequence(img)): ... bg = max(frame.histogram()) ... npixels = len([x for x in frame.histogram() if x]) ... print bg, npixels For bogus-0.gif I got: 220259 33 217760 52 213225 96 182636 256 222500 1 For bogus-1.gif I got: 326518 5 322180 9 322817 7 280174 11 314741 10 It seems that the frame with the fewest white pixels (or the fewest pixels in the most frequently used palette position) is a decent indicator of the frame with the most useful pixels. I also tried this more expensive test at the shell: % for f in bogus-1-?.png ; do echo "*** $f ***" pngtopnm $f | ocrad | wc -c done *** bogus-1-0.png *** 8 *** bogus-1-1.png *** 31 *** bogus-1-2.png *** 18 *** bogus-1-3.png *** 1219 *** bogus-1-4.png *** 340 The fourth frame does indeed have the most text. This didn't work for the bogus-0.png file because this save loop in Python didn't work properly: >>> img = Image.open("bogus-0.gif") >>> for (i, frame) in enumerate(ImageSequence(img)): ... frame.save(open("bogus-0-%d.png" % i, "wb")) The first frame saved has the proper palette. The other saved frames are just black-and-white. (Can someone with more PIL experience explain why this is so and how to get around it?) I can imagine a spammer putting together a palette where there are 248 not-quite-white palette entries making up an essentially white background and a few entries devoted to displaying text. I'm sure PIL has something we could use to quantize the palette down to 16 colors or so, then use that palette to compute histograms, so I'm not all that worried about that scheme. The spammers do seem to be adapting very quickly (take a look at the frames in bogus-1.gif to see what I mean). I find it hard to believe it's in response to what we're doing here. I'm sure some other much bigger groups must be doing OCR analysis of image-based spam these days. BTW, don't worry too much if your mail program won't display the two images properly. XEmacs didn't like them at all, but Mozilla displayed them just fine. Skip

2 2

Ocrad vs Tesseract OCR
by Tony Meyer Sept. 5, 2006

Sept. 5, 2006

I plan to do this myself at some point, but thought Skip (or someone else) might want to beat me to it: Google/UNLV have (re)released an open-source* OCR engine, which they claim is better than any other open-source OCR engine. So it would be interesting to compare the classification with this to that with ocrad. http://google-code-updates.blogspot.com/2006/08/announcing-tesseract- ocr.html =Tony.Meyer * The license is a bit vague, unfortunately. They state it can be freely used/distributed for research/development, and that for commercial use you have to contact the authors. However, they don't cover the middle ground (non-commercial non-research), which SpamBayes falls under.

2 5

Latest CVS update, Ocrad for Windows
by skip＠pobox.com Sept. 4, 2006

Sept. 4, 2006

I updated the OCR capabilities a bit more today. I added more intelligent assembly of split images into a single image after noticing that the spammers don't simply chop up multi-part GIF images horizontally. I also added a couple extra options (ocrad_scale and ocrad_charset) which control the image scaling factor (default is 2) and character set (default is "ascii") Ocrad uses. Scaling the image by a factor of 2 was a pretty obvious win: false positive percentages 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied won 0 times tied 5 times lost 0 times total unique fp went from 0 to 0 tied mean fp % went from 0.0 to 0.0 tied false negative percentages 4.213 4.213 tied 1.404 0.843 won -39.96% 3.371 2.809 won -16.67% 2.528 2.247 won -11.12% 4.213 3.652 won -13.32% won 4 times tied 1 times lost 0 times total unique fn went from 56 to 49 won -12.50% mean fn % went from 3.14606741573 to 2.75280898876 won -12.50% Scaling by a factor of three was even better in the false negative department but regressed a bit in the false positive category so I checked Options.py in with a default scaling factor of 2. A couple things could stand to be further tested: * I have no idea how good Ocrad's scaling algorithm is. It's possible that PIL or NetPBM's scaling code is better. If so, it would make sense to scale the images before feeding to Ocrad. * The images I've see so far were all plain English, so I blindly made ascii the default charset. The other choices were iso-8859-9 and iso-8859-15. I simply assumed ascii would be the most appropriate default, but didn't test it. Finally, I put together a really simpleminded Ocrad-for-Windows release based upon the ocrad.exe binary that Tony built. Check the Files section of the SpamBayes project site: http://sourceforge.net/project/showfiles.php?group_id=61702 and grab ocrad-cygwin. There are a few caveats: 1. I don't do Windows. (No, really, I don't, strange as that may seem.) This is no fancy-schmancy point-and-shoot Windows installer. It's just a simple zip file with the Ocrad 0.15 distribution, Tony's .exe file and the patch he applied to the source. 2. I don't do Windows. The code I've written so far has been done entirely on my Mac. I've made no obvious concessions to portability. That said, I hope portability issues won't be daunting for any early adopters. 3. I don't do Windows. If you have problems it won't do you any good to mail me directly. Post about problems on the SpamBayes bug tracker: http://sourceforge.net/tracker/?group_id=61702&atid=498103 4. If you do Windows you will need PIL to take advantage of the recent changes: http://www.pythonware.com/products/pil/ (unless you want to put hair on your chest and build NetPBM on Windows). Fredrik Lundh provides prebuilt Windows versions of PIL. Grab the one appropriate for the version of Python you have installed. 5. If you do Windows (or any other platform for that matter), feedback to the lists about successes and failures would be helpful. Cheers, Skip

4 7