Latest CVS update, Ocrad for Windows
I updated the OCR capabilities a bit more today. I added more intelligent assembly of split images into a single image after noticing that the spammers don't simply chop up multi-part GIF images horizontally. I also added a couple extra options (ocrad_scale and ocrad_charset) which control the image scaling factor (default is 2) and character set (default is "ascii") Ocrad uses. Scaling the image by a factor of 2 was a pretty obvious win: false positive percentages 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied won 0 times tied 5 times lost 0 times total unique fp went from 0 to 0 tied mean fp % went from 0.0 to 0.0 tied false negative percentages 4.213 4.213 tied 1.404 0.843 won -39.96% 3.371 2.809 won -16.67% 2.528 2.247 won -11.12% 4.213 3.652 won -13.32% won 4 times tied 1 times lost 0 times total unique fn went from 56 to 49 won -12.50% mean fn % went from 3.14606741573 to 2.75280898876 won -12.50% Scaling by a factor of three was even better in the false negative department but regressed a bit in the false positive category so I checked Options.py in with a default scaling factor of 2. A couple things could stand to be further tested: * I have no idea how good Ocrad's scaling algorithm is. It's possible that PIL or NetPBM's scaling code is better. If so, it would make sense to scale the images before feeding to Ocrad. * The images I've see so far were all plain English, so I blindly made ascii the default charset. The other choices were iso-8859-9 and iso-8859-15. I simply assumed ascii would be the most appropriate default, but didn't test it. Finally, I put together a really simpleminded Ocrad-for-Windows release based upon the ocrad.exe binary that Tony built. Check the Files section of the SpamBayes project site: http://sourceforge.net/project/showfiles.php?group_id=61702 and grab ocrad-cygwin. There are a few caveats: 1. I don't do Windows. (No, really, I don't, strange as that may seem.) This is no fancy-schmancy point-and-shoot Windows installer. It's just a simple zip file with the Ocrad 0.15 distribution, Tony's .exe file and the patch he applied to the source. 2. I don't do Windows. The code I've written so far has been done entirely on my Mac. I've made no obvious concessions to portability. That said, I hope portability issues won't be daunting for any early adopters. 3. I don't do Windows. If you have problems it won't do you any good to mail me directly. Post about problems on the SpamBayes bug tracker: http://sourceforge.net/tracker/?group_id=61702&atid=498103 4. If you do Windows you will need PIL to take advantage of the recent changes: http://www.pythonware.com/products/pil/ (unless you want to put hair on your chest and build NetPBM on Windows). Fredrik Lundh provides prebuilt Windows versions of PIL. Grab the one appropriate for the version of Python you have installed. 5. If you do Windows (or any other platform for that matter), feedback to the lists about successes and failures would be helpful. Cheers, Skip
Hi Skip,
Scaling the image by a factor of 2 was a pretty obvious win:
false positive percentages 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied
I'm playing a little with the new code and am trying to get things working with outlook. I'm a little stuck working out how to get some test data (and it doesn't help I'm a little rusty wrt to spambayes :) I'm trying to run the testtools code. The Outlook code that sets up the Data/Ham, Data/Spam directories etc just exports the text body of the message, but completely ignores 'attachments'. I'm out of time for tonight - can you offer any quick clues how your test environment is setup? Thanks, Mark.
Mark> I'm trying to run the testtools code. The Outlook code that sets Mark> up the Data/Ham, Data/Spam directories etc just exports the text Mark> body of the message, but completely ignores 'attachments'. I'm Mark> out of time for tonight - can you offer any quick clues how your Mark> test environment is setup? Quick clue: I'm not using Outlook or Windows. ;-) I don't know what to do given that Outlook shreds email so completely. Maybe this stuff can only be tested on Unix-y machines. Maybe the image analysis code won't even work because there's no such thing as an attachment with MIME content-type image/*... in Outlook. As for actual setup, it's done in what I think is the "usual" way. I start with two or more Unix mbox format files (at least one full of ham, one full of spam). I then run utilities/splitndirs.py to allocate them to the desired number of Data/{Ham,Spam}/SetN directories. I then make a series of runs like so: # control run python testtools/timcv.py ... args ... > std.txt python testtools/rates.py std.txt # one or more test runs with various parameters changed python testtools/timcv.py ... slightly different args ... > testN.txt python testtools/rates.py testN.txt python testtools/cmp.py stds.txt testNs.txt My guess is there's an easier way to run the tests and summarize the results, but it had been awhile since I'd done any testing either. This was the first "working" setup I stumbled upon, and thanks to my enormous bash command history buffer, I just recall the commands as I need them, so the pain of re-remebering is small. HTH, Skip
Quick clue: I'm not using Outlook or Windows. ;-)
Yep, I know that :) My mail was sent fairly late, so I didn't explain very well.
I don't know what to do given that Outlook shreds email so completely. Maybe this stuff can only be tested on Unix-y machines. Maybe the image analysis code won't even work because there's no such thing as an attachment with MIME content-type image/*... in Outlook.
I can manage all of that. What I need to know is in what format your Ham and Spam directories are. Currently mine are in plain-text. A quick look at the code showed that these were *not* expected to be a dump of a mime message, but instead a simple "word stream" - which didn't seem to fit with the binary data inside attachments. I was guessing they had already been processed to some degree, but gave up before digging deeper.
As for actual setup, it's done in what I think is the "usual" way. I start with two or more Unix mbox format files (at least one full of ham, one full of spam). I then run utilities/splitndirs.py to allocate them to the desired number of Data/{Ham,Spam}/SetN directories. I then make a series of runs like so:
hrm - so maybe they *are* just the complete dump of the message including the encoded image data and mime boundaries etc - I'll play a little more and look inside splitndirs. Thanks, Mark
I don't know what to do given that Outlook shreds email so completely. Maybe this stuff can only be tested on Unix-y machines. Maybe the image analysis code won't even work because there's no such thing as an attachment with MIME content-type image/*... in Outlook.
I can manage all of that. What I need to know is in what format your Ham and Spam directories are.
They're RFC2822. So for mail in a .pst, presumably the job (of export_messages.py) would be to get the attachments and insert them into the messages (encoded in base64 or whatever) with the appropriate headers. I planned to write code to do this at some point last year, but don't recall getting around to it (and then I switched to Mail as my main email client).
hrm - so maybe they *are* just the complete dump of the message including the encoded image data and mime boundaries etc
Yup. Is that accessible in Outlook? I had the feeling it wasn't. If you can get the attachments then it's easy enough to use the email package to build up the message with those and the plain text. =Tony.Meyer
Mark> hrm - so maybe they *are* just the complete dump of the message Mark> including the encoded image data and mime boundaries etc - I'll Mark> play a little more and look inside splitndirs. Yup, plain old RFC 2822 messages... Skip
skip@pobox.com ha scritto:
I updated the OCR capabilities a bit more today. I added more intelligent assembly of split images into a single image after noticing that the spammers don't simply chop up multi-part GIF images horizontally. I also added a couple extra options (ocrad_scale and ocrad_charset) which control the image scaling factor (default is 2) and character set (default is "ascii") Ocrad uses. Scaling the image by a factor of 2 was a pretty obvious win:
false positive percentages 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied
won 0 times tied 5 times lost 0 times
total unique fp went from 0 to 0 tied mean fp % went from 0.0 to 0.0 tied
false negative percentages 4.213 4.213 tied 1.404 0.843 won -39.96% 3.371 2.809 won -16.67% 2.528 2.247 won -11.12% 4.213 3.652 won -13.32%
won 4 times tied 1 times lost 0 times
total unique fn went from 56 to 49 won -12.50% mean fn % went from 3.14606741573 to 2.75280898876 won -12.50%
Scaling by a factor of three was even better in the false negative department but regressed a bit in the false positive category so I checked Options.py in with a default scaling factor of 2. A couple things could stand to be further tested:
* I have no idea how good Ocrad's scaling algorithm is. It's possible that PIL or NetPBM's scaling code is better. If so, it would make sense to scale the images before feeding to Ocrad.
* The images I've see so far were all plain English, so I blindly made ascii the default charset. The other choices were iso-8859-9 and iso-8859-15. I simply assumed ascii would be the most appropriate default, but didn't test it.
Finally, I put together a really simpleminded Ocrad-for-Windows release based upon the ocrad.exe binary that Tony built. Check the Files section of the SpamBayes project site:
http://sourceforge.net/project/showfiles.php?group_id=61702
and grab ocrad-cygwin.
There are a few caveats:
1. I don't do Windows. (No, really, I don't, strange as that may seem.) This is no fancy-schmancy point-and-shoot Windows installer. It's just a simple zip file with the Ocrad 0.15 distribution, Tony's .exe file and the patch he applied to the source.
2. I don't do Windows. The code I've written so far has been done entirely on my Mac. I've made no obvious concessions to portability. That said, I hope portability issues won't be daunting for any early adopters.
3. I don't do Windows. If you have problems it won't do you any good to mail me directly. Post about problems on the SpamBayes bug tracker:
http://sourceforge.net/tracker/?group_id=61702&atid=498103
4. If you do Windows you will need PIL to take advantage of the recent changes:
http://www.pythonware.com/products/pil/
(unless you want to put hair on your chest and build NetPBM on Windows). Fredrik Lundh provides prebuilt Windows versions of PIL. Grab the one appropriate for the version of Python you have installed.
5. If you do Windows (or any other platform for that matter), feedback to the lists about successes and failures would be helpful.
Cheers,
Skip
_______________________________________________ spambayes-dev mailing list spambayes-dev@python.org http://mail.python.org/mailman/listinfo/spambayes-dev
Hi, I'm very interested in this OCR and in the way SpamBayes analyzes image spam. Now there is a new kind of image spam using animated images and I've received a lot of "animated spam" lately so it's possible they could be very common in a brief period. Here you can find a brief description about this: http://www.viruslist.com/en/weblog?weblogid=196822613 I would like to ask you how your OCR manages this kind of images. Thank you a lot for your time. Regards -- Michele Belloli Research & Development Dept. Symbolic - Network Security Distributor http://www.symbolic.it eXtensiveControl La nuova soluzione di Content Filtering per la PMI http://www.extensivecontrol.it/
Michele> Now there is a new kind of image spam using animated images and Michele> I've received a lot of "animated spam" lately so it's possible Michele> they could be very common in a brief period. Here you can find Michele> a brief description about this: Michele> http://www.viruslist.com/en/weblog?weblogid=196822613 Michele> I would like to ask you how your OCR manages this kind of Michele> images. Right now it doesn't. I'm aware of the shortcoming. I just haven't had time to work on the problem. Skip
participants (4)
-
Mark Hammond -
Michele Belloli -
skip＠pobox.com -
Tony Meyer