Mailman 3 December 2006 - spambayes-dev

FAQ 6.5
by Geo. Jan. 2, 2007

Jan. 2, 2007

"FAQ 6.5 Why can't I bounce spam back to the sender?" The reasons you give are of course perfectly valid. However, to my mind the most obvious reason for not bouncing spam and one that is seldom acknowledged is the much more important consideration of traffic congestion. The internet is already groaning under the strain of billions of spam messages. If every one of them were to be bounced spam-mail traffic would in effect be doubled and the system would probably be unable to cope. At the very least, it would necessitate a great increase in world-wide server capacity the cost of which would almost certainly be passed on by ISPs to their clients. It is unfortunate that some of your competitors proudly proclaim as a 'feature', the ability of their programme to bounce spam messages. I can only conclude that they are either very shortsighted or irresponsible morons. Please consider the foregoing purely as comment. It is not intended to be critical of SpamBayes in any way. Perhaps you could expand the answer to FAQ 6.5 to reflect the above, e.g. something along the lines "In addition, bouncing spam messages simply increases email traffic leading to even further congestion on the internet." might suffice. Regards, Geo.

2 1

Re: [spambayes-dev] [Spambayes] Images of commercial text with decoy text are mushing my index
by skip＠pobox.com Dec. 29, 2006

Dec. 29, 2006

Jamie> I am being assaulted by a new sort of spam where the commercial Jamie> message is contained in an image, and the text of the message is Jamie> just there as a decoy ... The current alpha of SpamBayes has some OCR capability to deal with this. We've yet to make an official release. Mark Hammond was working with it a week or so ago and seemed to think it was working for him on Windows. I developed the new code in a Unix (actually MacOSX) environment, so once Mark is satisfied perhaps we can turn the knobs on an official release. -- Skip Montanaro - skip(a)pobox.com - http://www.mojam.com/ "The hippies and the hipsters did some great stuff in the sixties, but the geeks pulled their weight too." -- Billy Bragg

1 0

Stats are very slow
by Mark Hammond Dec. 28, 2006

Dec. 28, 2006

Hi guys, I'm playing a little more with integrating the outlook addin with Skip's ocrad code and making some progress. However, I noticed that while spambayes was taking much less than a second to load my (bsddb) databases, it takes nearly 30 seconds to load the stats! Specifically, this line in addin.py is the culprit: self.stats = bayes_stats.Stats(bayes_options, self.classifier_data.message_db) I've not even looked inside that module yet, but that seems quite extreme, to the point I'm not sure the feature is worth that cost... I guess the code is reading each record of my message DB (which is 85MB) - but does anyone have any insights? Cheers, Mark

3 6

image cracking results
by Mark Hammond Dec. 22, 2006

Dec. 22, 2006

I've finally managed to get something working with the Outlook addin and Skip's cool new ocrad stuff. the results look promising! :) A summary of my results are below. The runs are 'Tokenizer:x-image_size' and 'Tokenizer:x-crack_images' both set to False, versus both set to True. It looks like a 13% improvement in false negatives, which is nothing to sneeze at! I've never been an expert at reading these results though, so let me know if there is anything interesting I missed or neglected to send. Cheers, Mark false positive percentages <snip 10 lines of zeros> won 0 times tied 10 times lost 0 times total unique fp went from 0 to 0 tied mean fp % went from 0.0 to 0.0 tied false negative percentages 5.859 5.455 won -6.90% 5.410 4.363 won -19.35% 5.794 5.234 won -9.67% 3.008 2.820 won -6.25% 5.588 4.817 won -13.80% 4.469 3.166 won -29.16% 5.051 4.646 won -8.02% 5.829 5.647 won -3.12% 6.842 5.789 won -15.39% 7.356 5.964 won -18.92% won 10 times tied 0 times lost 0 times total unique fn went from 293 to 254 won -13.31% mean fn % went from 5.52048164035 to 4.79002154629 won -13.23% ham mean ham sdev 0.00 0.00 +(was 0) 0.05 0.05 +0.00% 0.03 0.04 +33.33% 0.48 0.70 +45.83% 0.06 0.04 -33.33% 0.82 0.53 -35.37% 0.08 0.08 +0.00% 1.53 1.53 +0.00% 0.00 0.00 +(was 0) 0.01 0.01 +0.00% 0.08 0.09 +12.50% 1.89 1.90 +0.53% 0.00 0.00 +(was 0) 0.08 0.08 +0.00% 0.00 0.00 +(was 0) 0.08 0.07 -12.50% 0.01 0.01 +0.00% 0.17 0.17 +0.00% 0.01 0.01 +0.00% 0.21 0.18 -14.29% ham mean and sdev for all runs 0.03 0.03 +0.00% 0.83 0.82 -1.20% spam mean spam sdev 90.19 90.67 +0.53% 24.58 23.81 -3.13% 91.43 91.74 +0.34% 22.99 22.24 -3.26% 91.51 91.85 +0.37% 23.66 22.79 -3.68% 93.62 93.94 +0.34% 18.73 17.90 -4.43% 90.62 91.11 +0.54% 23.83 22.99 -3.52% 91.07 91.66 +0.65% 22.71 21.31 -6.16% 90.85 91.31 +0.51% 23.28 22.58 -3.01% 89.74 90.19 +0.50% 25.29 24.57 -2.85% 89.49 90.15 +0.74% 25.99 24.73 -4.85% 88.57 89.45 +0.99% 26.84 25.00 -6.86% spam mean and sdev for all runs 90.72 91.21 +0.54% 23.91 22.90 -4.22% ham/spam mean difference: 90.69 91.18 +0.49

1 1

anyone able to rebuild docs?
by Mark Hammond Dec. 21, 2006

Dec. 21, 2006

I just checked in some changes to the docs to fix some broken links that have been reported. However, I have failed miserably in the process of regenerating the HTML and uploading to sourceforge. Would it be possible for someone else (presumably on Linux :) to update the website dir and execute the process (which appears to simply be "make; make install")? Thanks! Mark

1 0

SpamBayes
by Donna Walden Dec. 20, 2006

Dec. 20, 2006

Good Afternoon, Email trained as safe domain and working properly for a few months, has all of a sudden begun going into the Spam folder. What would cause this? Thank you, Donna Donna Walden Information System Manager Georgia HAP Administrators, Inc. 2296 Henderson Mill Road, Suite 306 Atlanta, GA 30345-2739 (770)939-3939, ext. 2014 (770)939-3886 (Fax) dwalden(a)ghaca.org

2 1

Applying SpamBayes to website spamming
by skip＠pobox.com Dec. 20, 2006

Dec. 20, 2006

I'm sure many of you are aware that spamming of the submission forms on blogs and other websites is a large and increasing problem. The Mojam and Musi-Cal concert websites suffered from the same malady. I originally considered implementing some sort of CAPTCHA scheme: http://en.wikipedia.org/wiki/Captcha but that has limitations and would have required changes to all submission forms on the websites. I decided to instead implement a SpamBayes-based solution in our XML-RPC server. It has a couple distinct advantages: * It has none of the CAPTCHA gotchas. * It is implemented at a single point in the system. * No changes to the Web interface were required so users don't have to learn something new. I'll give you a quick sketch of what I did to solve this problem. If you'd like more details, drop me a note. When someone submits concert dates to our sites the submission is represented as a simple dictionary. A valid submission will have information about who's performing, a date in the future, valid location information, etc. In contrast, when someone spams the submission forms the dictionary often contains bogus information or is missing some fields altogether. For example, if the spammer puts something in the date fields it's likely to be garbage which won't parse properly, resulting in a default date of 1900-01-01. Similarly, the city/state/country is likely to be invalid, so we won't be able to find lat/long info. The dictionary is preprocessed into a string of tokens which includes the obvious text which was part of the submission, but which also contains synthetic tokens. Here's a spammer's entry represented as text: Bradyn Maximus Ty jordan(a)e-mailanywhere.com 1900-01-01 Jerald kwds:False kwds-private:False Malcom 1900-01-01 Jarod date:ancient perflen:1 infolen:1 hasphone:False hasprice:False city:unknown venue:present Here's a valid entry represented as text: Anchorage skip(a)mojam.com 2006-10-07 kwds:True kwds-private:True .bl.1348 .ra LaVette,Bettye 2006-10-07 AK Discovery Theatre date:current perflen:1 infolen:0 hasphone:False hasprice:False city:known venue:present The synthetic tokens that suggest problems are such huge red flags for the classifier that after training on just a couple of these bad boys the rejection rate of spam submissions seems to be 100%. Of course, this sort of spamming is probably still in its infancy, so I expect we might eventually see some sort of arms race develop as has been true for email spam. I'm not too worried about that though because for the most part I think the spammers' primary target is the blogosphere with its ubiquitous comment feature, not specialized websites like ours. The tokenizer class is quite simple. I post it here in its entirety. Note that major bits of it were just pasted from the default tokenizer. from spambayes.tokenizer import log2, Tokenizer, numeric_entity_re, \ numeric_entity_replacer, crack_urls, breaking_entity_re, html_re, \ tokenize_word class Tokenizer(Tokenizer): def tokenize(self, text): maxword = 20 # Replace numeric character entities (like a for the letter # 'a'). text = numeric_entity_re.sub(numeric_entity_replacer, text) # Normalize case. text = text.lower() # Get rid of uuencoded sections, embedded URLs, <style gimmicks, # and HTML comments. for cracker in (crack_urls,): text, tokens = cracker(text) for t in tokens: yield t # Remove HTML/XML tags. Also  . <br> and <p> tags should # create a space too. text = breaking_entity_re.sub(' ', text) # It's important to eliminate HTML tags rather than, e.g., # replace them with a blank (as this code used to do), else # simple tricks like # Wr<!$FS|i|R3$s80sA >inkle Reduc<!$FS|i|R3$s80sA >tion # can be used to disguise words. <br> and <p> were special- # cased just above (because browsers break text on those, # they can't be used to hide words effectively). text = html_re.sub('', text) # Tokenize everything in the body. for w in text.split(): n = len(w) # Make sure this range matches in tokenize_word(). if 3 <= n <= maxword: yield w elif n >= 3: for t in tokenize_word(w): yield t The only thing I found a bit frustrating was that I had to override the Hammie class to provide an alternate tokenizer class: class Hammie(hammie.Hammie): def __init__(self, bayes): hammie.Hammie.__init__(self, bayes) self.tokenizer = Tokenizer() def _scoremsg(self, msg, evidence=False): return self.bayes.spamprob(self.tokenizer.tokenize(msg), evidence) def train(self, msg, is_spam, add_header=False): self.bayes.learn(self.tokenizer.tokenize(msg), is_spam) I think it would be a bit more general if the Hammie class accepted an optional tokenizer to avoid this. So far I've trained on 338 hams and 26 spams. (I have a new guy I'm breaking in who is not experienced with SpamBayes. I see a number of entries in the ham data I would not have added there. I expect I can probably reduce the ham data size to 200 or less.) The BerkDB file containing the token database is a whopping 86KBytes. I somewhat arbitrarily set the ham cutoff at 0.15 and the spam cutoff at 0.60. The true spam seems to so far all land in the 0.95-1.0 range. I see some "possible spam" in the 0.16-0.2 range. Most of the time that's because the submitter forgot to enter a date (or entered it incorrectly) or misspelled the city. Skip

1 0

Problem with struct.unpack in oe_mailbox.py
by Spandex Dec. 15, 2006

Dec. 15, 2006

Hi, I previously sent this mail to the spambayes users list, without response. Apologies for the repost... I'm hoping it's more appropriate here:- I'm running spambayes (1.0.4-3) on Debian unstable with Python 2.4.4c0 and a custom compiled 2.6.17 kernel. I'm using an AMD64 chip. sb_server starts up ok and proxies pop3 and smtp connections ok. I can train from the commandline ok. The problem comes when I try to train it from the web interface (using either mbox or dbx format). It bombs with the following error:- ---------------- Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/spambayes/Dibbler.py", line 470, in found_terminator getattr(plugin, name)(**params) File "/usr/lib/python2.4/site-packages/spambayes/UserInterface.py", line 494, in onTrain content = self._convertToMbox(content) File "/usr/lib/python2.4/site-packages/spambayes/UserInterface.py", line 536, in _convertToMbox content = oe_mailbox.convertToMbox(content) File "/usr/lib/python2.4/site-packages/spambayes/oe_mailbox.py", line 444, in convertToMbox if header.isValid() and header.isMessages(): File "/usr/lib/python2.4/site-packages/spambayes/oe_mailbox.py", line 117, in isValid return self.getEntry(0) == dbxFileHeader.MAGIC_NUMBER File "/usr/lib/python2.4/site-packages/spambayes/oe_mailbox.py", line 126, in getEntry self.dbxBuffer[dbxEntry * 4:(dbxEntry * 4) + 4])[0] error: unpack str size does not match format ---------------- I'm wondering whether this is something to do with my machine architecture and the sizes of datatypes? But I'm stabbing in the dark. I can easily disable dbx support by commenting out.. content = oe_mailbox.convertToMbox(content) .. around line 536 of UserInterface.py, and this does enable me to train on mbox format via the web interface, but I'd rather keep dbx support if possible. I don't speak Python so commenting out the offending code was about as far as I could go. Any ideas? Thanks, Matt

1 0

Dec 14 cvs seg faults on python-2.5: _weakref.so
by sean darcy Dec. 14, 2006

Dec. 14, 2006

Made the mistake of updating to python-2.5 :( That seg faulted sb. So I updated to cvs. Rebuilt and installed. Same result: python -v /usr/bin/sb_server.py # installing zipimport hook import zipimport # builtin # installed zipimport hook # /usr/lib64/python2.5/site.pyc matches /usr/lib64/python2.5/site.py import site # precompiled from /usr/lib64/python2.5/site.pyc # /usr/lib64/python2.5/os.pyc matches /usr/lib64/python2.5/os.py import os # precompiled from /usr/lib64/python2.5/os.pyc import posix # builtin ................ import spambayes.Version # precompiled from /usr/lib/python2.5/site-packages/spambayes/Version.pyc # /usr/lib/python2.5/site-packages/spambayes/ProxyUI.pyc matches /usr/lib/python2.5/site-packages/spambayes/ProxyUI.py import spambayes.ProxyUI # precompiled from /usr/lib/python2.5/site-packages/spambayes/ProxyUI.pyc SpamBayes POP3 Proxy Version 1.1a3 (August 2006) import bsddb # directory /usr/lib64/python2.5/bsddb # /usr/lib64/python2.5/bsddb/__init__.pyc matches /usr/lib64/python2.5/bsddb/__init__.py import bsddb # precompiled from /usr/lib64/python2.5/bsddb/__init__.pyc dlopen("/usr/lib64/python2.5/lib-dynload/_bsddb.so", 2); import _bsddb # dynamically loaded from /usr/lib64/python2.5/lib-dynload/_bsddb.so # /usr/lib64/python2.5/bsddb/dbutils.pyc matches /usr/lib64/python2.5/bsddb/dbutils.py import bsddb.dbutils # precompiled from /usr/lib64/python2.5/bsddb/dbutils.pyc # /usr/lib64/python2.5/bsddb/db.pyc matches /usr/lib64/python2.5/bsddb/db.py import bsddb.db # precompiled from /usr/lib64/python2.5/bsddb/db.pyc # /usr/lib64/python2.5/weakref.pyc matches /usr/lib64/python2.5/weakref.py import weakref # precompiled from /usr/lib64/python2.5/weakref.pyc dlopen("/usr/lib64/python2.5/lib-dynload/_weakref.so", 2); import _weakref # dynamically loaded from /usr/lib64/python2.5/lib-dynload/_weakref.so Segmentation fault sean

1 0

Question for Spambayes
by Jim Klein Dec. 1, 2006

Dec. 1, 2006

How do I do a silent automated install of Spambayes?

1 0