Mailman 3 December 2003 - spambayes-dev

Reduced training test results
by T. Alexander Popiel Dec. 29, 2003

Dec. 29, 2003

Training on just those messages whose score isn't 0.00 or 1.00 (rounded) seems to be a huge win over training on everything. Not so much because the accuracy is better (though accuracy does seem to be improved by neglecting those messages that it's already certain about), but because of a hugely reduced training set (and thus database). Specifically, training on everything yielded a database with 70,000 messages, while training only on the non-extreme put only about 3,500 messages into the database. Unfortunately, I don't have firm numbers on token counts. Also of significant interest is that the classifier doesn't seem to decay as badly over time. With training on everything, the unsure rate in particular (and fn to a much lesser extent) goes up significantly after about 200 days worth of traffic, though the fp rate stays low. With just training on those things that aren't already certain, the unsure rate climbs much more slowly after 200 days (with the cumulative rate staying relatively flat), while the fp and fn rates stay at very low values. Details of my experiment parameters: I've got about 77000 messages in my dataset, covering a span of 418 days. Of these, about 21500 are ham, and nearly 56000 are spam. I include virus/worm messages in my spam, and the "latest windows update" worm makes its presence felt around day 360. I divided my dataset into 10 subsets, and ran the incremental.py harness over these 10 times, excluding 1 set each time, as per normal cv-ish behaviour. Thus, each of my measurements is replicated 10 times, with slightly different input data. Finally, I did the above-mentioned 10 runs using both the 'perfect' and 'nonedge' regimes. The 'perfect' regime trains on every message using the proper ham/spam classification, while the 'nonedge' regime trains only on those messages that were not correctly classified with 0.00 or 1.00 (rounded) scores. I've plotted the both cumulative and 7-day average values for error rates (fp, fn, and unsure) and training counts (ham and spam). Pictures (and a copy of this writeup) are on my website at: http://www.wolfskeep.com/~popiel/spambayes/nonedge - Alex PS. Sorry this took so long, but running the perfect regime on such a large dataset took a couple days on my machine... I need more memory! ;-)

4 4

Re: error training on dbx file
by Paul Sorenson Dec. 28, 2003

Dec. 28, 2003

Please ignore this error. The user has been deleted :-)

1 0

error training on dbx file
by Paul Sorenson Dec. 28, 2003

Dec. 28, 2003

With code I checked out from CVS in the last 24 hours or so, I got the error below when trying to train a dbx file via the web interface. I have python 2.3.3 installed and I the box involved is running Windows XP. oe_mailbox.py doesn't appear to import time. 500 Server error Traceback (most recent call last): File "C:\usr\spambayes\spambayes\Dibbler.py", line 457, in found_terminator getattr(plugin, name)(**params) File "C:\usr\spambayes\spambayes\UserInterface.py", line 479, in onTrain content = self._convertToMbox(content) File "C:\usr\spambayes\spambayes\UserInterface.py", line 521, in _convertToMbox content = oe_mailbox.convertToMbox(content) File "C:\usr\spambayes\spambayes\oe_mailbox.py", line 465, in convertToMbox dbxBuffer += "From spambayes(a)spambayes.org %s\n%s" \ NameError: global name 'strftime' is not defined

1 0

Re: [spambayes-dev] New sort+group.py
by T. Alexander Popiel Dec. 28, 2003

Dec. 28, 2003

In message: <LNBBLJKPBEHFEDALKOLCCELFIAAB.tim.one(a)comcast.net> "Tim Peters" <tim.one(a)comcast.net> writes: >[Tim] >>> Ah, fiddlesticks -- does someone here still give a rip about Python >>> 2.2? > >[T. Alexander Popiel] >> Yeah, I do. There are no more recent versions packaged for Debian >> stable. Sorry. This will likely change when sarge makes it to >> stable... but that's likely not going to happen for at least 6 months. > >Heh. You're on a Linux system and can't upgrade a package? Makes me glad >I'm running Windows, where others don't dictate what I can run on my own >machine <wink>. Eh, it's not that I can't... it that if I do, I either have to go through a lot of hassle to package it myself, or I have to go through a lot of hassle to make all the packages that depend on python ignore the fact that there's no python package listed as installed. A pain either way. (I _might_ be able to grab a package version out of sarge and recompile to avoid a library-version-incompatibility cascade which would require me to upgrade half my system to possibly broken versions... but still, a nuisance). >I made sort+group.py 2.2.3-friendly, far as I can tell, but since 2.3 came >out I don't use 2.2 for anything anymore -- if I introduce more >incompatibilities, I won't know. *nod* I'll tell ya if you broke it for me. ;-) - Alex

1 0

Code changes
by Tim Peters Dec. 28, 2003

Dec. 28, 2003

The meaning of Outlook2000/export.py's -n option has changed. Here's the checkin comment: INCOMPATIBLE CHANGE: the -n option now gives the number of Set subdirectories desired, instead of a number of msgs per Set subdir "to shoot for". If you want to run, e.g., 10-fold cross-validation, you have to have exactly 10 Set folders, and the # of msgs per folder is of much less importance. Also added a note recommending to run rebal.py afterwards. rebal is the expert in setting up randomized Set subdirectories, and the export.py script probably should have stuck to just extracting msgs from Outlook. utilities/rebal.py has grown a -t option, which makes it (once again) easy to use with a standard test setup. It was originally easy to use that way, but grew -r and -s options, presumably added by someone with a non-standard test setup. Unfortunately, those with a standard test setup had to use them too, and they're both clumsy and error-prone to use with a standard test setup. -t can't be used in the same run with -r or -s. Those with a standard test setup no longer need to worry about -r or -s, just -t; vice versa for those with a non-standard test setup. The changes to testtools/sort+group.py discussed here have been checked in, after fiddling to play nice with Python 2.2.3 too.

1 0

Re: [spambayes-dev] New sort+group.py
by T. Alexander Popiel Dec. 28, 2003

Dec. 28, 2003

In message: <LNBBLJKPBEHFEDALKOLCAEKHIAAB.tim.one(a)comcast.net> "Tim Peters" <tim.one(a)comcast.net> writes: >[T. Alexander Popiel] >> Trying... but it seems to have major problems with python2.2. > >Ah, fiddlesticks -- does someone here still give a rip about Python 2.2? Yeah, I do. There are no more recent versions packaged for Debian stable. Sorry. This will likely change when sarge makes it to stable... but that's likely not going to happen for at least 6 months. >Not your fault: it's not possible to find out time of day under 2.2 either ><wink>. *pthbbbt* - Alex

2 1

Re: [spambayes-dev] New sort+group.py
by T. Alexander Popiel Dec. 27, 2003

Dec. 27, 2003

In message: <LNBBLJKPBEHFEDALKOLCOEJDIAAB.tim.one(a)comcast.net> "Tim Peters" <tim.one(a)comcast.net> writes: > >Attached is a major rewrite of testtools/sort+group.py. Here's a patch to make it work with python2.2. It appears that the 'for line in fh:' syntax for filereading in 2.2 buffered a bunch of lines which were then unavailable for use to subsequent similar loops in the case of the first loop terminating early. Also, enumerate() didn't seem to exist, so I just maintained the count manually. Enjoy. - Alex --- sort+group.py.noworky Sat Dec 27 11:58:04 2003 +++ sort+group.py Sat Dec 27 11:57:37 2003 @@ -26,20 +26,21 @@ def get_time(fpath): fh = file(fpath, "rb") # Find first Received header. - for line in fh: + line = fh.readline() + while line != "\r\n" and line != "\n" and line != "": if line.lower().startswith("received:"): break + line = fh.readline() else: print "\nNo Received header found." fh.close() return None # Paste on the continuation lines. received = line - for line in fh: - if line[0] in ' \t': - received += line - else: - break + line = fh.readline() + while line[0] in ' \t': + received += line + line = fh.readline() fh.close() # RFC 2822 says the date-time field must follow a semicolon at the end. i = received.rfind(';') @@ -90,7 +91,9 @@ if loud: print "Renaming second pass ..." earliest = data[0][0] # timestamp of earliest msg received - for i, (when_received, name) in enumerate(data): + i = 0 + for when_received, name in data: + i = i + 1 dirname = os.path.dirname(name) basename = os.path.basename(name) extension = os.path.splitext(basename)[-1]

1 0

Re: [spambayes-dev] New sort+group.py
by T. Alexander Popiel Dec. 27, 2003

Dec. 27, 2003

In message: <LNBBLJKPBEHFEDALKOLCOEJDIAAB.tim.one(a)comcast.net> "Tim Peters" <tim.one(a)comcast.net> writes: > >Attached is a major rewrite of testtools/sort+group.py. Yay! I'll be happy to admit I just sort of threw that together. >Anyone who uses that, please give it a try. Trying... but it seems to have major problems with python2.2. It barfs on enumerate(), and it doesn't seem to be picking up continuation lines, either, so I suspect the file reading style you're using isn't grokked correctly, either. >The major thrust of the changes is to order msgs by full-precision UTC >timestamp. It was sorting just by date (not time), and wasn't accounting >for that different ISPs may be in different time zones. Oops. Doh. Thanks for catching that. All my mail gets received and timestamped by my local machine, so the timezones weren't an issue... but ignoring time of day entirely is rather embarassing. - Alex, who is now trying to get it to work with python2.2...

2 1

NEWTRICKS
by David Abrahams Dec. 27, 2003

Dec. 27, 2003

I keep getting quite a few spams which fit the descriptions below (from NEWTRICKS.txt): - Punctuation sometimes gets inserted in otherwise spammy words or phrases, e.g.: "Ch-eck ou=t ou-r sel)ection _of grea)t R_X -emgffj". It might be helpful to try stripping punctuation. (Idea from Paul Sorenson) - Similarly, some letters get replaced by numbers, e.g.: "V1agra" instead of "Viagra". Mapping numbers to suitable letters might help in some situations. Since "this file is for ideas that have or have not yet been tried", I'd love to know what constitutes "trying". Is there some official testing procedure or corpus we can test against? I'd like to know whether any change I make is worth proposing. Of course I can try it on my own databases of Ham and Spam first... -- Dave Abrahams Boost Consulting www.boost-consulting.com

3 3

Two SB on One Computer
by Stephen Anderson Dec. 27, 2003

Dec. 27, 2003

Hi, I searched through the archives and I couldn't find anything conclusive on this. Please don't hesistate to point me back to the archives if you know I've missed something. I'm using the sb_server (pop3proxy) on an XP computer as a service. I'd like to install it as two separate services and use two separate databases and separate web management ports so two different users can each have their own customized spam filter. I tickled through the service script and eye-balled the sb_server but I'm not sure what all assumptions are made that would make two instances of SB overlap. Can anybody give me some insight on what things they think I will have to watch out for? Thank you! Stephen Anderson <stephena(a)HiWAAY.net> =========================================================================== http://wecanstopspam.org

2 1