Mailman 3 December 2003 - spambayes-dev

New sort+group.py
by Tim Peters Dec. 27, 2003

Dec. 27, 2003

Attached is a major rewrite of testtools/sort+group.py. Anyone who uses that, please give it a try. If nobody gripes, I'll check it in. (If you're on Linux, the attached probably has Windows line ends, and you may need to change that.) It's used exactly the same way as before, and creates filenames with the same pattern as before, *except* that any pre-existing extension (like ".txt" on Windows) is preserved. Extensions are necessary for sane life on Windows, but the code currently checked in strips extensions as part of renaming. The major thrust of the changes is to order msgs by full-precision UTC timestamp. It was sorting just by date (not time), and wasn't accounting for that different ISPs may be in different time zones. It also failed to parse many of the Received headers in my email, partly because Comcast's Received headers don't make any attempt to keep the date-time part on a single line. Other failures were due to "unusual" spellings in the date-time part. Instead email.Utils.parsedate_tz() is used to parse this stuff, and that didn't fail on any of the email I've tried so far. Almost all Received headers I see have hour:minute:second info, and since I do incremental training during the day, as email comes in, it's important to me that the email be ordered at finer granularity than "a day". A second should be good enough <wink>. My various ISPs are in different time zones too, and normalizing to UTC should help model that, e.g., the first time I see a new spam campaign it's much more likely to arrive from my MSN account than from my Comcast account.

1 0

Re: [spambayes-dev] Reduced training test results
by T. Alexander Popiel Dec. 27, 2003

Dec. 27, 2003

In message: <LNBBLJKPBEHFEDALKOLCCEIPIAAB.tim.one(a)comcast.net> "Tim Peters" <tim.one(a)comcast.net> writes: >[T. Alexander Popiel] > >> Also of significant interest is that the classifier doesn't seem >> to decay as badly over time. With training on everything, the >> unsure rate in particular (and fn to a much lesser extent) goes >> up significantly after about 200 days worth of traffic, > >That's peculiar. Did you try this with different starting dates, and find >that "about 200 days" was invariant across starting dates -- or did you try >a single starting date, and note that something funny happened about 200 >days after that single starting date. I think the latter, in which case >it's natural to speculate that something significant changed around then in >your ham and/or spam mix. It was in fact the latter, and I'm just now prepping for spinning my dataset by 80 and 160 days to revalidate. Even odder things are happening at specific times in the expiry stuff, and I want to see if it's specific real times, or time after training commences... - Alex

1 0

Re: [spambayes-dev] RE: [Spambayes] How low can you go?
by T. Alexander Popiel Dec. 27, 2003

Dec. 27, 2003

In message: <LNBBLJKPBEHFEDALKOLCEEIJIAAB.tim.one(a)comcast.net> "Tim Peters" <tim.one(a)comcast.net> writes: > >So what may be more important now, building on Alex's incremental testers, >isn't the sheer number of messages so much as the span of time they cover. I've been having this supposition, too, but was afraid of scaring people off by voicing it. After all, I don't know if anyone else has been anal enough to have been maintaining growing corpora for over a year... >OTOH, I don't have enough exhaustive personal email saved away to measure >anything other than how the system performs across a few days, and a scheme >that "learns fast" starting from nothing *may* also be slow to adapt to >changes over time (we all know a bright kid who never outgrew their >6th-grade worldview, right <wink>?). Heh. I could be convinced to run the bigram scheme over my dataset after I'm done with my current set of tests... though I may need a gig of memory to do it. ;-) My current 256 meg is dying under the load of TOE with expiry. - Alex

1 0

Re: [spambayes-dev] RE: [Spambayes] How low can you go?
by T. Alexander Popiel Dec. 27, 2003

Dec. 27, 2003

In message: <LNBBLJKPBEHFEDALKOLCCEIJIAAB.tim.one(a)comcast.net> "Tim Peters" <tim.one(a)comcast.net> writes: >[Tim] >>> ... >>> Ah, I've noted before that I throw away half my Unsures unclassified, >>> because I can't tell whether they're ham or spam >>> ... >>> No part of the testing framework can be talked into believing that >>> Unsure is the *desired* outcome for a msg, though ... > >[T. Alexander Popiel] >> Hrm. Good point. Perhaps we should fix this, adding a third branch >> to the testing framework's data directory tree, ... >> And then we'd have the six error states ... > >The added complication is unattractive -- I'm OK with guessing "the right" >category, even while believing that doesn't make sense <wink>. Oh, good. That wasn't a bit of hackery I was looking forward to. If you're OK with ignoring it, then I certainly am. - Alex

1 0

Re: [spambayes-dev] RE: [Spambayes] How low can you go?
by T. Alexander Popiel Dec. 27, 2003

Dec. 27, 2003

In message: <MHEGIFHMACFNNIMMBACACEAPHAAA.nobody(a)spamcop.net> "Seth Goodman" <nobody(a)spamcop.net> writes: > >One thing I do that may or may not be typical is that I let Outlook rules >take care of all the mailing list traffic. That includes almost no spam and >so I don't train or classify it (the list admins do a good job). Therefore, >I _don't_ include it in my ham corpus. Reasonable. >This gives me a roughly 1:5 ham/spam corpus, instead of roughly even, but >that's the mail stream that SpamBayes sees. This is the stuff I'd tend to use for the testing, as opposed to your equal-sized training sets. >At present, my corpus is about 7,500 messages total. This may not be enough >to "divide into ten sets", etc. Or is it? I think we did our first classifier shootouts with a minimum of 2,000 messages, so you should be fine. You may not have enough to see some of the longer-term effects I'm now witnessing (with inflection points at 120 and 200+ days), but you should be able to get started, at least. And heck, those inflection points (or the timing thereof) may be peculiarities of my own data. It'd be good to see. - Alex

2 1

Re: [spambayes-dev] RE: [Spambayes] How low can you go?
by T. Alexander Popiel Dec. 27, 2003

Dec. 27, 2003

In message: <LNBBLJKPBEHFEDALKOLCIEBMHOAB.tim.one(a)comcast.net> "Tim Peters" <tim.one(a)comcast.net> writes: > >> Sounds like _you're_ arguing for expiration of whole messages :) > >Oh yes, I do want that -- eventually. We have no experience with that in >this project, though; we have a lot of experience with the consequences of >hapaxes, and I have no fears remaining about picking on them. Actually, there have been experiments done (by me) with expiry of whole messages. I invite you to look at the 'expire4months' regime for my incremental testing harness. Performance was worse than remembering everything, but significantly better than mistake-based training (with the 'fpfnunsure' regime). I have not done any experiments with just nuking hapaxes; I didn't see any reason to do a partial job instead of a full one. >> I know you're not arguing that, but if there were bidirectional msg_id >> <-> feature_ID maps, it would be fairly easy to expire whole >> messages. >> That would obviate the need to track last time seen for every token. > >Only if you don't want also to be able to expire tokens on their own. No... just find the most recent message that the token appeared in, which would be a quick search through a few message times. A really quick search if you're only looking to expire hapaxes. - Alex

5 15

Re: [spambayes-dev] RE: [Spambayes] How low can you go?
by T. Alexander Popiel Dec. 27, 2003

Dec. 27, 2003

In message: <LNBBLJKPBEHFEDALKOLCAEFAIAAB.tim.one(a)comcast.net> "Tim Peters" <tim.one(a)comcast.net> writes: > >Ah, I've noted before that I throw away half my Unsures unclassified, >because I can't tell whether they're ham or spam >No part of the testing framework can be talked into believing that >Unsure is the *desired* outcome for a msg, Hrm. Good point. Perhaps we should fix this, adding a third branch to the testing framework's data directory tree, and then convincing the test code to use messages in that third branch in the classify phase, but not in the train phase. And then we'd have the six error states of ham->spam, ham->unsure, unsure->ham, unsure->spam, spam->ham, and spam->unsure. Hrm. Not for me to do today, though... I'm still running more variations of the stuff I posted about earlier. Redoing the fpfnunsure test that I did last March (with my new dataset so it's comparable), and then adding in 200 day message expiry to my nonedge regime. - Alex

2 1

Re: [spambayes-dev] NEWTRICKS
by T. Alexander Popiel Dec. 26, 2003

Dec. 26, 2003

In message: <uekur8mw8.fsf(a)boost-consulting.com> David Abrahams <dave(a)boost-consulting.com> writes: > >Since "this file is for ideas that have or have not yet been tried", >I'd love to know what constitutes "trying". Is there some official >testing procedure or corpus we can test against? I'd like to know >whether any change I make is worth proposing. Of course I can try it >on my own databases of Ham and Spam first... Heh. We just went through this question with Seth Goodman. Basic summation of the last week or so of advice is: Grab the latest CVS image, then read README-DEVEL.txt and incremental.HOWTO.txt. Lots of good info in there. Collect your own ham & spam corpora, put them into the appropriate directory structure, then run the testing tools over them with different options/classifiers/tokenizers/whatnot. Post results and enough explanation so that people can try to replicate your results using their own corpora. - Alex

1 0

Re: [spambayes-dev] Reduced training test results
by T. Alexander Popiel Dec. 26, 2003

Dec. 26, 2003

In message: <16364.18236.225460.401395(a)montanaro.dyndns.org> Skip Montanaro <skip(a)pobox.com> writes: > > Alex> Also of significant interest is that the classifier doesn't seem > Alex> to decay as badly over time. With training on everything, the > Alex> unsure rate in particular (and fn to a much lesser extent) goes up > Alex> significantly after about 200 days worth of traffic, though the fp > Alex> rate stays low. With just training on those things that aren't > Alex> already certain, the unsure rate climbs much more slowly after 200 > Alex> days (with the cumulative rate staying relatively flat), while the > Alex> fp and fn rates stay at very low values. > > Alex> Details of my experiment parameters: > > Alex> I've got about 77000 messages in my dataset, covering a span of > Alex> 418 days. Of these, about 21500 are ham, and nearly 56000 are spam. > Alex> I include virus/worm messages in my spam, and the "latest windows > Alex> update" worm makes its presence felt around day 360. > >Is it possible that the ham/spam ratio isn't as bad when you don't train on >everything? Eyeballing the graphs, it seems that the ratio is slightly _more_ unbalanced for the nonedge regime, rather than less. Also, from looking closer at the 7-day span graphs, I see that the inflection point is at about 120 days, not 200. - Alex

1 0

A new and altogether different bsddb breakage
by Richie Hindle Dec. 25, 2003

Dec. 25, 2003

In response to Skip's question about hapax ratios, I ran his script and received an error. I boiled the problem down to this: >>> print [db[k] for k in db] Traceback (most recent call last): File "hapaxes.py", line 3, in ? print [db[k] for k in db] File "C:\Python23\lib\shelve.py", line 118, in __getitem__ f = StringIO(self.dict[key]) File "C:\Python23\lib\bsddb\__init__.py", line 86, in __getitem__ return self.db[key] KeyError: 'pics' Excuse me? Er, so how many of these things are there? >>> len([k for k in db if db.get(k, None) is None]) 306 And what do they look like? >>> from pprint import pprint as p >>> p([k for i, k in enumerate(db) if db.get(k, None) is None and i % 50 == 0]) ['magnetism', 'url:mlqnuvs', 'from:addr:wi872u', 'autograph.', 'url:ff-programs', 'motels,'] So they have nothing obvious in common. Looking through the full list it's obvious that they don't all come from one message. Some are obviously ham clues and some are obviously spam. I'm probably winging my way towards a DBRunRecovery error, unless someone can explain what's going on? -- Richie Hindle richie(a)entrian.com

9 45