Mailman 3 December 2003 - spambayes-dev

RE: [spambayes-dev] Experimental SpamBayes build available
by Tony Meyer Dec. 31, 2003

Dec. 31, 2003

[Mark] > Otherwise, did anyone else try this build? Either for > Outlook or sb_server? I fear I may have "disclaimed" the > build a little too much, as this is the only reply I got, and > I see no new bugs etc. I briefly tried it, and all seemed ok to me (Outlook XP and the others on WinXP). I've also been trying various experimental builds of my own (both on the XP box and on Win98 at home). I can't see how they would be different to your build since it's using the same process. (Latest CVS each time). Sorry I didn't report back earlier - didn't your message say that you were away for a couple of weeks and wouldn't be looking at anything until then? I got a 'not urgent' impression from it :) No bugs to report - the install has worked fine for me, and I've fixed anything wrong I've found with the source itself. I've made some improvements to the documentation, too - is it sufficient for a 8.5 release, do you think? =Tony Meyer

2 1

RE: [spambayes-dev] pop3proxy_tray error
by Tony Meyer Dec. 31, 2003

Dec. 31, 2003

[Kenny on the 5th of December] > I tried to stop SpamBayes from the right-click menu, and then > start it again. Here's the output I got when it tried to > restart SpamBayes. [...] > serverStrings = ["%s:%s" % (s, p) for s, p in self.servers] > TypeError: iteration over non-sequence I finally got around to finding this and fixing it :) I think it's also the problem that Richie found. Fixed in sb_server 1.16, I hope. =Tony Meyer

1 0

Experimental SpamBayes build available
by Mark Hammond Dec. 30, 2003

Dec. 30, 2003

Hi all, I have just uploaded an installer for a new experimental binary of SpamBayes. This binary includes *both* the Outlook addin and the sb_server applications. The installer attempts to detect the most appropriate one to install. Everything is built from CVS sources as of today. Hopefully, this will mean the Outlook addin has a number of bugs fixed over the 0.8 release. However, it is possible there are a number of bugs *not* in 0.8, and even the possiblility it will not work at all for many people (as this is released with different 'python->.exe' technology than previous versions) The sb_server application suite all seem to work fine too, so non-outlook users are also encouraged to try this version. Note that it comes with almost no documentation (as there is none!) and that this is the first release of such a binary, so this too is bleeding edge. Thus, only brave people willing to test out stuff with almost no release notes should try it :) To further dissuade you, I am leaving for a week or so holiday, and will not be in a position to respond to any mail or bugs relating to this build. That said, it works well for me and the testing I have done on a number of machines. If anyone is keen, please visit http://starship.python.net/crew/mhammond/spambayes/ Happy holidays! Mark.

4 6

Re: [spambayes-dev] RE: [Spambayes] How low can you go?
by T. Alexander Popiel Dec. 30, 2003

Dec. 30, 2003

In message: <LNBBLJKPBEHFEDALKOLCAEBHIBAB.tim.one(a)comcast.net> "Tim Peters" <tim.one(a)comcast.net> writes: >[T. Alexander Popiel] >> ... >> Yup. I have a nice picture now of the ratio over time at the bottom >> of the report at: >> http://www.wolfskeep.com/~popiel/spambayes/nonedge > >Hmm. That appears to be using a log scale for the Y (ratio) axis, so what >*appears* to be straight-line growth in the ratio after about day 150 is >really exponential growth. That could get bad over time <wink>. Yeah, I used log scale for the ratio... log makes more sense to me for ratios. I can trivially replot on linear scale if you want. ;-) >Oh, there are billions of things that could be tried. Who knows what might >pay? Aye, there are. I don't have billions of CPU-days to burn, though, so I'm trying to winnow down to stuff that's likely to pay off. Theoretical beauty is one measure that sort of appeals. ><wink -- but "two decimal digits" is just an artifact of how scores get >displayed>. No argument there. I have no particular love for that rule, either. >Asymmetric bounds also have some attraction, since, e.g., in mistake-based >training "by hand" I always end up moving the ham cutoff closer to 0 than >the spam cutoff is to 1. One thing that's occurred to me is to have the training cutoffs at N sigma from mean (where N == .5?) for the two populations; how you'd bootstrap that is an open question, of course. - Alex

2 1

test_storage.py failing
by Skip Montanaro Dec. 30, 2003

Dec. 30, 2003

I ran the spambayes/test/test_storage.py this morning for the first time (on a fresh CVS checkout) and got several instances of the same error. Here's one example: ERROR: testHapax (__main__.DBStorageTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "test_storage.py", line 137, in setUp return _StorageTestBase.setUp(self) File "test_storage.py", line 20, in setUp self.classifier = self.__class__.StorageClass(self.db_name) TypeError: __init__() takes exactly 1 argument (2 given) I inserted print self.__class__.StorageClass right above the class call and in the error case it's always instantiating DBDictClassifier which does take a db_name argument, so I'm a bit confused about why this is generating an error. My brain is not in a high enough gear to see why. I think I'll just go do a little last minute Christmas shopping and let someone else figure it out. Skip

4 5

Re: [spambayes-dev] Full re-initialization
by T. Alexander Popiel Dec. 30, 2003

Dec. 30, 2003

In message: <PAEIKFCFJBPAOLNKOEEMEEHMCIAA.ltnieh(a)earthlink.net> "Luther Nieh" <ltnieh(a)earthlink.net> writes: > >Hello SpamBayes Tech support, Well, we're not really tech support here... it's just the people who wrote the code and us hangers-on who heckle from the sides. >One day, as I was cleaning up the M/S Outlook folders, I accidentally >deleted the spam email folder. Now, SpamBayes does not seem to work >anymore. It said it couldn't send spam emails to that folder, even >though I had manually re-created the spam email folder. In this case, Outlook is "helpful" and remembers that you "moved" the spam mail folder to the trash. What you should do is go back to the configuration panel where you set the spam mail folder, and point it back to your re-created spam mail folder. Once that's done, all should be happy again. - Alex (who doesn't use Outlook and thus can't get super-specific)

1 0

Full re-initialization
by Luther Nieh Dec. 30, 2003

Dec. 30, 2003

Hello SpamBayes Tech support, Thank you for developing this useful software. I have used it for a few weeks and I feel it has been doing it's job as described in your documentation. One day, as I was cleaning up the M/S Outlook folders, I accidentally deleted the spam email folder. Now, SpamBayes does not seem to work anymore. It said it couldn't send spam emails to that folder, even though I had manually re-created the spam email folder. I even attempted to remove and re-install SpamBayes without any improvement. Please let me know what one does to have the program re-initialize itself as if it was a new installation. Thank you for your help. Regards, Luther Nieh ltnieh(a)sunnydesign.com 12/29/03 --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.552 / Virus Database: 344 - Release Date: 12/15/2003

1 0

Re: [spambayes-dev] RE: [Spambayes] How low can you go?
by T. Alexander Popiel Dec. 29, 2003

Dec. 29, 2003

In message: <LNBBLJKPBEHFEDALKOLCGEIJIAAB.tim.one(a)comcast.net> "Tim Peters" <tim.one(a)comcast.net> writes: > >In your (Alex's) recent "nonedge" incremental training experiment, it looks >like your training data grew to about a 5.5::1 spam::ham ratio after 400 >days. Yup. I have a nice picture now of the ratio over time at the bottom of the report at: http://www.wolfskeep.com/~popiel/spambayes/nonedge >I know my personal classifiers start acting flaky whenever I've let >them get imbalanced by more than 2::1 in either direction. Interestingly enough, though, the nonedge did better than TOE, despite a worse imbalance. >So if I had your data, I'd be curious to try variations that force better >balance. I'd love to... but I haven't been able to come up with anything which maintains the balance better without extreme artificiality. If you think of any regimes that make sense, I'd be more than happy to run them. >You have enough data that it >may well be more interesting to you to try variations including expiration *grin* That's part of what's been burning my CPU ever since I posted the last report. I'll have another report, including that, probably within 3 days. Still have more to test... and my runs are taking between 6 and 20 hours each, depending on the memory used by the classifiers. - Alex

2 1

Re: [spambayes-dev] Reduced training test results
by T. Alexander Popiel Dec. 29, 2003

Dec. 29, 2003

In message: <3FEFF5F6.1090004(a)hooft.net> Rob Hooft <rob(a)hooft.net> writes: >T. Alexander Popiel wrote: >> Training on just those messages whose score isn't 0.00 or 1.00 >> (rounded) seems to be a huge win over training on everything. > >Told you: >See the section "Train on Errors, Unsures, and non-obvious correct >decisions" at http://www.entrian.com/sbwiki/TrainingIdeas Hrm. I suppose that I ought to actually look at the wiki. ;-) Is there any way for me to upload my plots to go along with any discussion that I might add to the above page? I could just reference them on my machine, but it seems better to keep the wiki content all in one place. >> Not so much because the accuracy is better (though accuracy >> does seem to be improved by neglecting those messages that it's >> already certain about), but because of a hugely reduced training >> set (and thus database). > >Both are effects I can feel in practice! FWIW, using this training style with my nightly retrains cut my database size in half (from 21 meg to 10 meg). This is with a 4-month horizon, too, so the difference would likely be even greater with a longer span. - Alex

3 2

Re: [spambayes-dev] Reduced training test results
by T. Alexander Popiel Dec. 29, 2003

Dec. 29, 2003

In message: <16368.13462.404757.694070(a)montanaro.dyndns.org> Skip Montanaro <skip(a)pobox.com> writes: > > Rob> T. Alexander Popiel wrote: > >> Training on just those messages whose score isn't 0.00 or 1.00 > >> (rounded) seems to be a huge win over training on everything. > > Rob> Told you: > Rob> See the section "Train on Errors, Unsures, and non-obvious correct > Rob> decisions" at http://www.entrian.com/sbwiki/TrainingIdeas > >I think we need to split that page into multiple chunks. Agreed. I think that using the subpage mechanism would be good. >Anybody got some pretty graphs to break up the text? Several, now... though I may need to see if I can rescale them to something less than full-page. Hrm. Also, a few of these training ideas are already represented by regimes for the incremental harness... and I've got a couple more to check in. I also recognize that my names for the regimes are, umm, less than optimal; if people have better names for such, please speak up. As an example, I'm probably going to rename the 'perfect' regime to 'TrainOnEverything'. Suggestions for capitalization style? Should it be TrainOnEverything, train_on_everything, or something else? - Alex

2 1