Mailman 3 November 2002 - SpamBayes

RE: [Spambayes] Important information for Outlook users
by Moore, Paul Nov. 27, 2002

Nov. 27, 2002

From: Moore, Paul > As I said, no time to do much more now. I'll do a full retest of all > this tomorrow. Right, I did a complete rebuild of everything - effectively, clearing out all traces of the addin as best I could, and then installing from scratch like a new user. (I trained on existing spam and ham, rather than starting from an empty database). Here's my log of what I did... ---- Start here ---- Unregister addin Delete fields twice, to ensure 0 fields deleted second time Delete directory Copy in new version Register addin Start Outlook Move 51 new mails to Unsure folder (temporarily) Train on Inbox and Spam folders 385 ham, 999 spam Moved new mails back Define filters, but don't enable yet Outlook has Spam columns in Inbox, Spam, and Unsure - first in Spam shows "0%", first in Inbox shows "0" (?!) - the one in the inbox came in while I was training... Filter now, all messages, Inbox and Spam, only score. Inbox showing as numbers. Spam as %. Filter unread (51 msgs) performing all actions. 42 spam, 9 good, 1 unsure. Look reasonable. Unsure folder is showing Spam field as a number, not a %. Delete unsure message as spam. Field shows as a % in spam folder. Enable filtering, and wait and see how it goes... Trace log: Collecting Python Trace Output... Outlook Spam Addin module loading SpamAddin - Connecting to Outlook Created new configuration file 'C:\Applications\Spambayes\Outlook2000\default_configuration.pck' Either bayes database or message database is missing - creating new Bayes database initialized with 0 spam and 0 good messages Checked 386 in folder Inbox - 385 new entries found. Checked 999 in folder Spam - 999 new entries found. AntiSpam: Watching for new messages in folder Inbox AntiSpam: Watching for new messages in folder Spam Training on message - trained as spam Training on message '' - already was trained as spam -------------------- The only real issue specific to me (Exchange) here that I can see is that mail arrived *during* the training process. I can't stop that happening (working offline breaks all sorts of things, and Exchange delivers mail as it feels - there is no "get mail on request only" facility). Otherwise, I see no reason why the Inbox and Unsure folders should show the Spam field as numbers rather than as percentages. I've left it for now - I can, I assume, delete and recreate the field as a percentage in the Field Chooser, but I don't want to change anything that might provide evidence :-) Filtering seems to be OK, working on the "real" values rather than the formatted ones (if you see what I mean - the *100 factor isn't mucking things up). One good thing, this time I got a much better result than yesterday. I think that deregistering the addin before deleting the fields was the key here, based on what Mark said about the field being created when the addin starts up... Hope this helps, Paul.

4 3

Re: [Spambayes] Re: Guidance re pickles versus DB for Outlook
by T. Alexander Popiel Nov. 27, 2002

Nov. 27, 2002

In message: <w53el9882i2.fsf(a)woozle.org> Neale Pickett <neale(a)woozle.org> writes: >So then, jeremy(a)alum.mit.edu (Jeremy Hylton) is all like: > >> I'm a big fan of unittests. We should probably develop some. > >I couldn't agree more. Err, before we start writing unit tests, shouldn't we have some specifications on what everything is actually supposed to do? While we're at it, it would probably be good to gather requirements and lay out interfaces for the various elements... Here's a first cut at what I see as requirements for the code: R1. The code should be written in a uniform language, as much as is possible. For historical reasons, this language is probably going to be python version 2.2.1. R2. The code (in training mode) should accept email messages and their classifications as inputs, and record relevant data for later classification. R3. The code (in classification mode) should accept email messages as input and offer trinary (ham/spam/unsure) classification as output. R4. Raw RFC 822 messages should be acceptable as input email messages. R5. For the MS users, the botch that Outlook turns messages into should be acceptable as input email messages. Despite my opinion of it. ;-) R6. Classifications should be output as a line of the form: X-Spambayes-Classification: ham/spam/unsure where only one of ham/spam/unsure should be present, without /. R7. Classifications may be added to the headers of a raw rfc822 message, in which case the whole (annotated) message should be echoed as output. R8. In no case should more than one classification be present in a message. R9. If a message provided as training input already has been trained with a classification, then it should be untrained from the old classification before training with the new classification. R10. There should be a classifier front-end usable as a procmail filter. R11. There should be a classifier front-end usable as a pop3 proxy. R12. There should be a classifier front-end usable as an Outlook plugin. R13. There should be training front-ends appropriate to each of the classifier front-ends. R14. The classifier front-ends should use a common internal classifier module/class/whatnot to do all work not specifically related to managing input and output. R15. The training front-ends should also use a common internal training module/class/whatnot to do all work not specifically related to managing input and output. R16. There should not be any costly 'process all the data' operation associated with either training or classification. R17. The internal database for the knowledge gleaned from training should be stored in persistent form between invocations. R18. Changes to the internal database should be reflected in the persistent store in a timely manner. R19. Changes to the persistent representation of the database should be done with an eye towards recoverability of the data in the case that a power outage (or similar catastrophic event) interrupts the update. R20. The classification method should consist of some combination scheme applied to ham/spam probabilities associated with tokens derived from parsing the email messages. R21. The chi-square combination method should be an allowed combination scheme. R22. The gary-combining method may be an allowed combination scheme. R23. The modified Graham probability computation (without biases, with bayesian adjustment, etc.) should be an allowed probability computation schere. R24. The tokenization scheme should be based on the recent spambayes tokenizer code, which I don't feel like describing in sufficient detail at this time. etc... - Alex

7 8

Re: [Spambayes] Re: Guidance re pickles versus DB for Outlook
by T. Alexander Popiel Nov. 27, 2002

Nov. 27, 2002

In message: <15843.63278.932180.923883(a)montanaro.dyndns.org> Skip Montanaro <skip(a)pobox.com> writes: > > Alex> Unfortunately, python 2.2.1 is the most recent version of python > Alex> packaged for debian-stable. Requiring a version more recent than > Alex> that will be a hassle for many people (myself included). > >(I think we travelled this road before, but just in case...) Yes, we have ( ;-) ), but just to rehash: >Isn't it possible for you to grab and expand the 2.2.2 tar file, then >execute > > configure --prefix=$HOME/local > make install > >and then put $HOME/local/bin in front of your PATH? Yes, it's possible, but then I have to pay more attention to upgrades than my normal method of primarily listening to the debian security announcement list. For some people, the local-python-in-the-home-dir approach won't work because they've got draconian disk quotas to contend with. For yet other people, the local copy of python won't work because the machine involved is not under their control and they don't have permission to create executables. - Alex

2 2

merging tomorrow--big code changes
by Neale Pickett Nov. 26, 2002

Nov. 26, 2002

Tomorrow I'm going to merge the hammie-playground branch back in to HEAD. A lot of stuff has changed and I'm late for a very important date so I can't go over it now. Tim Stone and I have tried to make sure everything still works, but a few of the more noticeable changes will be: * You'll have to re-train your databases (pickles or dbms) * some pop3proxy and hammie options have new names in the bayescustomize.ini file I'll post a more detailed explanantion of what will change later tonight when I get back, but I wanted to give interested parties a heads up as soon as possible. If you want to check for yourself, you can use cvs update -r hammie-playground to try all the new stuff (back up your trained databases first though). Neale

8 9

Guidance re pickles versus DB for Outlook
by Mark Hammond Nov. 26, 2002

Nov. 26, 2002

Hi everyone (and tim1 <wink>) I've been thinking about the "database" to use for the Outlook plugin. I see two reasonable choices today: pickles and whatever anydbm picks up on Windows. My understanding is that the main trade-offs are that pickles are slow to load, but lightening to use, whereas a database is fast(er) to load, but slow to use. IIRC, updating the probabilities was a real killer for a DB, but this has recently died. To be honest, my main motivation in even thinking about this is the terrible things we are doing to Outlook's startup time. My decent machine is taking quite a few seconds longer to get outlook started - and this cost is worn every time *any* application uses Outlook for anything at all. If we do any sort of training, we also pay this penalty shutting down, saving the pickle. If we crash, we lose all recent training data. So, I see two basic routes I can take: * Move to a DB, but stick with a fully synchronous model. We still wear the DB load time at startup, but this should be reduced significantly. We wear the performance costs at runtime associated with the scoring, and do all such scoring in the "foreground", and saving of the DB as necessary. * Stick with pickles, but move to a threaded asynchronous model. Messages can be "queued" for scoring/training. At startup, we spin a new thread to load the pickle. Any "missed" messages at startup, and all messages as they arrive are queued for scoring and filtering. If the pickle is loaded, then it will generally appear synchronous, otherwise new messages may sit in your inbox for a few seconds before they are removed. When the pickle is modified, a background thread copies the data, and starts writing. We do some smarts with renaming the previous versions, as Tim1 implicated. There would be support for synchronous calls too (eg, "show spam clues"), but in general, asynch could be used. I would appreciate some comments on this. I am leaning towards the asynch model, but it is clearly more complicated. However, if moving to a DB simply means we will have perf issues, just not at startup, then the complexity would be warranted. Any thoughts? Fairy god-mothers? Magic answers? Thanks, Mark.

15 55

Re: [Spambayes] Re: Guidance re pickles versus DB for Outlook
by Tim Stone - Four Stones Expressions Nov. 26, 2002

Nov. 26, 2002

11/26/2002 3:06:29 PM, Neale Pickett <neale(a)woozle.org> wrote: >So then, jeremy(a)alum.mit.edu (Jeremy Hylton) is all like: > >> And sorry to complain so loudly today. It's good that someone is >> maintaining and improving the code. The fault is probably more in our >> development process. I didn't say "Hey, check that you don't break >> the pspam code" and there was no obvious way for you to test it. > >> I'm a big fan of unittests. We should probably develop some. > >I couldn't agree more. I found myself wishing we had some when I was >making all the changes. The only code that has something approaching a >test is the pop3proxy. I'll bring that up on the list. Awwww... that's not quite right... FileCorpus has a fairly large test harness that exercises just about everything in that side of the universe... - TimS > >And please, keep complaining! I want stuff to work too :) > >Thanks > >Neale > >_______________________________________________ >Spambayes mailing list >Spambayes(a)python.org >http://mail.python.org/mailman/listinfo/spambayes > > c'est moi - TimS www.fourstonesExpressions.com

4 4

Re: [Spambayes] Guidance re pickles versus DB for Outlook
by T. Alexander Popiel Nov. 26, 2002

Nov. 26, 2002

In message: <15843.43038.341350.515691(a)slothrop.zope.com> Jeremy Hylton <jeremy(a)alum.mit.edu> writes: >I just did a cvs update in spambayes and tried to restart my pspam >code, but nothing is working anymore :-(. I'm sorry I haven't had >time to read every message on the proposed changes, but there's been a >flurry of activity and I've got a day job. Yeah, there's been a bunch of changes, mostly revolving around the removal of update_probabilities. >Anyway, here's a traceback. Can anyone suggest quickly how I would >fix this? The pspam code calls learn() for a bunch of messages and >then calls update_probabilities() at the end. Is that the default >now? Or is that a discontinued feature? Are the APIs documented anywhere? You need to remove the _last_ argument passed to learn(), dealing with whether or not to run update_probabilities(). Judging by later mail, it sounds like you removed the second-to-last argument (ham vs. spam), and are now calling everything ham. You should also remove the call to update_probabilities(), since it's toast, but it sounds like you already did that. The API is unfortunately not documented yet; until someone goes back through and updates it, the code itself will be the best documentation. Addressing a concern from a later mail, the MetaInfo class exists to make it easier to derive subclasses of the classifier and its parts, without having to touch everything. For the two counts that it currently holds, it is certainly overkill, but it provides a nice way to expand that set without having to go mucking around in as many places. You are correct that the revision number it holds is useless; it is cruft left over from one of the interim schemes for doing away with update_probabilities(), and as such the revision number should be removed. Perhaps I'll patch that. - Alex

2 2

packaging (was Re: [Spambayes] Guidance re pickles versus DB for Outlook )
by jm＠jmason.org Nov. 26, 2002

Nov. 26, 2002

Richie Hindle said: > On the subject of packaging: I've used InnoSetup before and been very > impressed. Someone mentioned Install Shield - I don't believe there's a > credible free version of that, whereas InnoSetup is completely free. Just to chime in on this one: I can second that. InnoSetup is quite good, and since it runs from 1 text format file, isn't too foreign for UNIXish folks ;) --j.

2 1

Re: [Spambayes] Guidance re pickles versus DB for Outlook
by papaDoc Nov. 26, 2002

Nov. 26, 2002

Hi, > On the subject of packaging: I've used InnoSetup before and been very > impressed. Someone mentioned Install Shield - I don't believe there's a > credible free version of that, whereas InnoSetup is completely free. You are right Install Shield is not free but I'm free to use it since we (my company) paid the full price for it ;-) > I already gave that one a spin, I think TimS was allmost by the SF staff by > trying to use something they don't use it in yesterday and got booed off the > planet... lol... seems that sourceforge has their own distribution mechanism. I can take a look at how SF packages thing or how InnoSetup work. papaDoc

1 0

Pop3proxy - doesn't save pickle when reviewing?
by Paul Moore Nov. 26, 2002

Nov. 26, 2002

Just switched pop3proxy over to pickle format. One thing I notices, which surprises me, is that when I review messages and train, the pickle is *not* saved. I have to manually save. Is this intentional? With my way of working, I imagine it could result in me forgetting to save. Surely a save isn't too much of an overhead - the review screen shows "Training... Trained on X messages... Updating probabilities... Done" messages already, so adding "Saving..." would fit into the UI fine. It should just be a change to onReview(), changing self.push("Done.</b></p>") to self.doSave() Paul. -- This signature intentionally left blank

2 1