Mailman 3 July 2005 - spambayes-dev

Re: [spambayes-dev] Integrating SpamBayes with a Mail.app plugin
by Tony Meyer Aug. 1, 2005

Aug. 1, 2005

> 1. Is Classifier.*_spamprob() thread-safe? I will be calling them > from multiple threads. Anything I should be aware of? 1. You'll want to be calling spamprob() itself, not one of the variants. 2. As long as you are not *training* from multiple threads (or training and classifying at the same time), you should be fine. (i.e. stick to reading only). > 2. I've looked around and tried to figure out what would be saved > into SB's database. Am I correct in saying that it only saves the > classification results and IDs, not the raw message source? The 'message info' database saves various things, but you won't need any of those if you're not any of the existing SpamBayes applications. The hammie (token) database saves: 1. The total number of ham and spam trained. 2. (token, hamcount, spamcount) entries. The 'hamcount' and 'spamcount' in the latter are the number of times that particular token has been seen in ham/spam. The raw message source is not stored. IMO, it would be best to use the existing SpamBayes infrastructure to manage the SpamBayes database (writing a new storage class if necessary). [...] > 3. At least as the first cut, I'll provide a very dumb-down > UI for SB (i.e., no user configuration UI will be provided). [...] > Any other default settings I should be aware of? Or using the > defaults in Options.py is all I need to do? All defaults will be fine. =Tony.Meyer

1 0

Integrating SpamBayes with a Mail.app plugin
by Benjamin Han July 31, 2005

July 31, 2005

Hi list, I just started working on integrating SpamBayes 1.0.4 into JunkMatcher, a Mail.app plugin I wrote on OS X. I've heard a lot of good things about SB and I'd love to have it work for me. :-) In JunkMatcher messages are filtered based on conducting a list of tests, and these include both "property tests" and "pattern tests". The latter are regex matching, and all other kinds of tests are property tests (including blacklists querying, etc.). (more info about JunkMatcher: http://junkmatcher.sf.net ) My plan is to integrate SB as a property test. I've downloaded the code and took a brief look. It seems that I only need to replicate the sb_filter.py so SpamBayes would kick in when checking. I'd appreciate if someone can do a bit hand-holding for me - here are a few newbie questions: 1. Is Classifier.*_spamprob() thread-safe? I will be calling them from multiple threads. Anything I should be aware of? 2. I've looked around and tried to figure out what would be saved into SB's database. Am I correct in saying that it only saves the classification results and IDs, not the raw message source? This is because JunkMatcher already saved raw email sources and I'd like to avoid duplicating data. 3. At least as the first cut, I'll provide a very dumb-down UI for SB (i.e., no user configuration UI will be provided). I already know I need to give some default settings as to where to deposit hammie.db. Any other default settings I should be aware of? Or using the defaults in Options.py is all I need to do? Thanks in advance, Ben

1 0

Shopping for software? Now in your language & currency!
by Sara July 23, 2005

July 23, 2005

Software for System Builders, Resellers, and Hardware Purchasers Only. http://ptick.6da39n6h3y6d37o.ekisthmoidgj.info For what is liberty but the unhampered translation of will into act? If they give you ruled paper, write the other way.

1 0

SpamBayes for Document Categorization?
by Michael Murdock July 23, 2005

July 23, 2005

Hello, I am interested in using SpamBayes as the core classifier for a system I want to write that classifies document instances into categories. Instances might be formatted in Word, PDF, text, or html. Of course I don't expect SpamBayes to know how to read all these different formats. So for the sake of discussion, let's just say it could process the text of any document I throw at it. For the sake of discussion, let's say I have five categories with many document instances (training examples) from each of these five categories: Doc Category #1 - streaming media protocols Doc Category #2 - media format conversion tools Doc Category #3 - DirectShow Doc Category #4 - media content management systems Doc Category #5 - none of the above In my proposed system I drag a document instance into a watch folder, which causes a text classifier to open it, analyze it and "tag" it somehow to indicate to which of the five categories it belongs (say by moving it into one of five directories). Here are my five concerns. *1. Embedding the SpamBase code into my app. * My first concern is whether or not the SpamBayes training and classifier code is structured such that it can be embedded into this kind of tool. I'm pretty comfortable with Python. But rewriting major pieces of SpamBayes to do this app would not be fun, nor feasible. *2. SpamBayes for Non-Email-Types of Classification.* Does it even make sense to start with SpamBayes since my problem domain doesn't have anything like email headers or the presence of an attachment, etc. that SpamBayes probably uses in its core feature extraction? *3. Discriminatory Training* My next concern relates to the lack of discriminatory training between categories. I think the way SpamBayes works is my training on a particular class, say class 1, is building a model with which to make the discrimination: Is this document instance a member of class 1 or not? When I train the model for class 1 do I only include positive instances (the ham) of Category 1? Or do I also include negative instances from the other categories (spam)? If the model for Category-1 is only trained on positive instance from that category, then this trained model is independent of the trained models for categories 2 through 5. And when it comes time to make a classification the model that responds "loudest" is the one selected. But, and here's my concern, there has never been a proabability model created that *discriminates between* the categories. Does this make sense what I am describing? I guess I'm thinking about Maximum Likelihood training of acoustic models in a speech recognition system, which has this lack of discriminatory training and I'm wondering if multi-class naive-Bayes classifiers have this same kind of shortcoming. *4. Adding a New Document Category.* Let's say I have trained the models on my five classes (as described above) and everything is working fine and I decide to add a new document category. Do the first five models need to be trained from scratch (to include the negative instances in this new sixth category)? Or can SpamBayes models be "incrementally" trained by just training on these new class-6 negative examples? *5. Size of Training Sample* My final concern relates to the number of training documents I would need. I'm guessing that each of my documents, no matter how long or short, reduces to a single feature vector for training and classification. Is this correct? If so, it would seem that I would need *at least* hundreds of examples from each category and probably thousands. Yes? No? Thanks for any thoughts you might have on my concerns and questions. ~Michael.

1 0

Re: [spambayes-dev] Outlook Plug-In Bug?
by Tony Meyer July 23, 2005

July 23, 2005

> The following were tested on a PC running WINXP SP2 all patches and > Office 2003 SP1 all patches. Thanks for that. > FWIW: I tried Inboxer 2.1 since it's based on the Spambayes engine and > it doesn't hang. Interesting. That indicates that even if this is an Outlook bug as I suspect, there must be a way to work around it. Perhaps if we can't figure it out Sean (of Inboxer) might be willing to contribute the fix. =Tony.Meyer

2 1

Re: [spambayes-dev] Outlook Plug-In Bug?
by Meyer, Tony July 22, 2005

July 22, 2005

> I'd be more than happy to try older versions. Old versions are (for the moment at least) here: <http://www.massey.ac.nz/~tameyer/SpamBayes-Outlook-Setup-001.exe> <http://www.massey.ac.nz/~tameyer/SpamBayes-Outlook-Setup-002.exe> <http://www.massey.ac.nz/~tameyer/SpamBayes-Outlook-Setup-003.exe> <http://www.massey.ac.nz/~tameyer/SpamBayes-Outlook-Setup-004.exe> <http://www.massey.ac.nz/~tameyer/SpamBayes-Outlook-Setup-005.exe> <http://www.massey.ac.nz/~tameyer/SpamBayes-Outlook-Setup-006.exe> <http://www.massey.ac.nz/~tameyer/SpamBayes-Outlook-Setup-007.exe> <http://www.massey.ac.nz/~tameyer/SpamBayes-Outlook-Setup-008.exe> <http://www.massey.ac.nz/~tameyer/spambayes-1.0a9.exe> (After 1.0a9 was just 1.0b1 and 1.0rc1 IIRC, which were all essentially the same as 1.0, which you have already tried). I would suggest trying the oldest version first. If it has the same problem, then it's highly likely that they all do (so every version has had this problem), since it's unlikely that we accidentally fixed the problem in one version, then accidentally broke it again later. =Tony.Meyer

1 0

Re: [spambayes-dev] Outlook Plug-In Bug?
by Robert Mezzone July 21, 2005

July 21, 2005

The following were tested on a PC running WINXP SP2 all patches and Office 2003 SP1 all patches. I installed a version and tested. Uninstalled, rebooted, deleted c:\program files\spambayes as well as the User Profile under XP prior to installing a different version. 1.0 Outlook.exe process does not end gracefully (I found this on a sourceforge site) 1.0.3 Outlook.exe process does not end gracefully 1.0.4 Outlook.exe process does not end gracefully 1.1a.1 Outlook.exe process does not end gracefully FWIW: I tried Inboxer 2.1 since it's based on the Spambayes engine and it doesn't hang. If I have time later I will build a second machine with OfficeXP and run the above tests. hth Robert -----Original Message----- From: Meyer, Tony [mailto:T.A.Meyer@massey.ac.nz] Sent: Tuesday, July 19, 2005 8:58 PM To: Robert Mezzone; spambayes-dev(a)python.org Subject: RE: [spambayes-dev] Outlook Plug-In Bug? > User opens Word or any application that supports the Send-To feature [...] I suspect that this is this bug: [ 1103976 ] Outlook.exe will not terminate when launched via mailto <http://sourceforge.net/tracker/index.php?func=detail&aid=1103976&group_ id=61702&atid=498103> My money is on this being an Outlook bug. It's possible we can work around it somehow, of course. Since this only appears to effect Outlook 2003, and I'm using Outlook 2002, it's pretty difficult for me to look into ways around this (not that I have the time right at the moment, anyway). Kenny is assigned the above bug report, and is going to look into it when he has time, I believe (which won't be for a while as well, IIRC). Whether or not this was a problem in earlier versions of the plug-in is an interesting question, to which I don't know the answer. It's possible that it was, especially since the problem doesn't effect old versions of Outlook. If you (or anyone else) wants to try old plug-in versions, I have copies of them all (or could unhide them on sourceforge temporarily). If the problem doesn't occur with old versions, then that would certainly be valuable information for the bug report, since that gives an idea of where to find a solution. =Tony.Meyer

1 0

Re: [spambayes-dev] Outlook Plug-In Bug?
by Robert Mezzone July 20, 2005

July 20, 2005

I'd be more than happy to try older versions. I should have saved my copies. I have to do some further testing but I can't recall having the problem with 1.1a1 and I've been using it for months. I want to try upgrading one of the 1.0.x machines tomorrow to see what happens. I downloaded Process Explorer from sysinterinals. Perhaps I can figure out what is causing the process to not end gracefully. Robert -----Original Message----- From: Meyer, Tony [mailto:T.A.Meyer@massey.ac.nz] Sent: Tue 7/19/2005 8:58 PM To: Robert Mezzone; spambayes-dev(a)python.org Subject: RE: [spambayes-dev] Outlook Plug-In Bug? > User opens Word or any application that supports the Send-To feature [...] I suspect that this is this bug: [ 1103976 ] Outlook.exe will not terminate when launched via mailto <http://sourceforge.net/tracker/index.php?func=detail&aid=1103976&group_id=6…> My money is on this being an Outlook bug. It's possible we can work around it somehow, of course. Since this only appears to effect Outlook 2003, and I'm using Outlook 2002, it's pretty difficult for me to look into ways around this (not that I have the time right at the moment, anyway). Kenny is assigned the above bug report, and is going to look into it when he has time, I believe (which won't be for a while as well, IIRC). Whether or not this was a problem in earlier versions of the plug-in is an interesting question, to which I don't know the answer. It's possible that it was, especially since the problem doesn't effect old versions of Outlook. If you (or anyone else) wants to try old plug-in versions, I have copies of them all (or could unhide them on sourceforge temporarily). If the problem doesn't occur with old versions, then that would certainly be valuable information for the bug report, since that gives an idea of where to find a solution. =Tony.Meyer

1 0

Re: [spambayes-dev] Outlook Plug-In Bug?
by Meyer, Tony July 20, 2005

July 20, 2005

> User opens Word or any application that supports the Send-To feature [...] I suspect that this is this bug: [ 1103976 ] Outlook.exe will not terminate when launched via mailto <http://sourceforge.net/tracker/index.php?func=detail&aid=1103976&group_id=6…> My money is on this being an Outlook bug. It's possible we can work around it somehow, of course. Since this only appears to effect Outlook 2003, and I'm using Outlook 2002, it's pretty difficult for me to look into ways around this (not that I have the time right at the moment, anyway). Kenny is assigned the above bug report, and is going to look into it when he has time, I believe (which won't be for a while as well, IIRC). Whether or not this was a problem in earlier versions of the plug-in is an interesting question, to which I don't know the answer. It's possible that it was, especially since the problem doesn't effect old versions of Outlook. If you (or anyone else) wants to try old plug-in versions, I have copies of them all (or could unhide them on sourceforge temporarily). If the problem doesn't occur with old versions, then that would certainly be valuable information for the bug report, since that gives an idea of where to find a solution. =Tony.Meyer

1 0

Outlook Plug-In Bug?
by Robert Mezzone July 20, 2005

July 20, 2005

Hi, We noticed the following problem with version 1.0.3 and 1.0.4 of the Outlook Plug-in All machines are WindowsXP SP2 all latest patches All machines are Office 2003 SP1 all latest patches Scenario User logs into windows User opens Word or any application that supports the Send-To feature User sends an email using the Send-To As Attachment feature. Message is delivered Try to send a second attachment Error "Operation Failed" Look in Task Manager - there are two OUTLOOK.EXE process' running. The first process did not end gracefully when the first message was sent. For some reason the two processes are causing a problem that prevents messages from being delivered from an external application. Disable the Spambayes COM Add-In within outlook and the OUTLOOK.EXE process ends gracefully every time. Scenario 2 User logs into Windows Opens Outlook Opens Word Uses the Send-To As Attachment feature User can send documents all day long provided they don't close Outlook. Every indication points to the Spambayes program causing the Operation Failed error. I haven't been able to reproduce this with 1.1a1 Is this a known bug in the older versions? Thanks. Robert Mezzone

1 0