Mailman 3 May 2003 - spambayes-dev

More testing on the common db
by T. Alexander Popiel May 31, 2003

May 31, 2003

Here's some more results from testing with the common db and my own private db: Testing a selection of messages 4-9 months old: Ham (2052 msgs): ham unsure spam common 2011 36 5 popiel 2041 8 3 Spam (3838 msgs): ham unsure spam common 5 53 3773 popiel 8 75 3748 Testing only the most recent 500 messages of each type: Ham (500 msgs): ham unsure spam common 488 11 1 popiel 495 5 0 Spam (500 msgs): ham unsure spam common 1 21 478 popiel 1 10 489 I find it rather interesting that the common db did better on the old spam than my personal one did; I think this is evidence of mail mutations having a real effect on accuracy (since my personal db only contains info from the most recent 4 months), but it could also be attributable to other things... such as differences between Skip's training regime and my own. For the most recent mail, the personal db was a clear win over the common db. - Alex

1 0

cvs resets
by bill parducci May 31, 2003

May 31, 2003

i see this a lot: $ cvs up -Pd cvs [update aborted]: end of file from server (consult above messages if any) is the python cvs server overloaded frequently, or is it just me? (usually by the fifth or sixth try i get through.) thanks b

2 1

Re: [spambayes-dev] Proposed fourth X-Spambayes-Classification headervalue
by T. Alexander Popiel May 31, 2003

May 31, 2003

In message: <LNBBLJKPBEHFEDALKOLCAEFEEIAB.tim.one(a)comcast.net> Tim Peters <tim.one(a)comcast.net> writes: >[T. Alexander Popiel] >> ... >> Personally, I'd like to see us have a simpler parser > >I don't expect this would help. Decoding base64 and quoted-printable are >important, but base64 if and only if it's a text section. In order to >identify this stuff requires decoding the MIME structure too. Decoding >charsets probably isn't important for *me*, because virtually all my ham is >in 7-bit ASCII English, but for non-English users I can easily believe it's >vital. Etc -- the email package does a lot of stuff, and it's valuable. I see I wasn't clear... I only want the simpler parser for handling the classification headers annotation and the cases where the email package barfs. Definitely keep the email package for what we've got it for now... because yes, it helps immensely. - Alex

1 0

Re: [spambayes-dev] Proposed fourth X-Spambayes-Classification header value
by T. Alexander Popiel May 31, 2003

May 31, 2003

In message: <oprpzxmqwi34x2gn(a)mail.fourstonesexpressions.com> Tim Stone <tim(a)fourstonesexpressions.com> writes: >I'm working on this problem that crops up in pop3proxy and hammie where >malformed headers cause the parser to raise an uncaught exception, >rendering spambayes more helpless than a baby seal. It's no problem to >catch the exception, but what to do with the message is the issue. I have >two suggested approaches: > >1. Assume that the mail is spam. > >2. Add a new possible value to the classification header, something like >'unclassifiable'. > >Other alternatives, votes, or general booing and hissing? 3. Mark it as unsure. - Alex

2 1

Re: [spambayes-dev] Proposed fourth X-Spambayes-Classification header value
by T. Alexander Popiel May 31, 2003

May 31, 2003

In message: <16087.60959.324302.344655(a)montanaro.dyndns.org> Skip Montanaro <skip(a)pobox.com> writes: > > >> would it be possible to catch the exception and move to the next line > >> in the header (and/or payload) for parsing? > > Alex> Alas, we're at the wrong level to do that sort of thing. To do > Alex> that level of granularity properly, we'd have to be in the guts of > Alex> the parser... > >This is just a shot in the dark, but would it be possible to modify the >email parser sufficiently so that it gave more detail about where it was >when the error condition was detected (e.g., what body line number, header >and/or MIME part)? That might allow Spambayes to tweak the message in the >right spot and retry the parse. It is my belief that tweaking the message intelligently (as opposed to just forcing the entire body to be treated as plain text by blowing away the MIME headers) would require more intelligence than doing the parsing in the first place. After all, you'd be permuting the data to make it parse, which means that you understand all about the parsing. If we're going to get that smart, we might as well not use the email package... which has already been circled around a few times. Personally, I'd like to see us have a simpler parser which just understood headers vs. body, and didn't try to decode the individual headers (for charset, or anything like that). Ideally, we'd give this simple parser the message (as a string) and a list of headers to remove from the message, and it would return a modified message (again as a string). We could use this simpler parser both for blowing away the MIME headers (as alluded to above for dealing with malformed messages) and for annotating the message with the classification results (blow away all the classification headers, then prepend the new ones (properly formatted) to the message). Of course, that would take about two hours of work, and I'm lucky to get two consecutive minutes right now... - Alex

3 2

New faq.txt
by Skip Montanaro May 31, 2003

May 31, 2003

Folks, I just checked in faq.txt, default.css, Makefile and scripts/make.rules, and cvs removed faq.ht. Together, this means you should edit faq.txt now to update the faq. The last question (4.6) has some info about what you need to install to rebuild faq.html from faq.txt. Thanks for the feedback. We now return you to your regularly scheduled programming. Skip

1 0

Next to no feedback on the trial faq
by Skip Montanaro May 31, 2003

May 31, 2003

(spambayes-dev now seems to have enough people on it to form a quorum of sorts (26), so I'm excluding spambayes...) I posted URLs yesterday for a trial version of the Spambayes FAQ built using Docutils tools: http://spambayes.sourceforge.net/faq-rest.txt http://spambayes.sourceforge.net/faq-rest.html So far, the only person who responded was Bill Parducci. It's not surprising that he responded, because he's the other person who seems to be in FAQ maintenance hell at the moment. Still, I thought one or two other people would have responded. The Docutils version has the advantage that it will automagically generate the table of contents. Its main drawback is that there is no explicit faq generator script in Docutils, so the format of the questions is a bit constrained. Before I make an executive decision and simply adopt this new stuff (I believe it will be much easier to maintain than the status quo), I would like some feedback: * Does anyone have a problem that it doesn't follow the ht2html format for the spambayes site as a whole? I can probably worm around this by extracting the body of the faq to faq.ht then let the usual ht->html dependency work its magic. * Should anyone wanting to update the FAQ be required to install a recent version of Docutils or should I somehow worm around that? (This is especially disconcerting because when you install Docutils it doesn't install the html.py front-end script used to generate the faq.html file, so I can't really assume it will be in PATH.) If either of these is a big deal, please let me know. Skip

4 4

[ spambayes-Bugs-706520 ] assert fails in classifier
by SourceForge.net May 31, 2003

May 31, 2003

Bugs item #706520, was opened at 2003-03-19 12:46 Message generated for change (Comment added) made by leobru You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=498103&aid=706520&group_i… Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Adam Glass (adamglass) Assigned to: Nobody/Anonymous (nobody) Summary: assert fails in classifier Initial Comment: This morning, I noticed that my emails no longer had a X-Spambayes-Classification header, so I looked through my procmail logs, and sure enough, hammiefilter.py is giving a traceback when an assertion fails. This happens on all messages now; it is not specific to a single message, or intermittent. Therefore, I suspect my .hammiedb is corrupted... I can supply it to anyone who would like to investigate it for debugging purposes. I am using Spambayes 1.0a2, installed on a system with Python 2.2.1, with the new version of the email library (as per the install docs.) Please contact me if you require any further details. Example of how to generate the error follows, along with traceback: adam$ /usr/local/bin/hammiefilter.py -f -d $HOME/.hammiedb < example Traceback (most recent call last): File "/usr/local/bin/hammiefilter.py", line 179, in ? main() File "/usr/local/bin/hammiefilter.py", line 175, in main action(msg) File "/usr/local/bin/hammiefilter.py", line 113, in filter return h.filter(msg) File "/usr/local/lib/python2.2/site-packages/spambayes/hammie.py", line 108, in filter prob, clues = self._scoremsg(msg, True) File "/usr/local/lib/python2.2/site-packages/spambayes/hammie.py", line 38, in _scoremsg return self.bayes.spamprob(tokenize(msg), evidence) File "/usr/local/lib/python2.2/site-packages/spambayes/classifier.py", line 217, in chi2_spamprob clues = self._getclues(wordstream) File "/usr/local/lib/python2.2/site-packages/spambayes/classifier.py", line 441, in _getclues prob = self.probability(record) File "/usr/local/lib/python2.2/site-packages/spambayes/classifier.py", line 304, in probability assert spamcount <= nspam AssertionError ---------------------------------------------------------------------- Comment By: Leonid (leobru) Date: 2003-05-30 17:36 Message: Logged In: YES user_id=790676 This happens, e.g., if a forced re-training was performed on a non-empty database, thus screwing up the message counts - this is for sure, I was bitten by it myself; or, potentially, if hammiefilter.py -t and mboxtrain.py were running at the same time ??? To avoid: do not do it (I do not use hammiefilter.py -t to be on the safe side). To fix, once it happens: start from scratch. Good to have in the next version: a database validator and corrector. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=498103&aid=706520&group_i…

1 0

Re: [spambayes-dev] Proposed fourth X-Spambayes-Classification header value
by T. Alexander Popiel May 30, 2003

May 30, 2003

In message: <3ED7C994.3090709(a)parducci.net> bill parducci <bill(a)parducci.net> writes: >Tim Stone wrote: >> I'm working on this problem that crops up in pop3proxy and hammie where >> malformed headers cause the parser to raise an uncaught exception, >> rendering spambayes more helpless than a baby seal. It's no problem to >> catch the exception, but what to do with the message is the issue. I >> have two suggested approaches: >> >> 1. Assume that the mail is spam. >> >> 2. Add a new possible value to the classification header, something like >> 'unclassifiable'. >> >> Other alternatives, votes, or general booing and hissing? > >would it be possible to catch the exception and move to the next line in >the header (and/or payload) for parsing? Alas, we're at the wrong level to do that sort of thing. To do that level of granularity properly, we'd have to be in the guts of the parser... and while I'm sure that Barry would love for us to come up with a way to recover inside the parser, I think it's a bit out of scope for the something external to the parser. - Alex

2 1

[ spambayes-Bugs-740843 ] No Disk Error with Outlook 2000 on startup
by SourceForge.net May 30, 2003

May 30, 2003

Bugs item #740843, was opened at 2003-05-20 18:39 Message generated for change (Comment added) made by portola You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=498103&aid=740843&group_i… Category: Outlook Group: None Status: Open Resolution: None Priority: 5 Submitted By: Sam Snow (snowsam) Assigned to: Mark Hammond (mhammond) Summary: No Disk Error with Outlook 2000 on startup Initial Comment: After installing SpamBayes-Outlook-Setup-002.exe I am now getting an error dialog on Outlook startup. The box says: (Header) Inbox - Microsoft Outlook:OUTLOOK.EXE - No Disk (Body) There is no disk in the drive. Please insert a disk into drive \Device\Harddisk0\DR0. (Buttons) Cancel, Try Again, Continue I am able to click cancel or continue several times and then outlook goes ahead and opens up. I just installed this evening, so I am not sure if the filtering is still working correctly. I was able to train the program sucessfully. I am using Office 2000 SP3 on Win 2000. I will try to attach a jpg of the dialog box. My error log says the following: SpamAddin - Connecting to Outlook Loaded bayes database from 'C:\Documents and Settings\Snow1\Application Data\SpamBayes\default_bayes_database.db' Loaded message database from 'C:\Documents and Settings\Snow1\Application Data\SpamBayes\default_message_database.db' Bayes database initialized with 0 spam and 0 good messages Loaded databases in 4.64165ms AntiSpam: Watching for new messages in folder Inbox AntiSpam: Watching for new messages in folder Spam Processing 0 missed spam in folder 'Inbox' took 31.9599ms pythoncom error: Python error invoking COM method. Traceback (most recent call last): File "E:\src\pythonex\com\win32com\server\policy.py", line 275, in _Invoke_ File "E:\src\pythonex\com\win32com\server\policy.py", line 280, in _invoke_ File "E:\src\pythonex\com\win32com\server\policy.py", line 601, in _invokeex_ File "E:\src\pythonex\com\win32com\server\policy.py", line 541, in _invokeex_ File "E:\src\spambayes\Outlook2000\addin.py", line 203, in OnItemAdd File "E:\src\spambayes\Outlook2000\addin.py", line 163, in ProcessMessage File "E:\src\spambayes\Outlook2000\filter.py", line 15, in filter_message File "E:\src\spambayes\Outlook2000\manager.py", line 440, in score File "e:\src\spambayes\spambayes\classifier.py", line 217, in chi2_spamprob File "e:\src\spambayes\spambayes\classifier.py", line 465, in _getclues File "e:\src\spambayes\spambayes\classifier.py", line 316, in probability exceptions.AssertionError: ---------------------------------------------------------------------- Comment By: Dennis Austin (portola) Date: 2003-05-30 14:44 Message: Logged In: YES user_id=787905 I have noted an additional piece of information. There is no alert first time I start Outlook after logging in. If Outlook is closed and reopened, the alert appears and requires three Cancel clicks before it goes away. (Unless I put a disk in the CD drive.) I'm using Outlook 2002 sp2 on Windows XP sp1. I have two CD drives on the secondary IDE channel and the error appears on the second drive. ---------------------------------------------------------------------- Comment By: Dennis Austin (portola) Date: 2003-05-27 10:16 Message: Logged In: YES user_id=787905 I also usually see this error when I start Outlook, although not every time. I also see it at the end of running the installer. In my configuration it shows up as "No disk in drive E:". E: is CD-ROM 1 on this machine. I can get past the error either by clicking Cancel several times, or by putting any old CD in the drive and clicking Try Again. The error does not seem to affect any function of the add-on. ---------------------------------------------------------------------- Comment By: Ferruccio Barletta (fgb) Date: 2003-05-25 07:40 Message: Logged In: YES user_id=786210 I may have found the root cause of this problem. When I brought up disk management on my notebook I noticed that my hard drive was Disk1 and the SD media drive was Disk0. When I disabled the SD drive and rebooted, the hard drive became Disk0 and the problem disappeared. ---------------------------------------------------------------------- Comment By: Ferruccio Barletta (fgb) Date: 2003-05-24 18:30 Message: Logged In: YES user_id=786210 I get the same error with Office 2002 SP1 on Windows XP SP1 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=498103&aid=740843&group_i…

1 0