[Spambayes] More Error Messages in log file?

[Tim Peters]

[Tim Peters]
>> Outlook 2003 introduced a NewMailEx event which *may* be a
>> reliable way (and a more efficient way) to learn about new messages, but
>> it looks like that fires before rules (and, of course, isn't available
>> before OL2003).

[VBCoder]
> I assumed that this meant that this issue was related to OL2K.  This
> leads me to believe that there is a compatibility issue with OL2K or at
> least a more significant issue with OL2K then any newer releases.

Sorry, I don't follow that logic.  When MS introduces a new API function,
and especially one that ends with "Ex", it almost always means that there
are problems in all previous versions of the API that cannot be fixed short
of introducing the new function.  If, e.g., ItemAdd had been, or could have
been, fixed in OL2000 or OL2002 or OL2003, there would have been no need for
the new function.  SpamBayes doesn't use NewMailEx, BTW, and ItemAdd isn't
any more predictable in OL2003 or OL2002 than in OL2000.

> I can assure you that if it was mentioned in the download area as an
> issue, I would not have tried it.

Please suggest specific text you would have liked to see.  You're the only
one to have a complaint here so far, and I really don't know what would have
made you happy.  The download page certainly isn't the place for a
discussion of bugs and workarounds in Outlook's event system, so the text
needs to be brief and easily comprehensible to non-programmers.

> I am not in the business of blaming and pointing fingers, just lending a
> little constructive criticism.  In my opinion, this issue is big enough
> to warrant a warning to those that would like to try it.

Based on our experience so far, the background filtering option made this a
non-issue for almost everyone.  You're the "almost" <wink>.

> I tried Tony's suggestion (yours too) and selected a few top level
> folders (not the very top as that is not allowed) and checked the
> subfolders box. Then enabled the timers.  It was almost 12 hours before I
> could use the computer.

Yikes!  That sucks.  Since nobody has tried a thousand (or more) folders
before, the cause could be anywhere (in Outlook, in MAPI, in SpamBayes).

> It seems that it takes over the whole PC and not just Outlook.  I was
> completely locked out of the entire PC for that time period.

That *suggests* it's going crazy in a system DLL.  Nothing in the Python
code changes thread priority.

> When it was finished, I found that most if not all the Spam fields had
> a value in it.  Now the problem was that none of my Outlook
> rules fired.

There's nothing SpamBayes can do to prevent Outlook rules from firing:  that
part of Outlook isn't exposed in the Outlook object model, so there's
nothing we can do to influence it (let alone stop it).  My best guess is
that adding handlers for ItemAdd events on that many folders isn't something
Microsoft ever tried either, and that Outlook just can't handle the load.

> All of the messages that did not fall into Spam or Unsure where still
> in the Inbox.

I don't know exactly what you did, so won't speculate.

> Needless to say, I have turned all of that off again.  I am now stumbling
> along with this product until I can find one more suitable for my
> environment.

Do let us know if you find one!  For people who don't like SpamBayes for
whatever reason, I usually recommend trying the POPFile addin:

    http://www.vargonsoft.com/Outclass/

I'm not sure you can live with that either, though (specifically, its docs
don't appear to say anything about how it does or doesn't work with Outlook
rules, or with a massive number of folders).

> Suggestion:  This plug-in should be coded to give up control periodically
> (frequently) so the OS and Outlook can do other tasks.

When doing massive amounts of scoring, SpamBayes can make Outlook
unresponsive (because SpamBayes runs *in* Outlook's process space), but
isn't capable of preventing the OS from task-switching.  Some deeper-level
software must be doing that part, possibly in the bowels of MAPI.

> Once again, The SpamBayes Outlook plug-in is a great idea.  It needs to
> be able to insure that it processes before any Outlook rules are fired as
> timing is essential here.  Your timer/Folder-subFolder work around seems
> to work under a small work load but is not usable when there is a large
> work load.

I'm not sure what "large" means to you.  I routinely get, e.g., 300 new
messages at a time, at cable-modem speed.  SpamBayes has no problem with
that on any of the 4 machines I routinely run it on (3 with OL2000, 1 with
OL2003; 2 Win98SE, 1 Win2K and 1 WinXP).  But I've only got a dozen .pst
files, and most of them contain only 1 or 2 non-system folders.  Scoring
tens of thousands of msgs at one time (via the Filtering menu item) makes
Outlook completely unusable for the duration (it's a modal dialog), but
doesn't prevent OS task-switching either.

As I said before, there is no known way to convince Outlook to run SpamBayes
before its own rules.  On OL2003 and later, it may be possible to exploit
NewMailEx to do so, but nobody has tried that yet.  I should also note that
*most* people think having SpamBayes run after rules is "a feature", not "a
bug".  For example, some people do whitelisting this way (by, e.g., using an
Outlook rule to move msgs claiming to be from their boss into a folder
SpamBayes doesn't watch).