[spambayes-dev] Semi-results for TOE, TOAE, and expiry

[T. Alexander Popiel]

Yes, a few days ago I promised a further report on various things,
including the effects of alternate start points in my dataset and
expiry on the train_on_everything and train_on_almost_everything
regimes.  Unfortunately, all I have at the moment is some
preliminary results and a big heap of frustration pointed at
my computer.

1. There doesn't appear to be anything particularly magical
   about 120 days after start.  Rotating my data forward or
   backward 80 days shows that (a) there was a particular
   event/change in my data at about 120 days after I started
   collecting that affects the accuracy of further 
   classifications, and (b) the general curve of getting
   better for a few months then decaying for the rest of
   time still holds even when the data is rotated... but
   the curve is not as distinct when not reinforced by (a).

2. Expiry (as I implemented it) appears to be a very bad thing
   for long-term TOAE.  I implemented it to expire trained
   messages after 120 days, without completely rebuilding the
   classifier.  This resulted in significantly degraded accuracy
   after about 250 days, though that may just be due to an ever
   increasing spam/ham imbalance.

   There was a sharp drop in the amount of spam training for
   about 30 days after the initial expiry date, and then a net
   spam training rate about equivalent to non-expiring TOAE
   until the "latest windows update" worm, after which spam
   training about doubled the non-expiry version.  This
   seems to show that spam mutation has a stong effect on
   4-month expiry for TOAE.

   On the other hand, net ham training was fairly consistently
   slightly negative after expiry commenced, showing that
   once it got a good idea of what ham was and threw out the
   oddballs that got trained on initially, it didn't need much
   categorize ham.

   By the end of the mess (at 418 days) the spam:ham ratio was
   over 15:1, and the unsure rate was around 3% (compared to
   non-expiring with 4.5:1 and 1%).

3. Expiry for TOE seems neutral (compared to non-expiring TOE),
   to the best of my ability to eyeball the three runs that
   actually completed.

The graphs I have are at:
http://www.wolfskeep.com/~popiel/spambayes/plots/expire.html

My primary machine (cashew.wolfskeep.com) unfortunately doesn't
seem have the capability to maintain reliable service while
running these tests anymore.  They're just too big, and it
doesn't have the memory/CPU to do everything all at once
(including running my web server, a mysql engine, my mail feed,
etc.).  Plus, it appears that Linux 2.4.18 doesn't take too
kindly to multiple processes trying to access/manipulate a
single directory with over 100,000 files in it; anything that
touches that directory after things have started going wonky
just hangs in disk-wait.  I'm suspecting a deadlock in the
filesystem layer on extended directory operations... probably
due to not enough file cache (see my memory problems) to hold
the entire structure at once.  I haven't poked deep enough into
the ext2 drivers to be sure, though.

Anyway, I'm not going to be able to do all that much until
I get this straightened out.  I'll add graphs and stuff to the
wiki as I have time, but that's likely to be all for a bit...

- Alex