[Spambayes] Upgrading from 1.0a2
gary at inauspicious.org
Fri Dec 5 09:08:09 EST 2003
Richie Hindle wrote:
> Gary Benson wrote"
> > I note that many of the changelog entries are for tokeniser
> > improvements. Would I have to do a retrain to get these
> > improvements into my database?
> In one way yes, because your current database is the result of
> running the emails through the 1.0a2 tokeniser. So say you had an
> email containing "via<hide>gra" (which the token now understands,
> but didn't used to) then you'll have a "via" and a "gra" token
> instead of one "viagra" token. But in another way no, because new
> emails will go through the new tokeniser. Since you probably have a
> decent spam score for "viagra" already, any new "via<hide>gra" email
> will get a hit for "viagra".
> If you're getting good results, I wouldn't worry about retraining.
For the past few months I've been getting a hit rate of about 97% on
my home account and with only a couple of false positives in that time
(all automated stuff along the lines of 'thanks for signing up to our
website'). My work account is not so good, 94%, but I've not been
using spambayes on it for so long so I imagine that the database is
less well trained.
I worked out that I could drop the spam_cutoff from the default of
0.90 to about 0.65 on my home account, which should bump it up for
next month, but I see Paul Graham quoting hitrates of 99.7% or
whatever and I lust ;) And I have copies of every email it was trained
on, so a retrain would simply be a matter spending of half an hour or
so writing a script.
On that note, is there anything I can do to the training set to
improve the generated database? I've seen things like ensuring the
number of spams and hams is roughly equal, for example; is there any
truth in that? Another thing is that I have some twelve months' worth
of mail in my home training set: should I use it all, or cull some of
the older stuff?
[ gary at inauspicious.org ][ GnuPG 85A8F78B ][ http://inauspicious.org/ ]
More information about the Spambayes