[spambayes-dev] Strange performance dip andDBRunRecoveryErrorretreat

[Richie Hindle]

[Richie]
> I've committed the script as testtools/hammer.py,

[Tony]
> Actually utilities/hammer.py :)

Like I said, it was 2am.  8-)

> Richie - around about how many messages could you do before it crashed?  It
> crashes early (~1700) for me with the 1.0a6 reopen-before-closing bug
> (sometimes a DB_RUNRECOVERY, sometimes a ham/spam count) but if I get it to
> always close or save the db before reopening, it just goes and goes - I got
> to 57900 before Python crashed *.

With the bug it would get to a few tens of thousands, same as you.
Without it went to 250,000 before I killed it.

> If I get it to close before reopening, then interrupt (ctrl-c) it at some
> point after its done its first save, and restart it (without it deleting the
> existing db file) it will chug along for a while, and then at some later
> point die with a RUNRECOVERY.  Restarting it then will provoke an immediate
> RUNRECOVERY.  This suggests (I think) that reopening the db without closing
> it can cause a RUNRECOVERY error at some later point - even several
> reopenings later - rather than immediately like I expected.

Agreed, though I never noticed the "even several reopenings later" part,
probably due to impatience...

[Tony]
> Maybe this is one of the causes for the RUNRECOVERY errors - the user
> doesn't close sb_server properly, so the db isn't properly closed/saved.
> Some time later the error occurs.  It would explain why they are less
> frequent with the plug-in, because the plug-in saves the db much more often
> (after every "delete as spam"/"recover from spam" event, and IIRC after any
> incremental training).

[Tim]
> I think you may be on to something here!

Sadly not.  sb_server saves the db after ever train as well, out of
paranoia.  The page should always say "Training... Saving... Done".  If
there's a way of training without saving, maybe that's the problem, but I
don't believe there is...?

-- 
Richie Hindle
richie at entrian.com