[Richie Hindle]

As part of trying to reproduce the DBRunRecoveryError problems (a task that I'm giving up on for now - see below) I've written a script to hammer the core SpamBayes code, repeatedly training and classifying using faked-up messages. It manages about 40 train-and-classify loops per second on my 2.4GHz P4, *except* between about 100 and 400 messages, when the performance drops to about a tenth of that and then recovers.

I've done enough investigation to know that the time is being spent in the core SpamBayes code and not my script,

Is that a true dichotomy? That is, do you know, for example, that the time is being spent in the core spambayes code as distinct from the Berkeley database library, or distinct from random network traffic other programs are engaging in? Or is it that you just know it's not in your script, and you divide the universe into "my script" and "the core SpamBayes code" here?

that it's only the occasional message that takes a long time (around a second in a few cases) and that it can be either training or classifying that slows down.

I've committed the script as testtools/hammer.py, and I offer this as a curiosity to anyone interested. I'm not going to pursue this myself because I've never seen a similar complaint about real-world SpamBayes use.

Well, Python certainly doesn't make any real-time guarantees, and I doubt Sleepycat, or even your OS, do either. So long as it recovers, I don't think there's anything worth investigating. It could be Python resizing a large dict, or an all-generations garbage collection cycle, or Sleepycat rearranging its memory allocation, or the OS rearranging swap space, ..., there's just no limit on what it *could* be.

.... I don't think the script is going to be a lot of use in tracking down DBRunRecoveryErrors - it *will* reproduce them as it is, but only by mimicking a bug that was fixed in 1.0a6, and people have still been complaining about DBRunRecoveryErrors in 1.0a6 and 1.0a7.

Thanks for the effort! Maybe somebody else can complicate it now in a way that does provoke DBRunRecoveryErrors. It's never what you expect <wink>.

Having read up on full-mode bsddb, and bsddb-backed ZODB (including the phrases "The underlying Berkeley database technology requires maintenance, careful system resource planning, and tuning for performance." and "BerkeleyDB never deletes "old" log files. Eventually, if you do not maintain your Berkeley database by deleting "old" log files, you will run out of disk space") I've given up - for the moment at least - on trying to use full-mode bsddb (with or without ZODB).

That's par for the course for "a real" database. Even plain FileStorage-backed ZODB requires ongoing maintenance, including periodic "packing" to prevent unbounded growth, and religiously observed backups. It's all this extra hair that makes a real database robust against most of the things that can go wrong. But also for that reason, it's unusual to see "a real database" solution in consumer-grade applications. We could write our own database specialized to our project's specific needs, and probably get that working faster and better than any general-purpose beast. But my interest in that was fully satisifed by pickling a giant dict <wink>.

sb_server users should use a pickle and be done with it.

I've been saying that for a decade <wink>. Before you get too sick of it, you might also want to investigate Neil Schemenauer's adaptation of spambayes for cdb. cdb is an efficient and essentially worry-free disk-based database. It buys this at the cost of *not* being incrementally updatable: you can replace the whole thing atomically, in one giant gulp, but that's it. If you don't need incremental training with instantly-visible effects, I bet it's an excellent approach. There are no worries about synchronizing concurrent reads and writes, simply because there are no writes. Looks like there *are* worries about concurrent reads, though: http://cr.yp.to/cdb/reading.html Beware that these functions may rely on non-atomic operations on the fd ofile, such as seeking to a particular position and then reading. Do not attempt two simultaneous database reads using a single ofile. Robust all-purpose database implementation is damned hard.

Maybe we should change the default. Maybe it's five to two and I should be in bed.

It's a pit, isn't it? If it's any consolation, even Unix mboxes get corrupted, and nothing is simpler than "append at the end".