on Sun Oct 21 2007, skip-AT-pobox.com wrote:

David> The result was that, when I was doing server-side filtering, David> cache pickles would be read while being written, and SpamBayes David> wasn't prepared to deal with the corruption.

David,

Can you explain how the pickle would be read and written simultaneously? Are we talking multiple apps or the same app from multiple threads? (Haven't looked at your patch yet...)

I assume it's the same app from multiple processes. I don't know how my server hands messages off to procmail -- it could well be running them in parallel. And then there's my cron job that does training once daily, which could happen at the same time as any message arrives. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com

on Mon Oct 22 2007, skip-AT-pobox.com wrote:

>> Can you explain how the pickle would be read and written >> simultaneously?

Dave> I assume it's the same app from multiple processes.... And then Dave> there's my cron job that does training once daily, which could Dave> happen at the same time as any message arrives.

That would be it.

Well, the cron job does tte with a fresh database, so it's only the various caches that can get into trouble.

There is no locking of the various databases between processes. I suppose we could implement something, but the vagaries of network file systems and cross-platform demands have always made that task problematic. Maybe we should expand your safe_pickle code into safe_pickle_read and safe_pickle_write with a corresponding bit of file locking code they can invoke (which will probably be ugly as sin - but hidden away in a dark corner - once it's complete).

Sounds fine to me. I doubt it would be so terribly ugly either. Somebody should have already written a file lock library that encapsulates it, so maybe we can use that. And if it doesn't exist, it should! -- Dave Abrahams Boost Consulting http://www.boost-consulting.com

skip@pobox.com wrote:

Dave> Somebody should have already written a file lock library that Dave> encapsulates it, so maybe we can use that. And if it doesn't Dave> exist, it should!

It does exist, in several flavors, as my query to python-dev indicates.

Sounds like a good addition to python, but Im not sure its the solution in this case. The cron job probably should tte using a different database filename, then atomically swap the new database in place of the old. This ensures you dont leave a partial database in place if the machine crashes/is shutdown/process is killed during the tte run. It also allows procmail to run on the old database concurrently with the training run, which I guess a lock file would prevent. Or am I confused here. Ive just re-read David's original post, and you say this the problem is caches rather than the database. Which caches are these? -- Toby Dickenson (happy procmail user)