
Over in the spambayes project, we get reports of database corruption from people using Sleepycat bsddb. The most recent comment on a bug report is revealing: http://sf.net/tracker/?func=detail&atid=498103&aid=788051&group_id=61702 """ Date: 2003-08-18 04:09 Sender: fufsource i did some experimenting with various bsddb3 versions: - with db-3.3.11 the python segfaults and core is dumped - with db-3.2.9 the database is corrupted - with db-4.1.25 everything works as it should (no db corruption) """ spambayes makes elementary use of a Berkeley DB, just accessing via the dict interface -- inserts, deletes and lookups, but no cursors, no transactions, nothing "fancy". I don't have time to dig into this, but assuming the report is correct, how can we "encourage" Unix weenies to use db-4.1.25? (On Windows, db-4.1.25 is shipped with the installer.) If the problems with older versions are so severe, maybe the Python wrapper should do a version check and refuse to run if it finds an old version?

Tim> I don't have time to dig into this, but assuming the report is Tim> correct, how can we "encourage" Unix weenies to use db-4.1.25? (On Tim> Windows, db-4.1.25 is shipped with the installer.) If the problems Tim> with older versions are so severe, maybe the Python wrapper should Tim> do a version check and refuse to run if it finds an old version? I'm using it just fine with 3.3.11 on Mac OS X. I'm not using pop3proxy though, just hammiefilter (for both scoring and incremental training) or hammie (for training from scratch). According to the pybsddb project's README.txt file (updated five weeks ago): This wrapper should be compatible with BerkeleyDB releases going back to 3.1.17 up to and including DB 4.1.25. I'm not saying the poster is wrong, just that his problem is not trivially confirmed. Skip

[Skip Montanaro]
Berkeley version numbers seem partially insane, though: if you have 3.3.11, that doesn't tell us whether you've installed either, neither, or both of the available Sleepycat patches *to* 3.3.11. I don't know whether those patches fix potential corruption problems in 3.3.11, but IIRC Sleepycat patches don't bump the version number regardless.

Tim> Berkeley version numbers seem partially insane, though: if you have Tim> 3.3.11, that doesn't tell us whether you've installed either, Tim> neither, or both of the available Sleepycat patches *to* 3.3.11. I Tim> don't know whether those patches fix potential corruption problems Tim> in 3.3.11, but IIRC Sleepycat patches don't bump the version number Tim> regardless. That's a good question. There is a patch file in the /sw/fink directory which patches several files in the distribution: db-3.3.11/build_vxworks/db.h Thu Dec 20 15:03:34 2001 db-3.3.11/db185/db185.c Thu Dec 20 15:03:30 2001 db-3.3.11/db185/db185_int.in Thu Dec 20 15:03:30 2001 db-3.3.11/dist/configure Thu Dec 20 15:04:25 2001 db-3.3.11/include_auto/db185_ext.in Thu Dec 20 15:03:30 2001 db-3.3.11/include_auto/db185_uext.in Thu Dec 20 15:03:30 2001 db-3.3.11/include/mutex.h Fri Oct 25 21:57:35 2002 I think the dates are patch dates. Still, there's nothing explicit - like a README file - which says, "we installed the foo and bar patches. Comparing the above files with http://www.sleepycat.com/update/3.3.11/patch.3.3.11.html suggests that I have both applied. Neither looks like it fixes any runtime bugs which we'd encounter. One's to fix a compile problem. The other's specific to the vxworks platform. Skip

On Mon, 2003-08-18 at 12:36, Tim Peters wrote:
Correct. That ain't the way Sleepcat does things. :)
A quick scan of the 3.3.11 patches doesn't indicate anything that would fix the reported problems. In the dim recesses of my memory, I recall similar unexplainable corruptions and core dumps when using 3.3.11. -Barry

On Tue, 18 Aug 2003, Barry Warsaw wrote:
Before the 2.3 release, I was testing all of 3.3.11, 4.0.14 and 4.1.25 builds of bsddb3 on FreeBSD. The 3.3.11 build always coredumps in the same place in regression test, while both 4.0.14 & 4.1.25 complete the test. I didn't have time to pursue the problem to see whether it was the FreeBSD build of DB 3.3.11 was the problem. I also forgot to file a bug report about the fact that the bsddb3 regression test doesn't clean out the DB environment it uses before getting down to business - a crash leaving environment info around (which is useful for debugging) screws a subsequent test run. Regards, Andrew. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia

"Tim Peters" <tim@zope.com> writes:
It's not clear to me that the problems are severe. One would have to analyse the problems (in particular, the crashes). Perhaps _bsddb is using the underlying API in a not-really-supported fashion... For example, strange things used to happen if you close the database while still having cursors around. It might be that the observed crashes have similar causes, and that more recent versions of Sleepycat got more robust against misuse. Then, we could fix the problems by making _bsddb more robust in itself. Regards, Martin

Tim> I don't have time to dig into this, but assuming the report is Tim> correct, how can we "encourage" Unix weenies to use db-4.1.25? (On Tim> Windows, db-4.1.25 is shipped with the installer.) If the problems Tim> with older versions are so severe, maybe the Python wrapper should Tim> do a version check and refuse to run if it finds an old version? I'm using it just fine with 3.3.11 on Mac OS X. I'm not using pop3proxy though, just hammiefilter (for both scoring and incremental training) or hammie (for training from scratch). According to the pybsddb project's README.txt file (updated five weeks ago): This wrapper should be compatible with BerkeleyDB releases going back to 3.1.17 up to and including DB 4.1.25. I'm not saying the poster is wrong, just that his problem is not trivially confirmed. Skip

[Skip Montanaro]
Berkeley version numbers seem partially insane, though: if you have 3.3.11, that doesn't tell us whether you've installed either, neither, or both of the available Sleepycat patches *to* 3.3.11. I don't know whether those patches fix potential corruption problems in 3.3.11, but IIRC Sleepycat patches don't bump the version number regardless.

Tim> Berkeley version numbers seem partially insane, though: if you have Tim> 3.3.11, that doesn't tell us whether you've installed either, Tim> neither, or both of the available Sleepycat patches *to* 3.3.11. I Tim> don't know whether those patches fix potential corruption problems Tim> in 3.3.11, but IIRC Sleepycat patches don't bump the version number Tim> regardless. That's a good question. There is a patch file in the /sw/fink directory which patches several files in the distribution: db-3.3.11/build_vxworks/db.h Thu Dec 20 15:03:34 2001 db-3.3.11/db185/db185.c Thu Dec 20 15:03:30 2001 db-3.3.11/db185/db185_int.in Thu Dec 20 15:03:30 2001 db-3.3.11/dist/configure Thu Dec 20 15:04:25 2001 db-3.3.11/include_auto/db185_ext.in Thu Dec 20 15:03:30 2001 db-3.3.11/include_auto/db185_uext.in Thu Dec 20 15:03:30 2001 db-3.3.11/include/mutex.h Fri Oct 25 21:57:35 2002 I think the dates are patch dates. Still, there's nothing explicit - like a README file - which says, "we installed the foo and bar patches. Comparing the above files with http://www.sleepycat.com/update/3.3.11/patch.3.3.11.html suggests that I have both applied. Neither looks like it fixes any runtime bugs which we'd encounter. One's to fix a compile problem. The other's specific to the vxworks platform. Skip

On Mon, 2003-08-18 at 12:36, Tim Peters wrote:
Correct. That ain't the way Sleepcat does things. :)
A quick scan of the 3.3.11 patches doesn't indicate anything that would fix the reported problems. In the dim recesses of my memory, I recall similar unexplainable corruptions and core dumps when using 3.3.11. -Barry

On Tue, 18 Aug 2003, Barry Warsaw wrote:
Before the 2.3 release, I was testing all of 3.3.11, 4.0.14 and 4.1.25 builds of bsddb3 on FreeBSD. The 3.3.11 build always coredumps in the same place in regression test, while both 4.0.14 & 4.1.25 complete the test. I didn't have time to pursue the problem to see whether it was the FreeBSD build of DB 3.3.11 was the problem. I also forgot to file a bug report about the fact that the bsddb3 regression test doesn't clean out the DB environment it uses before getting down to business - a crash leaving environment info around (which is useful for debugging) screws a subsequent test run. Regards, Andrew. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia

"Tim Peters" <tim@zope.com> writes:
It's not clear to me that the problems are severe. One would have to analyse the problems (in particular, the crashes). Perhaps _bsddb is using the underlying API in a not-really-supported fashion... For example, strange things used to happen if you close the database while still having cursors around. It might be that the observed crashes have similar causes, and that more recent versions of Sleepycat got more robust against misuse. Then, we could fix the problems by making _bsddb more robust in itself. Regards, Martin
participants (5)
-
Andrew MacIntyre
-
Barry Warsaw
-
martin@v.loewis.de
-
Skip Montanaro
-
Tim Peters