Barry Warsaw wrote:

On Thu, 2004-08-26 at 14:38, Kevin McCann wrote:

...

Of course, from my point of view, the discussion is really about Mailman 3.0 and future plans. ;-)

And I must apologize for being so totally incommunicado these last few weeks. I've been completely absorbed by work issues. Things should calm down soon and I plan on spending a lot of time on Mailman 3 this fall.

That's great news, Barry. As you know, we're completely behind you and will do whatever we can from our end (people and money) to help a SQL-able MM3 become a reality. We'll be in touch.

Cheers, Kevin

On Thu, 26 Aug 2004, Barry Warsaw wrote:

I've been thinking that, what with all the problems associated with BerkeleyDB, it might not make sense to switch to SQLite as the embedded, default database for MM3.

The biggest problem with BerkeleyDB is that it REQUIRES that the file system support memory mapping the files. This means that you cannot guarantee correctness if these files are located on an NFS mount.

http://www.sleepycat.com/docs/ref/env/remote.html

-Dale

On Thu, 26 Aug 2004, Brad Knowles wrote:

Given the sorts of things we're talking about doing, I can't imagine that NFS could possibly be a good solution.

I don't contest that conclusion. I do worry, though, that mailman may often times get used by people that don't know enough to realize that. While it's reasonable to expect that deployment to run slowly, I don't think it's reasonable to expect that deployment to randomly break.

Do we have "DON'T use on AFS/NFS" warnings in prominent enough places to prevent that type of deployment from happening?

-Dale

On Thu, 2004-08-26 at 16:53, Brad Knowles wrote:

I must confess that I didn't know that we were making any use of BerkeleyDB. However, I would have recommended that we use it as an alternative to the Python pickle format, if we weren't already using it.

Only in MM3, with experimental, unsupported MemberAPI backend for MM2. Definitely nothing official.

Except for the mmap()/NFS issue, IMO BerkeleyDB is the fastest, most robust, open source database solution around. No, it's not SQL. But it is the key technology differentiator between MySQL and MaxSQL, and is what allows MaxSQL to finally satisfy all of the ACID criteria.

But it's not SQL.

It is a requirement that a BerkeleyDB backend be possible for MM3.

However, it is not a requirement that such a backend be the default, and highest possible performance from the default configuration is not as important to me as simplicity of use. So far, SQLite wins on that count. All this would mean is that I would be personally responsible for the SQLite backend. I would do everything possible to help facilitate community contribution of a BerkeleyDB backend, and I would include it in the standard distro as long as there were volunteers to maintain it.

-Barry

Brad Knowles wrote:

At 4:10 PM -0400 2004-08-26, Dale Newfield wrote:

...

The biggest problem with BerkeleyDB is that it REQUIRES that the file system support memory mapping the files. This means that you cannot guarantee correctness if these files are located on an NFS mount.

True enough.  That's a known issue with BerkeleyDB.

But why would you be putting any of this stuff on NFS anyway? And 

how would you deal with all the file locking issues? And cross-platform issues? I've been doing NFS for a very long time, and I have yet to see a mail-related environment where NFS is a good choice or works well. Given the sorts of things we're talking about doing, I can't imagine that NFS could possibly be a good solution.

I agree, Brad.

Barry has redirected some of the discussion over to mailman3-dev@python.org. I'll crosspost for this message, but maybe we should move over to mm3-dev? In any event, further to NFS, it seems SQLite has issues, too. From the concurrency article I pointed to:

"SQLite uses POSIX advisory locks to implement locking on Unix. On windows it uses the LockFile(), LockFileEx(), and UnlockFile() system calls. SQLite assumes that these system calls all work as advertised. If that is not the case, then database corruption can result. One should note that POSIX advisory locking is known to be buggy or even unimplemented on many NFS implementations (including recent versions of Mac OS X) and that there are reports of locking problems for network filesystems under windows. Your best defense is to not use SQLite for files on a network filesystem."

• Kevin

Barry Warsaw wrote:

On Thu, 2004-08-26 at 15:00, Kevin McCann wrote:

...

That's great news, Barry. As you know, we're completely behind you and will do whatever we can from our end (people and money) to help a SQL-able MM3 become a reality. We'll be in touch.

While we're at it, let me mention in passing that I've been doing a lot of work lately with SQLite. I've been thinking that, what with all the problems associated with BerkeleyDB, it might not make sense to switch to SQLite as the embedded, default database for MM3. I think it would allow us to re-use a lot of the code, since it would be mostly DBAPI 2 compatible.

Might not make sense? Or might make sense? I have played with SQLite a bit but not extensively, so I'm not aware of any major drawbacks. I do recall at the sprint lunch that SQLite was suggested by that fellow who was doing the email sprinting (can't remember his name, but he seemd to be a sharp cookie). SQLite, if it's solid, is attractive to me because it's built into PHP 5. Thinking down the road, it starts to look pretty good.

• Kevin

On Thu, 2004-08-26 at 16:14, Brad Knowles wrote:

Problems with BerkeleyDB? Can you elaborate?

Briefly:

• Installation and compatibility problems. Lots of hoops to go through to get compatible versions of libdb, the bsddb module, etc. all working together.

• Verbosity/complexity/uniqueness of API. Okay, SQL is icky but at least it's a shared common ickiness. ;) The actual DBAPI2 interface is pretty simple and I don't think Mailman's SQL needs are much more complex than some inserts and selects with maybe a join here and there. bsddb's API is huge and complicated.

• Uncertainty of stability. IME Berkeley has been pretty stable, but I still hear lots of tails, gossip, voodoo, whatever about stability of the underlying database.

• For an embedded database, it still has a fairly high administrative overhead. My primary goal for default database is that it be drop dead simple to administer. BerkeleyDB in transactional mode is anything but. It's also complex to configure and tune.

On the plus side, most *nix OSes either come with libdb or can get them pretty easily, and Python 2.3/2.4 (yes, MM3 will likely require Python 2.4 when available) comes with support out of the box. OTOH, I've always found it extremely simple to install SQLite.

-Barry