Debate about Mailman on BytesForAll

Hello from India:
I am a Free Software enthusiast and co-founder of BytesForAll.org
Recently, there was an interesting debate taking place on our discussion list, about Mailman 2.0 and future plans. Many are users of Mailman, and I would request those developers with the inclination to kindly check on the same at
http://groups.yahoo.com/group/bytesforall_readers
Regards, FN
Frederick Noronha (FN) Near Convent, SALIGAO 403511 GOA India Freelance Journalist Tel: +91-832-2409490 MOBILE: 9822122436 fred at bytesforall.org fredericknoronha at vsnl.net http://fn.swiki.net (FN's swiki) http://www.ilug-goa.tk (GNULinux Users Group Goa)

Frederick Noronha (FN) wrote:
Of course, from my point of view, the discussion is really about Mailman 3.0 and future plans. ;-)
To give context (seeing how I originally posted to bytes4all, not to mailman-developers), we (Bellanet and other orgs) are working on a project which is essentially an Open Sourcification of a service we run called Dgroups (www.dgroups.org). Dgroups is similar in many ways to YahooGroups, but without the noisy ads. Unfortunately, it is based on proprietary components - Lyris/MS SQL/ColdFusion.
Many of you on this list might recall discussions we had last year about the SQL-ability of Mailman. Many of you know that I have been lobbying for this for a very, very long time. I was quite happy when Barry started warming up to this in late 2003, early 2004. And we really made some headway at the developers sprint in March, as it became clear that Maki, Dale, myself and Barry were all very keen on a Mailman 3 that would support SQL data storage. Making Mailman 3 much more modular was certainly one of the other many goals.
The debate happening on bytes4all seems to be whether or not pushing for a SQL-able Mailman (ala Mailman 3) is the way to go or whether it is better to try to hack something together with Mailman 2 now, despite its problems: pickle file storage, Pipermail archives, one-to-one list/member relationship, etc. I say "nay" to Mailman 2 and instead support the advancement of Mailman 3. Our organization, along with another funder, has earmarked significant funds for the Mailman 3 effort and look forward to sicing a few developers on Barry!
So, there's a bit of context for those of you who are interested in the "debate."
Best, Kevin

On Thu, 2004-08-26 at 14:38, Kevin McCann wrote:
Of course, from my point of view, the discussion is really about Mailman 3.0 and future plans. ;-)
And I must apologize for being so totally incommunicado these last few weeks. I've been completely absorbed by work issues. Things should calm down soon and I plan on spending a lot of time on Mailman 3 this fall.
-Barry

On Thu, 2004-08-26 at 15:00, Kevin McCann wrote:
While we're at it, let me mention in passing that I've been doing a lot of work lately with SQLite. I've been thinking that, what with all the problems associated with BerkeleyDB, it might not make sense to switch to SQLite as the embedded, default database for MM3. I think it would allow us to re-use a lot of the code, since it would be mostly DBAPI 2 compatible.
Thoughts? -Barry

On Thu, 26 Aug 2004, Barry Warsaw wrote:
The biggest problem with BerkeleyDB is that it REQUIRES that the file system support memory mapping the files. This means that you cannot guarantee correctness if these files are located on an NFS mount.
http://www.sleepycat.com/docs/ref/env/remote.html
-Dale

At 4:10 PM -0400 2004-08-26, Dale Newfield wrote:
True enough. That's a known issue with BerkeleyDB.
But why would you be putting any of this stuff on NFS anyway?
And how would you deal with all the file locking issues? And cross-platform issues? I've been doing NFS for a very long time, and I have yet to see a mail-related environment where NFS is a good choice or works well. Given the sorts of things we're talking about doing, I can't imagine that NFS could possibly be a good solution.
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
SAGE member since 1995. See <http://www.sage.org/> for more info.

On Thu, 26 Aug 2004, Brad Knowles wrote:
Given the sorts of things we're talking about doing, I can't imagine that NFS could possibly be a good solution.
I don't contest that conclusion. I do worry, though, that mailman may often times get used by people that don't know enough to realize that. While it's reasonable to expect that deployment to run slowly, I don't think it's reasonable to expect that deployment to randomly break.
Do we have "DON'T use on AFS/NFS" warnings in prominent enough places to prevent that type of deployment from happening?
-Dale

At 4:47 PM -0400 2004-08-26, Dale Newfield wrote:
Do we have "DON'T use on AFS/NFS" warnings in prominent enough places to prevent that type of deployment from happening?
I must confess that I didn't know that we were making any use of
BerkeleyDB. However, I would have recommended that we use it as an alternative to the Python pickle format, if we weren't already using it.
Except for the mmap()/NFS issue, IMO BerkeleyDB is the fastest,
most robust, open source database solution around. No, it's not SQL. But it is the key technology differentiator between MySQL and MaxSQL, and is what allows MaxSQL to finally satisfy all of the ACID criteria.
But it's not SQL.
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
SAGE member since 1995. See <http://www.sage.org/> for more info.

On Thu, 2004-08-26 at 16:53, Brad Knowles wrote:
Only in MM3, with experimental, unsupported MemberAPI backend for MM2. Definitely nothing official.
It is a requirement that a BerkeleyDB backend be possible for MM3.
However, it is not a requirement that such a backend be the default, and highest possible performance from the default configuration is not as important to me as simplicity of use. So far, SQLite wins on that count. All this would mean is that I would be personally responsible for the SQLite backend. I would do everything possible to help facilitate community contribution of a BerkeleyDB backend, and I would include it in the standard distro as long as there were volunteers to maintain it.
-Barry

On Thursday, Aug 26, 2004, at 16:47 US/Eastern, Dale Newfield wrote:
I would say no. From INSTALL:
- If you plan on running your MTA and web server on different machines, sharing Mailman installations via NFS, be sure that the clocks on those two machines are synchronized closely.
Not that Mailman 3 has an install document yet, but there is plenty of stuff in 2.x that seems geared toward this kind of use. I can recall at least one instance of hearing from someone on the list using NFS to split up MTA and webserver.
--Robby

Brad Knowles wrote:
I agree, Brad.
Barry has redirected some of the discussion over to mailman3-dev@python.org. I'll crosspost for this message, but maybe we should move over to mm3-dev? In any event, further to NFS, it seems SQLite has issues, too. From the concurrency article I pointed to:
"SQLite uses POSIX advisory locks to implement locking on Unix. On windows it uses the LockFile(), LockFileEx(), and UnlockFile() system calls. SQLite assumes that these system calls all work as advertised. If that is not the case, then database corruption can result. One should note that POSIX advisory locking is known to be buggy or even unimplemented on many NFS implementations (including recent versions of Mac OS X) and that there are reports of locking problems for network filesystems under windows. Your best defense is to not use SQLite for files on a network filesystem."
- Kevin

At 4:57 PM -0400 2004-08-26, Kevin McCann wrote:
I'm not really a developer, but I'll subscribe to it anyway since
I do feel there might be some architectural issues on which I might have some useful advice.
Your best defense is to not use SQLite for files on a network filesystem."
Fair enough. Are we looking for something where use on a network
filesystem is an issue?
Or do we care more about other issues, such as performance (where
I think BerkeleyDB would probably win), or interface methods/language (where SQL might be preferred)?
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
SAGE member since 1995. See <http://www.sage.org/> for more info.

Barry Warsaw wrote:
Might not make sense? Or might make sense? I have played with SQLite a bit but not extensively, so I'm not aware of any major drawbacks. I do recall at the sprint lunch that SQLite was suggested by that fellow who was doing the email sprinting (can't remember his name, but he seemd to be a sharp cookie). SQLite, if it's solid, is attractive to me because it's built into PHP 5. Thinking down the road, it starts to look pretty good.
- Kevin

At 4:01 PM -0400 2004-08-26, Barry Warsaw wrote:
Problems with BerkeleyDB? Can you elaborate?
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
SAGE member since 1995. See <http://www.sage.org/> for more info.

On Thu, 2004-08-26 at 16:14, Brad Knowles wrote:
Problems with BerkeleyDB? Can you elaborate?
Briefly:
Installation and compatibility problems. Lots of hoops to go through to get compatible versions of libdb, the bsddb module, etc. all working together.
Verbosity/complexity/uniqueness of API. Okay, SQL is icky but at least it's a shared common ickiness. ;) The actual DBAPI2 interface is pretty simple and I don't think Mailman's SQL needs are much more complex than some inserts and selects with maybe a join here and there. bsddb's API is huge and complicated.
Uncertainty of stability. IME Berkeley has been pretty stable, but I still hear lots of tails, gossip, voodoo, whatever about stability of the underlying database.
For an embedded database, it still has a fairly high administrative overhead. My primary goal for default database is that it be drop dead simple to administer. BerkeleyDB in transactional mode is anything but. It's also complex to configure and tune.
On the plus side, most *nix OSes either come with libdb or can get them pretty easily, and Python 2.3/2.4 (yes, MM3 will likely require Python 2.4 when available) comes with support out of the box. OTOH, I've always found it extremely simple to install SQLite.
-Barry

On Thu, 26 Aug 2004, Kevin McCann wrote:
|> Barry Warsaw wrote:
|>
|> >On Thu, 2004-08-26 at 14:38, Kevin McCann wrote:
|> >
|> >>Of course, from my point of view, the discussion is really about Mailman
|> >>3.0 and future plans. ;-)
|> >
|> >And I must apologize for being so totally incommunicado these last few
|> >weeks. We have a very major release due at the end of next week so I've
|> >been completely absorbed by that. Things should calm down after that
|> >and I plan on spending a lot of time on Mailman 3 this fall.
|>
|> That's great news, Barry. As you know, we're completely behind you and
|> will do whatever we can from our end (people and money) to help a
|> SQL-able MM3 become a reality. We'll be in touch.
<call-from-the-coke-stand> And don't forget LDAP support, either! </call-from-the-coke-stand>
Michael
|> _______________________________________________ |> Mailman-Developers mailing list |> Mailman-Developers@python.org |> http://mail.python.org/mailman/listinfo/mailman-developers |> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/miranda%40uranus.c...

Frederick Noronha (FN) wrote:
Of course, from my point of view, the discussion is really about Mailman 3.0 and future plans. ;-)
To give context (seeing how I originally posted to bytes4all, not to mailman-developers), we (Bellanet and other orgs) are working on a project which is essentially an Open Sourcification of a service we run called Dgroups (www.dgroups.org). Dgroups is similar in many ways to YahooGroups, but without the noisy ads. Unfortunately, it is based on proprietary components - Lyris/MS SQL/ColdFusion.
Many of you on this list might recall discussions we had last year about the SQL-ability of Mailman. Many of you know that I have been lobbying for this for a very, very long time. I was quite happy when Barry started warming up to this in late 2003, early 2004. And we really made some headway at the developers sprint in March, as it became clear that Maki, Dale, myself and Barry were all very keen on a Mailman 3 that would support SQL data storage. Making Mailman 3 much more modular was certainly one of the other many goals.
The debate happening on bytes4all seems to be whether or not pushing for a SQL-able Mailman (ala Mailman 3) is the way to go or whether it is better to try to hack something together with Mailman 2 now, despite its problems: pickle file storage, Pipermail archives, one-to-one list/member relationship, etc. I say "nay" to Mailman 2 and instead support the advancement of Mailman 3. Our organization, along with another funder, has earmarked significant funds for the Mailman 3 effort and look forward to sicing a few developers on Barry!
So, there's a bit of context for those of you who are interested in the "debate."
Best, Kevin

On Thu, 2004-08-26 at 14:38, Kevin McCann wrote:
Of course, from my point of view, the discussion is really about Mailman 3.0 and future plans. ;-)
And I must apologize for being so totally incommunicado these last few weeks. I've been completely absorbed by work issues. Things should calm down soon and I plan on spending a lot of time on Mailman 3 this fall.
-Barry

On Thu, 2004-08-26 at 15:00, Kevin McCann wrote:
While we're at it, let me mention in passing that I've been doing a lot of work lately with SQLite. I've been thinking that, what with all the problems associated with BerkeleyDB, it might not make sense to switch to SQLite as the embedded, default database for MM3. I think it would allow us to re-use a lot of the code, since it would be mostly DBAPI 2 compatible.
Thoughts? -Barry

On Thu, 26 Aug 2004, Barry Warsaw wrote:
The biggest problem with BerkeleyDB is that it REQUIRES that the file system support memory mapping the files. This means that you cannot guarantee correctness if these files are located on an NFS mount.
http://www.sleepycat.com/docs/ref/env/remote.html
-Dale

At 4:10 PM -0400 2004-08-26, Dale Newfield wrote:
True enough. That's a known issue with BerkeleyDB.
But why would you be putting any of this stuff on NFS anyway?
And how would you deal with all the file locking issues? And cross-platform issues? I've been doing NFS for a very long time, and I have yet to see a mail-related environment where NFS is a good choice or works well. Given the sorts of things we're talking about doing, I can't imagine that NFS could possibly be a good solution.
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
SAGE member since 1995. See <http://www.sage.org/> for more info.

On Thu, 26 Aug 2004, Brad Knowles wrote:
Given the sorts of things we're talking about doing, I can't imagine that NFS could possibly be a good solution.
I don't contest that conclusion. I do worry, though, that mailman may often times get used by people that don't know enough to realize that. While it's reasonable to expect that deployment to run slowly, I don't think it's reasonable to expect that deployment to randomly break.
Do we have "DON'T use on AFS/NFS" warnings in prominent enough places to prevent that type of deployment from happening?
-Dale

At 4:47 PM -0400 2004-08-26, Dale Newfield wrote:
Do we have "DON'T use on AFS/NFS" warnings in prominent enough places to prevent that type of deployment from happening?
I must confess that I didn't know that we were making any use of
BerkeleyDB. However, I would have recommended that we use it as an alternative to the Python pickle format, if we weren't already using it.
Except for the mmap()/NFS issue, IMO BerkeleyDB is the fastest,
most robust, open source database solution around. No, it's not SQL. But it is the key technology differentiator between MySQL and MaxSQL, and is what allows MaxSQL to finally satisfy all of the ACID criteria.
But it's not SQL.
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
SAGE member since 1995. See <http://www.sage.org/> for more info.

On Thu, 2004-08-26 at 16:53, Brad Knowles wrote:
Only in MM3, with experimental, unsupported MemberAPI backend for MM2. Definitely nothing official.
It is a requirement that a BerkeleyDB backend be possible for MM3.
However, it is not a requirement that such a backend be the default, and highest possible performance from the default configuration is not as important to me as simplicity of use. So far, SQLite wins on that count. All this would mean is that I would be personally responsible for the SQLite backend. I would do everything possible to help facilitate community contribution of a BerkeleyDB backend, and I would include it in the standard distro as long as there were volunteers to maintain it.
-Barry

On Thursday, Aug 26, 2004, at 16:47 US/Eastern, Dale Newfield wrote:
I would say no. From INSTALL:
- If you plan on running your MTA and web server on different machines, sharing Mailman installations via NFS, be sure that the clocks on those two machines are synchronized closely.
Not that Mailman 3 has an install document yet, but there is plenty of stuff in 2.x that seems geared toward this kind of use. I can recall at least one instance of hearing from someone on the list using NFS to split up MTA and webserver.
--Robby

Brad Knowles wrote:
I agree, Brad.
Barry has redirected some of the discussion over to mailman3-dev@python.org. I'll crosspost for this message, but maybe we should move over to mm3-dev? In any event, further to NFS, it seems SQLite has issues, too. From the concurrency article I pointed to:
"SQLite uses POSIX advisory locks to implement locking on Unix. On windows it uses the LockFile(), LockFileEx(), and UnlockFile() system calls. SQLite assumes that these system calls all work as advertised. If that is not the case, then database corruption can result. One should note that POSIX advisory locking is known to be buggy or even unimplemented on many NFS implementations (including recent versions of Mac OS X) and that there are reports of locking problems for network filesystems under windows. Your best defense is to not use SQLite for files on a network filesystem."
- Kevin

At 4:57 PM -0400 2004-08-26, Kevin McCann wrote:
I'm not really a developer, but I'll subscribe to it anyway since
I do feel there might be some architectural issues on which I might have some useful advice.
Your best defense is to not use SQLite for files on a network filesystem."
Fair enough. Are we looking for something where use on a network
filesystem is an issue?
Or do we care more about other issues, such as performance (where
I think BerkeleyDB would probably win), or interface methods/language (where SQL might be preferred)?
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
SAGE member since 1995. See <http://www.sage.org/> for more info.

Barry Warsaw wrote:
Might not make sense? Or might make sense? I have played with SQLite a bit but not extensively, so I'm not aware of any major drawbacks. I do recall at the sprint lunch that SQLite was suggested by that fellow who was doing the email sprinting (can't remember his name, but he seemd to be a sharp cookie). SQLite, if it's solid, is attractive to me because it's built into PHP 5. Thinking down the road, it starts to look pretty good.
- Kevin

At 4:01 PM -0400 2004-08-26, Barry Warsaw wrote:
Problems with BerkeleyDB? Can you elaborate?
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
SAGE member since 1995. See <http://www.sage.org/> for more info.

On Thu, 2004-08-26 at 16:14, Brad Knowles wrote:
Problems with BerkeleyDB? Can you elaborate?
Briefly:
Installation and compatibility problems. Lots of hoops to go through to get compatible versions of libdb, the bsddb module, etc. all working together.
Verbosity/complexity/uniqueness of API. Okay, SQL is icky but at least it's a shared common ickiness. ;) The actual DBAPI2 interface is pretty simple and I don't think Mailman's SQL needs are much more complex than some inserts and selects with maybe a join here and there. bsddb's API is huge and complicated.
Uncertainty of stability. IME Berkeley has been pretty stable, but I still hear lots of tails, gossip, voodoo, whatever about stability of the underlying database.
For an embedded database, it still has a fairly high administrative overhead. My primary goal for default database is that it be drop dead simple to administer. BerkeleyDB in transactional mode is anything but. It's also complex to configure and tune.
On the plus side, most *nix OSes either come with libdb or can get them pretty easily, and Python 2.3/2.4 (yes, MM3 will likely require Python 2.4 when available) comes with support out of the box. OTOH, I've always found it extremely simple to install SQLite.
-Barry

On Thu, 26 Aug 2004, Kevin McCann wrote:
|> Barry Warsaw wrote:
|>
|> >On Thu, 2004-08-26 at 14:38, Kevin McCann wrote:
|> >
|> >>Of course, from my point of view, the discussion is really about Mailman
|> >>3.0 and future plans. ;-)
|> >
|> >And I must apologize for being so totally incommunicado these last few
|> >weeks. We have a very major release due at the end of next week so I've
|> >been completely absorbed by that. Things should calm down after that
|> >and I plan on spending a lot of time on Mailman 3 this fall.
|>
|> That's great news, Barry. As you know, we're completely behind you and
|> will do whatever we can from our end (people and money) to help a
|> SQL-able MM3 become a reality. We'll be in touch.
<call-from-the-coke-stand> And don't forget LDAP support, either! </call-from-the-coke-stand>
Michael
|> _______________________________________________ |> Mailman-Developers mailing list |> Mailman-Developers@python.org |> http://mail.python.org/mailman/listinfo/mailman-developers |> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/miranda%40uranus.c...
participants (7)
-
Barry Warsaw
-
Brad Knowles
-
Dale Newfield
-
Frederick Noronha (FN)
-
Kevin McCann
-
Michael Chang
-
Robby Griffin