Pickle-induced integration issues
Hi folks,
I manage Helix Community, which makes pretty heavy use of Mailman as integrated into GForge. I'm moderately familiar with Mailman code, and pretty familiar with GForge. Of course, Perl is my native tongue, so I'm equally uncomfortable in both codebases ;-)
Anyway, one thing that my users are constantly clamoring for is features that would come from tighter integration between GForge and Mailman. I've been mulling this over for a while, and it seems that deep integration is limited by the way that most Mailman data is stored in pickles. Since GForge is coded and pretty heavily bound to PHP.
My options for integration between the two seem to be:
- Implement alternate GForge authentication mechanism in Mailman
- Implement alternate, PHP-compatible storage mechanism (e.g. SQL database) for everything currently stored in pickles
- Implement pickle support in PHP
- Implement IPC mechanism (e.g. SOAP, XML-RPC) as a Mailman control daemon, and call that from GForge
I'm wondering if there are recommendations from this group. I'm also wondering if there have been any movements/efforts in any of these directions already.
This is one of many problems that I plan to work on over time, and plan to contribute patches to the extent that they are useful to a larger community. If I'm lucky, I'll get budget to hire developers for this, but for now, I'll be doing this on the cheap.
Thoughts? Rob
Rob Lanphier, Development Support Manager - RealNetworks Helix Community: http://helixcommunity.org Development Support: http://www.realnetworks.com/products/support/devsupport
On Thu, 2004-09-30 at 19:24, Rob Lanphier wrote:
Anyway, one thing that my users are constantly clamoring for is features that would come from tighter integration between GForge and Mailman. I've been mulling this over for a while, and it seems that deep integration is limited by the way that most Mailman data is stored in pickles. Since GForge is coded and pretty heavily bound to PHP.
Do you care about list data, user data, or both?
If it's user data, then you can mostly write a backend for the MemberAdaptor interface that shoves the data into whatever GForge uses. If it's list data you want to integrate, that's harder <wink>.
I'm wondering if there are recommendations from this group. I'm also wondering if there have been any movements/efforts in any of these directions already.
This is what MM3 is all about. Every piece of data will be stored in a backend through an interface, including messages, user data, and list data. MM3 will come with some persistent database on the backend by default, but I hope/envision/expect that there will be backends for other databases (e.g. MySQL, LDAP, etc.).
-Barry
At 4:24 PM -0700 2004-09-30, Rob Lanphier wrote:
Anyway, one thing that my users are constantly clamoring for is features that would come from tighter integration between GForge and Mailman. I've been mulling this over for a while, and it seems that deep integration is limited by the way that most Mailman data is stored in pickles. Since GForge is coded and pretty heavily bound to PHP.
See <http://www.list.org/devs.html>. In particular:
Mailman uses an inefficient persistency system based on
Python pickles, and every mailing list has its own pickled
state. This has several disadvantages, including poor
scalability, difficulty in doing cross-mailing list
operations, and the virtual host limitation on list names.
Mailman 3.0 will use a real database, perhaps based on ZODB
or BerkeleyDB. Again, the goal is to generalize the
interface to the backend database so that others can be
used, but choose one and ship it by default.I'm wondering if there are recommendations from this group. I'm also wondering if there have been any movements/efforts in any of these directions already.
This is one of many problems that I plan to work on over time, and plan to contribute patches to the extent that they are useful to a larger community. If I'm lucky, I'll get budget to hire developers for this, but for now, I'll be doing this on the cheap.
You could funnel your work into Mailman3, and if you could get some budget to make that happen, then the project will be able to be shipped that much faster.
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755SAGE member since 1995. See <http://www.sage.org/> for more info.
On Fri, 2004-10-01 at 02:34 +0200, Brad Knowles wrote:
See <http://www.list.org/devs.html>. In particular:
Mailman uses an inefficient persistency system based on Python pickles, and every mailing list has its own pickled state. This has several disadvantages, including poor scalability, difficulty in doing cross-mailing list operations, and the virtual host limitation on list names. Mailman 3.0 will use a real database, perhaps based on ZODB or BerkeleyDB. Again, the goal is to generalize the interface to the backend database so that others can be used, but choose one and ship it by default.
Ah, very cool. Thanks Brad and Barry for your quick answers. I'll look through the MM3 plans, and hopefully I'll be able to contribute a little.
I'm somewhat tempted to knock something out quick and dirty based on the MemberAdapter work, but by the time I get around to doing something like that, MM3 will probably be a lot closer to reality. I'll also post something on helixcommunity.org. Since many of the people complaining are competent developers, maybe I can enlist them ;-)
Rob
Rob Lanphier, Development Support Manager - RealNetworks Helix Community: http://helixcommunity.org Development Support: http://www.realnetworks.com/products/support/devsupport
participants (3)
-
Barry Warsaw -
Brad Knowles -
Rob Lanphier