[Mailman-Developers] LTMP for incoming mail

[Barry Warsaw]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 28, 2006, at 9:21 AM, Brad Knowles wrote:

> At 8:12 AM -0400 9/28/06, Barry Warsaw wrote:
>
> Does it have to be GPL?  Is a Berkeley-type license not okay?

GPL would be best, but Berkeley is probably okay.  We'd probably want  
to get confirmation of that from the FSF.  The key thing is that it  
has to be compatible with the GPL (and the Python Software LIcense --  
see below) so that we can combine the whole kit and kaboodle.

> Checking the source for sendmail 8.13.8, I find that there is an  
> official part of the package which includes the LDA "mail.local",  
> which is LMTP-capable, among other things.  It can also do user  
> mailbox hashing, based on the username.  You can either hash  
> directly to a path like /var/mail/u/s/user or use an MD5 hash of  
> the username in a base64 representation (changing "/" to "_"), and  
> you can control how many levels of hashing are to be used.
>
> Seems to me like this would be a pretty obvious candidate.

So a sketch of the architecture would be to embed Python via its C  
API into mail.local, adding callbacks at each point in the LMTP  
protocol.  From there, we'd write the rules in Python so that they'd  
do the message parsing, sanity checking, etc, returning status codes  
which mail.local would use to respond to the LMTP command.  There has  
to be that Python hook, otherwise you can't get to Mailman's data  
structures.

>>        Postfix has an lmtp server, but it seems fairly heavyweight
>> (being tied into the smtp server) and it's not clear to me we could
>> combine our GPL code with Postfix's license.
>
> Please check out the sendmail mail.local stuff and tell me if this  
> is a better alternative.  If you need a different license, please  
> let me know -- I've known Eric for many years (since way before the  
> company existed).  While I won't make any guarantees, I will say  
> that if we need a different license, I imagine that I can get a  
> more sympathetic ear than you might otherwise be able to find.

Cool, that's good to know.

>
>>  ISTM that the trade-off then is rolling our own LMTP server vs.  
>> doing
>>  maildir delivery.  Are we confident that we can implement a high
>>  performance enough server that would give us better throughput than
>>  maildir would?  In Python?
>
> Dunno about doing it in Python, but I will say that going to  
> Maildir as an additional queue-on-disk mechanism on top of  
> everything else we're already doing seems to be a big step backward  
> in terms of potential performance issues and I don't really see any  
> significant positive benefit.

I don't think it's an additional queue-on-disk mechanism, certainly  
in comparison to what we're doing today.  In fact, thinking about it  
more, a maildir approach would be better than what we have today not  
only because it eliminates the extra process fork/exec, but because  
we could segregate queue/in files into per-list subdirectories.  That  
way, you're not dumping all message destined for Mailman into one  
directory.  Not as good as directory hashing, but better than what we  
have today.

I'll grant you that LMTP delivery has the potential to be the most  
efficient mechanism by which messages get from the MTA into Mailman.   
But it's certainly more work and more complicated than maildir;  will  
you grant that maildir is better than what we have today?  Think of  
it as a waystation on the road to the ultimate uber-performing list  
server. :)

Let me just say that ideally, I think LMTP would be a great way to  
go.  It's not my top priority though.  I'm looking for ways to get  
more developers involved in the project, and this seems like a  
perfect thing for someone seeking Mailman fame and fortune <wink>.   
So, anyone care to take the challenge?

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRRvXmHEjvBPtnXfVAQLdQAP8DRQdxi/rnHPIB4I+q3xReOpq2yeW7Wsp
y+1AerklGjAhy9+NHOAV2h28rj9YBaS9cp0euJuSWTAoceJfeqBvYM6voL/mfs0I
elTntMROq5diyzytfTxBz9qMDM+QAfuutcp7nuxlPCuYv7CskeAySZll7+v0P8eD
1unF56C5Dns=
=R86i
-----END PGP SIGNATURE-----