[Mailman-Developers] Re: [Mailman-Users] optimizing mail delivery

[J C Lawrence]

On Wed, 17 Nov 1999 20:37:58 -0500 (EST) 
Barry A Warsaw <bwarsaw@cnri.reston.va.us> wrote:

> So do you think the Mailman way is better or worse?  I'm curious
> because I'm trying to decide whether I should port Mailman 1.0's
> bulk mailer code to the new message pipeline.

MailMan's current code is definitely in the "good enough"
category.  

> In the current 1.2 code base (available via the anonCVS), I
> os.popen() sendmail[1] passing the entire recipient list on the
> command line, then I pipe the message text to stdin and let the
> MTA do the rest.  There's one complication; if the length of the
> recipients list is greater than a certain length (current 3000), I
> chop it up into multiple popens, but I don't do any sorting of the
> recipient list.

My view is that appropriate and efficient handling of mail for
delivery is the domain of the MTA, and should not be the domain of
MUA's (in which camp MailMan sorta falls).  As such domain sorting
is pleasant, it really only acts to further bolster technically
lagging mail servers which are in need of new life anyways.

> The advantage I see of this is that sendmail can do it's thing
> asychronously, without keeping the list object locked the entire
> time.  The disadvantage is that Mailman is only aware of delivery
> problems if the delivery bounces.

Which of course requires a local MTA, which the current design
doesn't.

What actually needs to happen (presuming my current understanding of
the source is correct) is that the abstraction of MailMan's internal
mail queue needs to be finished.  Then, a mail broadcast attempt
would only stuff messages (cheaply) into the MailMan queue
mechanism, and then return, unlocking the list objects.  The queue
object of course can be lock free (lockless DB's aren't difficult)
and you then merely need to run a could of MailMan queue runners to
pipe it to the MTA.

One the messages are stuffed, you can then fork a queue runner,
which, if there are already sufficient queue runners immediately
dies.  If more queue runners are needed however, it proceeds to grab 
messages and stuff them at the MTA in the normal fashion over SMTP.

Note:  If you do this adding VERP support becomes a doddle.

> I'm leary though of stepping on too much of the MTA's toes -- a
> good MTA should just do the right thing.

This is perhaps the best point.  Spending time to further bolster
fading technologies when better services are freely available hardly 
seems worth it.  

> Another alternative, which would be less work and delegates all
> delivery to the MTA, is to just pump all the recips to the local
> smtpd via smtplib.py.  The advantage here is that again we're MTA
> independent, but the disadvantage is that Mailman's delivery is
> synchronous with the smtpd.  We'd have to be very sure to unlock
> the list object during this transaction (but watching out for race
> conditions, locking again if failure status's are handled
> directly, etc.)  More code, more opportunity for bugs.

No no no no no no.  MailMan oeprates asynchronously of the MTA, and
because as a list server it generates unusual loads in itself, it
cannot be subject to the perfoamcen vagaries of the MTA.

Consider the case I used to commonly run into:

  Message arrives and is delivered to MailMan.

  MailMan explodes that message into a couple thousand more messages 
  (100K+ subscribers).

  MailMan attempts to hand off messages to MTA.

  MTA refuses connections as system load is too high.

  MailMan bitches.

  Meanwhile another message arrives at MailMan to be exploded.

> Comments?

Nope!

-- 
J C Lawrence                              Internet: claw@kanga.nu
----------(*)                            Internet: coder@kanga.nu
...Honorary Member of Clan McFud -- Teamer's Avenging Monolith...


Return-Path: <owner-mailman-developers@python.org>
Delivered-To: mailman-developers@dinsdale.python.org
Received: from python.org (parrot.python.org [132.151.1.90])
	by dinsdale.python.org (Postfix) with ESMTP id 568F31CD3E
	for <mailman-developers@dinsdale.python.org>; Wed, 17 Nov 1999 21:14:51 -0500 (EST)
Received: from caesar.elte.hu (caesar.elte.hu [157.181.42.42])
	by python.org (8.9.1a/8.9.1) with ESMTP id VAA20080;
	Wed, 17 Nov 1999 21:14:48 -0500 (EST)
Received: from localhost (gorgo@localhost)
	by caesar.elte.hu (8.9.0/8.9.0) with SMTP id DAA50042;
	Thu, 18 Nov 1999 03:14:16 +0100
Date: Thu, 18 Nov 1999 03:14:15 +0100 (MET)
From: Gergely Madarasz <gorgo@caesar.elte.hu>
X-Sender: gorgo@caesar.power.elte.hu
To: "Barry A. Warsaw" <bwarsaw@cnri.reston.va.us>
Cc: Paul Tomblin <ptomblin@xcski.com>, J C Lawrence <claw@cp.net>,
	Klaubert Herr da Silveira <klaubert@bcb.gov.br>,
	mailman-users@python.org, mailman-developers@python.org
Subject: Re: [Mailman-Developers] Re: [Mailman-Users] optimizing mail delivery
In-Reply-To: <14387.22646.76313.150613@anthem.cnri.reston.va.us>
Message-ID: <Pine.A32.3.96.991118030855.28574H-100000@caesar.power.elte.hu>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: mailman-developers-admin@python.org
Errors-To: mailman-developers-admin@python.org
X-BeenThere: mailman-developers@python.org
X-Mailman-Version: 1.1+
Precedence: bulk
List-Id: Mailman mailing list developers <mailman-developers.python.org>

On Wed, 17 Nov 1999, Barry A. Warsaw wrote:

> So do you think the Mailman way is better or worse?  I'm curious
> because I'm trying to decide whether I should port Mailman 1.0's bulk
> mailer code to the new message pipeline.

Actually I kinda like the mailman bulk mailer. Before I switched from
majordomo, I used an external bulkmail package (called from aliases, what
mailman doesn't do on delivery so I can't use it with mailman) to help out
sendmail. Changing MTA's on a high traffic site can be a real PITA, I
wouldn't like it if I had to change it because mailman can't help it out
anymore :( So I think it would be nice if you left it as an option.

-- 
Madarasz Gergely           gorgo@caesar.elte.hu         gorgo@linux.rulez.org
      It's practically impossible to look at a penguin and feel angry.
          Egy pingvinre gyakorlatilag lehetetlen haragosan nezni.
                    HuLUG: http://mlf.linux.rulez.org/