[Mailman-Users] Mailman performance / sends per hour
brad.knowles at skynet.be
Sat Jul 26 20:28:53 CEST 2003
At 9:43 AM -0400 2003/07/26, Jon Carnes wrote:
> Here is the section from the Release Notes that is pertinent to our
> "pissing contest":
> Add parallel queue runner code. Allows multiple queue runners per work
> group (one or more queues in a multi-queue environment
> collected together) to process the same work list at the
> same time.
>> Actually, what postfix does is handle multiple copies of the
>> message being transmitted to separate domains in parallel. This
>> helps ensure that fast sites further down the list don't get hung up
>> by slower sites that come earlier. However, there is a limit to this
>> parallelism. Mailman could help this process by tracking the average
>> delivery time per recipient, and then sorting the recipient list when
>> handing the messages to postfix -- fastest first, slowest last.
> Actually Brad, it looks like your knowledge of Sendmail is rather
> dated. Sendmail has been doing this since 2001.
You just said "this". You didn't say what "this" you were
talking about. I (naturally) assumed you meant to reference the most
recent part of the paragraph you were responding to, which was the
issue of the recipient sorting feature based on previous average
delivery times -- a feature that neither sendmail nor postfix has,
and which would be a notable improvement for mailman.
If you had wanted to refer to the issue of having multiple queue
runners going in parallel, you should have trimmed the paragraph at
the appropriate point, or you should have been more specific.
Either way, I would then have mentioned that this feature did not
work correctly when originally added to the system in version 8.10
(dated 2000/03/01) with "multiple queue directories":
Support multiple queue directories. To use multiple queues, supply
a QueueDirectory option value ending with an asterisk. For
example, /var/spool/mqueue/q* will use all of the
directories or symbolic links to directories beginning with
'q' in /var/spool/mqueue as queue directories. Keep in
mind, the queue directory structure should not be changed
while sendmail is running. Queue runs create a separate
process for running each queue unless the verbose flag is
given on a non-daemon queue run. New items are randomly
assigned to a queue. Contributed by Exactis.com, Inc.
The problem is that there was no way you could keep sendmail from
firing off a queue runner per multiple queue, and if you wanted to
have a large directory hierarchy of multiple levels of queues with
lots of queues at each level (a la postfix, but better), this could
cause thousands or millions of queue runners to be started --
obviously, a highly undesirable result. This caused no end of
problems for us at a previous employer, end I ended up turning off
all of sendmail's control over multiple queues and managing them
I was also an early adopter of running multiple queue runners in
the same directory, some with "QueueSortOrder=host", some with
"QueueSortOrder=time", some with "QueueSortOrder=random", etc... so
as to try to clear the queue as best as possible but with as little
lock contention as possible. I was doing this from ~1995.
Besides, with limiting the number of recipients per envelope
(either within the MLM or within the MTA) and then allowing multiple
processes to handle each chunk separately, you get pretty much the
> Re-read my statement above and maybe look into the old archives of this
> list. You will discover that due to Mailman's current design (well
> really a limitation of Python) large lists can be very slow to maintain
> and process. the solution is to move the MAILMAN (not sendmail you
> oaf!) list databases into a RAM drive. And (duh!) a battery backed up
> RAM drive sure would be best.
List databases or the MTA mqueues, it doesn't matter. The same
requirement for reliability is there. If you choose to be so casual
with the management of your mailing lists, it sure makes me wonder
about your competence in other areas.
Brad Knowles, <brad.knowles at skynet.be>
"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
-Benjamin Franklin, Historical Review of Pennsylvania.
GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)
More information about the Mailman-Users