[Mailman-Users] Mailman / Sendmail horribly slow performance..suggestions please?!?

[Richard Barrett]

On Monday, August 25, 2003, at 02:28  pm, Andrew Falgout wrote:

> The error log is at 0 bytes.  I have not overridden the Defaults.py  
> which
> shows Defaults.py:MTA_ALIASES_STYLE = 'sendmail'.  So I assume that  
> means
> that sendmail is the MTA.
You will be using SMTP to the local MTA (which you say is Sendmail) if  
the following MM config variables are set as follows in either  
mm_cfg.py or the Defaults.py file. The following are th standard MM  
2.0.13 default values:

DELIVERY_MODULE = 'SMTPDirect'
SMTPHOST = 'localhost'

Also are the following default values still in place:

SMTP_MAX_RCPTS = 500
MAX_DELIVERY_THREADS = 0

>
> Aug 25 08:08:31 2003 (27798) smtp for 2 recips, completed in 1.045  
> seconds
> Aug 25 08:08:31 2003 (27798) smtp for 2 recips, completed in 1.149  
> seconds
> Aug 25 08:12:47 2003 (27798) smtp for 1 recips, completed in 0.208  
> seconds
> Aug 25 08:12:47 2003 (27798) smtp for 1 recips, completed in 0.241  
> seconds
> Aug 25 08:14:25 2003 (31184) smtp for 1 recips, completed in 0.701  
> seconds
> Aug 25 08:14:25 2003 (31184) smtp for 1 recips, completed in 0.716  
> seconds
> Aug 25 08:17:21 2003 (31184) smtp for 1 recips, completed in 0.167  
> seconds
> Aug 25 08:17:21 2003 (31184) smtp for 1 recips, completed in 0.179  
> seconds
> Aug 25 08:19:34 2003 (31184) smtp for 1 recips, completed in 3.962  
> seconds
> Aug 25 08:19:34 2003 (31184) smtp for 1 recips, completed in 4.234  
> seconds
>
> This is just a few of the lines in my /var/log/mailman/smtp file.   
> There are
> a few bounces from time to time, but not on great frequency.  I did  
> send an
> email to a test list, and attempted to time when I would see my email
> address in the post log, but after 10 minutes I have not seen it.  This
> really confuses me, because the mail system on this server delivers  
> mail
> directly to a mailbox in seconds.  Why the delay?
>
I am going to assume that you had not made the edit to  
$prefix/Mailman/Handlers/SMTPDirect.py I suggested in one of my  
response and that these log entries are the regular ones?

Bear in mind that the number in brackets in these log entries - (27798)  
for instance - is the process id of the qurnner handling the message  
and the same qrunner is handling multiple messages.

The smtp log entries are interesting if you have a large backlog of  
messages in the qfiles directory because:

1. only a few recipients are being delivered for each log entry. Are  
your mailing list that sparely populated for subscribers? Do these smtp  
log entries tie in with post entries for lists with few members and  
/var/log/mail entries for outgoing mail through the MTA?

2. quite long periods - 2, 3, 4 minutes in the sample log entries you  
give - elapse between log entries. But the log entries are for the same  
process so it has been continuing to run during these pregnant pauses.  
But what is it doing?

Some thoughts. Mailman's qrunner can get slow if it is:

a. archiving and

b. the messages for a list are large messages and/or

c.  the number of messages in the current archive of a list is large.

On the other hand, the pregnant pauses could be because MM is  
encountering long delays when opening SMTP connections to the MTA.

What sort of shape are the traffic/lists that MM is handling.

If you can catch the qrunner at it then maybe strace will give you some  
idea of what is doing: waiting on a socket open to the MTA maybe.

Another possibility is if by some means a bunch of lists have got  
orphaned locks over them so that the qrunner is spinning its wheels  
trying to find a list it can operate on. Does anything indicative of  
this show up in the $prefix/locks directory.

> ----- Original Message -----
> From: "Richard Barrett" <r.barrett at ftel.co.uk>
> To: "Andrew Falgout" <Andrew.Falgout at usm.edu>
> Cc: <mailman-users at python.org>
> Sent: Sunday, August 24, 2003 3:36 PM
> Subject: Re: [Mailman-Users] Mailman / Sendmail horribly slow
> performance..suggestions please?!?
>
>
>>
>> On Sunday, August 24, 2003, at 12:51  am, Andrew Falgout wrote:
>>
>>> I'm currently running mailman-2.0.13-1 eith sendmail-8.11.6-25.72.
>>> Posts
>>> are processed very fast, but delivery is more than slow.  It takes  
>>> 1-2
>>> minutes per post to delivery mail.  Evaluating what is going on in  
>>> the
>>> queue is a little more than cumbersome.  Over the course of 3-4 days
>>> mails
>>> have been piling up in the qfiles directory to the tune of 4000 mails
>>> and
>>> waiting.  There are mails 3 days old waiting to go out.  I'm looking
>>> for
>>> ways to evaluate what is in the queue, why it's in the queue, and how
>>> to
>>> tell qrunner to report more information on what it's thinking.  Like
>>> perhaps the msgid of the email it's trying to process at this time.
>>>
>> Are you using the local sendmail instance as the outgoing SMTP server?
>>
>> Regardless, what do the Mailman post, smtp and smtp-failure logs show
>> about mail being handled by the system. How does this tie in with the
>> MTA's mail log.
>>
>> If your outgoing mail is being handled by the local sendmail MTA then  
>> I
>> would normally expect Mailman to handoff to the MTA without much  
>> delay.
>> If MM is having problems handing off to sendmail then you might find
>> some evidence of this in the smtp-failure log.
>>> --  
>>> Thank You,
>>> Andrew
>>>
>>> -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+- 
>>> +-
>>> +-+-+-
>>> -      Andrew Wilson Falgout (RHCE)               Linux, E Pluribus
>>> UNIX!  -
>>> -      Systems Admin  II, TIU               Office:  (601) 266-5603
>>>     -
>>> -      University of Southern Mississippi
>>>     -
>>> -                                              Fax:  (601) 266-6788
>>>     -
>>> -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
>>> +-+-+-+
-----------------------------------------------------------------------
Richard Barrett                               http://www.openinfo.co.uk