We just moved a very old mailman infrastructure mailman to Solaris and sendmail. It seems, when a message is posted, the "out" directory doesn't seem to clear out fast enough.
1K archive 1K bounces 1K commands 1K in 1K news 403K out 1K retry 1K shunt 1K virgin
The size just stays like that forever and slowly decrements. When I look in /var/log/syslog (mail log of sendmail), I see deliveries going through. I do a 'truss' of the OutgoingRunner python process and I see:
% truss -p 11997 recv(10, 0x00748454, 8192, 0) (sleeping...) ... %
What is it waiting for?
Anil Jangity wrote:
We just moved a very old mailman infrastructure mailman to Solaris and sendmail. It seems, when a message is posted, the "out" directory doesn't seem to clear out fast enough.
1K archive 1K bounces 1K commands 1K in 1K news 403K out 1K retry 1K shunt 1K virgin
The size just stays like that forever and slowly decrements. When I look in /var/log/syslog (mail log of sendmail), I see deliveries going through.
Look at Mailman's 'smtp' log. You undoubtedly have a huge backlog in the out queue because you aren't processing outgoing SMTP fast enough to keep up with demand. In the smtp log you will see entries like
Jan 20 09:41:14 2012 (17283) <message-id> smtp to listname for 223 recips, completed in 3.500 seconds
You will observe that each entry's timestamp is equal to that of the previous entry plus this entry's 'completed in' time indicating no delay between messages. Each entry represents processing of one out/ queue entry. You will probably also note that the processing rate is much less than the 64 recipients/second in the above message which is from a full VERP installation.
I do a 'truss' of the OutgoingRunner python process and I see:
% truss -p 11997 recv(10, 0x00748454, 8192, 0) (sleeping...) ... %
What is it waiting for?
Most likely an SMTP reply from sendmail. See the FAQ's at <http://wiki.list.org/x/roA9> and <http://wiki.list.org/x/q4A9>. This is almost certainly a case of sendmail not keeping up with Mailman's demand and if so, will require sendmail tuning to fix.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
It seems it is doing about 6000 users in 35 minutes. The problem seems to be that sendmail is doing one email connection per recipient.
I have these set in mm_cfg.py:
SMTP_MAX_RCPTS = 15 SMTP_MAX_SESSIONS_PER_CONNECTION = 50
It seems Qrunner isn't honoring those?
Look at Mailman's 'smtp' log. You undoubtedly have a huge backlog in the out queue because you aren't processing outgoing SMTP fast enough to keep up with demand. In the smtp log you will see entries like
Jan 20 09:41:14 2012 (17283) <message-id> smtp to listname for 223 recips, completed in 3.500 seconds
You will observe that each entry's timestamp is equal to that of the previous entry plus this entry's 'completed in' time indicating no delay between messages. Each entry represents processing of one out/ queue entry. You will probably also note that the processing rate is much less than the 64 recipients/second in the above message which is from a full VERP installation.
Yes. What exactly does that option do/mean? Does mailman actually (if it works) send to 15 recipients at a time - per SMTP session?
On Feb 1, 2012, at 5:02 PM, Carl Zwanzig wrote:
On 2/1/2012 3:57 PM, Anil Jangity wrote:
I have these set in mm_cfg.py:
SMTP_MAX_RCPTS = 15 SMTP_MAX_SESSIONS_PER_CONNECTION = 50
It seems Qrunner isn't honoring those?
Did you restart the qrunners after changing those settings?
z!
Actually, looking at the code... it seems it might be because I have these set for some of the lists.
VERP_PERSONALIZED_DELIVERIES = 1 OWNERS_CAN_ENABLE_PERSONALIZATION = 1
Thanks
On Feb 1, 2012, at 9:01 PM, Anil Jangity wrote:
Yes. What exactly does that option do/mean? Does mailman actually (if it works) send to 15 recipients at a time - per SMTP session?
On Feb 1, 2012, at 5:02 PM, Carl Zwanzig wrote:
On 2/1/2012 3:57 PM, Anil Jangity wrote:
I have these set in mm_cfg.py:
SMTP_MAX_RCPTS = 15 SMTP_MAX_SESSIONS_PER_CONNECTION = 50
It seems Qrunner isn't honoring those?
Did you restart the qrunners after changing those settings?
z!
Anil Jangity wrote:
Does mailman actually (if it works) send to 15 recipients at a time - per SMTP session?
Mailman's use of the term session is not consistent with the RFCs. What Mailman calls SMTP_MAX_SESSIONS_PER_CONNECTION is actually in RFC 821, 2821, 5321 terms, max transactions per session. I.e. one session is one connection from HELO/EHLO to QUIT and one transaction is one MAIL FROM and everything that follows up to the next MAIL FROM.
Unless you have some specific MTA reason for setting SMTP_MAX_SESSIONS_PER_CONNECTION, you should leave it at the default value of 0 meaning unlimited.
If a message is personalized or VERPd, SMTP_MAX_RCPTS is ignored because you can't send a personalized or VERPd message to more than one recipient. Otherwize, SMTP_MAX_RCPTS is the maximum number of RCPT TO recipients in one MAIL FROM transaction.
Note that 6000 users per 35 minutes is less than 3 per second. This is extremely slow. I typically see 30 to over 60 recipients per second on a rather ordinary production server with full VERP.
See my reply to your original post in this thread at <http://mail.python.org/pipermail/mailman-users/2012-January/072778.html> and look at the FAQs linked in that post for hints. In particular, do not do DNS verification of RCPT TO domains during SMTP from Mailman.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark, thanks for the very detailed info below. I am still stuck on this.
This sendmail server is setup to relay the mail to smarthost. When I inject a test mail on port 25 with RCPT and FROM to be a made up junk domain, the email is still delivered *immediately* to the smarthost, there is no resolver issues.
If sendmail was really doing DNS lookups, I would think it would be stuck locally or throw a bounce/error right away. It just seems like it's running serially, and not dequeueing multiple messages to the smarthost at a time.
Anil PS: Putting a local DNS cacheing server is a possibility, but not yet.
On Feb 1, 2012, at 10:56 PM, Mark Sapiro wrote:
Anil Jangity wrote:
Does mailman actually (if it works) send to 15 recipients at a time - per SMTP session?
Mailman's use of the term session is not consistent with the RFCs. What Mailman calls SMTP_MAX_SESSIONS_PER_CONNECTION is actually in RFC 821, 2821, 5321 terms, max transactions per session. I.e. one session is one connection from HELO/EHLO to QUIT and one transaction is one MAIL FROM and everything that follows up to the next MAIL FROM.
Unless you have some specific MTA reason for setting SMTP_MAX_SESSIONS_PER_CONNECTION, you should leave it at the default value of 0 meaning unlimited.
If a message is personalized or VERPd, SMTP_MAX_RCPTS is ignored because you can't send a personalized or VERPd message to more than one recipient. Otherwize, SMTP_MAX_RCPTS is the maximum number of RCPT TO recipients in one MAIL FROM transaction.
Note that 6000 users per 35 minutes is less than 3 per second. This is extremely slow. I typically see 30 to over 60 recipients per second on a rather ordinary production server with full VERP.
See my reply to your original post in this thread at <http://mail.python.org/pipermail/mailman-users/2012-January/072778.html> and look at the FAQs linked in that post for hints. In particular, do not do DNS verification of RCPT TO domains during SMTP from Mailman.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Anil Jangity wrote:
This sendmail server is setup to relay the mail to smarthost. When I inject a test mail on port 25 with RCPT and FROM to be a made up junk domain, the email is still delivered *immediately* to the smarthost, there is no resolver issues.
If sendmail was really doing DNS lookups, I would think it would be stuck locally or throw a bounce/error right away. It just seems like it's running serially, and not dequeueing multiple messages to the smarthost at a time.
This issue is not a Mailman issue per se. It is either a Sendmail configuration issue or something to do with the interaction of Sendmail with the smarthost.
We have given you all the information we have to help you with this. If that isn't sufficient, you'll have to pursue this using Sendmail support resources. See <http://www.sendmail.com/sm/open_source/support/>.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Mark,
Thanks. We're getting better performance now once we added this to our sendmail .mc file:
FEATURE(`nocanonify')
I wanted to see if we can do further improvements. Right now our email flow is like this for lists:
Internet -> ServerA:25 (/etc/aliases mailman) -> ServerA -> SmartHostA
Mailman receives the mail and then posts to the same local MTA on the host which sends it out to Smart host. I am not sure we can't just have mailman send directly to the smarthost?
What are the implications (on the web site as well as how Mailman functions) if I change DEFAULT_EMAIL_HOST to be SmartHostA?
Thanks, Anil
PS: Now I am getting about 3000 recips in about 5 minutes. Still not the greatest but it's better than before.
On Feb 22, 2012, at 6:53 PM, Mark Sapiro wrote:
Anil Jangity wrote:
This sendmail server is setup to relay the mail to smarthost. When I inject a test mail on port 25 with RCPT and FROM to be a made up junk domain, the email is still delivered *immediately* to the smarthost, there is no resolver issues.
If sendmail was really doing DNS lookups, I would think it would be stuck locally or throw a bounce/error right away. It just seems like it's running serially, and not dequeueing multiple messages to the smarthost at a time.
This issue is not a Mailman issue per se. It is either a Sendmail configuration issue or something to do with the interaction of Sendmail with the smarthost.
We have given you all the information we have to help you with this. If that isn't sufficient, you'll have to pursue this using Sendmail support resources. See <http://www.sendmail.com/sm/open_source/support/>.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Anil Jangity wrote:
Mailman receives the mail and then posts to the same local MTA on the host which sends it out to Smart host. I am not sure we can't just have mailman send directly to the smarthost?
What are the implications (on the web site as well as how Mailman functions) if I change DEFAULT_EMAIL_HOST to be SmartHostA?
Most likeley, SmartHostA will refuse to relay the mail because Mailman hasn't authenticated to it as an authorized user.
See <https://bugs.launchpad.net/mailman/+bug/558281> for a patch that implements SMTP authentication for Mailman.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Actually, this is a internal SmartHost that doesn't require SMTP auth. I think a while back I had configured Mailman with --with-mailhost=localhost, a bunch of pages also showed "localhost" as the email domain for some of the list details. That's not what I wanted.
I assume there won't be issues if all I do is change that variable to SmartHostA in the mm_cfg.py file. I would like everything else to be in tact, and only want the outgoing mail to be directed to smart host. I will test it out in our UAT.
Thanks! Anil
On Mar 8, 2012, at 2:23 PM, Mark Sapiro wrote:
Anil Jangity wrote:
Mailman receives the mail and then posts to the same local MTA on the host which sends it out to Smart host. I am not sure we can't just have mailman send directly to the smarthost?
What are the implications (on the web site as well as how Mailman functions) if I change DEFAULT_EMAIL_HOST to be SmartHostA?
Most likeley, SmartHostA will refuse to relay the mail because Mailman hasn't authenticated to it as an authorized user.
See <https://bugs.launchpad.net/mailman/+bug/558281> for a patch that implements SMTP authentication for Mailman.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Anil Jangity wrote:
Actually, this is a internal SmartHost that doesn't require SMTP auth. I think a while back I had configured Mailman with --with-mailhost=localhost, a bunch of pages also showed "localhost" as the email domain for some of the list details. That's not what I wanted.
The --with-mailhost config sets DEFAULT_EMAIL_HOST in Defaults.py. This has nothing to do with the MTA used for sending.
I assume there won't be issues if all I do is change that variable to SmartHostA in the mm_cfg.py file. I would like everything else to be in tact, and only want the outgoing mail to be directed to smart host. I will test it out in our UAT.
You want to set SMTPHOST (and SMTPPORT if you want other than 25) in mm_cfg.py. That's all you need to change.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Anil Jangity -
Carl Zwanzig -
Mark Sapiro