Patch for use of Postfix VERP support
I've just installed Mailman version 2.1.11.
I was surprised to find that the standard distribution had no support for the Postfix VERP feature, which can create VERP return paths much more efficiently than when Mailman does it itself.
I did find a patch for the FreeBSD port of an earlier Mailman version (at http://lists.freebsd.org/pipermail/freebsd-ports-bugs/2007-October/128777.ht...), and changed it to match the standard 2.1.11 distribution.
It seems to work fine.
If anybody should be interested, I've placed the modified patch, directly usable for 2.1.11, at http://www.dybdal.dk/mailman-2.1.11-postfix-verp.patch
Note that even with the patch, this feature is disabled by default; it is turned on by setting VERP_STYLE = 'Postfix'. It also needs to be allowed explicitly in the Postfix configuration (e.g., "smtpd_authorized_verp_clients=127.0.0.0/8").
I have tested the patch only with the feature turned on.
Jesper Dybdal, Denmark. http://www.dybdal.dk (in Danish).
Hi Jesper,
I've done the following, with little success so far:
- applied your patch to Mailman/Defaults.py and Mailman/Handlers/SMTPDirect.py
- added the 2 configuration lines in Mailman/mm_cfg.py
- restarted mailman :-)
- added to /etc/postfixmain.cf the line smtpd_authorized_verp_clients=127.0.0.0/8
- reloaded postfix
But no luck: I still receive messages with Return-Path: <test-bounces@rezo.net>
a small HOWTO would be very useful.
On Sat, Jan 3, 2009 at 1:06 AM, Jesper Dybdal <jd-mailman-users@dybdal.dk> wrote:
I was surprised to find that the standard distribution had no support for the Postfix VERP feature, which can create VERP return paths much more efficiently than when Mailman does it itself.
Hi Fil,
On Sat, 3 Jan 2009 16:07:20 +0100, Fil <fil@rezo.net> wrote:
I've done the following, with little success so far:
- applied your patch to Mailman/Defaults.py and Mailman/Handlers/SMTPDirect.py
- added the 2 configuration lines in Mailman/mm_cfg.py
- restarted mailman :-)
- added to /etc/postfixmain.cf the line smtpd_authorized_verp_clients=127.0.0.0/8
- reloaded postfix
But no luck: I still receive messages with Return-Path: <test-bounces@rezo.net>
What you've done so far is to ensure that when and if Mailman uses VERP, it does so by asking Postfix to create the VERP return path, instead of doing it by itself (which would require it to deliver the message to Postfix once for every recipient).
You also need to tell Mailman when to actually use VERP. My mm_cfg.py contains:
VERP_PROBES = Yes VERP_PASSWORD_REMINDERS = Yes VERP_PERSONALIZED_DELIVERIES = Yes VERP_DELIVERY_INTERVAL = 1 VERP_CONFIRMATIONS = Yes VERP_STYLE = 'Postfix'
Jesper Dybdal, Denmark. http://www.dybdal.dk (in Danish).
On Sat, Jan 3, 2009 at 4:16 PM, Jesper Dybdal <jd-mailman-users@dybdal.dk> wrote:
VERP_PROBES = Yes VERP_PASSWORD_REMINDERS = Yes VERP_PERSONALIZED_DELIVERIES = Yes VERP_DELIVERY_INTERVAL = 1 VERP_CONFIRMATIONS = Yes VERP_STYLE = 'Postfix'
Yay, it works, takk.
One thing to note: local addresses such as mine (fil@rezo.net is defined in /etc/postfix/virtual as an alias for recifs@gmail.com) will be VERPed to test-bounces+recifs=gmail.com and not test-bounces+fil=rezo.net
So I now guess we have the README file ready. Will you give it an official home and link it from the patch file?
-- Fil
Another difference to mention in the README is that the postfix logs will show the normal (unVERPed) addresses instead of the VERPed addresses:
Jan 3 16:28:09 alan postfix/qmgr[17364]: 4754D3B05C7: from=<test-bounces@rezo.net>, size=3082, nrcpt=4 (queue active) Jan 3 16:28:10 alan postfix/smtp[27894]: 4754D3B05C7: to=<recifs@gmail.com>, orig_to=<fil@rezo.net>, relay=gmail-smtp-in.l.google.com[216.239.59.27]:25, delay=0.72, delays=0.11/0.02/0.14/0.45, dsn=2.0.0, status=sent (250 2.0.0 OK 1230996360 t2si31343988gve.31)
-- Fil
[My apologies to Fil for first sending this to him only - it was intended for the list]
On Sat, 3 Jan 2009 16:29:19 +0100, Fil <fil@rezo.net> wrote:
One thing to note: local addresses such as mine (fil@rezo.net is defined in /etc/postfix/virtual as an alias for recifs@gmail.com) will be VERPed to test-bounces+recifs=gmail.com and not test-bounces+fil=rezo.net
Yes. I have discussed this on the Postfix-users mailing list during the past week, and it sounds as if that will be corrected at some time. Until it is changed in Postfix, it can be a problem if a local address bounces, because Mailman will probably not be able to determine which subscriber it was that bounced - but fortunately, local addresses do not bounce very often.
So I now guess we have the README file ready. Will you give it an official home and link it from the patch file?
I am afraid there is nothing "official" about any home I can give to anything.
Jesper Dybdal, Denmark. http://www.dybdal.dk (in Danish).
Jesper Dybdal wrote:
I am afraid there is nothing "official" about any home I can give to anything.
Post the patch and a brief write-up to the tracker at <https://bugs.launchpad.net/mailman>.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hello,
I'm trying to document the VERP stuff, and I'm at a loss finding where scripts/bounces exploits the VERP information on bounces. scripts/bounces mentions VERP but does not actually process the info
-- Fil
Fil wrote:
I'm trying to document the VERP stuff, and I'm at a loss finding where scripts/bounces exploits the VERP information on bounces. scripts/bounces mentions VERP but does not actually process the info
All scripts/bounces does is queue the bounce message for BounceRunner. The actual processing of the bounce and detection of VERP is in the BounceRunner._dispose() method in Mailman/Queue/BounceRunner.py.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi,
the VERP system is good to detect bounces, but it has issues when lots of subscribers are on the same server:
(delivery temporarily suspended: connect to smtp.xxxx.xxx.xxx[xxxxxxxxx]: Connection timed out) (host mx01.xxxx.xxx[xxxxx] refused to talk to me: 421 4.3.2 Connection rate limit exceeded.)
As the emails are sent one by one the connection rate to these systems can get very high, and we get delayed.
Ideas?
-- Fil
Fil wrote:
As the emails are sent one by one the connection rate to these systems can get very high, and we get delayed.
Ideas?
Having postfix do your VERP'ing for you isn't going to help. Mailman needs to know what VERP was used going out, so that it can match that when the bounce comes back in. Even if you could patch Mailman so that it would automatically understand the postfix-style VERP and properly apply that, it still wouldn't help you.
Delivering larger amounts of messages from Mailman to postfix and then having postfix do the VERP'ing isn't going to keep your delivery rates to remote sites at a lower level -- if anything, it's going to increase it.
This is a postfix problem. Find the sites that cause the problems, and set up separate queues for them, with different parallelism and re-queueing/re-try schedules.
-- Brad Knowles <brad@shub-internet.org> If you like Jazz/R&B guitar, check out LinkedIn Profile: my friend bigsbytracks on YouTube at <http://tinyurl.com/y8kpxu> http://preview.tinyurl.com/bigsbytracks
Brad Knowles wrote:
This is a postfix problem. Find the sites that cause the problems, and set up separate queues for them, with different parallelism and re-queueing/re-try schedules.
Brad,
Normally, I would defer to you for all things MTA as your knowledge and experience in this area are far greater than mine, but I have been under the impression that if an MTA has two or more messages to deliver to the the same remote MX, that it can and perhaps will deliver them in one SMTP session as follows
connect HELO (or EHLO) MAIL FROM RCPT TO DATA . MAIL FROM RCPT TO DATA . ... QUIT
I.e., each message will be sent with a separate MAIL FROM, RCPT TO and DATA, but more than one message will be sent within a single connect, HELO, QUIT session.
If this is actually done (or can be done) by Postfix, it might help Fil, since the complaint in Fil's post was
(host mx01.xxxx.xxx[xxxxx] refused to talk to me: 421 4.3.2 Connection rate limit exceeded.)
in response to a connect request. Thus, if more messages could be sent over one connection, that would reduce the connection rate even though it wouldn't reduce the message rate.
Also, I think from earlier messages in this thread (<http://mail.python.org/pipermail/mailman-users/2009-January/064588.html> et seq) that Fil is already successfully using Postfix VERPing with Mailman, so the only question here is can Postfix be convinced to send multiple messages to the same MX over a single connection.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
on 5/14/09 7:58 PM, Mark Sapiro said:
Normally, I would defer to you for all things MTA as your knowledge and experience in this area are far greater than mine, but I have been under the impression that if an MTA has two or more messages to deliver to the the same remote MX, that it can and perhaps will deliver them in one SMTP session as follows
My understanding is that the issue here isn't so much the single SMTP connection, which may or may not get re-used, depending on the queueing strategy of the sender.
If you want to increase your efficiency in this area, then you make sure that your submission actually just puts the message into the queue in deferred mode and doesn't make an immediate delivery attempt, and of course you turn off all spam and other access control checks on the outbound side, since you should have already done all those on inbound. You then allow later queue runs to come along and pick that up, along with any other messages bound for that destination.
I think we cover this in the various "performance" articles in the FAQ Wiki.
My understanding is that the issue here is whether or not the VERP'ed format will be understood and usable by Mailman if there is a bounce, and how many unique messages will be delivered, as opposed to how many connections will be created to deliver the messages.
-- Brad Knowles <brad@shub-internet.org> If you like Jazz/R&B guitar, check out LinkedIn Profile: my friend bigsbytracks on YouTube at <http://tinyurl.com/y8kpxu> http://preview.tinyurl.com/bigsbytracks
On Sat, 3 Jan 2009 16:29:19 +0100, Fil <fil@rezo.net> wrote:
[About using Postfix' built-in VERP mechanism with Mailman]
One thing to note: local addresses such as mine (fil@rezo.net is defined in /etc/postfix/virtual as an alias for recifs@gmail.com) will be VERPed to test-bounces+recifs=gmail.com and not test-bounces+fil=rezo.net
This has now been corrected in Postfix 2.6 Snapshot 20090125: that version uses the orignal destination address as known by Mailman to create the VERP'ed sender addresses.
Jesper Dybdal, Denmark. http://www.dybdal.dk (in Danish).
- Jesper Dybdal <jd-mailman-users@dybdal.dk> wrote:
If anybody should be interested, I've placed the modified patch, directly usable for 2.1.11, at http://www.dybdal.dk/mailman-2.1.11-postfix-verp.patch
Uh oh. I remember that patch, I've even tested it (German: http://www.incertum.net/archives/307-Postfix,-Mailman-und-XVERP.html), but it was incomplete (I found that out only weeks after I had written that test).
I don't remember any specific details, but there were definitely problems with double VERP'ifying. Something with the monthly reminders. Whatever you do, don't simply go live with this one, perform more testing first: Password reminders, monthly membership reminders, moderation messages, reject messages and so on.
Don't get me wrong, being able to reduce the number of queue files created can be a major performance increase on busy servers, the above patch just doesn't cut it.
Cheers Stefan
Stefan Förster http://www.incertum.net/ Public Key: 0xBBE2A9E9
participants (5)
-
Brad Knowles -
Fil -
Jesper Dybdal -
Mark Sapiro -
Stefan Förster