Re: [Mailman-Developers] Re: way to minimize IO load with MTA supported VERP
On Thu, 6 Dec 2001 18:00:52 -0800 (PST) Dan Mick <dmick@utopia.West.Sun.COM> wrote:
MAIL FROM: sender@sender.dom XVERP
Cheap prediction:
Mailman is going to end up with a set of MTA-specific and internally configurable VERP configurations to chose from.
--
J C Lawrence
---------(*) Satan, oscillate my metallic sonatas.
claw@kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
"JCL" == J C Lawrence <claw@kanga.nu> writes:
JCL> Mailman is going to end up with a set of MTA-specific and
JCL> internally configurable VERP configurations to chose from.I actually don't think that MTA-directed VERPing helps us out much. Sure, it can give us an envelope sender that we can use for better bounce detection[*], but I think that the much more interesting personalization is content personalization. I.e. inserting into the message body, footers, headers, RFC 2822 headers, etc. information specific to the recipient. Only Mailman knows that data and how to interpolate it into the message body. AFAIK, there's no way to coordinate this with the MTA, so content personalization is always going to be performed by Mailman.
The VERP-like approach in MM2.1 toward the envelope sender is mostly just gravy. (I try to be careful to describe it as VERP-like since it isn't technically VERP.)
BTW, see VERP_REGEXP and VERP_FORMAT for how the envelope sender is composed and parsed.
-Barry
[*] VERP helps with knowing exactly which address on which list is bouncing, but I don't think it helps much with knowing the severity of the bounce.
On Thu, Dec 06, 2001 at 10:14:35PM -0500, Barry A. Warsaw wrote:
[*] VERP helps with knowing exactly which address on which list is bouncing, but I don't think it helps much with knowing the severity of the bounce.
Sure, but the beauty is that there's almost no way that a broken MTA can bounce in such a way as to not know where the message went. Autoresponders, on the other hand, suck rocks through straws. So you can afford to handle a lot more bounces, and implement an ezmlm-like probe when it gets critical, all in the certianty that you're (a) not bouncing messages to the list admins that they have to read and (b) if you miss any messages over a long time you can reset your counter and let the user know what they've missed. These are 2 really nice ezmlm features...
-- The 5 year plan: In five years we'll make up another plan. Or just re-use this one.
On Fri, 2001-12-07 at 03:57, Peter C. Norton wrote:
Sure, but the beauty is that there's almost no way that a broken MTA can bounce in such a way as to not know where the message went.
I just *love* this boundless optimism. Never underestimate the ability of MTA writers and postmasters (ie those configuring their MTA) to do something *really* obscure to screw you. [VERP-ed header addresses, folks? :-) ]
So you can afford to handle a lot more bounces, and implement an ezmlm-like probe when it gets critical, all in the certianty that you're (a) not bouncing messages to the list admins that they have to read and (b) if you miss any messages over a long time you can reset your counter and let the user know what they've missed.
Yeah - some of the MLMs are encoding the message number (ie a sequential number per mailing list message) into the VERP. That opens up interesting possibilities.
[Hmm... are there systems that are broken by long sender addresses?]
Nigel.On Thu, 6 Dec 2001 22:14:35 -0500 Barry A Warsaw <barry@zope.com> wrote:
"JCL" == J C Lawrence <claw@kanga.nu> writes:
JCL> Mailman is going to end up with a set of MTA-specific and JCL> internally configurable VERP configurations to chose from.
I actually don't think that MTA-directed VERPing helps us out much.
Its a question of publics and their needs. There are large and desperately unfilled needs for body-customisation in list servers to be sure. That's an untapped market. Its also a non-traditional MLM market. VERP, for Mailman, solves fundamental problems in bounce handling and membership roll maintenance, and the majority of Mailman's current and traditional userbase are going to be wildly interested in that, not in message body customisation.
Which doesn't mean we can't or shouldn't do both, just that (at least initially) the target consumers are rather different.
Sure, it can give us an envelope sender that we can use for better bounce detection[*], but I think that the much more interesting personalization is content personalization. I.e. inserting into the message body, footers, headers, RFC 2822 headers, etc. information specific to the recipient. Only Mailman knows that data and how to interpolate it into the message body. AFAIK, there's no way to coordinate this with the MTA, so content personalization is always going to be performed by Mailman.
Yup. Not a whole lot of argument going on at this end.
The VERP-like approach in MM2.1 toward the envelope sender is mostly just gravy. (I try to be careful to describe it as VERP-like since it isn't technically VERP.)
<nod>
Technical precision terminology: always bites you in the arse when you're trying to say something useful.
BTW, see VERP_REGEXP and VERP_FORMAT for how the envelope sender is composed and parsed.
Will do. I'm behind on my 2.1 reading (been looking into a CRM system to bolt Mailman into).
[*] VERP helps with knowing exactly which address on which list is bouncing, but I don't think it helps much with knowing the severity of the bounce.
It doesn't. I'm strongly tempted to treat all bounces as hard, unless we can cheaply _and_ conclusively determine that they are soft.
--
J C Lawrence
---------(*) Satan, oscillate my metallic sonatas.
claw@kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
"Barry A. Warsaw" wrote:
"JCL" == J C Lawrence <claw@kanga.nu> writes:
JCL> Mailman is going to end up with a set of MTA-specific and JCL> internally configurable VERP configurations to chose from.I actually don't think that MTA-directed VERPing helps us out much. Sure, it can give us an envelope sender that we can use for better bounce detection[*],
...which is clearly all it's good for. I have a significant percentage of users whose bounces are useless without VERP, and it's tiring dealing with the uncaught bounces. If I can get VERP with little extra load (because it's done in the MTA) then I'm happy. (Been running for the last 8-10 hours with Postfix doing the VERPing, with a small hack to SMTPDirect.py, and some logging in BounceRunner.py, and it looks really nice so far.)
but I think that the much more interesting personalization is content personalization. I.e. inserting into the message body, footers, headers, RFC 2822 headers, etc. information specific to the recipient. Only Mailman knows that data and how to interpolate it into the message body. AFAIK, there's no way to coordinate this with the MTA, so content personalization is always going to be performed by Mailman.
Sure. But that's *expensive*. It's stunning how much more time that all uses up. Granted, I haven't done the various spool vs. syslog optimizations etc., but the point is I don't have to until we get to one-MLM-to-MTA-transaction-per-user. That's when it gets brutal.
Anyway, the VERP-in-the-MTA thing seems like a useful featurelet, even if it has limited benefit. Hopefully more MTAs will follow.
On Fri, 2001-12-07 at 11:41, Dan Mick wrote:
Anyway, the VERP-in-the-MTA thing seems like a useful featurelet, even if it has limited benefit. Hopefully more MTAs will follow.
I've kicked off discussion on this on the exim lists. It would be nice if there was some form of spec for it though - there appears to be one for a VERP extension on the courier site, but thats using a VERP keyword and not XVERP (presumably they are just about the same, but one you don't need to clear with the ESMTP spec gurus).
Nigel.Nigel Metheringham wrote:
It would be nice if there was some form of spec for it though - there appears to be one for a VERP extension on the courier site, but thats using a VERP keyword and not XVERP (presumably they are just about the same, but one you don't need to clear with the ESMTP spec gurus).
The bottom of http://www.courier-mta.org/ links that VERP proposal behind the name XVERP. I'm sure that the X is there simply to allow its presence and use before the RFC reaches some consensus.
A thing I hadn't thought about: if the local MTA senses that the remote MTA *also* supports XVERP, it doesn't have to do the expansion itself; it can simply send on one XVERP-ed multirecipient message, and let the final MTA handle the envelope multiplication.
On Fri, Dec 07, 2001 at 04:00:11AM -0800, Dan Mick wrote:
A thing I hadn't thought about: if the local MTA senses that the remote MTA *also* supports XVERP, it doesn't have to do the expansion itself; it can simply send on one XVERP-ed multirecipient message, and let the final MTA handle the envelope multiplication.
Which would be the True Win... but *absolutely requires* an RFC for verp syntax, which will have to be extensible. No, really. It will. Take my word for it. :-)
*I* think that sending a VERP template, with a spot marked for the destination MTA to put the destination email address, would be the most flexible approach.
Cheers, -- jra
Jay R. Ashworth jra@baylink.com Member of the Technical Staff Baylink RFC 2100 The Suncoast Freenet The Things I Think Tampa Bay, Florida http://baylink.pitas.com +1 727 647 1274
"If you don't have a dream; how're you gonna have a dream come true?" -- Captain Sensible, The Damned (from South Pacific's "Happy Talk")
"Jay R. Ashworth" wrote:
On Fri, Dec 07, 2001 at 04:00:11AM -0800, Dan Mick wrote:
A thing I hadn't thought about: if the local MTA senses that the remote MTA *also* supports XVERP, it doesn't have to do the expansion itself; it can simply send on one XVERP-ed multirecipient message, and let the final MTA handle the envelope multiplication.
Which would be the True Win... but *absolutely requires* an RFC for verp syntax, which will have to be extensible. No, really. It will. Take my word for it. :-)
Oh, absolutely.
*I* think that sending a VERP template, with a spot marked for the destination MTA to put the destination email address, would be the most flexible approach.
? You're talking about some other ESMTP extension then?
On Fri, Dec 07, 2001 at 01:21:00PM -0800, Dan Mick wrote:
*I* think that sending a VERP template, with a spot marked for the destination MTA to put the destination email address, would be the most flexible approach.
? You're talking about some other ESMTP extension then?
I was not aware that ESMTP already *had* an extension for VERP; so, "yes", I guess. :-)
Cheers, -- jra
Jay R. Ashworth jra@baylink.com Member of the Technical Staff Baylink RFC 2100 The Suncoast Freenet The Things I Think Tampa Bay, Florida http://baylink.pitas.com +1 727 647 1274
"If you don't have a dream; how're you gonna have a dream come true?" -- Captain Sensible, The Damned (from South Pacific's "Happy Talk")
Speaking of Exim.. one thing that really bothers me about Exim is the message(s) it sends when it has to wait to deliver the message... these would be interpreted as bounces, although they really are not. I've seen a few such messages, only to have the message delivered normally. That would trigger some bounce logic to at least pick up on what really isn't (yet) a bounce.
Bob
At 14:18 -0500 12/7/2001, Bob Puff@NLE wrote:
Speaking of Exim.. one thing that really bothers me about Exim is the message(s) it sends when it has to wait to deliver the message... these would be interpreted as bounces, although they really are not. I've seen a few such messages, only to have the message delivered normally. That would trigger some bounce logic to at least pick up on what really isn't (yet) a bounce.
Exim's delayed delivery warnings are "orderly" enough that it would be quite easy to ignore them in bounce processing. In several ways, they don't look like failure messages.
Meanwhile, they let a user who has mistyped an address know about it sooner than without the messages. For an Exim running in support of "live" users.
Assuming, of course, that the Exim administrator has resisted the temptation to "improve" the delayed delivery message. If she's done that, users are likely to fall off lists. One hopes that the Exim admin and the Mailman admin are speaking to each other.
If the Exim is running just to handle Mailman, the warning message can be done away with entirely (and the retry intervals can be adjusted, and several other such adjustments made). Trivially. But of course, you can't configure away messages generated by remote sites you don't control.
--John
On Sat, 2001-12-08 at 03:16, John W Baxter wrote:
Exim's delayed delivery warnings are "orderly" enough that it would be quite easy to ignore them in bounce processing. In several ways, they don't look like failure messages.
The other thing is that there is a condition on the generation of delay warning messages. The default configuration for this condition for a few years now has been that mailing list mail (identified by the Precedence: header) does not have delay warnings generated for it.
See http://www.exim.org/exim-html-3.30/doc/html/spec_11.html#SEC206
If an admin is running an exim old enough that this conditional is not available or is not defaulted, then take a baseball bat to them until they see the error of their ways.
If an admin has changed the condition such that it now sends delay warnings in response to mailing list mail, then add some nails to your baseball bat and wield it appropriately on said admin.
Nigel.
On Thu, Dec 06, 2001 at 10:14:35PM -0500, Barry A. Warsaw wrote:
I actually don't think that MTA-directed VERPing helps us out much. Sure, it can give us an envelope sender that we can use for better bounce detection[*]
How robust is the bounce detection? Even with VERP and/or good MTAs, is there enough smarts in the system to prevent a black hat from connecting to the MTA on the mailman server and using fake bounce messages to knock someone off a list without their knowledge?
, but I think that the much more interesting personalization is content personalization. I.e. inserting into the message body, footers, headers, RFC 2822 headers, etc. information
Also RFC 2369 List-* headers and in-body subscription management links. :-)
specific to the recipient. Only Mailman knows that data and how to interpolate it into the message body.
Yep. I'm glad to hear you considering this as an option, though I imagine a lot of folks, for good reason, want the current efficient behavior as a choice.
[*] VERP helps with knowing exactly which address on which list is bouncing, but I don't think it helps much with knowing the severity of the bounce.
Or the authenticity. If Mailman did VERP-like customizations itself, you could do something like my crypto-VERP proposal, where if you sent message number 1234 to me, the unique return path would look something like peterw-usa-net-1234-033fe9dbe554a34839e1b82ec4eb5ab0-list-owner@example.com or maybe list-owner+peterw-usa-net-1234-033fe9dbe554a34839e1b82ec4eb5ab0@example.com where 033fe9dbe554a34839e1b82ec4eb5ab0 is the MD5 hash of peterw-usa-net-1234-secret (the MM install routine would pick a random phrase to be used as the secret, which would probably be long). This way, mailman could be quite certain if a bounce was legit, and in response to a recent message delivery attempt (valid bounces for old messages [> 14 days?] could be ignored; alternately MM could use time_t instead of a message number, making calculations easier). Thoughts?
-Peter
-- I am what I am 'cause I ain't what I used to be. - S Bruton & J Fleming
On Fri, Dec 07, 2001 at 02:36:39PM -0500, Peter W wrote:
On Thu, Dec 06, 2001 at 10:14:35PM -0500, Barry A. Warsaw wrote:
I actually don't think that MTA-directed VERPing helps us out much. Sure, it can give us an envelope sender that we can use for better bounce detection[*]
How robust is the bounce detection? Even with VERP and/or good MTAs, is there enough smarts in the system to prevent a black hat from connecting to the MTA on the mailman server and using fake bounce messages to knock someone off a list without their knowledge?
You can avoid this by is by sending a test message to them and use a cookie in the envelope-from that is a hash of a saved secret value that you can compare to on the bounce. If you get a bounce to the address that has the proper hash, then you can pretty safely disable them (unless their postmaster is out to get them. But you can't save them from that). If you don't get the message bounced back then that email address isn't really (or at least always) bouncing.
-- The 5 year plan: In five years we'll make up another plan. Or just re-use this one.
On 12/7/01 6:40 PM, "Peter C. Norton" <spacey-mailman@lenin.nu> wrote:
If you don't get the message bounced back then that email address isn't really (or at least always) bouncing.
Unless, of course, the postmaster implements bounces using a "not here any more" mailbot, which will strip all of the headers, not return as a bounce, and probably not be parsable by anything but a human, and maybe not then.
This is called "being helpful", of course.
On Fri, Dec 07, 2001 at 08:01:57PM -0800, Chuq Von Rospach wrote:
On 12/7/01 6:40 PM, "Peter C. Norton" <spacey-mailman@lenin.nu> wrote:
If you don't get the message bounced back then that email address isn't really (or at least always) bouncing.
Unless, of course, the postmaster implements bounces using a "not here any more" mailbot, which will strip all of the headers, not return as a bounce, and probably not be parsable by anything but a human, and maybe not then.
Sure it will. The message will go back to the envelope sender, which will be a cookied address. I guess to be sure you'd want to put the address in the header from, too.
-- The 5 year plan: In five years we'll make up another plan. Or just re-use this one.
Peter C. Norton wrote:
Sure it will. The message will go back to the envelope sender, which will be a cookied address. I guess to be sure you'd want to put the address in the header from, too.
How far do you take that? I have seen people put the "From:" address in their addressbook, then try to send email to that. You don't want to interpret their poor attempts at posting to be bounces.
Bob
They put in the header from most of the time, don't they? If so, not a problem.
On Fri, Dec 07, 2001 at 11:27:53PM -0500, Bob Puff@NLE wrote:
Peter C. Norton wrote:
Sure it will. The message will go back to the envelope sender, which will be a cookied address. I guess to be sure you'd want to put the address in the header from, too.
How far do you take that? I have seen people put the "From:" address in their addressbook, then try to send email to that. You don't want to interpret their poor attempts at posting to be bounces.
Bob
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
-- The 5 year plan: In five years we'll make up another plan. Or just re-use this one.
Chuq Von Rospach wrote:
Unless, of course, the postmaster implements bounces using a "not here any more" mailbot, which will strip all of the headers, not return as a bounce, and probably not be parsable by anything but a human, and maybe not then.
This is called "being helpful", of course.
I -just- experienced one of those! Had a bunch of them in my mailman-owner mailbox from this one guy on an active list. Sigh.
Bob
On Fri, Dec 07, 2001 at 06:40:15PM -0800, Peter C. Norton wrote:
On Fri, Dec 07, 2001 at 02:36:39PM -0500, Peter W wrote:
How robust is the bounce detection? Even with VERP and/or good MTAs, is there enough smarts in the system to prevent a black hat from connecting to the MTA on the mailman server and using fake bounce messages to knock someone off a list without their knowledge?
You can avoid this by is by sending a test message to them and use a cookie in the envelope-from that is a hash of a saved secret value that you can compare to on the bounce.
Right. That's what I'm suggesting, that maybe such a cookie plan should be
implemented. I like my idea of the cookie being a hash of both the
recipient address and something like a time value, so that "replay"
attacks are less feasible. You shouldn't be able to pick up a disk drive
that Barry W discarded a year earlier and get a cookie that still lets you
unsubscribe him from this list. :-)
If you get a bounce to the address that has the proper hash, then you can pretty safely disable them (unless their postmaster is out to get them. But you can't save them from that).
Or if someone gets to their saved messages, right.
If you don't get the message bounced back then that email address isn't really (or at least always) bouncing.
Eaxctly. Sounds like we're in basic agrement about the potential value of a cookie-laden envelope?
-Peter
-- I am what I am 'cause I ain't what I used to be. - S Bruton & J Fleming
On Fri, Dec 07, 2001 at 11:23:02PM -0500, Peter W wrote:
Right. That's what I'm suggesting, that maybe such a cookie plan should be implemented. I like my idea of the cookie being a hash of both the recipient address and something like a time value, so that "replay"
attacks are less feasible. You shouldn't be able to pick up a disk drive that Barry W discarded a year earlier and get a cookie that still lets you unsubscribe him from this list. :-)
Throw in a saved secret per list or per test message, too. The recipient address is known, and time values can probably be guessed if you have a known config and the attacker is generating the "bounces". The attacker could probably brute force the right address within 300 messages (5 minute timespan).
If you get a bounce to the address that has the proper hash, then you can pretty safely disable them (unless their postmaster is out to get them. But you can't save them from that).
Or if someone gets to their saved messages, right.
If you don't get the message bounced back then that email address isn't really (or at least always) bouncing.
Eaxctly. Sounds like we're in basic agrement about the potential value of a cookie-laden envelope?
It makes my life easier when I use ezmlm. I think it would be a good addition to mailman.
-- The 5 year plan: In five years we'll make up another plan. Or just re-use this one.
participants (10)
-
barry＠zope.com -
Bob Puff＠NLE -
Chuq Von Rospach -
Dan Mick -
J C Lawrence -
Jay R. Ashworth -
John W Baxter -
Nigel Metheringham -
Peter C. Norton -
Peter W