Re: [Mailman-Users] Bounce issues with Yahoo
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Greg,
Like most people who've been running mailing lists for 20+ years, I have strong opinions about list policy. Please address the mechanism I asked for instead of seeking to discuss the policy issues.
You have a policy problem with your MTA, not your MLM.
Which part of "I don't think we should talk about the policy issues" do you not understand? Please stop wasting your own time.
Hey don't do what you are complaining about others doing. I understand you got some crummy replies, but take a moment to understand what I am saying. It was meant to be constructive.
Do you really have a *policy* to accept messages that you will never deliver, save them to disk, and then generate reject messages for them? Or is that simply the way your MTA+MLM is configured by default? Is that *policy* really any different from your standpoint, from rejecting undeliverable messages during SMTP?
Real users who send from the wrong email account, for example, would still have their messages returned to them either way. The only difference is that the bounce will be generated by their MTA, and not yours. You can still include the reason the message was rejected.
Junk from spammers will never land on your disks, you will not be flooding yahoo users with incomprehensible "reject" notices for stuff they didn't send, or annoying yahoo.com's postmaster with a blizzard of messages to non-existant accounts.
It would not be inconsistent with your current MLM policy. It is simply a better mechanism for implementing it, a better MTA policy, and better netizenship to boot.
- H-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin)
iD4DBQFEB1KgOy/dHTCUq6oRArwjAJ0Y1UNtC7xviyu61C9uVaWvyKQfcgCYpJKL mv7r+x+uQ02tuszPPi1zPw== =j0UH -----END PGP SIGNATURE-----
At 12:16 PM -0800 2006-03-02, Harold Paulson wrote:
Do you really have a *policy* to accept messages that you will never deliver, save them to disk, and then generate reject messages for them?
How could the MTA possibly know that the MLM would choose to reject those messages because the sender is not a subscriber? Or that the MLM might choose to hold those messages for moderation, because the sender is not a subscriber.
These are concepts that are totally beyond the scope of the MTA. There is nothing the MTA can do for you here.
Or is that simply the way your MTA+MLM is configured by default?
As far as the MTA is concerned, they are *ALL* configured this way -- assuming that all the appropriate anti-spam and authorization checks have been passed, they detect that the recipient is a mailing list, then hand the message over to the MLM.
That's it. That's pretty much all that an MTA is capable of at this level.Isthat *policy* really any different from your standpoint, from rejecting undeliverable messages during SMTP?
The policy you are proposing is null and void, because the software in question is not physically capable of acting in the manner you suggest.
Before chastising others so violently in public, you might do well to better learn how MTAs and MLMs really work, so that you can at least make accurate claims about what you believe that the software in question should be configured to do.
It would not be inconsistent with your current MLM policy. It is simply a better mechanism for implementing it, a better MTA policy, and better netizenship to boot.
Go back to basics, learn how MTAs and MLMs actually work.
Then you are free to come back and tell us what you think the right solution is.
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755LOPSA member since December 2005. See <http://www.lopsa.org/>.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Brad,
Do you really have a *policy* to accept messages that you will never deliver, save them to disk, and then generate reject messages for them?
How could the MTA possibly know that the MLM would choose to reject those messages because the sender is not a subscriber? Or that the MLM might choose to hold those messages for moderation, because the sender is not a subscriber.
Better MTAs can do this.
These are concepts that are totally beyond the scope of the MTA. There is nothing the MTA can do for you here.
It is much better to do this at the MTA level for the reasons I previously outlined:
- Do not accept stuff you will not deliver
- Do not send rejection messages to spoofed senders
- Do not annoy other postmasters (or more likely their automated blacklists) by inundating them with junk
- Do not be a backscatter source.
Or is that simply the way your MTA+MLM is configured by default?
As far as the MTA is concerned, they are *ALL* configured this way -- assuming that all the appropriate anti-spam and authorization checks have been passed, they detect that the recipient is a mailing list, then hand the message over to the MLM.
Of course they are all configured this way by default. Doesn't mean they should stay this way.
Isthat *policy* really any different from your standpoint, from rejecting undeliverable messages during SMTP?
The policy you are proposing is null and void, because the software in question is not physically capable of acting in the manner you suggest.
You have the sender and the recipient right at the beginning of the SMTP transaction. Check a map to see if it's deliverable. Mailman already maintains such a map. You just need some glue.
Before chastising others so violently in public,
Uh, "violently"?
you might do well to better learn how MTAs and MLMs really work, so that you can at least make accurate claims about what you believe that the software in question should be configured to do.
Check out:
http://www.postfix.org/SMTPD_POLICY_README.htmlfor instance. There are already policy servers written in Python, one could use as a starting place. I'll bet a Python hacker could write a mailman-subscriber-checker in very short order.
Surely other MTAs give admins a way to make decisions during SMTP as well?
It would not be inconsistent with your current MLM policy. It is simply a better mechanism for implementing it, a better MTA policy, and better netizenship to boot.
Go back to basics, learn how MTAs and MLMs actually work.
Then you are free to come back and tell us what you think the right solution is.
I think the right solution is to reject junk, instead of accept-and-bounce. I think accept-and-bounce is a blight on the internet, that lowers the quality and reputation of email. I think it is a waste of resources.
- H-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin)
iD8DBQFEB3fhOy/dHTCUq6oRAiBeAKD05nqBD2x6XM7aD+bvPsWmLRMnUQCgsciI fz37JE3NOB4aXs/A5SpfnSQ= =ra1X -----END PGP SIGNATURE-----
At 2:55 PM -0800 2006-03-02, Harold Paulson wrote:
How could the MTA possibly know that the MLM would choose to reject those messages because the sender is not a subscriber? Or that the MLM might choose to hold those messages for moderation, because the sender is not a subscriber.
Better MTAs can do this.
I've been specializing in Internet e-mail for over a decade, I was the comp.mail.sendmail FAQ maintainer for years, I was heavily involved in the postfix community from back when it was still being called VMailer, and I have yet to run into any MTAs that can do this.
Please provide evidence for your claims.The policy you are proposing is null and void, because the software in question is not physically capable of acting in the manner you suggest.
You have the sender and the recipient right at the beginning of the SMTP transaction. Check a map to see if it's deliverable. Mailman already maintains such a map. You just need some glue.
That would be nice, but Mailman does not, in fact, maintain such a map. It has a procedure that it goes through to determine if the user is a subscriber or should otherwise be moderated or the message should be rejected, but there is no easy "map" for this process.
Try checking the Mailman source code.Check out:
http://www.postfix.org/SMTPD_POLICY_README.html
for instance. There are already policy servers written in Python, one could use as a starting place. I'll bet a Python hacker could write a mailman-subscriber-checker in very short order.
Saying that it would be possible to write such a beast is not the same thing as saying that such things already exist, and that people should configure their software to use the pre-existing functionality.
That's a very cheap shot, frankly.Surely other MTAs give admins a way to make decisions during SMTP as well?
Sure, anyone can write an SMTP or LMTP-like engine that internally goes through the processes that the MLM would normally go through, and be able to provide that information to the MTA through a proxy service, but the MTA would also have to be configured to hold the sender open while checking the proxy -- again, most MTAs are not configured to do this, and for lots of good reasons the MTA authors suggest in the strongest possible terms that you not make the mistake of configuring their software in this way.
At the very least, if you hold open the sender like this, you run the serious risk of violating the "minimize acceptance delay" requirements of RFCs 1123 and 1047. Yes, I know these are really old RFCs, but they are just as valid today as they were on the day they were published, and RFCs 1122 and 1123 still comprise STD-0003, which is a required Internet standard.
I think the right solution is to reject junk, instead of accept-and-bounce. I think accept-and-bounce is a blight on the internet, that lowers the quality and reputation of email. I think it is a waste of resources.
Fine. Then write the Python code to do all the magic work and contribute that back to the Mailman community, and get that incorporated into an upcoming release.
Once you've done that, you'll have the right to encourage people to use it.
But again, this is an MLM problem, not an MTA problem. And you shouldn't be chastising people to make use of a supposedly pre-existing functionality that does not actually exist.
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755LOPSA member since December 2005. See <http://www.lopsa.org/>.
At 12:26 PM +0100 2006-03-03, Brad Knowles wrote:
I think the right solution is to reject junk, instead of accept-and-bounce. I think accept-and-bounce is a blight on the internet, that lowers the quality and reputation of email. I think it is a waste of resources.
Fine. Then write the Python code to do all the magic work and contribute that back to the Mailman community, and get that incorporated into an upcoming release.
Note that an SMTPD_POLICY daemon is not going to be enough. There are things that Mailman can do based on the content of the body, or based on what the anti-spam filtering system has done to the message, so having just the envelope sender and recipient information isn't enough.
To make this work right, what you're going to have to do is write an LMTP delivery interface for Mailman, so that you have Mailman go through all the appropriate steps of handling a message based on the envelope, header, and body content, meanwhile keeping the MTA open the entire time. Your alternative is to write an SMTP proxy interface where your code goes through all those same processes but then throws away the result after it passes the return code to the caller -- if you're going to go through all that work anyway, you might as well save the result and go ahead and deliver the message.
See <http://www.postfix.org/CONTENT_INSPECTION_README.html> and <http://www.postfix.org/SMTPD_PROXY_README.html> for examples of doing this as an SMTP proxy instead of an LMTP delivery interface, but note that all the performance issues are going to be the same. Let me quote:
external, medium-weight, real-time
This method inspects mail BEFORE it is stored in the queue,
and uses the SMTP protocol. Although this approach appears
to be the more attractive one, it really combines the worst
of the other two. Because mail is inspected before it is
queued, content inspection software must finish in a
limited amount of time, and must run in a limited amount
of memory. If content inspection needs too much time then
incoming mail deliveries will time out, and if content
inspection needs too much memory then software will crash
under a peak load. Before-queue inspection limits the peak
load that your system can handle, and limits the
sophistication of the content filter that you can use.
Details are in the SMTPD_PROXY_README document. This
approach is available only with Postfix version 2.1 and
later.
Moreover, an SMTPD_POLICY daemon would be a postfix-specific solution. An LMTP delivery engine would be a generally applicable solution which could be applied to any MTA that supports LMTP.
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755LOPSA member since December 2005. See <http://www.lopsa.org/>.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Brad,
I've been specializing in Internet e-mail for over a decade, I was
the comp.mail.sendmail FAQ maintainer for years, I was heavily
involved in the postfix community from back when it was still being
called VMailer, and I have yet to run into any MTAs that can do this.
My bad. I failed to check the archives:
http://mail.python.org/pipermail/mailman-users/2006-February/ 049346.html http://mail.python.org/pipermail/mailman-users/2005-July/045672.html http://mail.python.org/pipermail/mailman-users/2004-February/ 034534.html http://mail.python.org/pipermail/mailman-users/2003-July/029947.html
...for your resume first, and was suggesting a solution I thought was
both useful, and more...correct. I did meet Eric Allman once. He
seemed nice.
Please provide evidence for your claims.
You will have to forgive me because I have only been on the `net for
a minute or two, and never worked for AOL, but don't you sort of
water this down below and - at the perilous risk of putting words in
your mouth - agree that this is entirely possible, but, maybe a
little work?
The policy you are proposing is null and void, because the
software in question is not physically capable of acting in the manner
you suggest.You have the sender and the recipient right at the beginning of the SMTP transaction. Check a map to see if it's deliverable. Mailman already maintains such a map. You just need some glue.
That would be nice, but Mailman does not, in fact, maintain such a
map. It has a procedure that it goes through to determine if the
user is a subscriber or should otherwise be moderated or the
message should be rejected, but there is no easy "map" for this
process.
Ok, maybe it's not an MTA-ready map, but Mailman looks up the sender
in it's subscriber database when it gets a message to a members only
list, right? That code exists? And could be adapted, in a fairly
straight-forward manor to do exactly the same thing a few seconds
earlier, before your message hits disk?
Probably my misunderstanding.
Surely other MTAs give admins a way to make decisions during SMTP
as well?Sure, anyone can write an SMTP or LMTP-like engine that internally
goes through the processes that the MLM would normally go through,
and be able to provide that information to the MTA through a proxy
service, but the MTA would also have to be configured to hold the
sender open while checking the proxy -- again, most MTAs are not
configured to do this, and for lots of good reasons the MTA authors
suggest in the strongest possible terms that you not make the
mistake of configuring their software in this way.
Seems like you are imagining a lot of unnecessary plumbing.
When you have the sender address during SMTP If it's going to a known Mailman list address If the sender address can't be found in the list subscriber db Say no thanks...(or say please try again later)
Mailman will do all of this a second or two later anyway, right?
I think the right solution is to reject junk, instead of accept- and-bounce. I think accept-and-bounce is a blight on the internet, that
lowers the quality and reputation of email. I think it is a waste of
resources.Fine. Then write the Python code to do all the magic work and
contribute that back to the Mailman community, and get that
incorporated into an upcoming release.
Well you know of course I was trying to goad you into writing it. I
don't know Python, or have a problem with backscatter. :)
- H-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin)
iD8DBQFECIKHOy/dHTCUq6oRAiAKAKD4tiNXVX0RO9idCNMLk5+ni3AG/wCg5RhS U+Woiw8Nn0HVfXBqtrfmTxY= =W7Ix -----END PGP SIGNATURE-----
At 11:08 AM -0800 2006-03-03, Harold Paulson wrote:
...for your resume first, and was suggesting a solution I thought was both useful, and more...correct.
You were saying that a non-existant magic mechanism should be used for the MTA to know exactly how the MLM would react to a given message.
That's what I was shooting down.I did meet Eric Allman once. Heseemed nice.
I've met Eric on several occasions, and I like to think that I've had a pretty good relationship with him and the company -- since before the company existed.
Please provide evidence for your claims.
You will have to forgive me because I have only been on the `net for a minute or two, and never worked for AOL, but don't you sort of water this down below and - at the perilous risk of putting words in your mouth - agree that this is entirely possible, but, maybe a little work?
Not really. As I said, a policy daemon solution won't work very well -- you have to use an SMTPD_PROXY or LMTP solution instead.
In either case, you don't have a pre-existing magic mechanism where the MTA automatically knows precisely how the MLM will react to a given message. Instead, you've got a lot more work ahead of you to make use of a mechanism that MTAs may make available, but which has never existed for Mailman.
That would be nice, but Mailman does not, in fact, maintain such a map. It has a procedure that it goes through to determine if the user is a subscriber or should otherwise be moderated or the message should be rejected, but there is no easy "map" for this process.
Ok, maybe it's not an MTA-ready map, but Mailman looks up the sender in it's subscriber database when it gets a message to a members only list, right?
That's a small part of the process, yes. There are lots of other parts that you are ignoring.
That code exists? And could be adapted, in a fairlystraight-forward manor to do exactly the same thing a few seconds earlier, before your message hits disk?
Could be adapted -- yes.
Fairly straightforward manner? I don't know.
I'm not a programmer, but I think it would probably be "Non-Trivial" to adapt that code. However, I would be happy to be proven wrong.
Sure, anyone can write an SMTP or LMTP-like engine that internally goes through the processes that the MLM would normally go through, and be able to provide that information to the MTA through a proxy service, but the MTA would also have to be configured to hold the sender open while checking the proxy -- again, most MTAs are not configured to do this, and for lots of good reasons the MTA authors suggest in the strongest possible terms that you not make the mistake of configuring their software in this way.
Seems like you are imagining a lot of unnecessary plumbing.
Demonstrate to me that you have a level of understanding of the problem equal to Eric Allman, Wietse Venema, Barry Warsaw, Tokio Kikuchi, or Mark Sapiro, and I might be willing to believe you.
Otherwise, you're going to have to prove it to me by actually doing it. Then tell me what you did, and how easy it was.
When you have the sender address during SMTP If it's going to a known Mailman list address If the sender address can't be found in the list subscriber db Say no thanks...(or say please try again later)
Mailman will do all of this a second or two later anyway, right?
Again, the connection would have been dropped by then. That's why you have to cook up a complete LMTP interface for Mailman.
Well you know of course I was trying to goad you into writing it. I don't know Python, or have a problem with backscatter. :)
I'm not a programmer. That's something I've said more than a few times over.
I do have a fairly deep operational understanding of Internet e-mail and mailing list management systems, and I can give you plenty of tips and pointers of things you might need to do if you were going to try to build a Yahoo! or AOL-size mail system, or a mailing list management system of that scale.
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755LOPSA member since December 2005. See <http://www.lopsa.org/>.
"Harold" == Harold Paulson <haroldp@sierraweb.com> writes:
Harold> Do you really have a *policy* to accept messages that you
Harold> will never deliver, save them to disk, and then generate
Harold> reject messages for them?As I read his post, indeed he does. He wants legitimate posters to be told they need to sign up for the list before they can post. It's true that you can supply a reason for an MTA rejection, but (in my experience) it is unfortunately also true that many originating MTAs are not configured to report that reason. Others will substitute a stock text for the numerical code (especially in cases of internationalization). The most reliable way to give those legitimate users full information is for Mailman to create and send a bounce mail.
Because of the specific issue with Yahoo, he "deeply regrets that we must make a special exception for Yahoo."
When implementing that policy exception for Yahoo, it would make sense to do so at the MTA level, as you suggest, if his MTA provides the hooks and he can write the needed glue code. (It's a bit much to call that exercise "configuration", IMO.)
-- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Stephen,
On Mar 2, 2006, at 9:48 PM, Stephen J. Turnbull wrote:
"Harold" == Harold Paulson <haroldp@sierraweb.com> writes:
Harold> Do you really have a *policy* to accept messages that you Harold> will never deliver, save them to disk, and then generate Harold> reject messages for them?As I read his post, indeed he does. He wants legitimate posters to be told they need to sign up for the list before they can post. It's true that you can supply a reason for an MTA rejection, but (in my experience) it is unfortunately also true that many originating MTAs are not configured to report that reason. Others will substitute a stock text for the numerical code (especially in cases of internationalization). The most reliable way to give those legitimate users full information is for Mailman to create and send a bounce mail.
Ah, this is a good criticism: other MTA's bounce messages can't be
trusted to be readable by the sender. Mailman after all, is sending
a notice, not exactly generating a bounce. A better policy-server
might greylist non-member senders. 95% of the spammers will go
away. All of the normal posts would go right through. Members who
send from the wrong account would get the usual Mailman notice.
Occasionally wrong-account-posters would have to wait a long time on
their notice.
Because of the specific issue with Yahoo, he "deeply regrets that we must make a special exception for Yahoo."
When implementing that policy exception for Yahoo, it would make sense to do so at the MTA level, as you suggest, if his MTA provides the hooks and he can write the needed glue code. (It's a bit much to call that exercise "configuration", IMO.)
Yeah, I just bristle at a hack for one destination based solely on
it's size, when this is a general problem. It could be that other
ISPs blacklist servers based on the ratio of valid:invalid addresses
that you send their way (AOL does this). We just notice Yahoo or AOL
because they are big, and our phones start ringing when they
blacklist us.
This treats a symptom, not the problem.
The problem is that we are making net.messes by automatically
replying to junk. It would be nice to see a general fix.
- H-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin)
iD4DBQFEB+JFOy/dHTCUq6oRAnonAKCbxFOCNY4UXfHHugXrp8aVKzQRUgCYgcv1 bbAo8RhAMAotNPoHtljqUA== =2gHo -----END PGP SIGNATURE-----
"Harold" == Harold Paulson <haroldp@sierraweb.com> writes:
Harold> The problem is that we are making net.messes by
Harold> automatically replying to junk. It would be nice to see a
Harold> general fix.Well, there isn't one.[1] Take the case in point. Your solution is also a hack in that it depends on having an explicit whitelist.
Of course it's an excellent hack! It works in a *lot* of cases of interest to MLM admins, but in principle you don't want to keep whitelists (well, my Mom can spam me :-) or blacklists. It's content that you want to filter on (or out).
Footnotes: [1] Well, you could have a "bonded mail" solution: every message arrives with a PayPal credit, and the reader either waives it or the sender pays. My sister gets a permanent waiver, the taxman and the "performance enhancement" salesmen can pay up. ;-) Problem is, that's too easy to corrupt into a Goodmail solution where the ISP collects the gelt. There's also Hashcash, but that's not a terribly happy solution for mailing lists.
-- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.
Harold Paulson wrote:
away. All of the normal posts would go right through. Members who
send from the wrong account would get the usual Mailman notice.
Occasionally wrong-account-posters would have to wait a long time on
their notice.
For the sender, there is a very simple solution to "whitelist" his or her own e-mail addresses which I have used myself: subscribe the other e-mail address then set "Mail delivery" to "Disabled." And if it's that important to the sender that his or her messages get through, then set "Receive acknowledgment" to "Yes." In many ways, I think that is a better solution than expressly whitelisting addresses elsewhere.
Jonathan
On Fri, Mar 03, 2006 at 06:46:28AM -0500, Jonathan Dill wrote:
For the sender, there is a very simple solution to "whitelist" his or her own e-mail addresses which I have used myself: subscribe the other e-mail address then set "Mail delivery" to "Disabled."
Or the admin can add these additional addresses to accept_these_nonmembers.
The users who need this done generally don't even realize their address changed, and don't follow directions well enough to whitelist themselves. So my reject message for non-members says, "If you think you are a member of this list, email the list owner and he'll fix it."
BTW, irony of ironies, Yahoo Groups sends rejection messages to non-members. What's sauce for the goose is poison for the gander.
-- greg
participants (5)
-
Brad Knowles -
Greg Lindahl -
Harold Paulson -
Jonathan Dill -
Stephen J. Turnbull