On 2006-04-27 at 22:46-05 Brad Knowles brad@stop.mail-abuse.org wrote:
At 10:40 AM -0500 2006-04-27, Neal Groothuis wrote:
Again from RFC 2822 3.6.2, the Sender: header should contain the address of the agent responsible for transmitting the message, meaning that a person who sends mail to the address in that header should expect to reach said agent, not suggest to Mailman that a message bounced.
Right. Mailman is the agent responsible for transmitting the message, and this needs to be reflected.
This is not correct. Quoting RFC2822 section 3.6.2:
3.6.2. Originator fields
The originator fields indicate the mailbox(es) of the source of
the message. The "From:" field specifies the author(s) of the
message, that is, the mailbox(es) of the person(s) or system(s)
responsible for the writing of the message. The "Sender:" field
specifies the mailbox of the agent responsible for the actual
transmission of the message. For example, if a secretary were to
send a message for another person, the mailbox of the secretary
would appear in the "Sender:" field and the mailbox of the actual
author would appear in the "From:" field.The Sender header should be employed by the orignator of the message, and only the originator. Mailman is not the originator of a message sent to a list; it is merely a relay agent.
I will grant that the phrasing of the RFC is suboptimal here--it uses "transmission" when a better word choice would have been "submission". But the immediately proceeding example (of a secretary sending mail on behalf of another person) clarifies the intent beyond any claim of ambiguity.
[Outlook's behavior] is an MUA problem. See FAQ 2.3.
No, it's not. As much as it pains me to say it, Outlook's behavior matches perfectly the intent of the RFC. It is Mailman's behavior of rewriting the Sender header that is the problem.
However, we want to make sure to capture any potential messages that may be routed to the "Sender:" field and have them automatically processed through the part of the system that is designed to do that sort of thing.
Mailman's "processing" behavior is to treat a reply to the Sender as a bounce. This is incorrect behavior, because many mail clients will include address of the Sender header in a "reply-to-all" function, causing Mailman to treat the reply as a bounce.
So, in summary, the disadvantages of Mailman's behavior of rewriting the Sender header is that doing so is not in the intended spirit of RFC2822, causes subscription grief, and breaks Outlook. The advantage is that it helps Mailman detect bounces from a slim minority of brain-dead MTAs that send bounces to the Sender header.
The problem is that you said you wanted to implement an option to allow people to turn it off, not to rip this feature completely out of the system.
I would argue that the best course of action is to excise Sender header rewriting entirely and provide no option to turn it on. (Mailman has way too many options already.)
If, however, an option is created to control the behavior, it should definitely default to OFF (no Sender header rewriting), not on.
-- James Ralston, Information Technology Software Engineering Institute Carnegie Mellon University, Pittsburgh, PA, USA
At 5:32 PM -0400 2006-04-28, James Ralston wrote:
I will grant that the phrasing of the RFC is suboptimal here--it uses "transmission" when a better word choice would have been "submission". But the immediately proceeding example (of a secretary sending mail on behalf of another person) clarifies the intent beyond any claim of ambiguity.
I read through all of section 3.4 pretty carefully. Unfortunately, I haven't been able to find any references to automated agents acting on behalf of other parties. It is not exactly clear to me if a "relay agent" should be considered as a "user" (for which examples are given), or if it should be treated in some other manner.
There are cases of "forwarding" mentioned where it is not appropriate to make any modifications to any headers, but there are other cases where modifications are acceptable or even recommended.
Additional clarification would be very useful.So, in summary, the disadvantages of Mailman's behavior of rewriting the Sender header is that doing so is not in the intended spirit of RFC2822, causes subscription grief, and breaks Outlook. The advantage is that it helps Mailman detect bounces from a slim minority of brain-dead MTAs that send bounces to the Sender header.
On the whole, I'm coming around to the view that Mailman should be using "Resent-" headers for the things it would like to add to the message (e.g., Resent-From:, Resent-Date:, Resent-Message-ID:, etc...), as well as the standard "List-" headers.
However, it is not clear to me what the impact on the user community would be.
Yes, some users would stop seeing things that are confusing them, because we would stop rewriting/replacing the "Sender:" field. Well-known MTAs would not have any problems with these changes, but I do have to wonder what the negative impact would be from less common MTAs, not to mention all the different MUAs that are out there.
I would argue that the best course of action is to excise Sender header rewriting entirely and provide no option to turn it on. (Mailman has way too many options already.)
If, however, an option is created to control the behavior, it should definitely default to OFF (no Sender header rewriting), not on.
Right now, I'm probably somewhere around 75% convinced that we should not be rewriting/replacing the "Sender:" header, and should be using appropriate "Resent-" headers instead. At least, in the ideal world.
However, In the real world, I am much less convinced that we should be making a radical change like this, at least not without a lot more experience in how things are actually impacted.
I would support adding code to the system to make this change optional and to initially default to the old behaviour (allowing people who want to play with this option to do so), then in a future version to default to the proposed new behaviour (but still giving people the option to go back to the old method), and then finally to removing the option to revert to the old behaviour at some distant point in the future.
But I definitely would not support just ripping out the offending code.-- Brad Knowles, brad@stop.mail-abuse.org
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755LOPSA member since December 2005. See http://www.lopsa.org/.
On Fri, 2006-04-28 at 17:32 -0400, James Ralston wrote:
I would argue that the best course of action is to excise Sender header rewriting entirely and provide no option to turn it on. (Mailman has way too many options already.)
I agree that this is what we should hope for. Our data supporting the rewriting of Sender is many years out of date, so we need to gather more data. Who's brave enough to try it? :)
-Barry
"BAW" == Barry Warsaw barry@python.org writes:
BAW> I agree that this is what we should hope for. Our data
BAW> supporting the rewriting of Sender is many years out of date,
BAW> so we need to gather more data. Who's brave enough to try
BAW> it? :)If this can be done by modifying the list-specific pipeline, you got a volunteer. I'm willing to volunteer XEmacs's lists---our users can deal with it, and if need be I can revert without much lossage. We share the MM installation with others, though, so it has to be something done by substituting a modified Handler for our lists only.
You'll also need to tell me what I should be looking for; I have had no need to pay much attention to MM bounces for several years.
-- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.
Stephen J. Turnbull wrote:
If this can be done by modifying the list-specific pipeline, you got a volunteer. I'm willing to volunteer XEmacs's lists---our users can deal with it, and if need be I can revert without much lossage. We share the MM installation with others, though, so it has to be something done by substituting a modified Handler for our lists only.
This can be done by making a modified SMTPDirect.py and modifying the pipeline for the XEmacs lists. I'm pretty sure you know how to do this, but after I was recently reminded (by you I think - thank you) that this is a good way to do lots of things, I wrote a FAQ on it http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.067.htp. The specific changes to SMTPDirect.py would be at the beginning of bulkdeliver().
You'll also need to tell me what I should be looking for; I have had no need to pay much attention to MM bounces for several years.
You would be looking for bounces being returned to inappropriate places such as the original poster or the list itself. Thus, list members would have to be aware of the fact that there was a change and that they should try to observe and report any change in received bounces or other autoresponses to their posts. It might be tricky to filter out actual change from perceived change due to being more aware.
-- Mark Sapiro msapiro@value.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Sat, 2006-04-29 at 09:03 -0700, Mark Sapiro wrote:
This can be done by making a modified SMTPDirect.py and modifying the pipeline for the XEmacs lists. I'm pretty sure you know how to do this, but after I was recently reminded (by you I think - thank you) that this is a good way to do lots of things, I wrote a FAQ on it http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.067.htp. The specific changes to SMTPDirect.py would be at the beginning of bulkdeliver().
Nice entry Mark, thanks. I'll add two things:
* You could still add a handler to the global pipeline, but have
it do a test of the listname or domain first, and if it doesn't
match, do a quick return. You pay the cost of the handler for
every mailing list, but it should be fairly minimal in the
scheme of things
* We should probably define a more convenient way to extend the
pipeline for specific lists or domains (IOW, MM2.2 should have a
more useful hook mechanism).You would be looking for bounces being returned to inappropriate places such as the original poster or the list itself. Thus, list members would have to be aware of the fact that there was a change and that they should try to observe and report any change in received bounces or other autoresponses to their posts. It might be tricky to filter out actual change from perceived change due to being more aware.
Yep. -Barry
"BAW" == Barry Warsaw barry@python.org writes:
* You could still add a handler to the global pipeline, but have
it do a test of the listname or domain first, and if it doesn't
match, do a quick return. You pay the cost of the handler for
every mailing list, but it should be fairly minimal in the
scheme of thingsI once added a maybe-log-and-return handler at every point in the pipeline on our busiest list. No effect on load. I dunno, if you're handling a million posts a day you might notice that. But handler calling is way down in the scheme of CPU suckers compared to spam and virus filtering.
I'm unwilling to add to the global pipeline simply because there's no such thing as a 100% safe edit, not because I'm worried about function call overhead in Python.
-- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.
On Tue, 2006-05-02 at 02:56 +0900, Stephen J. Turnbull wrote:
I once added a maybe-log-and-return handler at every point in the pipeline on our busiest list. No effect on load. I dunno, if you're handling a million posts a day you might notice that. But handler calling is way down in the scheme of CPU suckers compared to spam and virus filtering.
True.
I'm unwilling to add to the global pipeline simply because there's no such thing as a 100% safe edit, not because I'm worried about function call overhead in Python.
Fair enough. Probably going with the extend.py stuff is your best bet at this point.
-Barry
On 2006-04-28 at 19:54-04 Barry Warsaw barry@python.org wrote:
On Fri, 2006-04-28 at 17:32 -0400, James Ralston wrote:
I would argue that the best course of action is to excise Sender header rewriting entirely and provide no option to turn it on. (Mailman has way too many options already.)
I agree that this is what we should hope for. Our data supporting the rewriting of Sender is many years out of date, so we need to gather more data. Who's brave enough to try it? :)
We're going to implement it, because we really don't have any choice. :/
At our site, we have around 550 lists and 15,700 subscribers in total. We process (send) around 16,708 messages per day. Our list owners have discovered that a non-trivial number of the recipients on our lists have mail clients that include the Sender address in the reply-to-all feature. As a result, we have a continual stream of subscribers who are accidentally hosing their subscriptions (causing them to be suspended or removed) by accidentally replying to Mailman's bounce processing address in the Sender header.
This problem is so bad for us that some list owners are demanding that we abandon Mailman in favor of some other list manager. (They're also incensed about Mailman's bounce processing options, but that's another topic.)
Since disabling the Sender header processing is literally a one-line patch, we're going to go ahead and do it. If it breaks horribly (even for a small percentage of users) we'll know; if it makes our users happy, we'll know that too.
James
On Mon, 2006-05-01 at 17:09 -0400, James Ralston wrote:
We're going to implement it, because we really don't have any choice. :/
...and later...
Since disabling the Sender header processing is literally a one-line patch, we're going to go ahead and do it. If it breaks horribly (even for a small percentage of users) we'll know; if it makes our users happy, we'll know that too.
Definitely let us know how it goes!
This problem is so bad for us that some list owners are demanding that we abandon Mailman in favor of some other list manager. (They're also incensed about Mailman's bounce processing options, but that's another topic.)
Please do start a new thread with your feedback here. Fixing the bounce processor is definitely one of the things I want to do for MM2.2.
-Barry
On 2006-05-01 at 17:19-04 Barry Warsaw barry@python.org wrote:
Since disabling the Sender header processing is literally a one-line patch, we're going to go ahead and do it. If it breaks horribly (even for a small percentage of users) we'll know; if it makes our users happy, we'll know that too.
Definitely let us know how it goes!
We haven't rolled it out yet (because we want to do at least some cursory testing first), but just FYI, I've attached the patch we came up with. (I'm not sure if Mailman will eat it; if not, I'll resend it inline.)
This problem is so bad for us that some list owners are demanding that we abandon Mailman in favor of some other list manager. ([Our list owners are] also incensed about Mailman's bounce processing options, but that's another topic.)
Please do start a new thread with your feedback here. Fixing the bounce processor is definitely one of the things I want to do for MM2.2.
Will do.
James
James Ralston wrote:
As a result, we have a continual stream of subscribers who are accidentally hosing their subscriptions (causing them to be suspended or removed) by accidentally replying to Mailman's bounce processing address in the Sender header.
This should only be happening if Mailman's VERP delivery is enabled so the Sender is "list-bounces+user=users.domain@lists.domain. In the non-VERP case, a post or reply should be an 'unrecognozed bounce' - still a major annoyance for the admin who elects to see unrecognized bounces, but at least not a problem that disables the user.
-- Mark Sapiro msapiro@value.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
As a result, we have a continual stream of subscribers who are accidentally hosing their subscriptions (causing them to be suspended or removed) by accidentally replying to Mailman's bounce processing address in the Sender header.
Curious: are your lists set up such that replies go to the list, or recipient?
Bob
"James" == James Ralston qralston+ml.mailman-developers@andrew.cmu.edu writes:
James> The Sender header should be employed by the orignator of
James> the message, and only the originator. Mailman is not the
James> originator of a message sent to a list;OK up to here.
James> it is merely a relay agent.That is false. Mailman edits the messages, both adding and removing body content, and compiles the message for digests. It may alter or even remove some original headers. The semantics of "originator" in the context of a mailing list are quite unclear to me, and there is not yet general agreement, as is evident from the long history of wrangling over Reply-To munging, Mail-Followups-To, and the like.
I think the "best" behavior would be to consider that if the body of the message is included unchanged (although header or footer may be added), the originator of the message is the author or the author's personal agent (original Sender). In a digest, Mailman is pretty clearly the Sender (since there must be at most one Sender mailbox).
In cases where HTML and/or MIME bodies are stripped, it's a tossup. Or would be, if we didn't have "Resent-*" headers.
Anybody know if Outlook groks Resent-* headers?
James> Mailman's "processing" behavior is to treat a reply to the
James> Sender as a bounce. This is incorrect behavior, because
James> many mail clients will include address of the Sender header
James> in a "reply-to-all" function, causing Mailman to treat the
James> reply as a bounce.s/incorrect/impractical/. Those clients are broken. If Mailman sends an RFC 1153 digest, it *must* be the Sender, and the individual messages presumably won't have them. Secretaries who are privy to their bosses' mail are reading it at the bosses address; those who aren't, should not be getting copies of replies to their boss. Etc, etc. Replying to Sender is dumb.
James> I would argue that the best course of action is to excise
James> Sender header rewriting entirely and provide no option to
James> turn it on. (Mailman has way too many options already.)Only if Mailman is taught to add a full complement of Resent-* headers.
Note that use of Resent-* headers has the serendipitous effect of providing an easily available UUID for naming messages when archiving (the content of the Resent-Message-Id field). If there were a List-Archive-Relative-URL header it could be copied there.
-- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.
At 11:15 PM +0900 2006-04-29, Stephen J. Turnbull wrote:
If Mailman sendsan RFC 1153 digest, it *must* be the Sender, and the individual messages presumably won't have them.
Actually, in the case of a digest, Mailman would be the originator of the message, and put its address in the "From:" field, leaving the "Sender:" field blank. The individual messages in the digest would have their own "From:" and possibly "Sender:" fields, but those would not be promoted to the digest itself.
There's just no way you can do digests using any other method.
But, most of what we've been talking about so far has to do with regular list activity with regards to individual messages, and not digests.
-- Brad Knowles, brad@stop.mail-abuse.org
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755LOPSA member since December 2005. See http://www.lopsa.org/.
participants (6)
-
Barry Warsaw -
Bob Puff -
Brad Knowles -
James Ralston -
Mark Sapiro -
Stephen J. Turnbull