On Thu, 2006-04-27 at 22:46 -0500, Brad Knowles wrote:

If the previous value of the "Sender:" field is being lost, then that should be corrected. At the very least, the value should be saved in an "Old-Sender:" or "Previous-Sender:" or some other suitable renamed sender field.

Probably Original-Sender:

-Barry

John W. Baxter wrote:

Probably, indeed. But what happens if that header was already "taken" in the process that brought the message to mailman for distribution to the list?

As I noted in my previous response, I believe that the correct field (if Mailman were to add a "Sender:" header) to add would be "Resent-Sender". Please see RFC 2822, section 3.6.6. The "Resent-Sender" field may be multivalued, so this isn't a problem. However, Mailman should not be modifying the Sender: field at all.

"Original-Sender" is a header used when gatewaying X.400 messages into RFC 822 messages for use in JNT mail networks. It would not be appropriate for use here.

At 7:50 PM -0400 2006-04-28, Barry Warsaw wrote:

On Fri, 2006-04-28 at 13:05 -0500, Neal Groothuis wrote:

...

As I noted in my previous response, I believe that the correct field (if Mailman were to add a "Sender:" header) to add would be "Resent-Sender". Please see RFC 2822, section 3.6.6.

Whatever else we decide, I don't agree, or at least, it won't help us. $3.6.6 says that Resent-* headers are to be added by a user. It also says that these are purely informational headers, so I don't see how adding them will instruct a receiving MTA usefully.

Siunce the RFC doesn't specifically talk about "relay agents" as 

separate from "users", I think we could argue that Mailman would qualify as a "user" in this context. Therefore, the Resent-* headers seem to be most appropriate.

But you are correct that these are purely informational and will 

be completely ignored by any MTA. If we need something that will be noticed by other MTAs beyond the envelope sender and the "Return-Path:" & "Errors-To:" headers, then we're going to have to carefully think about this.

I am still opposed to blindly making this change and letting the 

community find out what happens.

I think we need to gather a lot more information about the likely 

outcome from this change, and I think the best way to achieve this is through giving admins (either site admins or list admins) the ability to set an option and choose whether or not they want to see what happens.

-- Brad Knowles, <brad@stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

LOPSA member since December 2005. See <http://www.lopsa.org/>.

At 12:00 AM +0900 2006-04-30, Stephen J. Turnbull wrote:

 Brad> If we need something that will be noticed by other MTAs
 Brad> beyond the envelope sender and the "Return-Path:" &
 Brad> "Errors-To:" headers, then we're going to have to carefully
 Brad> think about this.

What's an Errors-To header? I can't find it in the usual suspects.

That's the oldest technique for handling bounces.  It has been 

deprecated for a while, but I would be inclined to continue to at least provide the appropriate information.

-- Brad Knowles, <brad@stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

LOPSA member since December 2005. See <http://www.lopsa.org/>.

On Sun, 2006-04-30 at 00:00 +0900, Stephen J. Turnbull wrote:

Sender doesn't instruct *conformant* MTAs at all, does it? AFAIK the only thing that a RFC 2821-conforming MTA looks at is the Return-Path header, and it's supposed to remove that.

So this is purely a matter of pragmatic self-defense against broken MTAs that do bounce to Sender.

Correct, and what we're trying to figure out is whether we need that self-defense any longer. The change to test this may be as simple as commenting out "msg['Sender'] = envsender" in bulkdeliver() inside SMTPDirect.py (a little more complicated if you want to do it just for one domain though -- you'd want to test for something like "if 'xemacs.org' in mlist.host_name")

Agreed. For a number of reasons, I think this information can be useful. As I mentioned elsewhere, the Resent-Message-Id field can be used to supply a UUID that we can trust (eg, for constructing canonical archive URLs). Unlike the Received headers, these are readable by humans who aren't wall-eyed, helpful in tracing delays, for example.

It's an intersting idea.

-Barry

I'd like to work up an unofficial diff to Mailman 2.1 for people like Stephen who are willing to give it a try on a live site.

I'm not sure this is even necessary.

Ezmlm doesn't touch the Sender: header at all, Majordomo sets it to the owner of the list, and (AFAICT) Listserv sets it to the list itself. This would seem to me to indicate that incidences of mail being returned inappropriately to the sender are infrequent, at worst.

The important question would seem to be "what's appropriate?" From RFC 2822, 3.6.2: "The originator fields indicate the mailbox(es) of the source of the message." Given this, the definition of the Sender and From headers, and the example given in this section, it seems that Outlook's interpretation of the fields ("SENDER on behalf of FROM") is reasonable. Mailman is not the originator of the message, so it should not be tampering with the From: or Sender: fields at all.

It might be appropriate for Mailman to add Resent-* headers, depending on how one reads RFC 2822, 3.6.6. I personally don't think it's necessary or useful, since list servers add their own List-* headers, per RFC 2369. The Resent-* headers seem to exist for individuals resending messages, not automated systems. This is supported by the RFC: "Resent fields are used to identify a message as having been reintroduced into the transport system by a user."

...

...

...

...

"William" == William D Tallman <wtallman@olypen.com> writes:

William> How does the RFC, or the writers thereof, define "user"?

They don't. IMHO (there are those more expert than I on this list) anything that is normally expected to touch the headers or body of a message is a "user" for the purpose of RFC 2822. What is excluded is the mail transport system (MTAs) which are specified in RFC 2821.

There is also a distinction between "originators" and "others". Certain headers ("From", "Subject", "Date", etc) are specified as for the use of the originator. Other headers are generally specified in terms of their semantics alone, and not according to who may use them.

William> An automated system is the tool of some deliberate
William> intent, implying (necessarily?)  the will of a "user", I
William> would think.

I don't think that is the way that RFC writers in general think.

Although there is a desire for RFC 2822 headers to be intelligible to humans, and many are very useful in day-to-day work, RFCs are in the end about machine-to-machine communication. Thus the focus is on syntax so that machines can parse them without knowing what they mean, and of semantics that allow machines to make a good guess at what the humans are going to want.

For example, if there is a Reply-To header, the From header should be ignored, and the Reply-To address used in composing the addressee list for a reply. However, one of the examples used IIRC is where the author of the original message is traveling and uses his own address as From (that's the identity the recipient recognizes) but Reply-To to direct the response to his host, whose email address he is borrowing. Now, a human who replies a week later will know that the boss is back home and want to reply to From but the mail client can't know that.

So a good mail client will initialize the address of a reply to the Reply-To, but provide a way for the user to override. The RFC only specifies the former, though, and only as a suggestion. Exactly how to handle this problem is a user interface issue, and the RFC remains silent on such issues.

William> Or is this relevant?

Yes. Sometimes such definitions are made explicitly. I don't think they exist in this case, but it's a very good idea to ask.

Disclaimer: this is the way I think about these things, and I've found it useful in understanding what RFCs do and don't say. Others will surely have different opinions.

-- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

...

...

...

...

"William" == William D Tallman <wtallman@olypen.com> writes:

William> On Wed, May 03, 2006 at 11:11:22PM +0900, Stephen
William> J. Turnbull wrote:

>> I don't think that is the way that RFC writers in general
>> think.

William> Yes, so I gather.

:-)

William> Which means that people really should learn to use email
William> systems intelligently, with the MUA of choice as the
William> means of control.

I firmly believe that, but unfortunately there are lots of MUAs that don't really permit intelligent use. Many people "inherit" an MUA either from their OS or maybe their brother-in-law, and do not have the desire or resources to change MUAs or even learn to use the one they've got effectively.

There are a number of ways to look at this, but what the people who write RFCs have come to insist is that you be strict with yourself (always follow the rules and best practices), while being lenient with others. You can think of this as simply aikido on the Internet (== self-defense), or some kind of generosity to those less clued than oneself, but the rule works well whyever you follow it. :-)

>> So a good mail client will initialize the address of a reply to
>> the Reply-To, but provide a way for the user to override.  The
>> RFC only specifies the former, though, and only as a
>> suggestion.  Exactly how to handle this problem is a user
>> interface issue, and the RFC remains silent on such issues.

William> Implication here is that 'user' is a real human being,
William> not a software agent.

In this particular case, user refers to "user of a good mail client", presumably human (but it could easily be an Emacs Lisp program or an expect script ... ok, ok, that's not "easy", that's "heroic" ... but it could be heroicly!) However, most of RFC 2822 doesn't refer to how the headers should be treated by a mail client, just to what they mean. That meaning could be useful to a human, or a mailing list server, or whatever.

William> RFC2822, section 3.6.6 discusses re-sending fields as
William> intended for use by a re-sending 'user'.  It also
William> specifies that these fields are for informational
William> purposes only, and MUST NOT be used to actively
William> manipulate the message.

"Automatically." There's nothing that says that you can't write a mail client that has an "bounce followup" feature which looks for sender, resent-sender and so on, and adds them to the "To" header, as well as formatting the body with a summary of the progress of the message by using Date and Resent-Date headers.

William> So an email list server cannot act as a re-sender, IIUC.

I don't see why not. I think you're overinterpreting the RFCs. Certainly, in this case "human user" is a leading interpretation. But if the actions described could be executed by a program, then there's no good reason not to interpret "user" as possibly being a program, unless the RFC explicitly says so.

William> The alternative is that the server actually initiates a
William> new message as a 'forwarding' agent.

I don't think either of the meanings of "forward" suggested in RFC 2822 section 3.6.6 apply here. ("New message with old message as body" clearly applies to digests, but I think we're more interested in non-digest delivery in this thread.)

[William gives an example] William> That means that the server must (MUST?) identify itself William> in the originator fields.

No, I think that's wrong. If the server wants to claim responsibility for injecting the message into the mail system, then it should put itself in the Sender field. This absolves the original Sender of all responsibility for misformatting of the message, misdirection to wrong addresses, etc, etc. If the server doesn't change the body at all, and only adds new headers, then I think it should not do this. In the grey areas like Mailman, it's unclear.

However, suppose Mailman is configured to leave the headers alone, and to leave the body alone, forwarding verbatim to the addresses on the mailing list (except for adding its List-XXXX headers, etc). I would argue that since Mailman doesn't automatically forward, but instead checks for spam, whether you're subscribed or not, whether the subscriber is already an addressee in the message, etc., it makes decisions about what to send where, and is therefore a "user" in the sense of section 3.6.6.

Mailman SHOULD add Resent-XXXX headers, because if delivery gets screwed up, bugs in its logic should be considered a candidate cause. Ie, those headers mean that Mailman accepts partial responsibility for misdirecting the reinjection of the message into the mail transport system.

For example, suppose Mailman hiccups and reinjects the same post twice. It would be useful to check whether the Resent-Message-Ids differ. If they do, you know for sure it was Mailman. If they don't, it doesn't prove it wasn't Mailman, but you do know that the phase where the error occurred was fairly late in Mailman's delivery pipeline.

William> IIUC, that is.  Which apparently I do not, having read
William> through the headers of a message from this list.

While I respect the Mailman authors for their efforts to conform to the RFCs, nobody's perfect. Specifically there is controversy about whether Mailman's treatment of the Sender field is conforming to RFCs. There may be other errors (I don't know of any, but I don't know everything :-).

William> There is no Sender: field.

This implies that whoever is in From is Sender.

William> The first field is apparently an unstructured field with
William> no identifier with the canonical following colon.

This was added by *your* MTA when it delivered to your mailbox. It was not present at any time before the message reached your host. This is a non-standard (but very frequently used) format for mailboxes called "Unix mbox".

Here "non-standard" does not mean "does not conform to the standard" for some value of "the". It means "this is neither defined nor prohibited by any formal standard."

William> It contains the sender (mailman-users-bounces...) and the
William> date, presumably of sending.

The date is a timestamp added when the message was delivered to your mailbox.

William> The second field is Return-Path: with an 'addr-spec'.

This was also added by your MTA. This is a standard header, specified by RFC 2821 (the RFC for the protocol spoken by one MTA to another).

William> The originator fields are untouched.

This is not always true. Mailman will sometimes alter the From header to preserve privacy, IIRC.

William> Which means that the list server is neither a re-sender
William> or a forwarder,

Neither of these have a definition in RFC 2822 (the RFC for the protocol spoken by MUAs, to each other and to human users). There are two suggested interpretations for forwarder, and nothing very specific about resender, except that it is not the kind of thing the MTAs do.

Note that although technically when Mailman adds a footer it has created a new message containing a copy of the original, I doubt *people* think of it that way. As long as there's a complete, verbatim copy of the original, the header and footer are considered decoration, not part of the message. Even if HTML parts or images are removed, I think *people* still consider the message "the same". So I don't think these operations necessarily disqualify Mailman as a resender. (If you convince me that people will often consider these operations to create a *new* message, then I'd have to rethink.)

As far as I can tell, a re-sender is any agent that (1) receives a message, (2) sends it elsewhere, and (3) adds Resent-XXXX headers to the resent message.

You could argue that there's a condition that Mailman, as the resender, be a "bona fide addressee" of the message. Well, (1) it's the envelope recipient and probably in the addressee headers as well, and (2) Mailman often keeps file copies (AKA mailing list archives). You could claim that (1) is just an accident of technology, but I don't think you can explain away (2).

To put all this another way, the only intermediary actually defined by these RFCs is the "mail transport agent" (MTA), which accepts a sender address, a list of recipients, and message data from another agent, and returns to that agent a promise to deliver. It then contacts other MTAs that it believes represent some of those recipients, and passes on the list of (a subset of) recipients, the sender address and the message data, or delivers the message data verbatim to mailboxes it has responsibility for. Once it has received promises to deliver for all of the recipients on the list it received, it is done, and can purge the message data and envelope information from its queues. This is not what Mailman does, so it's a "user".

Note that a quick search of the RFC site for "delivery agent" or "MDA" turns up nothing relevant, and a search for "mailing list" turns up only RFCs 2369 and 2919 (describing the List-XXXX headers, but not the kind of thing we're talking about) and RFC 1429 which describes an obsolete service called DISTRIBUTE that is something intermediate between a mailing list and a Usenet news server (and hosted on Bitnet *shiver*). There are no relevant standards yet.

William> my MUA (MDA, actually: Procmail) is forced to process
William> this message to its final destination in my mail system
William> illegally?

I don't really understand what you're trying to say. I can offer the following random comments:

"Illegal" of course is a slight overstatement. :-) The preferred word is "nonconforming".

But I don't think it's very useful to talk about "nonconforming delivery processing" here. Nothing about the delivery process is non-conforming for the simple reason that the *only* things that you can see in the message that reflect delivery processing are the Received and Return-Path headers, and the (nonstandard) Unix mbox "From " headline.

The other headers are all for the use of the MUAs and the users (including Mr. Procmail).

A general remark: You might find RFC 2369 useful as context (and in the future, to understand Mailman's use of those headers, though it's not relevant to this thread).

-- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Neal Groothuis wrote:

Mailman is not the originator of the message, so it should not be tampering with the From: or Sender: fields at all.

This is arguably not true. Mailman may add a list header and/or list footer to the body of the message as well as potentially filtering or scrubbing various MIME parts. The message sent by Mailman can be significantly different from the one originally received.

-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On Mon, 2006-05-01 at 13:27 -0500, Neal Groothuis wrote:

...

I'd like to work up an unofficial diff to Mailman 2.1 for people like Stephen who are willing to give it a try on a live site.

I'm not sure this is even necessary.

Ezmlm doesn't touch the Sender: header at all, Majordomo sets it to the owner of the list, and (AFAICT) Listserv sets it to the list itself. This would seem to me to indicate that incidences of mail being returned inappropriately to the sender are infrequent, at worst.

As I said, I think it's defensible to claim Mailman is the originator, but practicality beats purity, and I do think a list manager falls into a gray area here.

Having said all that, you might be right, in that we really don't need to do much except remove the addition of the Sender: header in bulkdeliver().

-Barry

It does not appear that Mailman modifies the "Sender:" field to comply with RFC 2822. The list-bounces address is not the mailbox of the agent responsible for transmitting the message, as required in section 3.6.2. The mailbox of the agent responsible for the transmission of the message would be the list-owner address.

Mailman's use the "Sender:" field does not seem to be in line with the intent of the RFC, nor with current usage of the field. The example given in the RFC is of a secretary sending an email on behalf of someone else. Outlook obviously interprets it this way. Some versions of Thunderbird display both the Sender: and From: lines to the user, which may prove confusing if the Sender: address is not a person or an obvious alias for one. Gmail uses it if you choose a "From" address that is not your gmail.com address.

Further, if Mailman is going to change the "Sender:" field, it should add Resent-* headers, per section 3.6.6 of RFC 2822; otherwise, the original sender information is lost. The RFC does say that this is to be used when "users" reintroduce a message into the system, further providing evidence that automated components of the mail routing system shouldn't be changing these. (Note that MTAs don't change the Sender: field, despite being, by their nature, agents responsible for transmission of messages.)

RFC 2369 provides headers which are to be used by mail list software to identify the various ways of interacting with the list, and Mailman already adds them. This makes adding this information to the Sender: field redundant.

Based on all of this, Mark's note that there are some MTAs which bounce to the Sender: address is the only reason that I've seen why this behavior should continue. Does anyone know what MTAs these are, or if they're even still in use? If these buggy MTAs are common, I would suggest that an option be added to the list to enable this behavior, marked as an accomodation for buggy MTAs, and defaulting to "off". I'll see if I can scrounge up the time to submit a patch to accomplish this, if it'll actually get included in a future release; otherwise I won't waste my time. If these MTAs are not in use, I stand by my original recommendation to comment out/remove the two lines responsible for the behavior.

At any rate, the "keep patching" suggestion is unhelpful. This is obviously a problem that many people are running into, enough that there's a FAQ entry about it. It should be addressed.

On Fri, 2006-04-28 at 14:08 -0400, Bob Puff@NLE wrote:

...Trouble similar to a current problem I am having with AOL: they are bouncing all email with the FROM: address of a specific AOL user, when mailman delivers the messages to -any- aol or cs.com address.

Have you tried turning on full personalization so that every recipient gets their own copy?

-Barry

Now that I have a few minutes to breath ;) I'll try to summarize my thoughts on this, and then perhaps go back later and follow up to specific points later in the thread.

I'm sympathetic to ripping out the Sender: field munging. It was always primarily a workaround for buggy MTAs. If the majority of MTAs out there now Do The Right Thing, then of course we want to be RFC compliant. Outlook's behavior has always been a wart but so far, the benefits have outweighed the costs. If the benefits are minimal now, then it should go.

The question that must be answered is: if we no longer munge Sender: header, what are the right headers to set to increase the likelihood that bounces will go where we want them to?

I don't care about the minority of still-deployed buggy MTAs that may send bounces to the wrong place as long as we can be reasonably assured they won't send them to /some/ wrong places. For example, I think it would be bad if they started spamming list owners with bounces, very bad if they spammed the original message authors, and worse if they spammed the mailing list with bounces. We have almost none of that now and if we removed the Sender: header and saw a huge increase in this bad behavior, then the benefits of munging the header go way up.

Of course, we might not know until we start deploying, which would be too late. I encourage those of you who want to get rid of Sender: munging to modify your Mailman 2.1 code now and see what happens. :) Of course, you'll let us know how that goes! My recommendation would be to record the results in the wiki.

I agree that the RFCs are a bit murky in this area, but it's relatively clear that the RFC 2822 'secretary' example illustrates the intent of Sender:. Our list owners are not the agents transmitting the messages, so listname-owner should clearly not go in Sender:.

We really want to increase the chances that Mailman will process all bounces. As I mention above, we absolutely can't be a vector for spamming people with bounces they can't do anything about. If some buggy MTAs throw bounces away though, tough luck for their users. There should be enough compliant MTAs out there that the added cost of keeping bogus addresses on a list because we never saw their bounces should be low.

I don't really like list or site options for this kind of thing. For one thing, we already have too many options and adding list options complicates the U/I and cognitive load for list admins who usually really don't want to be bothered with this nonsense. We should be reducing the options a list admin has to worry about, not increasing them.

This is not really appropriate for a site admin either because that's too coarse of a decision. A Mailman site talks to a huge array of MTAs and MUAs, so I don't think you could make a good choice. If anything, should we determine that Sender: munging has to stay, it should be a user option because only the user knows whether their MUA will present them with an ugly message display. A user could decide to turn off Sender: munging, but I suspect it's still more than the typical user will want to deal with. And of course per-user options get into all the personalization vs. performance issues.

Is listname-bounces confusing? Yes, and I wouldn't be opposed to changing this, although I'm not sure what we'd use. listname-processor? Well, let's hope we can just get rid of Sender: munging instead.

-Barry