Stephen J. Turnbull writes:

Your fact is presumably due to some error in observation, since Gmail acknowledges this behavior as a feature of Gmail. It is simply not possible to receive your own posts on Gmail; you can only keep the Sent folder copy.

I can confirm my observation is correct. I have at least three gmail users which I personally know that suffer from that "feature" on one of my lists and don't experience the same problem on another list which is mailman powered but hosted at a friend's server.

Other users started to experience the "feature" only when I migrated a mailing list from a server to another and they notified me just after the migration. I currently have a few examples of existing and running mailing lists powered by mailman where gmail subscribers do receive their own posts, this is one:

http://mailman.pinerolo.linux.it/mailman/listinfo/soci

Feel free to try subscribing to the above list and try posting from gmail. It's an italian language list dedicated to our local linux users group, but as far as you put an [OT] in the subject there are no problems.

Are you sure you don't mean a different issue such as FAQ 3.42?

http://wiki.list.org/pages/viewpage.action?pageId=4030690

Absolutely sure. Messages from the list are correctly delivered to gmail subscribers, except the ones themselves sent.

Please recheck; if there's a reasonable way to get around this, there are probably thousands of Gmail users who would love to do so. But I suspect you'll discover that Gmail users simply won't receive their own posts from the list.

Like I said, I suspect it depends on the list. My current best guess is that older lists (i.e. the ones that had been created before some time in the past) don't hit the infamous feature, while newer ones do.

I would imagine that you could change the Message-ID at the list, and that might do the trick. But that is strictly against the relevant standards for mail.

What's more against standards than throwing messages away like gmail does? I wouldn't care too much if the patch concerned only gmail subscribers.

It should be possible to do it for the particular message sent to the poster only, but that would be delicate.

Please elaborate. That would be the solution. I imagine something fairly simple and not very intrusive, like adding something like this somewhere in the mailman code:

msg = create_message_copy(message)

if msg.recipient == msg.sender && is_option_active(workaround_gmail_feature) && domain(msg.recipient) == "gmail" # do whatever delicate and nasty on msg # in order to have it delivered. # At worst, it won't be delivered anyway... endif

send_message(msg)

Brad Rogers writes:

Gmail has *always* been that way. There is a workaround. Maybe it is employed (if only by accident) on the lists you mention. It is required that the list be set up with the following;

Receive: list.foo.bar

Ok, I think the example fits my case (3rd level domain is "lists" in my case).

Send: smtp.foo.bar

Does that mean that I must have an external (from my mailman server point of view) smtp server? That would explain everything, but that would also leave me without a dime, because I can't reasonably identify another meaningful SMTP server to act as smart relay for my smtp server. And no, purposely setting one up is not an option, at least for me.

and then add the header;

Reply to: list.foo.bar

Already done since when I created my lists (I almost always set "reply_goes_to_list" option to the list itself, unrespectful of big fat warnings mailman shows about reply-to munging considered harmful).

I got this from another list I'm subbed to, have never done it myself so can't say for sure it's correct, but have no reason to doubt the source.

See http://lists.claws-mail.org/pipermail/users/2012-July/002752.html and a few surrounding messages for some context.

I've read the whole thread, thanks. Unfortunately it does not reply to my questions above, but maybe you have undestood the exact meaning of those words.

Lucio Crusca writes:

Feel free to try subscribing to the above list and try posting from gmail.

OK, but it will have to wait until tomorrow. I need to sleep after that last goal by Mexico. :-(

Like I said, I suspect it depends on the list. My current best guess is that older lists (i.e. the ones that had been created before some time in the past) don't hit the infamous feature, while newer ones do.

That's definitely wrong. I've been a gmail user for at least 5 years, occasionally posting to various mailing lists of which some have been in existence for more than 20 years, with the oldest Mailman installation being 13 years old (but Mailman has been upgraded regularly, of course).

What's more against standards than throwing messages away like gmail does? I wouldn't care too much if the patch concerned only gmail subscribers.

Sorry, but you're wrong. Gmail is an MUA, there are no RFCs for what it should do with messages it receives, only for the wire format to use when sending and interpret when receiving. If users don't like what it does, there are plenty of other MUAs.

To the extent that the RFCs apply, Gmail is perfectly within the standards. Two messages that have the same Message-ID are presumed to be the same message by the RFCs.

Gmail also provides MTA services. If someone uses a different MUA to send via Gmail's SMTP server, I believe they would receive that message via the list as the Gmail MUA wouldn't have a copy. Is that a possible explanation?

Please elaborate. That would be the solution. I imagine something fairly simple and not very intrusive, like adding something like this somewhere in the mailman code:

If you're willing to do it for all recipients but only for posts from Gmail users, you can do it in a Handler, pretty much anywhere in the pipeline. I consider that non-intrusive and relatively benign because everybody's copy will have the same Message-ID for threading, Gmane searching, and the like.

If you do it only for the Gmail-using subscriber, you'll probably screw up his threading because his copy of the message will have a different Message-ID from what everybody else refers to. I find that to be pretty intrusive.

Also, as far as I know, doing that would require doing surgery on the personalization code, after the handoff to the outgoing queue runner. I'm not willing to advise you on that, I don't know that part of the code at all.

Lucio Crusca writes:

Actually I already suspected that no RFC said what a MUA should do with messages. However Gmail is accessible via POP/IMAP also. AFAICT the same messages are lost also when accessing gmail via POP/IMAP, and in that case GMail is not only a MUA and it does break standards.

I don't think so. Perhaps "MUA" is the wrong term for a message store "in the cloud", but the fact is that Gmail is the final recipient as far as the RFCs are concerned. Eg, IMAP servers often implement SIEVE recipes and spam filtering, so some messages will be lost.

In any case, no messages are lost; only copies with different meta-data.

I don't really disagree with you that Gmail's behavior is horrible. My point is that if you think its behavior is non-conforming, you may be in for other, even less pleasant surprises in the future.

About the fact that you can always use a different MUA, unfortunately in many cases that's only theory.

I can't say I have a lot of sympathy. You get Gmail for free, you shouldn't think it comes with no strings attached.

No, I'm willing to do it only for gmail users that activate the relevant option in their mailman preferences.

I think that's best, but I can't help you with implementing that kind of thing (in principle I know what to do, but I don't have time to nail down the details for a while). For Mark or Barry it's probably quite straightforward, but of course their time is very valuable.

...

If you do it only for the Gmail-using subscriber, you'll probably screw up his threading because his copy of the message will have a different Message-ID from what everybody else refers to. I find that to be pretty intrusive.

You are absolutely right. Probably changing Message-Id is not the best way to workaround that "feature" when applied only to the copy for the original poster.

Well, unfortunately Gmail is closed-source and I don't know what the full algorithm is. Surely Message-Id is part of it, but evidently there are other aspects to it, or the behavior you and Brad R. describe wouldn't happen.

I wonder if there are any other headers that can be changed, only in the message copy for the original poster, that do the job.

I presume that there are. Probably Brad's discussion gives a hint.

Steve

On Thursday 09 August 2012, Lucio Crusca wrote:

I'd only like to slap gmail in the face if I could, by working around their wonderful feature, just for the taste of feeling smarter than they pretend to be. All in all, what is hacking about if not that?

Please do! Gmail user only because my ISP outsourced mail to them three years ago. Was helping a small discusion list move from an LSoft ListServe to Mailman at the time so suddenly missing my own posts back made testing impossible.

Was infuriated when I discovered there was no way to turn this stupid feature off. My work around is to post through my web hosts mail server. Most people do not have this option so a setting in Mailman for other would be great.

William

On Aug 8, 2012, at 11:11 PM, Stephen J. Turnbull stephen@xemacs.org wrote:

Well, unfortunately Gmail is closed-source and I don't know what the full algorithm is. Surely Message-Id is part of it, but evidently there are other aspects to it, or the behavior you and Brad R. describe wouldn't happen.

In the large-scale mail system design that I've done in the past, the tuple of (sender,recipient,message-id) was considered to be a pretty good index key for the mail database, albeit not a guaranteed unique key. Most greylisting implementations use a tuple of (sender,recipient,sending-IP) to determine if this particular message should be delayed or not.

I even did a single-instance-store message database design that did an SHA-1 hash of the message body content to see if the message contents really were unique, and if not then you could store the headers separate from the body and for the body you could just include a pointer to the existing message body that you already have. I believe that some implementations of Microsoft Exchange implement a similar algorithm.

If you wanted to go to the extreme, you could de-compose each message to the individual MIME bodyparts, and then do an SHA-1 hash on each of those. So, no matter how many copies of the latest Dilbert cartoon get mailed out, and no matter what text or other material might surround that, you'd still be able to reduce that to storing just one copy of the cartoon with multiple inbound links.

On the other hand, Nick Christensen (author of "Sendmail Performance Tuning", ISBN-13: 978-0321115706) and I discovered that you would be trading more disk I/O operations in order to try to save a relatively trivial amount of disk space, and that's the exact opposite of the trade-off you want to make given the way disk storage capacities have rapidly grown while I/O capacities have been relatively stagnant. We discussed all these issues in the invited talk "Design and Implementation of Highly Scalable E-mail Systems", see http://www.shub-internet.org/brad/papers/dihses/.

I happen to know the former SRE for gmail, but I don't think he'd be able to tell me anything useful on this subject.

I really don't think that this is a disk storage issue, I think this is much more likely to be a wrong-headed idea that this kind of thing will be beneficial to the users -- after all, they know that they sent the message and that copy is sitting in the outbox, so they don't need to have another copy sitting in the inbox.

And maybe for the majority of users, that decision might actually be helpful. But they need to give people a way to turn that option off, so that they don't break the ability to do debugging when testing the sending of messages to remote systems.

Of course, if people are on Google Groups, then this probably isn't an issue for them. And maybe that's the other part of the problem -- maybe Google sees this "feature" as being a competitive advantage for them with combining Google Groups and gmail working better together, and they don't see the benefit of making gmail be able to play better with the rest of the world.

If you think it's worthwhile, you could always try turning on personalization for the list, and then add a footer with unique information per recipient. That would cause the message-id to be unique as well as the message body, and wouldn't require any new code to be developed.

-- Brad Knowles brad@shub-internet.org LinkedIn Profile: http://tinyurl.com/y8kpxu