Hello *,
I'm new here. I run a few mailman lists on a Debian server. I've been using mailman for a few years now but I'm no expert (it's "too" easy, anyone can use it without understanding much of it).
Now I'm facing this problem:
http://tinyurl.com/c9pn9sk http://support.google.com/mail/bin/answer.py?hl=it&answer=6588
There's a fact that makes me think a solution has to exist: I've run a few mailman lists on a different debian server until a few months ago and they did not have this problem. A friend of mine is currently running a few lists on yet another debian server and those do not have the gmail "feature" as of today.
I suspect gmail applies that feature only to some lists/domains/servers. The question is: what's the rule? How can you be so lucky as to fall in the lists gmail does not "block"? I fear the reply is "you can't".
So, next question: is there any unofficial, untested, underground, YMMV patch for mailman that masks/forges the messages copy to the sender so that gmail does not recognize it as such and lets it through?
Thanks in advance Lucio.
Lucio Crusca writes:
Now I'm facing this problem:
http://tinyurl.com/c9pn9sk http://support.google.com/mail/bin/answer.py?hl=it&answer=6588
There's a fact that makes me think a solution has to exist: I've run a few mailman lists on a different debian server until a few months ago and they did not have this problem.
Your fact is presumably due to some error in observation, since Gmail acknowledges this behavior as a feature of Gmail. It is simply not possible to receive your own posts on Gmail; you can only keep the Sent folder copy.
Are you sure you don't mean a different issue such as FAQ 3.42?
http://wiki.list.org/pages/viewpage.action?pageId=4030690> A friend of mine is currently running a few lists on yet another
debian server and those do not have the gmail "feature" as of today.
Please recheck; if there's a reasonable way to get around this, there are probably thousands of Gmail users who would love to do so. But I suspect you'll discover that Gmail users simply won't receive their own posts from the list.
So, next question: is there any unofficial, untested, underground, YMMV patch for mailman that masks/forges the messages copy to the sender so that gmail does not recognize it as such and lets it through?
I would imagine that you could change the Message-ID at the list, and that might do the trick. But that is strictly against the relevant standards for mail. Among other things, it would break deduplication measures for all of your subscribers, not just posters. It should be possible to do it for the particular message sent to the poster only, but that would be delicate.
Steve
Stephen J. Turnbull writes:
Your fact is presumably due to some error in observation, since Gmail acknowledges this behavior as a feature of Gmail. It is simply not possible to receive your own posts on Gmail; you can only keep the Sent folder copy.
I can confirm my observation is correct. I have at least three gmail users which I personally know that suffer from that "feature" on one of my lists and don't experience the same problem on another list which is mailman powered but hosted at a friend's server.
Other users started to experience the "feature" only when I migrated a mailing list from a server to another and they notified me just after the migration. I currently have a few examples of existing and running mailing lists powered by mailman where gmail subscribers do receive their own posts, this is one:
http://mailman.pinerolo.linux.it/mailman/listinfo/soci
Feel free to try subscribing to the above list and try posting from gmail. It's an italian language list dedicated to our local linux users group, but as far as you put an [OT] in the subject there are no problems.
Are you sure you don't mean a different issue such as FAQ 3.42?
http://wiki.list.org/pages/viewpage.action?pageId=4030690
Absolutely sure. Messages from the list are correctly delivered to gmail subscribers, except the ones themselves sent.
Please recheck; if there's a reasonable way to get around this, there are probably thousands of Gmail users who would love to do so. But I suspect you'll discover that Gmail users simply won't receive their own posts from the list.
Like I said, I suspect it depends on the list. My current best guess is that older lists (i.e. the ones that had been created before some time in the past) don't hit the infamous feature, while newer ones do.
I would imagine that you could change the Message-ID at the list, and that might do the trick. But that is strictly against the relevant standards for mail.
What's more against standards than throwing messages away like gmail does? I wouldn't care too much if the patch concerned only gmail subscribers.
It should be possible to do it for the particular message sent to the poster only, but that would be delicate.
Please elaborate. That would be the solution. I imagine something fairly simple and not very intrusive, like adding something like this somewhere in the mailman code:
msg = create_message_copy(message)
if msg.recipient == msg.sender && is_option_active(workaround_gmail_feature) && domain(msg.recipient) == "gmail" # do whatever delicate and nasty on msg # in order to have it delivered. # At worst, it won't be delivered anyway... endif
send_message(msg)
On Tue, 7 Aug 2012 12:19:41 -0000 "Lucio Crusca" <lucio@sulweb.org> wrote:
Hello Lucio,
Like I said, I suspect it depends on the list. My current best guess is that older lists (i.e. the ones that had been created before some time in the past) don't hit the infamous feature, while newer ones do.
Gmail has *always* been that way. There is a workaround. Maybe it is employed (if only by accident) on the lists you mention. It is required that the list be set up with the following;
Receive: list.foo.bar Send: smtp.foo.bar
and then add the header;
Reply to: list.foo.bar
I got this from another list I'm subbed to, have never done it myself so can't say for sure it's correct, but have no reason to doubt the source.
See http://lists.claws-mail.org/pipermail/users/2012-July/002752.html and a few surrounding messages for some context.
-- Regards _ / ) "The blindingly obvious is / _)rad never immediately apparent" Kill joy, bad guy, big talking, small fry Death On Two Legs - Queen
Lucio Crusca writes:
Feel free to try subscribing to the above list and try posting from gmail.
OK, but it will have to wait until tomorrow. I need to sleep after that last goal by Mexico. :-(
Like I said, I suspect it depends on the list. My current best guess is that older lists (i.e. the ones that had been created before some time in the past) don't hit the infamous feature, while newer ones do.
That's definitely wrong. I've been a gmail user for at least 5 years, occasionally posting to various mailing lists of which some have been in existence for more than 20 years, with the oldest Mailman installation being 13 years old (but Mailman has been upgraded regularly, of course).
What's more against standards than throwing messages away like gmail does? I wouldn't care too much if the patch concerned only gmail subscribers.
Sorry, but you're wrong. Gmail is an MUA, there are no RFCs for what it should do with messages it receives, only for the wire format to use when sending and interpret when receiving. If users don't like what it does, there are plenty of other MUAs.
To the extent that the RFCs apply, Gmail is perfectly within the standards. Two messages that have the same Message-ID are presumed to be the same message by the RFCs.
Gmail also provides MTA services. If someone uses a different MUA to send via Gmail's SMTP server, I believe they would receive that message via the list as the Gmail MUA wouldn't have a copy. Is that a possible explanation?
Please elaborate. That would be the solution. I imagine something fairly simple and not very intrusive, like adding something like this somewhere in the mailman code:
If you're willing to do it for all recipients but only for posts from Gmail users, you can do it in a Handler, pretty much anywhere in the pipeline. I consider that non-intrusive and relatively benign because everybody's copy will have the same Message-ID for threading, Gmane searching, and the like.
If you do it only for the Gmail-using subscriber, you'll probably screw up his threading because his copy of the message will have a different Message-ID from what everybody else refers to. I find that to be pretty intrusive.
Also, as far as I know, doing that would require doing surgery on the personalization code, after the handoff to the outgoing queue runner. I'm not willing to advise you on that, I don't know that part of the code at all.
Brad Rogers writes:
Gmail has *always* been that way. There is a workaround. Maybe it is employed (if only by accident) on the lists you mention. It is required that the list be set up with the following;
Receive: list.foo.bar
Ok, I think the example fits my case (3rd level domain is "lists" in my case).
Send: smtp.foo.bar
Does that mean that I must have an external (from my mailman server point of view) smtp server? That would explain everything, but that would also leave me without a dime, because I can't reasonably identify another meaningful SMTP server to act as smart relay for my smtp server. And no, purposely setting one up is not an option, at least for me.
and then add the header;
Reply to: list.foo.bar
Already done since when I created my lists (I almost always set "reply_goes_to_list" option to the list itself, unrespectful of big fat warnings mailman shows about reply-to munging considered harmful).
I got this from another list I'm subbed to, have never done it myself so can't say for sure it's correct, but have no reason to doubt the source.
See http://lists.claws-mail.org/pipermail/users/2012-July/002752.html and a few surrounding messages for some context.
I've read the whole thread, thanks. Unfortunately it does not reply to my questions above, but maybe you have undestood the exact meaning of those words.
Stephen J. Turnbull writes:
Lucio Crusca writes:
Feel free to try subscribing to the above list and try posting from gmail.
OK, but it will have to wait until tomorrow. I need to sleep after that last goal by Mexico. :-(
Probably you don't need to try anymore (see Brad Rogers' post).
That's definitely wrong. I've been a gmail user for at least 5 years
You're right, Brad gave a clue about what's going on and his reply fits perfectly my observation.
What's more against standards than throwing messages away like gmail does? I wouldn't care too much if the patch concerned only gmail subscribers.
Sorry, but you're wrong. Gmail is an MUA,
Actually I already suspected that no RFC said what a MUA should do with messages. However Gmail is accessible via POP/IMAP also. AFAICT the same messages are lost also when accessing gmail via POP/IMAP, and in that case GMail is not only a MUA and it does break standards.
However that's not the real point. My sentence meant that if the patch were to violate standards only for gmail users that want it (e.g. users that activate an option in their mailman preferences panel), then that violation is less of a problem, at least from those users point of view, when compared to messages automatically being thrown away by gmail.
About the fact that you can always use a different MUA, unfortunately in many cases that's only theory. In practice changing your email address translates in a loss of time and often money. Either way, Gmail is not only a MUA.
Please elaborate. That would be the solution. I imagine something fairly simple and not very intrusive, like adding something like this somewhere
in the mailman code: If you're willing to do it for all recipients but only for posts from Gmail users,
No, I'm willing to do it only for gmail users that activate the relevant option in their mailman preferences. I think that doing it for anyone outside the users that explicitly ask for it is invasive and dangerous.
you can do it in a Handler, pretty much anywhere in the pipeline. I consider that non-intrusive and relatively benign because everybody's copy will have the same Message-ID for threading, Gmane searching, and the like.
I agree that could work well, but I don't understand email standards enough to feel comfortable in modifying message headers of outgoing mail for all subscribers.
If you do it only for the Gmail-using subscriber, you'll probably screw up his threading because his copy of the message will have a different Message-ID from what everybody else refers to. I find that to be pretty intrusive.
You are absolutely right. Probably changing Message-Id is not the best way to workaround that "feature" when applied only to the copy for the original poster. I wonder if there are any other headers that can be changed, only in the message copy for the original poster, that do the job.
Also, as far as I know, doing that would require doing surgery on the personalization code, after the handoff to the outgoing queue runner. I'm not willing to advise you on that, I don't know that part of the code at all.
Ok, before that I need to identify correct headers to change anyway, assuming they exist...
On Wed, 8 Aug 2012 09:09:22 +0200 Lucio Crusca <lucio@sulweb.org> wrote:
Hello Lucio,
Send: smtp.foo.bar Does that mean that I must have an external (from my mailman server
Brad Rogers writes: point of view) smtp server? That would explain everything, but that
I believe so, yes. I only run one list, and don't have sufficient access rights to "play" with this to find out. Like you, I have limited funds, so can't buy my list out of this situation.
Reply to: list.foo.bar Already done since when I created my lists (I almost always set "reply_goes_to_list" option to the list itself, unrespectful of big warnings mailman shows about reply-to munging considered harmful).
I don't set it, but there are arguments both for and against. It's up to each of us to make up our own minds.
See http://lists.claws-mail.org/pipermail/users/2012-July/002752.html and a few surrounding messages for some context. I've read the whole thread, thanks. Unfortunately it does not reply to my questions above, but maybe you have undestood the exact meaning of those words.
Yes, we were diverging from the real subject of the thread, and I never pursued the matter any further.
-- Regards _ / ) "The blindingly obvious is / _)rad never immediately apparent" Black man got a lot of problems, but he don't mind throwing a brick White Riot - The Clash
Lucio Crusca writes:
Actually I already suspected that no RFC said what a MUA should do with messages. However Gmail is accessible via POP/IMAP also. AFAICT the same messages are lost also when accessing gmail via POP/IMAP, and in that case GMail is not only a MUA and it does break standards.
I don't think so. Perhaps "MUA" is the wrong term for a message store "in the cloud", but the fact is that Gmail is the final recipient as far as the RFCs are concerned. Eg, IMAP servers often implement SIEVE recipes and spam filtering, so some messages will be lost.
In any case, no messages are lost; only copies with different meta-data.
I don't really disagree with you that Gmail's behavior is horrible. My point is that if you think its behavior is non-conforming, you may be in for other, even less pleasant surprises in the future.
About the fact that you can always use a different MUA, unfortunately in many cases that's only theory.
I can't say I have a lot of sympathy. You get Gmail for free, you shouldn't think it comes with no strings attached.
No, I'm willing to do it only for gmail users that activate the relevant option in their mailman preferences.
I think that's best, but I can't help you with implementing that kind of thing (in principle I know what to do, but I don't have time to nail down the details for a while). For Mark or Barry it's probably quite straightforward, but of course their time is very valuable.
If you do it only for the Gmail-using subscriber, you'll probably screw up his threading because his copy of the message will have a different Message-ID from what everybody else refers to. I find that to be pretty intrusive.
You are absolutely right. Probably changing Message-Id is not the best way to workaround that "feature" when applied only to the copy for the original poster.
Well, unfortunately Gmail is closed-source and I don't know what the full algorithm is. Surely Message-Id is part of it, but evidently there are other aspects to it, or the behavior you and Brad R. describe wouldn't happen.
I wonder if there are any other headers that can be changed, only in the message copy for the original poster, that do the job.
I presume that there are. Probably Brad's discussion gives a hint.
Steve
Stephen J. Turnbull writes:
I don't think so. Perhaps "MUA" is the wrong term for a message store "in the cloud", but the fact is that Gmail is the final recipient as far as the RFCs are concerned. Eg, IMAP servers often implement SIEVE recipes and spam filtering, so some messages will be lost.
Again, that's not the point and we basically agree gmail is bad, but... a standard is some set of commonly accepted rules. Be it written down into a RFC or not. The "standard" (expected by most people) behavior of a email final recipient software, if not MUA, is to receive emails, not to throw them away based on ill advised algorithms. Sieve recipes and spam filtering is something that users can disable and modify at will (at least that's the "standard" for MUAs). If a recipe or spam filter accidentally trashes a message, the user can always disable that recipe or filter. Gmail does break the standard (expected behavior) in that does not let users choose if they want to receive some messages that are not spam by any stretch of imagination. Imho.
In any case, no messages are lost; only copies with different meta-data.
However some information is actually lost (threading in the user's inbox and the acknowledgment that your message has actually reached the mailing list).
I don't really disagree with you that Gmail's behavior is horrible. My point is that if you think its behavior is non-conforming, you may be in for other, even less pleasant surprises in the future.
You see, there must be a reason why I decided to roll my own mail server after all... I'm prepared to surprises. I'm not a gmail user, though I do have a sleeping gmail account. I'd only like to slap gmail in the face if I could, by working around their wonderful feature, just for the taste of feeling smarter than they pretend to be. All in all, what is hacking about if not that?
I can't say I have a lot of sympathy. You get Gmail for free, you shouldn't think it comes with no strings attached.
Quite obvious, though I can't see what Gmail earns from that "feature", but I suspect it's me not foreseeing very far away.
On Thursday 09 August 2012, Lucio Crusca wrote:
I'd only like to slap gmail in the face if I could, by working around their wonderful feature, just for the taste of feeling smarter than they pretend to be. All in all, what is hacking about if not that?
Please do! Gmail user only because my ISP outsourced mail to them three years ago. Was helping a small discusion list move from an LSoft ListServe to Mailman at the time so suddenly missing my own posts back made testing impossible.
Was infuriated when I discovered there was no way to turn this stupid feature off. My work around is to post through my web hosts mail server. Most people do not have this option so a setting in Mailman for other would be great.
William
Lucio Crusca writes:
Again, that's not the point and we basically agree gmail is bad, but... a standard is some set of commonly accepted rules. Be it written down into a RFC or not.
It doesn't need to be in an RFC, but it must be written. "What is commonly accepted" is simply not a standard because it's impossible to know if you're conforming, or what you need to conform to.
On Aug 8, 2012, at 11:11 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Well, unfortunately Gmail is closed-source and I don't know what the full algorithm is. Surely Message-Id is part of it, but evidently there are other aspects to it, or the behavior you and Brad R. describe wouldn't happen.
In the large-scale mail system design that I've done in the past, the tuple of (sender,recipient,message-id) was considered to be a pretty good index key for the mail database, albeit not a guaranteed unique key. Most greylisting implementations use a tuple of (sender,recipient,sending-IP) to determine if this particular message should be delayed or not.
I even did a single-instance-store message database design that did an SHA-1 hash of the message body content to see if the message contents really were unique, and if not then you could store the headers separate from the body and for the body you could just include a pointer to the existing message body that you already have. I believe that some implementations of Microsoft Exchange implement a similar algorithm.
If you wanted to go to the extreme, you could de-compose each message to the individual MIME bodyparts, and then do an SHA-1 hash on each of those. So, no matter how many copies of the latest Dilbert cartoon get mailed out, and no matter what text or other material might surround that, you'd still be able to reduce that to storing just one copy of the cartoon with multiple inbound links.
On the other hand, Nick Christensen (author of "Sendmail Performance Tuning", ISBN-13: 978-0321115706) and I discovered that you would be trading more disk I/O operations in order to try to save a relatively trivial amount of disk space, and that's the exact opposite of the trade-off you want to make given the way disk storage capacities have rapidly grown while I/O capacities have been relatively stagnant. We discussed all these issues in the invited talk "Design and Implementation of Highly Scalable E-mail Systems", see <http://www.shub-internet.org/brad/papers/dihses/>.
I happen to know the former SRE for gmail, but I don't think he'd be able to tell me anything useful on this subject.
I really don't think that this is a disk storage issue, I think this is much more likely to be a wrong-headed idea that this kind of thing will be beneficial to the users -- after all, they know that they sent the message and that copy is sitting in the outbox, so they don't need to have another copy sitting in the inbox.
And maybe for the majority of users, that decision might actually be helpful. But they need to give people a way to turn that option off, so that they don't break the ability to do debugging when testing the sending of messages to remote systems.
Of course, if people are on Google Groups, then this probably isn't an issue for them. And maybe that's the other part of the problem -- maybe Google sees this "feature" as being a competitive advantage for them with combining Google Groups and gmail working better together, and they don't see the benefit of making gmail be able to play better with the rest of the world.
If you think it's worthwhile, you could always try turning on personalization for the list, and then add a footer with unique information per recipient. That would cause the message-id to be unique as well as the message body, and wouldn't require any new code to be developed.
-- Brad Knowles <brad@shub-internet.org> LinkedIn Profile: <http://tinyurl.com/y8kpxu>
Brad Knowles writes:
I really don't think that this is a disk storage issue, I think this is much more likely to be a wrong-headed idea that this kind of thing will be beneficial to the users -- after all, they know that they sent the message and that copy is sitting in the outbox, so they don't need to have another copy sitting in the inbox.
I agree it's not about disk storage, I think it's just de-duplication of the messages that users see. Back when Canter and Siegel first got started, we hated spam not because there was so much of it, but because it was so bloody annoying to see it in every newsgroup we subscribed to. I don't see why ordinary users wouldn't feel the same way (of course one dupe is far less annoying than Green Card lawyers in every group you read, but if you get a lot of them, the annoyance level would build up).
I disagree that it's wrongheaded, if Gmail is going to always do de-duplication with one algorithm. Gmail always stores the mail you sent, as you sent it. It is not necessarily the case that it will come back to you in one piece. After all, our favorite list distribution software is just bristling with settings determining what's going to be left of your post once it arrives at the subscriber's mailbox. Everybody can understand if they send out a PNG, it comes back from the list stripped or the mail gets dropped, and for some reason they don't have a copy of their original. OTOH, only a very few would know, let alone care, about missing RFC 2369 headers in a few copies they have locally!
I just think that users ought to have a choice of how de-duping is done. Or if it gets done at all.
If you think it's worthwhile, you could always try turning on personalization for the list, and then add a footer with unique information per recipient. That would cause the message-id to be unique as well as the message body, and wouldn't require any new code to be developed.
Small correction: the Message-Id will be the same for all copies. Mailman cannot go changing those, or it would play hell with all threading MUAs.
Steve
participants (5)
-
Brad Knowles -
Brad Rogers -
Lucio Crusca -
Stephen J. Turnbull -
William Bagwell