(relatively) new DMARC issues - and Gmail
I've been working with a From-rewriting tool with code taken from Mailman (thanks, Mark!) and discovered a couple of things which I thought deserved posting about. I expect that they peripherally affect Mailman, too.
At some point Amazon (amazon.com) started publishing a DMARC "p=quarantine" policy, which means that any email which gets redirected and hits my dmarc_shield piece is going to have its From address re- written to "postmaster@fmp.com" (fmp.com has a proper SPF record).
I don't know what Gmail's policy is with regard to "p=quarantine" - whether it rejects such email outright or relegates it to the recipient's spam folder. I know that if the sending site publishes "p=reject", redirected email is refused by Gmail at the front door. I'll have to test the "p=quarantine" behavior.
Here's the really annoying thing. My dmarc_shield processor rewrites the From header as per SOP for Mailman with the proper switch turned on. The From header address becomes "postmaster@fmp.com" with the original From address in the address comment (from xxx at yyz.com). If the email didn't already have a Reply-To address, the original From address is inserted as the Reply-To address. If a Gmail user replies to such an email, the reply goes to the Reply-To address, but Gmail **whitelists** the From address! Thereafter, any email which comes in with a munged From address is accepted, bypassing Gmail's otherwise pretty good spam filtering. I'm noticing a lot of spam email going out with From addresses for which a DMARC "p=reject" policy is published, which means that any such spam redirected to the Gmail user via FMP is also whitelisted. Bah! It's a fucking war zone out there!
The only possible solution here would be to randomize the username portion of the rewritten From address, which makes the email look more like spam, and the Gmail user would end up with a whole lot of useless whitelisted address which would need to be deleted. Not to mention the fact that FMP's mail server might be blocked from sending ANY email to Gmail.
--
Lindsay Haisley | "The first casualty when
FMP Computer Services | war comes is truth."
512-259-1190 |
http://www.fmp.com | -- Hiram W Johnson
On 3/31/18 2:31 PM, Lindsay Haisley wrote:
I've been working with a From-rewriting tool with code taken from Mailman (thanks, Mark!) and discovered a couple of things which I thought deserved posting about. I expect that they peripherally affect Mailman, too.
At some point Amazon (amazon.com) started publishing a DMARC "p=quarantine" policy, which means that any email which gets redirected and hits my dmarc_shield piece is going to have its From address re- written to "postmaster@fmp.com" (fmp.com has a proper SPF record).
I don't know what Gmail's policy is with regard to "p=quarantine" - whether it rejects such email outright or relegates it to the recipient's spam folder. I know that if the sending site publishes "p=reject", redirected email is refused by Gmail at the front door. I'll have to test the "p=quarantine" behavior.
Here's the really annoying thing. My dmarc_shield processor rewrites the From header as per SOP for Mailman with the proper switch turned on. The From header address becomes "postmaster@fmp.com" with the original From address in the address comment (from xxx at yyz.com). If the email didn't already have a Reply-To address, the original From address is inserted as the Reply-To address. If a Gmail user replies to such an email, the reply goes to the Reply-To address, but Gmail **whitelists** the From address! Thereafter, any email which comes in with a munged From address is accepted, bypassing Gmail's otherwise pretty good spam filtering. I'm noticing a lot of spam email going out with From addresses for which a DMARC "p=reject" policy is published, which means that any such spam redirected to the Gmail user via FMP is also whitelisted. Bah! It's a fucking war zone out there!
The only possible solution here would be to randomize the username portion of the rewritten From address, which makes the email look more like spam, and the Gmail user would end up with a whole lot of useless whitelisted address which would need to be deleted. Not to mention the fact that FMP's mail server might be blocked from sending ANY email to Gmail.
To me the issue sounds like why is fmp.com forwarding spam?
If this is a case of fmp.com offering forwarding mailboxes to users, who might be using gmail as a final destination, then yes, fmp needs to try to be as good at detecting spam as gmail or users need to accept the increased spam levels.
Another option is to deterministically munge the from address so every incoming email address gets a unique fmp address that it represents (it doesn't have to be absolutely unique, mostly unique is likely good enough), something like replace the at with _at_ and add a tail wart like _dmarc@fmp.com (so you can have other addresses an not worry about possible overlaps with those) and use that as the from address. Then a reply will only whitelist that specific original from address.
-- Richard Damon
On Sat, 2018-03-31 at 14:50 -0400, Richard Damon wrote:
To me the issue sounds like why is fmp.com forwarding spam?
If this is a case of fmp.com offering forwarding mailboxes to users, who might be using gmail as a final destination, then yes, fmp needs to try to be as good at detecting spam as gmail or users need to accept the increased spam levels.
If pigs could fly ....! I do the very best job I can of filtering spam from inbound email, and get about 90% of it, maybe more, but fighting spam is a forever job of whack-a-mole. I certainly wish that I could do as good a job of parsing spam from legit email as Gmail does, but I'm a one-person shop, and have many tasks. Gmail has dozens, perhaps hundreds of very smart people assigned to managing their spam filtering, and they do a very good job of it. I could _never_ hope to match their efficiency or accuracy, nor could most small operations such as FMP Computer Services.
The problem is that Gmail is whitelisting based on the From address, rather than the Reply-To address, which should be an _option_ open to users. On Google's scale of operation, I'm just a fly on a dog turd so any feature which might benefit my users and subscribers is pretty much a no-nevermind for them.
Another option is to deterministically munge the from address so every incoming email address gets a unique fmp address that it represents (it doesn't have to be absolutely unique, mostly unique is likely good enough), something like replace the at with _at_ and add a tail wart like _dmarc@fmp.com (so you can have other addresses an not worry about possible overlaps with those) and use that as the from address. Then a reply will only whitelist that specific original from address.
Which, as I noted in my original post, will cause the Gmail user's mail account to end up with a whole lot of useless whitelisted address which would need to be deleted, and FMP's server might well end up getting blacklisted as a result.
--
Lindsay Haisley | "The first casualty when
FMP Computer Services | war comes is truth."
512-259-1190 |
http://www.fmp.com | -- Hiram W Johnson
On 3/31/18 3:35 PM, Lindsay Haisley wrote:
On Sat, 2018-03-31 at 14:50 -0400, Richard Damon wrote:
To me the issue sounds like why is fmp.com forwarding spam?
If this is a case of fmp.com offering forwarding mailboxes to users, who might be using gmail as a final destination, then yes, fmp needs to try to be as good at detecting spam as gmail or users need to accept the increased spam levels. If pigs could fly ....! I do the very best job I can of filtering spam from inbound email, and get about 90% of it, maybe more, but fighting spam is a forever job of whack-a-mole. I certainly wish that I could do as good a job of parsing spam from legit email as Gmail does, but I'm a one-person shop, and have many tasks. Gmail has dozens, perhaps hundreds of very smart people assigned to managing their spam filtering, and they do a very good job of it. I could _never_ hope to match their efficiency or accuracy, nor could most small operations such as FMP Computer Services. But coming at least close is the job you sign up for in being a mail forwarder. You at least need to be good enough that you aren't seen by google as an uncaring domain, and maintain enough information that they can continue to do what they do well.
The problem is that Gmail is whitelisting based on the From address, rather than the Reply-To address, which should be an _option_ open to users. On Google's scale of operation, I'm just a fly on a dog turd so any feature which might benefit my users and subscribers is pretty much a no-nevermind for them. Which is why I was saying make a 1:1 mapping of From addresses to Reply-To addresses.
Another option is to deterministically munge the from address so every incoming email address gets a unique fmp address that it represents (it doesn't have to be absolutely unique, mostly unique is likely good enough), something like replace the at with _at_ and add a tail wart like _dmarc@fmp.com (so you can have other addresses an not worry about possible overlaps with those) and use that as the from address. Then a reply will only whitelist that specific original from address. Which, as I noted in my original post, will cause the Gmail user's mail account to end up with a whole lot of useless whitelisted address which would need to be deleted, and FMP's server might well end up getting blacklisted as a result. No more than if GMail did implement a white-list on Reply-To addresses.
-- Richard Damon
On Sat, 2018-03-31 at 17:57 -0400, Richard Damon wrote:
On 3/31/18 3:35 PM, Lindsay Haisley wrote:
On Sat, 2018-03-31 at 14:50 -0400, Richard Damon wrote:
To me the issue sounds like why is fmp.com forwarding spam?
If this is a case of fmp.com offering forwarding mailboxes to users, who might be using gmail as a final destination, then yes, fmp needs to try to be as good at detecting spam as gmail or users need to accept the increased spam levels.
If pigs could fly ....! I do the very best job I can of filtering spam from inbound email, and get about 90% of it, maybe more, but fighting spam is a forever job of whack-a-mole. I certainly wish that I could do as good a job of parsing spam from legit email as Gmail does, but I'm a one-person shop, and have many tasks. Gmail has dozens, perhaps hundreds of very smart people assigned to managing their spam filtering, and they do a very good job of it. I could _never_ hope to match their efficiency or accuracy, nor could most small operations such as FMP Computer Services.
But coming at least close is the job you sign up for in being a mail forwarder. You at least need to be good enough that you aren't seen by google as an uncaring domain, and maintain enough information that they can continue to do what they do well.
Rest assured, we "come at least close". This is not an option here, it's a necessity. Email redirection is a feature of my MTA (Courier) and has been offered since FMP went into business in the 1990s. It's a standard feature of many MTAs and many ESPs offer it.
I've had to deal with Gmail's honey-potting before, and I can do it again if necessary. I don't imagine that you've ever done commercial email administration, Richard, or you might have something constructive to say instead of just spewing admonitions to "do better".
The problem is that Gmail is whitelisting based on the From address, rather than the Reply-To address, which should be an _option_ open to users. On Google's scale of operation, I'm just a fly on a dog turd so any feature which might benefit my users and subscribers is pretty much a no-nevermind for them.
Which is why I was saying make a 1:1 mapping of From addresses to Reply-To addresses.
The From address _has_ to be from an address at fmp.com, which is the reason for From-munging in the first place. If you don't understand how DMARC works, or the problems it causes, Mark, or someone else on this list can send you to a reference on it. The Reply-To address is EITHER the original Reply-To address on the received email, or, if it had none, the ORIGINAL From address. Mapping the Reply-To address to the munged From address makes no sense at all.
Another option is to deterministically munge the from address so every incoming email address gets a unique fmp address that it represents (it doesn't have to be absolutely unique, mostly unique is likely good enough), something like replace the at with _at_ and add a tail wart like _dmarc@fmp.com (so you can have other addresses an not worry about possible overlaps with those) and use that as the from address. Then a reply will only whitelist that specific original from address. Which, as I noted in my original post, will cause the Gmail user's mail account to end up with a whole lot of useless whitelisted address which would need to be deleted, and FMP's server might well end up getting blacklisted as a result. No more than if GMail did implement a white-list on Reply-To addresses.
No, because the Reply-To address is the _original_ From address. Such a whitelisting would be useless as long as Gmail's policy with regard to DMARC rejection remains in place, but unless we get into some kind of meta-heading BS, it's the best we might do.
--
Lindsay Haisley | "The first casualty when
FMP Computer Services | war comes is truth."
512-259-1190 |
http://www.fmp.com | -- Hiram W Johnson
On Sat, 2018-03-31 at 17:33 -0500, Lindsay Haisley wrote:
I've had to deal with Gmail's honey-potting before, and I can do it again if necessary. I don't imagine that you've ever done commercial email administration, Richard, or you might have something constructive to say instead of just spewing admonitions to "do better".
And in any event, my original post was a FYI, mainly concerned with the interaction between Google's Gmail policies and _any_ system which munges headers to deal with DMARC, as MM does - combined with the fact that a lot of spammers these days seem to be sending mail over From addresses which publish a DMARC p=reject policy. This is a particular nasty combination which can effectively neutralize Gmail's spam blocking.
I'm not looking for advice from anyone on how to run FMP's mail system, nor for admonitions to do a better job of spam blocking, so if that's what you want to say, please email me personally because it's OT for this list.
If anyone has insights into this issue that _don't_ relate to FMP's email service, or which constitute constructive insights based on knowledge and experience, I'd love to hear them. Otherwise, please just leave it!
--
Lindsay Haisley | "The first casualty when
FMP Computer Services | war comes is truth."
512-259-1190 |
http://www.fmp.com | -- Hiram W Johnson
On 3/31/18 6:55 PM, Lindsay Haisley wrote:
I've had to deal with Gmail's honey-potting before, and I can do it again if necessary. I don't imagine that you've ever done commercial email administration, Richard, or you might have something constructive to say instead of just spewing admonitions to "do better". And in any event, my original post was a FYI, mainly concerned with the interaction between Google's Gmail policies and _any_ system which munges headers to deal with DMARC, as MM does - combined with the fact
On Sat, 2018-03-31 at 17:33 -0500, Lindsay Haisley wrote: that a lot of spammers these days seem to be sending mail over From addresses which publish a DMARC p=reject policy. This is a particular nasty combination which can effectively neutralize Gmail's spam blocking.
I'm not looking for advice from anyone on how to run FMP's mail system, nor for admonitions to do a better job of spam blocking, so if that's what you want to say, please email me personally because it's OT for this list.
If anyone has insights into this issue that _don't_ relate to FMP's email service, or which constitute constructive insights based on knowledge and experience, I'd love to hear them. Otherwise, please just leave it!
I see the issue as largely not as big of an issue for most mailing list than a more generic mail forwarding system. I suspect most mailing lists are setup for only subscribers posting, which cuts out a lot of the spam issues, and many of those also it is preferred to reply to list instead of poster so the munged from isn't as important.
I suspect the major use of non-subscriber lists would be support lists where the subscribers are the support team, so more able to handle the technical challenges of dealing with the spam issues.
For a mailing list, because reply-all is more often used, it make differentiated munged From's (if needed), you likely want to plus-hack the From line (listname+sender@listdomain), though that might also create duplicate posts.
-- Richard Damon
On Sat, 2018-03-31 at 17:57 -0400, Richard Damon wrote:
On 3/31/18 3:35 PM, Lindsay Haisley wrote:
On Sat, 2018-03-31 at 14:50 -0400, Richard Damon wrote:
To me the issue sounds like why is fmp.com forwarding spam?
If this is a case of fmp.com offering forwarding mailboxes to users, who might be using gmail as a final destination, then yes, fmp needs to try to be as good at detecting spam as gmail or users need to accept the increased spam levels. If pigs could fly ....! I do the very best job I can of filtering spam from inbound email, and get about 90% of it, maybe more, but fighting spam is a forever job of whack-a-mole. I certainly wish that I could do as good a job of parsing spam from legit email as Gmail does, but I'm a one-person shop, and have many tasks. Gmail has dozens, perhaps hundreds of very smart people assigned to managing their spam filtering, and they do a very good job of it. I could _never_ hope to match their efficiency or accuracy, nor could most small operations such as FMP Computer Services. But coming at least close is the job you sign up for in being a mail forwarder. You at least need to be good enough that you aren't seen by google as an uncaring domain, and maintain enough information that they can continue to do what they do well. Rest assured, we "come at least close". This is not an option here, it's a necessity. Email redirection is a feature of my MTA (Courier) and has been offered since FMP went into business in the 1990s. It's a standard feature of many MTAs and many ESPs offer it.
I've had to deal with Gmail's honey-potting before, and I can do it again if necessary. I don't imagine that you've ever done commercial email administration, Richard, or you might have something constructive to say instead of just spewing admonitions to "do better". I will admit, that I haven't had to do that sort of email administration. I have run mail servers for much smaller operations, and do understand the difficulties (one reason I don't anymore). Just
On 3/31/18 6:33 PM, Lindsay Haisley wrote: pointing out that if you have decided to go into that business, you really need a better story than 'its hard' to convince customers to use you if you can't meet there expectations and needs.
The problem is that Gmail is whitelisting based on the From address, rather than the Reply-To address, which should be an _option_ open to users. On Google's scale of operation, I'm just a fly on a dog turd so any feature which might benefit my users and subscribers is pretty much a no-nevermind for them. Which is why I was saying make a 1:1 mapping of From addresses to Reply-To addresses. The From address _has_ to be from an address at fmp.com, which is the reason for From-munging in the first place. If you don't understand how DMARC works, or the problems it causes, Mark, or someone else on this list can send you to a reference on it. The Reply-To address is EITHER the original Reply-To address on the received email, or, if it had none, the ORIGINAL From address. Mapping the Reply-To address to the munged From address makes no sense at all.
Another option is to deterministically munge the from address so every incoming email address gets a unique fmp address that it represents (it doesn't have to be absolutely unique, mostly unique is likely good enough), something like replace the at with _at_ and add a tail wart like _dmarc@fmp.com (so you can have other addresses an not worry about possible overlaps with those) and use that as the from address. Then a reply will only whitelist that specific original from address. Which, as I noted in my original post, will cause the Gmail user's mail account to end up with a whole lot of useless whitelisted address which would need to be deleted, and FMP's server might well end up getting blacklisted as a result. No more than if GMail did implement a white-list on Reply-To addresses. No, because the Reply-To address is the _original_ From address. Such a whitelisting would be useless as long as Gmail's policy with regard to DMARC rejection remains in place, but unless we get into some kind of meta-heading BS, it's the best we might do.
I think you aren't understanding the munging I am suggesting. If I sent a message that went through your system, (and my setup triggered your munging) would be something like:
richard_at_damon.family.org_dmarc@fmp.com
This, and exactly this would be the from address for every message I sent through your system to a gmail user. This would be the only address that would get white-listed due to my messages. There should be no additional whitelisting load due to this, unless I also contact them outside your system.
-- Richard Damon
On Sat, 2018-03-31 at 19:12 -0400, Richard Damon wrote:
I think you aren't understanding the munging I am suggesting. If I sent a message that went through your system, (and my setup triggered your munging) would be something like:
richard_at_damon.family.org_dmarc@fmp.com
This, and exactly this would be the from address for every message I sent through your system to a gmail user. This would be the only address that would get white-listed due to my messages. There should be no additional whitelisting load due to this, unless I also contact them outside your system.
This is an interesting idea.
-- Lindsay Haisley | "Humor will get you through times of no humor FMP Computer Services | better than no humor will get you through 512-259-1190 | times of humor." http://www.fmp.com | - Butch Hancock
Have you considered sending your message to the Mailop mailing list?
I know that there are a couple of Gmail admins / coworkers that are subscribed to Mailop and will respond to issues like this.
Plus, it might also be a better forum and get more engagement / suggestions / gratitude by others learning from your toils.
On 03/31/2018 12:31 PM, Lindsay Haisley wrote:
At some point Amazon (amazon.com) started publishing a DMARC "p=quarantine" policy, which means that any email which gets redirected and hits my dmarc_shield piece is going to have its From address re- written to "postmaster@fmp.com" (fmp.com has a proper SPF record).
I'm sure that Amazon is just one of /many/ companies that are working with DMARC. - Seeing as how some ~> more governments are (going to be) requiring DMARC, I expect that we will see more of this.
I don't know what Gmail's policy is with regard to "p=quarantine"
- whether it rejects such email outright or relegates it to the recipient's spam folder. I know that if the sending site publishes "p=reject", redirected email is refused by Gmail at the front door. I'll have to test the "p=quarantine" behavior.
I'm confident that Mailop subscribers can respond to this.
Here's the really annoying thing. My dmarc_shield processor rewrites the From header as per SOP for Mailman with the proper switch turned on. The From header address becomes "postmaster@fmp.com" with the original From address in the address comment (from xxx at yyz.com). If the email didn't already have a Reply-To address, the original From address is inserted as the Reply-To address. If a Gmail user replies to such an email, the reply goes to the Reply-To address, but Gmail **whitelists** the From address! Thereafter, any email which comes in with a munged From address is accepted, bypassing Gmail's otherwise pretty good spam filtering. I'm noticing a lot of spam email going out with From addresses for which a DMARC "p=reject" policy is published, which means that any such spam redirected to the Gmail user via FMP is also whitelisted. Bah! It's a fucking war zone out there!
I'm confident that Mailop subscribers can respond to this too. Probably including reasons as to why something is done.
I speculate that it's to prevent abuse of meaningless addresses being used in the From: address and causing replies to go somewhere other than back to the (purported) sender.
The only possible solution here would be to randomize the username portion of the rewritten From address, which makes the email look more like spam, and the Gmail user would end up with a whole lot of useless whitelisted address which would need to be deleted. Not to mention the fact that FMP's mail server might be blocked from sending ANY email to Gmail.
I initially thought about something like an MD5 hash of the (purported) From address. Though that still suffers from the multiple addresses being white listed. Despite that, I'd consider forwarding from a "forwarding" (sub)domain. Something to hopefully help articulate to the human looking at the complaints that the message is forwarded. Plus this I would expect this to help differentiate email reputation for fmp.com from the (sub)domain used for forwarding. (I don't know if a sub-domain would suffice or if it should be a different parallel / sibling domain, fmp-forwarding.com.)
-- Grant. . . . unix || die
On 03/31/2018 11:31 AM, Lindsay Haisley wrote:
At some point Amazon (amazon.com) started publishing a DMARC "p=quarantine" policy, which means that any email which gets redirected and hits my dmarc_shield piece is going to have its From address re- written to "postmaster@fmp.com" (fmp.com has a proper SPF record).
Why do you feel this is necessary?
I suppose it is possible that amazon publishises a DMARC policy and does NOT DKIM sign it's outgoing email but relies solely on SPF domain alignment to pass DMARC, but I think this would be a rare exception.
If the mail from Amazon is DKIM signed with an aligned domain and you make no transformations that would break that sig, i.e. you are a simple .forward or alias type forwarder, the DKIM sig will still validate at the receiver and you don't need to munge the From:
Here's the really annoying thing. My dmarc_shield processor rewrites the From header as per SOP for Mailman with the proper switch turned on. The From header address becomes "postmaster@fmp.com" with the original From address in the address comment (from xxx at yyz.com). If the email didn't already have a Reply-To address, the original From address is inserted as the Reply-To address. If a Gmail user replies to such an email, the reply goes to the Reply-To address, but Gmail **whitelists** the From address! Thereafter, any email which comes in with a munged From address is accepted, bypassing Gmail's otherwise pretty good spam filtering. I'm noticing a lot of spam email going out with From addresses for which a DMARC "p=reject" policy is published, which means that any such spam redirected to the Gmail user via FMP is also whitelisted. Bah! It's a fucking war zone out there!
The first question is why would the ultimate gmail recipient reply to the spam in the first place.
The next question is assuming it is spam, does it originate from an amazon server. If not, it should fail DMARC when you receive it and you should consider honoring the amazon DMARC police and not forward the mail.
And if it does originate from an amazon server with a valid DKIM sig are you making transformations that invalidate the DKIM sig?
Again, if not, if you are a simple forwarder, you shouldn't need to mung the From:.
I understand part of the intent is a heads-up to people like me, and in my case, I am not a simple forwarder, but I'm hopeful that eventually at least, ARC can help, but I still don't understand why this is an issue for you.
It seems your case is simple. If it fails DMARC when it reaches you, honor the p=quarantine and don't forward the mail. If it passes DMARC based on DKIM, forward it without munging the From:. That leaves only the case where it passes DMARC solely on SPF, and my guess is that this is an empty or almost empty set.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (4)
-
Grant Taylor -
Lindsay Haisley -
Mark Sapiro -
Richard Damon