Amazon SES and Verified Senders
Hello,
Does anyone have any ideas on how to deal with this dilemma: I am running Mailman+Postfix+Ubuntu in Amazon AWS, and using Amazon SES as a relay. Although, this problem isn't unique to just SES. This problem is common among many relay services, DynDNS to name another.
To prevent against spam and abuse, SES, DynDNS and other relay services require that you VERIFY each SENDER before you can send mail from that email address.
When running Mailman, each member of every list is the SENDER, and it is not practical or even possible to verify every sender.
I have two workarounds, but neither one is ideal.
Option 1) In Mailman, I can enable: "Hide the sender of a message, replacing it with the list address (Removes From, Sender and Reply-To fields) "
This will mean any post to a list will show only the list, and the list will be the return address (that is ok, even desirable). But the problem with this is, that unless the poster includes a signature, there is no way to know who it came from when the other list members receive the post. We need to know who posted to the lists, so we know who we're replying to, and if we need their email to take the conversation off-list, etc.
Option 2) In Postfix, maintain the canonical file so that each member address will be rewritten with a mailman domain address. Example:
In /etc/postfix/canonical: jondoe@hotmail.com jondoe.at.hotmail.com@mymailmandomain.com
Because I've approved the domain @mymailmandomain.com with DKIM in Amazon SES, and email from jondoe@hotmail.com will be rewritten as From: jondoe.at.hotmail.com@mymailmandomain.com, and Amazon SES will permit it. The problem with this is that it still doesn't accurately reflect the senders real email anywhere, and another list member might pull the bogus "jondoe.at.hotmail.com@mymailmandomain.com" address, and try to send to this person off-list, or add the bogus email address to their address book....not good. Also, a cronjob will have to regularly build and update the canonical addresses, which in itself isn't that a big deal, but is another point of failure.
Does anybody else have this problem, and how do you deal with it? Are there better solutions? Perhaps their is a better way to do #2 so that the From: address is rewritten to be acceptable to Amazon SES, but displays something that is more useful and friendly to recipients?
Thanks for any input!
DW
Duane Winner writes:
AFAICS, you lose, then. Specifically, if you obfuscate the sender, you are probably in violation of the AUP for the relay service you are using.
Have you tried working directly with Amazon SES to resolve the issue?
I wonder if a third possibility, namely encapsulating every message in another message sent by Mailman, would do the trick. Ie, require all subscribers to subscribe to the digest edition of the list.
Steve
Have you tried working directly with Amazon SES to resolve the issue?
I have not personally, but on their forums, others have posed have the same problem, and following is a reply directly from AWS:
If I knew how to replace the "Friendly name" with something else, that /might/ be another solution, but I haven't been able to figure out how to do that. I'm guessing that is a Postfix question.
That's not an option. We use lists for minute-by-minute round table conversations and tech support.
If find it hard to believe that this isn't a common issue, since many people need a 3rd party SMTP relay.... Is it possible that this just doesn't come up much because people who run Mailman run everything, including their own SMTP relay in-house?
-DW
Duane Winner wrote:
I could be wrong, but I don't think Postfix can do that, but you could easily do that with a custom handler in Mailman. See <http://wiki.list.org/x/l4A9>.
The difficulty with this approach is it makes it harder to reply only to the poster.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
Well, it would be pretty confusing, but you could put the real address in a comment:
From: "Stephen J. Turnbull" (stephen@xemacs.org) <fake@acceptable.com>
You could also do
From: "Stephen J. Turnbull" <fake@acceptable.com> Reply-To: stephen@xemacs.org, list@your-host.org
People would have to be careful to clean out the unneeded address, but everything they need is there.
This is not a practical comment, but... I am amazed Amazon is checking the header From instead of the header Sender and envelope sender... and recommending breaking standards to get around what they are doing. They not only break mailing lists but the types of forwarding that preserve the original From header. RFC 2822:
The "From:" field specifies the author(s) of the message, that is, the mailbox(es) of the person(s) or system(s) responsible for the writing of the message. The "Sender:" field specifies the mailbox of the agent responsible for the actual transmission of the message.
The latter, and the envelope sender, is what they should care about.
All right, enough, back to real life where you just have to deal with crazy stuff you can't control...
Joseph Brennan Columbia University Information Technology
Duane Winner writes:
Now that is a real WTF. "We suggest that you automatically obscure the sources of the spam you're passing through to your users so that it will pass our verification process."
But since that's what they recommend, you can probably sue them if they try to kick you off for doing it.
Actually, I would guess that the majority of people using Mailman to host lists are relatively unsophisticated and do so on a virtual host which provides Mailman for them. That wouldn't do them much good if their users couldn't post. Of course the folks who are answering questions on this list probably generally do have their own MXes.
I suppose this problem will come up more frequently as IaaS becomes more popular.
Stephen J. Turnbull writes:
It just occurred to me that it would be possible to do what XEmacs does (as a convenience/vanity feature): issue all users an address at your domain, eg of the form "closetotheedge%yahoo.com@example.com".[1] It should be fairly easy to set up the MTA to rewrite and forward mail to those addresses. The main question at this point would be how to link those addresses into the verification process.
In our case, we don't issue all users vanity addresses, only those who are committers, and it's a labor-intensive process involving correspondence about choosing and confirming the address. But it could be done for all users automatically as above.
Steve
Footnotes: [1] Or for a retro mood, example.com!yahoo.com!closetotheedge. ;-)
On Fri, Jan 11, 2013 at 09:27:23AM -0800, Duane Winner wrote:
Does anyone have any ideas on how to deal with this? [snip]
Amazon's cloud has been a prolific long-term source of spam and other forms of abuse (e.g., brute-force ssh attacks). Thus it's long since been a best practice to refuse all email from hosts in compute-1.amazonaws.com and compute.amazonaws.com subdomains, and no doubt unless serious efforts are made to address this, blocking of incoming SMTP connections from Amazon's cloud will eventually increase in both scope and coverage.
Not that this is your fault, of course. But unless you can convince Amazon to take an active interest in controlling *outbound* abuse from their operation, there's little you can do about it.
So my recommendation is to set up a VPN tunnel from your Mailman host to a (secure) SMTP relay outside their network space. (And of course outside other problematic network spaces; check Spamhaus and similar resources first.) Let the host inside Amazon do the heavy lifting of running Mailman and so on, let the one outside do the simple work of just relaying outbound traffic. OpenBSD+postfix+BIND on very low-end hardware should suffice, and as long as it only relays traffic handed off via the VPN, you should be okay.
(Incidentally, verifying senders has no anti-spam value. I get spam by the megabyte in my spamtraps all day, every day, from verified senders and from verified hosts.)
---rsk
Duane Winner writes:
AFAICS, you lose, then. Specifically, if you obfuscate the sender, you are probably in violation of the AUP for the relay service you are using.
Have you tried working directly with Amazon SES to resolve the issue?
I wonder if a third possibility, namely encapsulating every message in another message sent by Mailman, would do the trick. Ie, require all subscribers to subscribe to the digest edition of the list.
Steve
Have you tried working directly with Amazon SES to resolve the issue?
I have not personally, but on their forums, others have posed have the same problem, and following is a reply directly from AWS:
If I knew how to replace the "Friendly name" with something else, that /might/ be another solution, but I haven't been able to figure out how to do that. I'm guessing that is a Postfix question.
That's not an option. We use lists for minute-by-minute round table conversations and tech support.
If find it hard to believe that this isn't a common issue, since many people need a 3rd party SMTP relay.... Is it possible that this just doesn't come up much because people who run Mailman run everything, including their own SMTP relay in-house?
-DW
Duane Winner wrote:
I could be wrong, but I don't think Postfix can do that, but you could easily do that with a custom handler in Mailman. See <http://wiki.list.org/x/l4A9>.
The difficulty with this approach is it makes it harder to reply only to the poster.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
Well, it would be pretty confusing, but you could put the real address in a comment:
From: "Stephen J. Turnbull" (stephen@xemacs.org) <fake@acceptable.com>
You could also do
From: "Stephen J. Turnbull" <fake@acceptable.com> Reply-To: stephen@xemacs.org, list@your-host.org
People would have to be careful to clean out the unneeded address, but everything they need is there.
This is not a practical comment, but... I am amazed Amazon is checking the header From instead of the header Sender and envelope sender... and recommending breaking standards to get around what they are doing. They not only break mailing lists but the types of forwarding that preserve the original From header. RFC 2822:
The "From:" field specifies the author(s) of the message, that is, the mailbox(es) of the person(s) or system(s) responsible for the writing of the message. The "Sender:" field specifies the mailbox of the agent responsible for the actual transmission of the message.
The latter, and the envelope sender, is what they should care about.
All right, enough, back to real life where you just have to deal with crazy stuff you can't control...
Joseph Brennan Columbia University Information Technology
Duane Winner writes:
Now that is a real WTF. "We suggest that you automatically obscure the sources of the spam you're passing through to your users so that it will pass our verification process."
But since that's what they recommend, you can probably sue them if they try to kick you off for doing it.
Actually, I would guess that the majority of people using Mailman to host lists are relatively unsophisticated and do so on a virtual host which provides Mailman for them. That wouldn't do them much good if their users couldn't post. Of course the folks who are answering questions on this list probably generally do have their own MXes.
I suppose this problem will come up more frequently as IaaS becomes more popular.
Stephen J. Turnbull writes:
It just occurred to me that it would be possible to do what XEmacs does (as a convenience/vanity feature): issue all users an address at your domain, eg of the form "closetotheedge%yahoo.com@example.com".[1] It should be fairly easy to set up the MTA to rewrite and forward mail to those addresses. The main question at this point would be how to link those addresses into the verification process.
In our case, we don't issue all users vanity addresses, only those who are committers, and it's a labor-intensive process involving correspondence about choosing and confirming the address. But it could be done for all users automatically as above.
Steve
Footnotes: [1] Or for a retro mood, example.com!yahoo.com!closetotheedge. ;-)
On Fri, Jan 11, 2013 at 09:27:23AM -0800, Duane Winner wrote:
Does anyone have any ideas on how to deal with this? [snip]
Amazon's cloud has been a prolific long-term source of spam and other forms of abuse (e.g., brute-force ssh attacks). Thus it's long since been a best practice to refuse all email from hosts in compute-1.amazonaws.com and compute.amazonaws.com subdomains, and no doubt unless serious efforts are made to address this, blocking of incoming SMTP connections from Amazon's cloud will eventually increase in both scope and coverage.
Not that this is your fault, of course. But unless you can convince Amazon to take an active interest in controlling *outbound* abuse from their operation, there's little you can do about it.
So my recommendation is to set up a VPN tunnel from your Mailman host to a (secure) SMTP relay outside their network space. (And of course outside other problematic network spaces; check Spamhaus and similar resources first.) Let the host inside Amazon do the heavy lifting of running Mailman and so on, let the one outside do the simple work of just relaying outbound traffic. OpenBSD+postfix+BIND on very low-end hardware should suffice, and as long as it only relays traffic handed off via the VPN, you should be okay.
(Incidentally, verifying senders has no anti-spam value. I get spam by the megabyte in my spamtraps all day, every day, from verified senders and from verified hosts.)
---rsk
participants (5)
-
Duane Winner -
Joseph Brennan -
Mark Sapiro -
Rich Kulawiec -
Stephen J. Turnbull