On 05/19/2014 09:56 AM, Sergio Durigan Junior wrote:

I am not exactly sure, but I wanted to try "fixing" this and see if things got better. The real problem is Google/Gmail tagging the mailing list messages as Spam, but since the headers are fine, the cause must be something else.

My experience is that goggle sometimes will stop tagging emails as spam if they have a valid dkim signature. This probably means either:

1. rewriting the from to be your local mailman server and signing the outbound message with dkim OR
2. passing through any valid DKIM signature, which means running a recent version of mailman and avoid adding any headers/footers which would break the signature. OR
3. possibly having no DKIM signature at all (better than having a signature which fails verification)

If there is a dkim signature and it fails google will treat it as spam

Goggle and the other various FREEMAILs maintain a rating of the credibility of email coming from your mailserver. DKIM signing (with a signature that will verify correctly) will increase your rating.

I was getting messages like the following from gmail and they stopped when I implemented DKIM signatures (must be a keysize of at least 1024):

Jan 21 16:36:06 myserv postfix/smtp[5374]: 68972108248: host gmail-smtp-in.l.google.com[74.125.142.26] said: 421-4.7.0 [1.2.3.4] Our system has detected an unusual rate of 421-4.7.0 unsolicited mail originating from your IP address. To protect our 421-4.7.0 users from spam, mail sent from your IP address has been temporarily 421-4.7.0 blocked. Please visit http://www.google.com/mail/help/bulk_mail.html 421 4.7.0 to review our Bulk Email Senders Guidelines. ih5si10430009icb.36 - gsmtp (in reply to end of DATA command)

You may also benefit from using DMARC (probably not technically, but politically).

You may not be doing bulk mailings, but the information here may be helpful: https://support.google.com/mail/answer/81126

Also you can try things like https://duckduckgo.com/?q=gmail+flags+my+mail+as+spam&t=canonical <https://duckduckgo.com/?q=gmail+flags+my+mail+as+spam&t=canonical> Also check http://multirbl.valli.org/ to make sure your mail server is not on any DNS blocklists.

Nataraj

On 05/20/2014 05:17 AM, Stephen J. Turnbull wrote:

Natu writes:

...

If there is a dkim signature and it fails google will treat it as spam

Note that, taking your words literally, this is against the DKIM RFCs -- a failed signature is supposed to be treated the same as a lack of a signature.

That doesn't mean that Google can't or doesn't use it. In particular, if the percentage of spam in the "failed or no signature" class goes up (which it will if you start signing), your "failed or no signature" rating will slump. That effect might be enough to send unsigned mail from your site to the spambucket.

But you can be sure that a valid DKIM signature will improve your site's reputation.

I believe you are correct Stephen. When I had the problem, in addition to signing outbound messages, I also increased the level of spam control. In particular I have a small number of users that forward their mail to gmail, and gmail was blocking us because of the spam that was getting forwarded. I suspect that it may only have effected the forwarded email from those accounts. I also made efforts to increase spam detected and thus reduced the amount of forwarded spam.

With Yahoo, I was also getting complaints that I was sending them spam (i.e. rejected mail from their smtp server). I registered with yahoo's complaint feedback loop, where they claim they will forward to you (the registered Mail administrator for the domain) any email they receive from your domain that they think is spam or that their users have flagged as spam. After I both turned on dkim and registered, I never got a single complaint from them. Note that we do NOT send any kind of marketing mail from this mailserver.

One other thing that might cause problems with the freemail providers is as follows: If mailman sends a large number of messages to a freemail provider that are almost the same, i.e. the exact same email, but addressed to many different recipient, then they might think your are sending bulk unsolicited mail. Having your mailing list mail test positive in DCC is probably a good indication that this is happening.
There may be an option in mailman to ensure that outbound messages from a single post don't all look identical. If so, you could try turning that on.

Also, if you are dealing with lists that send high volumes of email to gmail, destination specific metering of your outbound mail, as well as adjusting the number of recipients in a single smtp transmission may help. If you are running postfix, discussions about how to implement this can be found in the postfix archives.

Nataraj

Natu writes:

When I had the problem, in addition to signing outbound messages, I also increased the level of spam control. In particular I have a small number of users that forward their mail to gmail, and gmail was blocking us because of the spam that was getting forwarded. I suspect that it may only have effected the forwarded email from those accounts. I also made efforts to increase spam detected and thus reduced the amount of forwarded spam.

It sometimes happens that there is a way past a host's inbound filters. This shouldn't happen, of course, so when running mailing lists you want to filter on the way in.

But it might be an "amusing" test to hook up the spam checker to the *outgoing* MTA -- if it catches anything on the way out, then you have a backdoor into your MTA somewhere. For example, I trusted my secondary MXes to filter (in fact the evidence suggests their filtering was better than mine), until one day the spamchecker at one of them fell over, and they continued to forward mail addressed to us -- *including* the spam.

I happened to be online when it happened, or shortly after, so only a few hundred (!) got through, mostly to invalid addresses (thank heavens for really stupid spammers! at least that time :-). It turned out that *for me* there was no degradation of service when I stopped whitelisting that MX, and I ended up just checking *all* mail always regardless of my trust in the MX. But YMMV, be careful -- spam checking is resource intensive!

Also, if you're an ESP and your users keep their addressbooks on Windows machines, you probably should be filtering outbound anyway. In that case it's probably a good idea to keep your lists on a separate IP from your other users, which allows you to filter only on the way in (much more efficient).

With Yahoo, I was also getting complaints that I was sending them spam (i.e. rejected mail from their smtp server). I registered with yahoo's complaint feedback loop, where they claim they will forward to you (the registered Mail administrator for the domain) any email they receive from your domain that they think is spam or that their users have flagged as spam. After I both turned on dkim and registered, I never got a single complaint from them.

Cool! Nice to hear that actually works (we've heard complaints here about Yahoo!s responsiveness in the past).

One other thing that might cause problems with the freemail providers is as follows: If mailman sends a large number of messages to a freemail provider that are almost the same, i.e. the exact same email, but addressed to many different recipient, then they might think your are sending bulk unsolicited mail. Having your mailing list mail test positive in DCC is probably a good indication that this is happening.
There may be an option in mailman to ensure that outbound messages from a single post don't all look identical. If so, you could try turning that on.

The option is "personalization". Many lists use it so the user gets a direct URL to their personal config page in the footer.

For the big ESPs like Yahoo!, just registering with their feedback loop or bulk mailer list often helps here, too. It may be worth experimenting with that, because turning on personalization may be a serious resource drain if you have hundreds of subscribers at one domain.

Also, if you are dealing with lists that send high volumes of email to gmail, destination specific metering of your outbound mail, as well as adjusting the number of recipients in a single smtp transmission may help. If you are running postfix, discussions about how to implement this can be found in the postfix archives.

All good advice! Thank you for an excellent summary!

Steve

On 05/19/2014 09:56 AM, Sergio Durigan Junior wrote:

I am not exactly sure, but I wanted to try "fixing" this and see if things got better. The real problem is Google/Gmail tagging the mailing list messages as Spam, but since the headers are fine, the cause must be something else.

My experience is that goggle sometimes will stop tagging emails as spam if they have a valid dkim signature. This probably means either:

1. rewriting the from to be your local mailman server and signing the outbound message with dkim OR
2. passing through any valid DKIM signature, which means running a recent version of mailman and avoid adding any headers/footers which would break the signature. OR
3. possibly having no DKIM signature at all (better than having a signature which fails verification)

If there is a dkim signature and it fails google will treat it as spam

Goggle and the other various FREEMAILs maintain a rating of the credibility of email coming from your mailserver. DKIM signing (with a signature that will verify correctly) will increase your rating.

I was getting messages like the following from gmail and they stopped when I implemented DKIM signatures (must be a keysize of at least 1024):

Jan 21 16:36:06 myserv postfix/smtp[5374]: 68972108248: host gmail-smtp-in.l.google.com[74.125.142.26] said: 421-4.7.0 [1.2.3.4] Our system has detected an unusual rate of 421-4.7.0 unsolicited mail originating from your IP address. To protect our 421-4.7.0 users from spam, mail sent from your IP address has been temporarily 421-4.7.0 blocked. Please visit http://www.google.com/mail/help/bulk_mail.html 421 4.7.0 to review our Bulk Email Senders Guidelines. ih5si10430009icb.36 - gsmtp (in reply to end of DATA command)

You may also benefit from using DMARC (probably not technically, but politically).

You may not be doing bulk mailings, but the information here may be helpful: https://support.google.com/mail/answer/81126

Also you can try things like https://duckduckgo.com/?q=gmail+flags+my+mail+as+spam&t=canonical <https://duckduckgo.com/?q=gmail+flags+my+mail+as+spam&t=canonical> Also check http://multirbl.valli.org/ to make sure your mail server is not on any DNS blocklists.

Nataraj

On 05/20/2014 05:17 AM, Stephen J. Turnbull wrote:

Natu writes:

...

If there is a dkim signature and it fails google will treat it as spam

Note that, taking your words literally, this is against the DKIM RFCs -- a failed signature is supposed to be treated the same as a lack of a signature.

That doesn't mean that Google can't or doesn't use it. In particular, if the percentage of spam in the "failed or no signature" class goes up (which it will if you start signing), your "failed or no signature" rating will slump. That effect might be enough to send unsigned mail from your site to the spambucket.

But you can be sure that a valid DKIM signature will improve your site's reputation.

I believe you are correct Stephen. When I had the problem, in addition to signing outbound messages, I also increased the level of spam control. In particular I have a small number of users that forward their mail to gmail, and gmail was blocking us because of the spam that was getting forwarded. I suspect that it may only have effected the forwarded email from those accounts. I also made efforts to increase spam detected and thus reduced the amount of forwarded spam.

With Yahoo, I was also getting complaints that I was sending them spam (i.e. rejected mail from their smtp server). I registered with yahoo's complaint feedback loop, where they claim they will forward to you (the registered Mail administrator for the domain) any email they receive from your domain that they think is spam or that their users have flagged as spam. After I both turned on dkim and registered, I never got a single complaint from them. Note that we do NOT send any kind of marketing mail from this mailserver.

One other thing that might cause problems with the freemail providers is as follows: If mailman sends a large number of messages to a freemail provider that are almost the same, i.e. the exact same email, but addressed to many different recipient, then they might think your are sending bulk unsolicited mail. Having your mailing list mail test positive in DCC is probably a good indication that this is happening.
There may be an option in mailman to ensure that outbound messages from a single post don't all look identical. If so, you could try turning that on.

Also, if you are dealing with lists that send high volumes of email to gmail, destination specific metering of your outbound mail, as well as adjusting the number of recipients in a single smtp transmission may help. If you are running postfix, discussions about how to implement this can be found in the postfix archives.

Nataraj

Natu writes:

When I had the problem, in addition to signing outbound messages, I also increased the level of spam control. In particular I have a small number of users that forward their mail to gmail, and gmail was blocking us because of the spam that was getting forwarded. I suspect that it may only have effected the forwarded email from those accounts. I also made efforts to increase spam detected and thus reduced the amount of forwarded spam.

It sometimes happens that there is a way past a host's inbound filters. This shouldn't happen, of course, so when running mailing lists you want to filter on the way in.

But it might be an "amusing" test to hook up the spam checker to the *outgoing* MTA -- if it catches anything on the way out, then you have a backdoor into your MTA somewhere. For example, I trusted my secondary MXes to filter (in fact the evidence suggests their filtering was better than mine), until one day the spamchecker at one of them fell over, and they continued to forward mail addressed to us -- *including* the spam.

I happened to be online when it happened, or shortly after, so only a few hundred (!) got through, mostly to invalid addresses (thank heavens for really stupid spammers! at least that time :-). It turned out that *for me* there was no degradation of service when I stopped whitelisting that MX, and I ended up just checking *all* mail always regardless of my trust in the MX. But YMMV, be careful -- spam checking is resource intensive!

Also, if you're an ESP and your users keep their addressbooks on Windows machines, you probably should be filtering outbound anyway. In that case it's probably a good idea to keep your lists on a separate IP from your other users, which allows you to filter only on the way in (much more efficient).

With Yahoo, I was also getting complaints that I was sending them spam (i.e. rejected mail from their smtp server). I registered with yahoo's complaint feedback loop, where they claim they will forward to you (the registered Mail administrator for the domain) any email they receive from your domain that they think is spam or that their users have flagged as spam. After I both turned on dkim and registered, I never got a single complaint from them.

Cool! Nice to hear that actually works (we've heard complaints here about Yahoo!s responsiveness in the past).

One other thing that might cause problems with the freemail providers is as follows: If mailman sends a large number of messages to a freemail provider that are almost the same, i.e. the exact same email, but addressed to many different recipient, then they might think your are sending bulk unsolicited mail. Having your mailing list mail test positive in DCC is probably a good indication that this is happening.
There may be an option in mailman to ensure that outbound messages from a single post don't all look identical. If so, you could try turning that on.

The option is "personalization". Many lists use it so the user gets a direct URL to their personal config page in the footer.

For the big ESPs like Yahoo!, just registering with their feedback loop or bulk mailer list often helps here, too. It may be worth experimenting with that, because turning on personalization may be a serious resource drain if you have hundreds of subscribers at one domain.

Also, if you are dealing with lists that send high volumes of email to gmail, destination specific metering of your outbound mail, as well as adjusting the number of recipients in a single smtp transmission may help. If you are running postfix, discussions about how to implement this can be found in the postfix archives.

All good advice! Thank you for an excellent summary!

Steve