On Wednesday 07 October 2009 21:00:52 Ian Eiloart wrote:

As far as I recall, Mailman removes DKIM signatures, yes and re-signs messages. not that I recall though the MTA is free to sign it on the way out and I encourage all list owners to do so.

You're saying that with ADSP, that's not adequate unless Mailman first rewrites the "From:" address. yes, as its easiest place in the whole signing verification scenario to make a change that benefits the most people without adversely effecting anyone significantly.

Some lists are configured to do this already, I didn't know this. Anyone know who these are and if they incur any problems as result of this rewrite?

the question is what to do about those that don't.

[not mailing list obligation to fix problem]..., but it is sensible to discuss how list managers could address the problem, thank you

It seems to me that it's sensible for the list software to test the DKIM signature before and after any changes it makes to the message. You can tell from the mailing list settings if it will break without revalidating it. Same policies apply though. And apply these policies:

Good before, good after: deliver the message as normal.

Bad before (broken DKIM sig, or missing a sig that ADSP says should be there), reject the message at SMTP time, but that's the MTA's job. yes. very easy to do. including dropping broken sigs etc where dkim=all?

Good before, "discardable" after: ....(cut).... Rejection at SMTP time The MTA would need to know whether the list was going to rewrite the From: header, I suppose. yes - requested feature just added for this: https://sourceforge.net/tracker/?func=detail&aid=2874043&group_id=269812&atid=1147704

Bad before, bad after: (DKIM or ADSP fail), reject the message at SMTP time if possible. That's the job of the border MTA, though. yes

Bad before, good after: presumably this is a list that rewrites From headers, but I don't think we want this message, so we should reject at SMTP time. agree

If the ADSP policy is "all", then I don't see a problem. The recipient should not reject a message from a mailing list just because it doesn't carry a matching signature. This is expected behaviour: we know the message was sent with a signature,

1. has a List-ID (or other signature) - must be a mailist. This allows email spoofers just to add List-ID tags or a simple email characteristic to bypass checking.
2. manage a whitelist of maillists that have subscribers in the domain, that can't be easily spoofed. Pretty easy for small domains but many thousand user bases requires more admin time or run the risk of a user whitelisting a spoofer IP address.
3. originator specified third party signatures - discussion (re)-starting on IETF WG list on this. Bit labour intensive on the sender part. (http://mipassoc.org/pipermail/ietf-dkim/2009q4/thread.html)

we know the list was going to break or remove the signature. We should, however, look for a signature from the mailing list, yes good. this falls nicely into #2 above, A strong signature of the email list to whitelist. More fully mailing lists should DKIM sign emails (after dropping invalid signatures).

we know the message came from a mailing list, this actually is the hard bit. Options for the recipient verifier are:

and we should apply initial reputation tests (and modifications) to the list, not the original sender. If the list has good reputation, we should assume that it tested ADSP, and found a good DKIM signature on the original message. We can, therefore continue and check (and modify) the reputation of the original sender.

If you have a method of determining if it a list in 1-3 above then this works as mailing list domains should gain good reputation quickly (though varying list policy between lists or large list servers like sourceforge.net could hide bad in the reputation of the majority).

If you are blindly assessing an email without knowledge that is a mailing list what do you see? You can check the reputation of the domain on a third party DKIM signature and treat this like a mail list above. What would it mean for a new list domain/third party sender? My assumption, based on not much experience with reputation services, is neutral reputation will allows the email though.

Scenario 1: Putting names on this. Phisher Mallory buys a domain friendly.org sets up DKIM on it. Mallory sends email to Bob saying that Bob needs to validate his credentials with Bank of Alice. The author of the email looks like Bank of Alice (who says has an ADSP policy of dkim=all). Bob's university who filters his email doesn't know about friendly.org, so checks it reputation: (pick an ending) a) it has a neutral reputation, so lets it though and Bob gets scammed. b) it has a negative reputation. and gets blocked after Mallory got detected sending lots of phishing email and gaining lots of money out of the Bank of Alice's customers all for the cost of a domain that wasn't visible to the end victim.

Sure this is all the receiver's problem. The receiver's problem would also be easier if, say in 4 years time, when mailman 3 and 4 are common place, and has been rewriting From addresses for years, and the end verifier has a small whitelist on third party signatures and most spoofing is dropped using ADSP.

That last paragraph makes the job of reputation assignment harder where mailing lists are concerned - but that's to be expected. The whole point of DKIM, as far as I'm concerned, is to allow more sophisticated assessment and assignment of reputation scores. Though it can contribute to reputation scores this is taking a strong cryptographic signature method and contributing it towards a spam score. This only goes so far defeating some forms of phishing.

The IETF WG Charter (http://tools.ietf.org/wg/dkim/charters) describes DKIM as "Taken together[signature and policy], these will assist receiving domains in detecting (or ruling out) certain forms of spoofing as it pertains to the signing domain."

not there yet..but I can't too many easy solutions either.

-- Daniel Black Infrastructure Administrator CAcert

--On 8 October 2009 00:21:08 +1100 Daniel Black <daniel@cacert.org> wrote:

...

It seems to me that it's sensible for the list software to test the DKIM signature before and after any changes it makes to the message. You can tell from the mailing list settings if it will break without revalidating it. Same policies apply though.

Doesn't that depend on what's signed, as well as what the list is going to alter? Agreed, though, this is an implementation detail.

-- Ian Eiloart IT Services, University of Sussex 01273-873148 x3148 For new support requests, see http://www.sussex.ac.uk/its/help/

--On 8 October 2009 00:21:08 +1100 Daniel Black <daniel@cacert.org> wrote:

...

That last paragraph makes the job of reputation assignment harder where mailing lists are concerned - but that's to be expected. The whole point of DKIM, as far as I'm concerned, is to allow more sophisticated assessment and assignment of reputation scores. Though it can contribute to reputation scores this is taking a strong cryptographic signature method and contributing it towards a spam score. This only goes so far defeating some forms of phishing.

DKIM helps you determine whether an email really was sent by the purported sending domain. If it wasn't, that's bad. If it was, that doesn't mean it's good, but it does allow you to check the sending domain (or sender) against your reputation database, and to modify your view of the sender's reputation based on the current email.

Currently, all we really have that's useful for reputation assignment are content (too complex) and IP source (too often shared between good and bad senders). I'm not saying they're not useful, and there are even some sender addresses that you can blacklist.

Without DKIM and SPF, you can't really build a reputation infrastructure for sender addresses, because for most spam you're checking or modifying the reputation of an innocent third party.

-- Ian Eiloart IT Services, University of Sussex 01273-873148 x3148 For new support requests, see http://www.sussex.ac.uk/its/help/

Stephen,

Thanks you for your responses.

In that case it is very often a violation of ... RFC 5322). Surely you already know that!

On Thursday 08 October 2009 17:07:30 Stephen J. Turnbull wrote: thanks for the reminder.

That's a *lot* of history of best practice that you are dismissing, There's lots I don't know and I am keen to learn too.

it's not going to be acceptable to a lot of folks, Apart from the assertions of mailing list software developers I'm yet to receive a strong assertion from list operators or users. Even less forthcoming is the reasons the lack of acceptability that could guide standards or implementations.

1. global dkim-friendly= true to disable list modifications parameters
2. From: rewriting

</RANT>

Wouldn't it be more straightforward (not to mention that it would work for many more lists) to have an LDSP RFC, whose first draft simply takes the ADSP RFC and substitutes "mailing list" for "author" everywhere, and "RFC 2369 and RFC 2919 headers" for "From"? (The point of multiple headers is that "active" headers like List-Subscribe could contain bogus URLs.) Doing so would allow List-* headers to be added by every spoofer, add their own signature and get immunity from spoofing every author domain while the end user doesn't notice because the List-* headers are hidden in the MUA (in most cases).

<RANT> and in general sucks for users of discussion lists. Personally, I'd much rather have my posts dropped. "Oh yes, your ISP regularly drops mail because they use broken spam-fighting practices. It's not just us, it's every list that conforms to one of the oldest Internet standards. If you want to receive your list mail, either subscribe with an address hosted at a decent ISP, or get your current one to fix their spam filters." Most of my users are well-informed, and quite sympathetic to that argument because they've seen it happen any number of times. I really would not appreciate it if "worst practices" were to become widespread because they cater to the unwashed who just don't want to receive spam more any phishing than spam. and don't care who pays the cost (as long as it's not them). there's a development cost that I'll consider contributing to but I'm not going to develop stuff that has no change of being accepted. I will consider writing patches for:

A second draft might add "If the list's host implements ADSP itself (as verification i'm assuming) , it could also sign the author headers relevant to ADSP."

A proposal for author domains to authorize third party lists is in development. http://mipassoc.org/pipermail/ietf-dkim/2009q4/012592.html

It does place a large burden on author domains to deploy DNS records pre- presenting every list their user base send email too.

Perhaps if it is known that the DKIM signature of the author's host is going to remain valid, this would be idea. you *don't* sign it, allowing the recipient to authenticate both the author and the list. Adding another list signature gives the mechanism to the recipient to validate the list and doesn't invalidate the original author signature.

The only real problem with this is getting the big ISPs to implement, but that's nothing new. In fact if it's as easy as adding routines to process the RFC 2369 + RFC 2919 set of headers "just like" ADSP handles "From:", I bet most would be happy to sign on. So this is for mailing list that send though big ISPs that can't apply their own DKIM signatures? Not totally sure what you mean here though signing is the easy part compared to applying filtering on verification results.

...

...

Some lists are configured to [rewrite From:] already,

I didn't know this. Anyone know who these are and if they incur any problems as result of this rewrite?

Announce lists .........fit into the ADSP framework quite well, so are basically irrelevant to your proposal.

ack.

Anonymous discussion lists are special-purpose lists used by folks like victims of domestic violence. These are a very good thing IMO, but again they are not a model for other lists. ack.

...

If you are blindly assessing an email without knowledge that is a mailing list what do you see?

If the list doesn't implement any of RFC 2369 (published 1998) or RFC 2919 (published 2001), the joke is on it. its more the consideration that, as a verifier, you shouldn't trust a List-ID as a way to bypass filtering.

Otherwise you shouldn't be blind. I think it is reasonable to assume that mailing lists are easily identifiable by the presence of those headers. For that precise reason, I propose that they be used instead of "From:" for ADSP-like authentication of mailing lists.

This is so obvious that I suspect there's some "good" reason why it won't work, as provided.

but as long as a harmful alternative is being suggested, may as well give it a try. please explain why this is harmful.

It seems that anonymous discussion lists have worked so slightly manipulated From addresses would also work. Just take the return-path variant, From: "Daniel Black via mailman-developers list" <mailman-developers- from+daniel=cacert.org@python.org> and add a reply-to: daniel@cacert.org if the header field doesn't exist for MUA compatibility.

-- Daniel Black Infrastructure Administrator CAcert

Daniel,

...

[hacking the from] is not going to be acceptable to a lot of folks,

Apart from the assertions of mailing list software developers I'm yet to receive a strong assertion from list operators or users.

Er, do you think we write open source purely out of charity? We are all operators and users ourselves. (Non-developer) users and list operators? See http://wiki.list.org/display/DOC/From+field+displayed+by+Microsoft+Outlook for what they think of the kind of display the munging you suggest produces. (The point is it's a FAQ; they *do not* like it.) And can you imagine what that will look like when combined with the munging you suggest Mailman do to fake out ADSP?

Even less forthcoming is the reasons the lack of acceptability that could guide standards or implementations.

1. Spoofing authorship is what the phishers and the spammers do. We're the good guys, remember? Note that the name of the standard you are trying to promote here is *Author* Domain Signing Policy. What that means is that the list's domain is claiming authorship of the post. This is problematic in any number of ways.

2. Authorship information is lost, or at best obscured. You point out that most MUAs don't present List-* headers. Well, they don't present Sender or In-Reply-To either (except for Outlook). You're either in the From header, or the users don't know about it.

3. Sophisticated users filter on From. This breaks the filters bigtime once, and makes them more fragile forever after.

4. It's ugly.

there's a development cost that I'll consider contributing to but I'm not going to develop stuff that has no change of being accepted. I will consider writing patches for:

1. global dkim-friendly= true to disable list modifications parameters
2. From: rewriting

1. dkim-friendly is CAN-SPAM unfriendly. Specifically, best practice (and maybe the law AIUI) requires you to put an unsubscribe notice in somewhere. As you point out, the List-Unsubscribe header just won't do....

2. From: rewriting. Thank you, but it is already a feature of most list software because it is required for common applications (announce lists, anonymous lists, and probably others I don't remember offhand). Adapting that to this purpose is nearly trivial.

...

Wouldn't it be more straightforward (not to mention that it would work for many more lists) to have an LDSP RFC, whose first draft simply takes the ADSP RFC and substitutes "mailing list" for "author" everywhere, and "RFC 2369 and RFC 2919 headers" for "From"? (The point of multiple headers is that "active" headers like List-Subscribe could contain bogus URLs.)

Doing so would allow List-* headers to be added by every spoofer, add their own signature and get immunity from spoofing every author domain while the end user doesn't notice because the List-* headers are hidden in the MUA (in most cases).

Nonsense.

1. ADSP is vulnerable, too. Besides the Safemail! hack described below, there are any number of obscure RFC 822 features (and yes, they're still present in RFC 5322) available:

   From: cs@yourbank.com (Customer Service) .phisher.net

2. Anybody who is sufficiently unsophisticated or distracted to get caught by any of the email phishing scams I've seen is not going to be checking ADSP on authors, anyway. So this is *not* about users, it's about ISPs/email providers. *They* see *all* the headers, and they either filter (based on reputation) or provide notification of authentication results to users.

   Eg, Google and other providers tag mail "This mail probably did not come from the purported author. Don't click on anything in it." Mail which is only LDSP-authenticated can be tagged "This mail failed author authentication. Do not provide personal information to its sender by replying to it or clicking on anything in it."

3. List mail by its very nature (lists are traffic aggregators) is unreliable. Obviously the amount of trust you put in LDSP- authenticated mail is *much* lower than that you put in ADSP- authenticated mail. Still, in the current environment I'll take any bonuses I can get, and as long as I behave, I suspect the big ISPs would be willing to allow me to build up some cred.

4. ADSP is just as meaningless as LDSP, unless it comes from a reputable party. As has been pointed out already in this thread, DKIM and ADSP are helpful only because they allow ISPs to keep a reputation database based on reliable identification of sender domains. You don't rely on something just because it's signed. LDSP is the same (but attenuated, #3 still applies).

A proposal for author domains to authorize third party lists is in development. http://mipassoc.org/pipermail/ietf-dkim/2009q4/012592.html

It does place a large burden on author domains to deploy DNS records pre- presenting every list their user base send email too.

So it certainly loses in my use case: I run support lists. I really don't think someone whose editor has crashed and wants to know if they can recover the data they were editing is going to wait for IT to deploy DNS records before posting.

...

Perhaps if it is known that the DKIM signature of the author's host is going to remain valid,

this would be idea.

Sure, but it's rarely the case. See "best practice and unsubscribe", above.

...

The only real problem with this is getting the big ISPs to implement, but that's nothing new. In fact if it's as easy as adding routines to process the RFC 2369 + RFC 2919 set of headers "just like" ADSP handles "From:", I bet most would be happy to sign on.

So this is for mailing list that send though big ISPs that can't apply their own DKIM signatures? Not totally sure what you mean here though signing is the easy part compared to applying filtering on verification results.

No, my mailing lists, and those like them, can deploy signatures and DNS easily enough. The question about the big ISPs is will they bother to do something to make things more comfortable for discussion lists.

...

...

If you are blindly assessing an email without knowledge that is a mailing list what do you see?

If the list doesn't implement any of RFC 2369 (published 1998) or RFC 2919 (published 2001), the joke is on it.

its more the consideration that, as a verifier, you shouldn't trust a List-ID as a way to bypass filtering.

Nonsense. It depends on the reputation of the list and the sending domain, just like From, which is just as easy to spoof, and reasonably easy to confuse the gullible.

please explain why this is harmful.

Ugly (makes every MUA look as broken as Outlook). The From header is an *Originator header*; it is a violation of the RFCs to change it.

It seems that anonymous discussion lists have worked so slightly manipulated From addresses would also work.

Apples and asteroids, for heaven's sake. *You* want them to work, so you call completely different contexts similar.

Just take the return-path variant, From: "Daniel Black via mailman-developers list" <mailman-developers-from+daniel=cacert.org@python.org> and add a reply-to: daniel@cacert.org if the header field doesn't exist for MUA compatibility.

Reply-To is also an originator field, munging it violates the RFCs. It's not obvious what the right thing to do if the Reply-To field exists. For example, I often set it to a list.

And, per your suggestion, let's just take that return-path variant and see what we can do with it.

From: "JP Morgan Customer Service via SafeMail!"
      <joe-victim+service=morgan.com@safemail.akamai.163.com>

Uh-oh. I *know* I've seen *that* domain somewhere before. Now, right now that looks suspicious. But once people get used to jumbles like

From: "Daniel Black via mailman-developers list"
      <mailman-developers-from+daniel=cacert.org@python.org>

they will stop trying to parse addresses at all (assuming the poor phish ever did in the first place).

Daniel Black writes:

quite right. sorry.

I'm sorry for being sharp about it. You gave it the old college try, but similar proposals have been floated before. I was also in a bad mood for reasons that have nothing to do with Mailman Developers and not really to do with Mailman at all.

The rest of the stuff below really isn't Mailman Developers-relevant, but I'm not sure if I can find time to accept your invitation to a more appropriate forum, so I'll post once more in hopes of seeding some memes.

good point. Though seeing it suggests that maybe ADSP could cover Sender: and at least some users with MUA's that display things a little ugly could tell the difference between a spoofed email on a mailing list they subscribe to.

I think users would probably not like that, or the proposals to display List-*, Sender, Reply-To and so on. I currently display a lot of unusual stuff (including In-Reply-To, References, X-Mailer, and User-Agent) but ironically none of those you suggest directly or indirectly. Most users, though, just want the usual: author, addressees, date.

I think that in most cases the MUA can do a pretty good job of summarizing authenticity based on the DKIM signatures, according to a scale something like

1. *author* = author is authorized to send from her apparent domain
2. *list-verified* = list is authorized to send from its apparent domain *and* it authenticated the author
3. *list* = list is authorized to send from its apparent domain

Users should by default be presented with *author* or not (boolean), because really anything less should not be trusted to be from your bank. Users who want the details could optionally be given the full scale.

...

Well, they don't
present Sender or In-Reply-To either (except for Outlook).

do you think MUAs should show Sender and/or Reply-To? if so would making mailman set these to the list address be acceptable for users?

No, and no. Users really don't want to know those most of the time, and I think the kind of thing that people who don't browse the Received headers would be able to recognize probably can be reliably computed automatically, per the above list. It's certainly true that sometimes lists or individuals will get caught by that (eg, stephen@xemacs.org will almost never be ADSP authenticated, because I send from my university, not from an xemacs.org host). But that should be handled by a personal signature, not by the DNS, I think.

Both Sender and Reply-To are originator headers. But, for example, I think it's reasonable for Mailman to munge Sender, since the user clearly has delegated the responsibility for transferring the message to Mailman, and any technical difficulties in distribution are probably best handled by Mailman. The only exception I can think of is the case where Mailman strips attachments by policy, and a recipient wants to request a copy. Normally From is a good place to ask for that. So really I don't see what value Sender has for spam-fighting. It could be set to any number of things, and interim hosts often have plausible reasons for changing it.

Reply-To ... you don't want to mix into that one.<wink> See http://turnbull.sk.tsukuba.ac.jp/Blog/Software/ReplyToMungingConsideredEmCar... for something I wrote for a different venue just today.

...

1. dkim-friendly is CAN-SPAM unfriendly. Specifically, best practice (and maybe the law AIUI) requires you to put an unsubscribe notice in somewhere. As you point out, the List-Unsubscribe header just won't do....

This can probably be accounted for by DKIM l= tags (the signed length is..) on the sender's behalf and still allow additions of unsubscribe notices. If I account for this provision would it be considerable?

Don't know. Many lists are configured to hack into HTML parts and add it *inside* the existing HTML BODY element, for example. Somebody more familiar with the use cases for editing the message body would have to answer for that.

true, though automated techniques may help http://tools.ietf.org/html/draft-kucherawy-dkim-reporting-05. I'll ask Murray about the future here.

Yeah. I worry about the security implications of something like that, though. Automatically munging the DNS on the basis of frequent requests from ordinary users is worrying.

I'm a lot more included to support your LDSP proposal scoped with security concerns

Oh, of course you need to scope it. However, ISTM you just map all the security concerns of ADSP to LDSP, then multiply "trust" by 0.5 (pick whatever figure you like; note that above I map it to 0 for the typical end user!)

...

No, my mailing lists, and those like them, can deploy signatures and DNS easily enough. The question about the big ISPs is will they bother to do something to make things more comfortable for discussion lists.

Is this a matter of:

1. technology standardisation - (LDSP, DKIM Third-Party Authorization Label, Sender DSP)
2. deployment practice standardisation - http://tools.ietf.org/html/draft- ietf-dkim-deployment-08#section-6.3
3. deployment promotion - inclusion in, for example, mailman installation manual / site administrator documentation
4. ensuring DKIM verification tools development supports this.
5. some marketing / guides for ISPs with respect to DKIM

All of the above, of course.

Many would love to have your involvement there if you're interested.

I'm interested, the question is making the time.....