Mailman 3 November 2013 - Mailman-Developers

Re: [Mailman-Developers] REST API call for creating archive view?
by Aurélien Bompard Nov. 13, 2013

Nov. 13, 2013

Hey Colin, sorry for not replying earlier There is a REST interface in HyperKitty, that will give you access to the archives if you want to implement another web interface on top of it. The good thing is that I know someone who already did that, with about the same needs as you, so it must be doable :-) The bad thing is that it was some months ago, and HyperKitty keeps evolving, while this part is not covered by unit tests yet. So it may be broken at the moment. But it may also be very easily fixable if needed. Aurélien -- http://aurelien.bompard.org ~~~~~~ xmpp:aurelien@bompard.org Recursion: (n.) See "Recursion".

2 1

Re: [Mailman-Developers] REST API call for creating archive view?
by Stephen J. Turnbull Nov. 10, 2013

Nov. 10, 2013

Colin Fleming writes: > Thanks for the comments, Steve - can you clarify what you mean by a > "write-only API"? "Mailman core has no way to retrieve those messages." For the rest, I think Barry's answer is clear and definitive.

2 1

Re: [Mailman-Developers] REST API call for creating archive view?
by Barry Warsaw Nov. 9, 2013

Nov. 9, 2013

On Nov 09, 2013, at 06:10 PM, Colin Fleming wrote: >Thanks for the comments, Steve - can you clarify what you mean by a >"write-only API"? As far as I can tell I could use that to insert messages >that Mailman now considers "posted to the list" into my own DBMS and then >present them however I like. I'll have a bunch of questions around >robustness and error handling, but is that not basically the case? It is. The IArchiver API is not part of the REST API; it's not an external API that clients can talk to. It's an internal API that Mailman uses exactly as you say - it's a generic way to get messages posted to the list into other services, structured around the needs of archivers. >I like the idea of Hyperkitty returning fairly plain HTML that I could >later style, although having a REST API that would allow me to get at >thread objects which contained a tree of message objects would probably be >ideal. I suspect that's fairly trivial for me to implement assuming my >assumptions about IArchiver are correct. IArchiver is part of Mailman core, not Hyperkitty. In fact, it's the way messages posted to mailing lists get *into* Hyperkitty. :) -Barry

1 0

REST API call for creating archive view?
by Stephen J. Turnbull Nov. 9, 2013

Nov. 9, 2013

Colin Fleming writes: > Is there a way to create an archive view (i.e. present a view of > all messages in a list and allow reading particular mails) using > the current API? No, and (sorry Barry, but I think you just added to the confusion -- IArchiver is really a write-only API) there won't be unless the architecture of Mailman 3 changes dramatically. Archiving (storage and retrieval of posted messages) is the responsibility of a separate module. What Mailman itself can tell you is the suggested permalink for individual archived messages. We do plan to provide such a module (the current implementation and leading candidate is called HyperKitty). I do not know if the current architecture of HyperKitty is sufficiently modular as to allow you to just grab messages and then format them yourself. I'm not sure why you'd want to grab raw messages, actually; I think the best idea would for HyperKitty core (or any other implementation of an archiver) to spit out a DIV element containing the (semantically) formatted email message, and provide a default CSS, plus a fairly trivial full webapp that wraps the DIV in a full page. I'm not sure if it does that yet, though, you need to talk to the HyperKitty devs about it. (Heck, if it doesn't, maybe I'll go into the archiver business myself. Look for CSSZenKitty at a DVCS host near you!) Steve

2 1

Re: [Mailman-Developers] Mailman DMARC Support (it's not what you think!)
by Franck Martin Nov. 8, 2013

Nov. 8, 2013

----- Original Message ----- > From: "Jim Popovitch" <jimpop(a)gmail.com> > To: Mailman-Developers(a)python.org > Sent: Thursday, November 7, 2013 11:49:30 AM > Subject: Re: [Mailman-Developers] Mailman DMARC Support (it's not what you think!) > > On Thu, Nov 7, 2013 at 1:27 PM, Franck Martin <franck(a)peachymango.org> wrote: > > To be noted, the procedure, in this patch, to find the policy record in the > > DNS, is not in line with the best current practice specified in the DMARC > > spec, far from it. > > Care to offer some insight? As they say at IETF: Read the spec! :P > > > Also, what about p=quarantine? > > I think that should be treated the same as p=reject. Don't send me > your junk for storage! :-) > Please amend the patch to reflect it.

3 2

Re: [Mailman-Developers] REST API call for creating archive view?
by Barry Warsaw Nov. 8, 2013

Nov. 8, 2013

On Nov 07, 2013, at 02:50 PM, Colin Fleming wrote: >I'm interested in using Mailman to set up a mailing list for my product. >I'd like to have it be fully integrated into my website, so I'm planning to >use Mailman 3 and the REST API. I've checked out mailman.client and poked >around, and also looked at >http://bazaar.launchpad.net/~mailman-coders/mailman/3.0/files/head:/src/mai…, >but the one thing I couldn't see was any way to access mail messages >themselves. Is there a way to create an archive view (i.e. present a view >of all messages in a list and allow reading particular mails) using the >current API? While we could (and should) add a REST API for accessing the IMessageStore, I think this will not be very useful for you. That API would just return the message text based on the Message-ID (or hash) that you want to look up. It wouldn't have any support for common archiving structure like threading, etc. It would be better for you to implement an IArchiver plugin to get the messages fed to you as they come in. Of course, then you'd have to implement the archiver yourself, or you can use one of the standard archiver plugins such as for HyperKitty or MHonArc. And actually, having a REST API to HyperKitty is probably the best long-term approach. Perhaps Aurélien can speak to that. -Barry

2 1

Re: [Mailman-Developers] Mailman DMARC Support (it's not what you think!)
by Franck Martin Nov. 8, 2013

Nov. 8, 2013

----- Original Message ----- > From: "Jim Popovitch" <jimpop(a)gmail.com> > To: "Franck Martin" <franck(a)peachymango.org> > Cc: Mailman-Developers(a)python.org > Sent: Thursday, November 7, 2013 4:59:38 PM > Subject: Re: [Mailman-Developers] Mailman DMARC Support (it's not what you think!) > > On Thu, Nov 7, 2013 at 7:44 PM, Franck Martin <franck(a)peachymango.org> wrote: > > > > ----- Original Message ----- > >> From: "Jim Popovitch" <jimpop(a)gmail.com> > >> To: Mailman-Developers(a)python.org > >> Sent: Thursday, November 7, 2013 11:49:30 AM > >> Subject: Re: [Mailman-Developers] Mailman DMARC Support (it's not what you > >> think!) > >> > >> On Thu, Nov 7, 2013 at 1:27 PM, Franck Martin <franck(a)peachymango.org> > >> wrote: > >> > To be noted, the procedure, in this patch, to find the policy record in > >> > the > >> > DNS, is not in line with the best current practice specified in the > >> > DMARC > >> > spec, far from it. > >> > >> Care to offer some insight? > > > > As they say at IETF: Read the spec! :P > > Well, as they say in production environments, I'll wait for the final cut. > ;-) Then wait for the final cut before submitting patches...

2 1

Re: [Mailman-Developers] Mailman DMARC Support (it's not what you think!)
by J. Trent Adams Nov. 8, 2013

Nov. 8, 2013

Jim - On 11/7/13 3:27 PM, Jim Popovitch wrote: > On Thu, Nov 7, 2013 at 5:12 PM, J. Trent Adams <jtrentadams(a)gmail.com> wrote: >> should provide an option to participate in as secure communication as possible. > Randomly applying security distinctions, to RFC de'jour, is not really > helping. Well, here we disagree. DMARC is far from random (in development since 2007), and it's actually helping. And not just on the margins. We're seeing significant improvement in the real world at Internet scale for millions of our customers. It'd be fantastic to see Mailman trumpet an option that allows those who want to take advantage of it to drop their old, beat-up MLM and join the party. > If you want true message security, then PGP/GPG is the only > universal way. If you are just looking to protect the integrity of > the pathway, might I suggest that a wrapper around 2 different > technologies (one being header reliability and the other being source > reliability) is just that... a wrapper (or as I say, a panacea). If > you truly wanted secure comms, DMARC would be mandating PGP and going > after MUAs.... Here, though, I totally agree with you! After all, I earned my degree in astrophysics, and we modeled everything using perfect theoretical spheres. Sadly, our models always fell short under close scrutiny. . . fortunately, as opposed to physics, the astro variety of the science didn't often come under close scrutiny. Talk about being left off the hook! That being said, in a more nuanced, complex, messy world, theoretical solutions don't always live up to the promise. So, yeah, it'd be super kewl to ask all our customers (and all the communication channels) to get on the bandwagon with encryption. I've got stacks and stacks of real world data and research that explains why that's simply not viable (today). DMARC, on the other hand, emerged as an empirical experiment on what works today, in the real world. It was a lot of trial and error until we happened to get the mix right. It's no panacea (it only stops one type of particularly nasty attack), but you're right, it's just a wrapper. Fortunately, it works, so the Lords of Kobol be praised! Finally, some defense against the maurauding Cylons (even though it's a mish-mash of low-tech bits and bobs). . . . amusingly, this reminds me of the LAN parties we used to have. It wasn't pretty, but the goal was to get the Sinclair to talk to the Commodore using whatever means necessary (air nets were particularly hilarious, albeit incredibly hard on the ears). > but I digress. Yeah, but aren't digressions fun? Happy to continue the conversation off-list so that we don't bore the rest of the crew. Cheers, Trent > > -Jim P.

1 0

Re: [Mailman-Developers] Mailman DMARC Support (it's not what you think!)
by J. Trent Adams Nov. 7, 2013

Nov. 7, 2013

Jim - On 11/7/13 12:58 PM, Jim Popovitch wrote: > On Thu, Nov 7, 2013 at 1:23 PM, J. Trent Adams <jtrentadams(a)gmail.com> wrote: [snip] > Now we're left participating on mailing lists using personal mail > accounts. As you can imagine, that makes many departments in the > organization really nervous for a variety of obvious reasons. > Which .... appears to be working well for you, mr > jtrentadams@<freemailer.tld> ;-) Indeed! . . . though let's keep that on the down-low given the concern about the apparent ease with which some entities seem to have access to these accounts. ;) [ snip ] > To be fair, Mailman doesn't modify the body, it does add footer > attachments and follows existing RFC requirements for appending > headers. I wouldn't necessarily classify that in the same way as > RFC5322 From spoofing. Yeah, that's a fair point about Mailman using appropriate mechanisms to append the footers. . . But if a sender signs the subject header, does the signature break if Mailman pre-pends (useful) info? At the end of the day, though, assuming SPF and DKIM are the most effective means of email authentication, DMARC is the only (current) way to effectively close the loop. And Mailman (my preferred MLM by far) has an excellent opportunity to leap ahead of the pack and keep that loop from being severed. > >> . . . unless someone can think of another clever way to maintain >> effective end-to-end email authentication in a novel way? Perhaps >> through the broader adoption of transitive trust (e.g. the Original >> Authentication Results header)? >> >> Anyway, thanks for your careful consideration, and I do look forward to >> continuing to discuss viable options for keeping email as trusted and >> secure as possible. > I'm not convince that email trust and security have any impact > whatsoever on Mailman or MLMs in general. I believe you can have > secure and trusted email (i.e. ibm.com) yet not break MLMs. > > -Jim P. Hmmm. . . that's an interesting take. A quick glance at ibm.com shows that they SPF -all, so at least their top level domain shouldn't be sending mail at all. Is that what you meant? Glancing at their subdomains (e.g. us.ibm.com), though, shows that they're protected by SPF ~all and not DMARC. Given that many major mailbox providers only soft fail ~all, it's still likely to see spoofed domain mail reach the recipient. . . Or am I missing something clever that they're doing (which I should probably know about)? Other than that, I believe that MLMs in general play a huge part in trusted and secure mail. Sure, they're not on the front line, but if they require a hobbling of security of their subscribers to participate, I see that as not so great. Now, it's not a perfect analogy, but what if in order to ride the city bus you were required to lick the handrails? OK, it's not pretty, but that'd be requiring you to do something inherently dangerous simply to take advantage of public transportation. I don't know about you, but I'd prefer another mode of transport. Anyway, I'd like to think that MLMs in general, and Mailman in specific as the Cadillac (or should we say Tesla now?) of MLMs, should provide an option to participate in as secure communication as possible. Someone's going to do it eventually, and I'm hoping that we can build on the momentum here given that there are a few reasonably visible lists willing to cut over to Mailman once their subscribers have a better experience. Thanks, by the way, for the discussion and consideration. - Trent

2 1

Re: [Mailman-Developers] Mailman DMARC Support (it's not what you think!)
by J. Trent Adams Nov. 7, 2013

Nov. 7, 2013

Jim - On 10/20/13 4:17 PM, Jim Popovitch wrote: > Hello, > > Having read the archives, I see that (at least 6 of) you are aware of > DMARC, or as I like to call it YAPFS. (Yet Another Panacea For Spam) > :-) Heh - If only DMARC was as successful at stopping spam as it is in shutting down spoofed domain mail! Wow, that'd be fantastic. . . I'd love to stop receiving unwanted solicitations. But that's not what it does. As it turns out, DMARC has proven so remarkably successful at eliminating spoofed domain phishing attacks. So much so that it's already saving us a huge, non-trivial, amount of money per year (with a trend line of savings that's continuing to increase). I wish we could share the actual dollar value we're saving by protecting our customers, but that information isn't public. Suffice it to say that it works (and works tremendously well). > > Earlier this year Mark asked me to run by MM-Dev a patch that Phil > Pennock and I collaborated on. Mark, thank you for your valuable > feedback, I have addressed all but 1 of those issues. > > Phil's and my take is that mailing lists, like MTAs, have no business > modifying the From header; nor should mailing lists accept mail that > they knowingly can't reflect. To that end, we have added support for > testing the From domain for a DMARC p=reject policy, and if it exists, > allowing lists to Accept/Hold/Reject/Discard the message. Here's my problem with this solution: it forces participants of mailing lists to send from domains that do not have adequate protection against spoofed domain attacks. Doing so then opens a hole in the defenses which can then be abused. I can speak to this from experience on email security for one of the most phished brands (the most phished, if you trust APWG's reporting). We set up an unprotected domain specifically for mailing lists, and we watched that as our DMARC protection across our other domains came online, the abuse on our unprotected domains increased. Finally, we'd locked everything down other than the mailing list domain, and then it became a primary target of abuse. . . which resulted in us having to shut it down. Now we're left participating on mailing lists using personal mail accounts. As you can imagine, that makes many departments in the organization really nervous for a variety of obvious reasons. Sure, we're slightly unique in that we're highly susceptable to phishing attacks, and a common abuse vector is domain spoofing. . . but as we lock down our doors, we're seeing shifts by the attackers to other, less protected targets. So, the problems that vex us today are beginning to nip at the heels of the victims of tomorrow. . . . and given the rampant problem of password reuse, any credentials phished from an unprotected domain can easily be used to attack a domain that is protected. Again, I wish I could share the numbers of these types of attacks, but they're staggeringly large. And it's incredibly annoying that our hands are tied since humans will do what humans do (e.g. reuse passwords everywhere). Not good, that. It's unfortunate that the real world infringes on the perfection of a pure "reflector" architecture. I wish that solutions were beautiful and unblemished. Sadly, reality is messy, and some consideration for compromise is necessary for solutions to be effective. Besides, mailing lists modify messages they "reflect" all the time. They pre-pend list identifiers to the subject line (which makes my life much easier), and also post-pend list management options (which is a regulatory requirement in some jurisdictions). So, if we want to be purists, no modifications should be allowed. But that's taking the argument to an extreme. Instead, we should consider a benefit analysis for each proposed change. And in this particular case, as more senders are adopting DMARC each and every day for entirely valid security and economic reasons, I believe that modifying the RFC5322.From field is a perfectly reasonable option. . . . unless someone can think of another clever way to maintain effective end-to-end email authentication in a novel way? Perhaps through the broader adoption of transitive trust (e.g. the Original Authentication Results header)? Anyway, thanks for your careful consideration, and I do look forward to continuing to discuss viable options for keeping email as trusted and secure as possible. Yours, Trent > > Here is the LP diff for your perusal: > http://bazaar.launchpad.net/~jimpop/mailman/dmarc-reject/revision/1379?reme… > > I will soon be porting this to 3.0, and I will return here for input > on that as well. > > Thank you everyone for your valued opinions, > > -Jim P. > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers(a)python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/jtrentadams%40gm… > > Security Policy: http://wiki.list.org/x/QIA9

2 1