Mailman 3 February 2007 - Mailman-Developers

Problems if shunting fails
by Mark Sapiro 26 Feb '07

26 Feb '07

There is a bug report at <https://sourceforge.net/tracker/index.php?func=detail&aid=1656289&group_id=…>. The basic issue here is there is an extremely large message in the 'in' queue. In followup emails with the submitter, he said "The file was nearly half a terabyte in size". Not surprisingly, IncomingRunner threw a MemoryError when SpamDetect tried to flatten() the message, but then Runner tried to shunt the message and Switchboard.enqueue() threw another MemoryError in the attempt to pickle the new shunt queue entry. The second MemoryError is uncaught in Runner._oneloop, so it causes the main loop in Runner.run to exit. Pre-Mailman 2.1.9, the master just restarts IncomingRunner - the message is lost, but everything else is OK. Because of the changes in 2.1.9 to prevent message loss in case of disaster, there is now a .bak file left in the 'in' queue. When IncomingRunner restarts, it recovers the .bak file and the whole scenario repeats until the master reaches MAX_RESTARTS on IncomingRunner and we are left with no IncomingRunner and the .bak file still in the 'in' queue. In order to fix this, I suggest we protect the shunt enqueue in a try. I have worked up a patch for this which is attached. This patch also adds a 'preserve' argument to Switchboard.finish such that if it is called with preserve=True, instead of removing the .bak file, it just renames it with a .psv extension. These changes ensure that IncomingRunner doesn't exit, and no .bak file is left to cause a subsequent problem while still preserving the original queue entry for further analysis if possible. I would like some feedback on whether or not this is the right approach. -- Mark Sapiro <msapiro(a)value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 4

Reg mailman to yahoo groups kind of interface
by mobiledreamers＠gmail.com 16 Feb '07

16 Feb '07

Hello I love and use python a lot. And I m trying to make mailman, into a yahoo groups kind of interface, In that I want to find out if any of you guys tried to do this and integrate it into another webservice? Does mailman uses coroutines to send email, because a coroutine based email gateway can support much more email messages perday than a thread based one Mark

2 1

Show real names in roster?
by Martin Schütte 14 Feb '07

14 Feb '07

Hello, is there a reason why the roster page does not show full names? Obvious concern is the privacy of the list members. But on the other hand: - the roster has its own access control, - the mail address itself is usually more valuable and can be used to get the real name. - entering the real name on subscription is optional. Thus IMHO showing real names does not disclose significant information, but makes things more convenient for legitimate users. I have attached a three-line patch against Mailman/HTMLFormatter.py -- Martin Schütte 96a97,99 > realname = Utils.uncanonstr(self.getMemberName(person), lang) > if realname: > showing += " (%s)" % Utils.websafe(realname)

1 0

Feature Suggestion: Digest Reply handling
by Peter Brooks 12 Feb '07

12 Feb '07

Anyone please correct me if this feature already exists, however I'm wondering if it's already been developed so that if a user reply's to a digest in their box then mailman will extract the relevant post they are replying to and also warn the user. Obviously there are a fair few metric with this, such as which post are they replying to. So is this already implemented somewhere or should I develop it? Regards

2 1

[ 1657458 ] feature request: batch deletion of subscription requests
by cmpalmer＠metalab.unc.edu 11 Feb '07

11 Feb '07

Per http://www.list.org/bugs.html I am sending an email to this list about the feature request I reported at: https://sourceforge.net/tracker/index.php?func=detail&aid=1657458&group_id=… It basically says that it would be nice to have another tickbox in the admindb page that lets you "delete all pending *subscription* *requests* marked /defer/" and not just a tickbox that does that for *messages* since we're getting a fair amount of spam in the form of subscription requests to lists that we manage. Thanks for your attention, -- Cristóbal M. Palmer ibiblio.org system administrator

1 0

Re: [Mailman-Developers] dkim-signature headers
by Barry Warsaw 09 Feb '07

09 Feb '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm not sure what the right answer is just yet, but I'll offer some of my thoughts FWIW. I think the fundamental question is whether the mailing list is the originator of the messages its members receive or whether the original author is. This question has come up in other contexts before, and I don't think it's ever been answered satisfactorily. A quick search through DKIM archives seems to indicate that this question has come up there too, and I think answering it will help us understand what we should be doing here. On Feb 1, 2007, at 3:00 PM, Mark Sapiro wrote: > Consider that while Mailman doesn't do all of these things to every > message, it can do any of the following: [munge the original message] From the DKIM FAQ: - What is the purpose of DKIM? DKIM lets an organization take responsibility for a message. The organization taking responsibility is a handler of the message, either as its originator or as an intermediary. Their reputation is the basis for evaluating whether to trust the message for delivery. I think you can make a legitimate case that Mailman is the originator of messages its members receive. The message is certainly different than the one the original poster sent to the list, and Mailman is clearly an intermediary. Perhaps the message has only been munged in very trivial ways, but it's also possible to munge it in ways that could potentially be viewed as spammy. For example, what if a site decides to put some advertisements in the footer? If you take this view then it seems reasonable to say that it is the mailing list's system that "take[s] responsibility for a message." Sure, the mailing list system could verify the DKIM headers on the message it receives, but ultimate, it is up to the mailing list system to decide whether that message (or some derivative of that message) gets transmitted to its recipients. Or looked at another way, if I send a message through a mailing list, I wouldn't want to vouch for whatever comes out the other end because I don't know what they're going to do to my original content. Maybe then, it's correct for the DKIM signature on the copied message to be broken because what recipients got was /not/ the message I sent, and I don't know how it was munged. But that view implies that I am the originator of the recipient's message. I am, sort of, but also sort of not. I'm not convinced that DKIM is really designed to handle the mailing list use case. It seems to me that it was designed to handle point- to-point messages, not messages that flow through an intermediary, because it's not an enveloping system. Contrast that with S/MIME or OpenPGP. I can sign the message I send from my mailer and that could be preserved through the transformations that Mailman performs, with Mailman wrapping my original in its own signature if it wanted to. Practically speaking, if we can't come up with a consensus on the interpretation of which "organization [should] take responsibility for" the actual message that recipients receive, then what would be the right thing to do? (Note that this answer is different depending on whether we're talking about Mailman 2.1 or some future version.) When this came up before I statement my preference not to make a "strip DKIM headers" selectable by the list owner. I still prefer this for Mailman 2.1 because doing so would clearly be a new feature. Maybe a future version could treat the DKIM header the way it treats the RFC 2369 headers, with a separate selector for List- Post. Ideally, we'd have a more general way to decide which headers get cleansed and which new ones get added. But that's for the future. One elaboration you /might/ be able to get away with in Mailman 2.1 occurs to me as I look at Mark's list: > - Add text to the beginning of the message body (msg_header) > - Add text to the end of the message body (msg_footer) > - Remove text from the beginning of the message body (Approved: line) > - Add additional MIME parts to a multipart message (msg_header, > msg_footer) > - Convert a single part message to multipart in order to add > msg_header/msg_footer > - Remove parts from a multipart message (content filtering) > - Convert an HTML part to plain text (content filtering) > - Decode a base64 or quoted-printable encoded part and perhaps > re-encode it with a different encoding. > - Change or delete various headers including Subject:, To:, From: > - Replace some MIME parts with URLs of where they were stored and > flatten the entire message into a single plain text message > (scrubber). > - Probably other things I'm overlooking. If you could identify the message transformations that break the signature, then you could remove the signature. If the signature of the outgoing message were still valid because Mailman didn't touch any part of the message affecting the signature, then you could keep it. The implementation of this would be fairly simple; the hard part is writing the code to verify a DKIM signature and parse the selectors (IIUC the specification) to figure out which of the above transformations would break the signature. That might be enough to not do it in Mailman 2.1. I'm not sure how much I like that anyway, so comments are definitely welcome. After mulling over this post for an hour ;) I'm starting to believe that it's the mailing list system that needs to vouch for the messages its recipients receive. Of course, it could be Mailman doing the DKIM signing, or it could be Mailman's outgoing MTA, etc. But, ISTM Mailman is ultimately deciding what goes into the list copy, so it is responsible for it. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (Darwin) iEYEARECAAYFAkXCvIgACgkQ2YZpQepbvXFnVgCfeUkfQ0+h/bBAKiwznDTdrHJ6 7V0An2O9TcUYBJYlFhYFpLtYUUGalabq =yCgT -----END PGP SIGNATURE-----

10 55

Re: [Mailman-Developers] dkim-signature headers
by Mark Sapiro 09 Feb '07

09 Feb '07

Michael Thomas wrote: > >On Wed, 7 Feb 2007, Mark Sapiro wrote: > >> Mike talks about the l= parameter allowing adding trailing content, but >> I don't see Y! and Gmail using it, and even if they did, how would we >> (could we) add a footer without breaking either the signature or the >> MIME structure of the message. > > l= is the number of canonical bytes added to the body hash. > If l=5, for example, anything past the 5th canonical byte will > not affect the verification of the signature. That's the reason > we get such high verify rates through mailing lists. My point is that for what I consider good reasons, Mailman will add the msg_footer to such a message by wrapping additional MIME structure around the original multipart/alternative message. I.e., the original multipart/alternative text/plain text/html message will be recast as multipart/mixed multipart/alternative text/plain text/html text/plain with the final text/plain part containing the footer. Given that the original content-type header is included in the signature, the signature is now broken. If we were to take a different approach with a signature containing l=, either the l= includes all the text/plain and at least part of the text/html, in which we can't add the footer to the text/plain alternative without breaking the signature, or the l= includes none of the text/html part in which case the signature is not very good at verifying the validity of the text/html part. This further assumes we even know how to add a footer to a text/html part. See <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.039.htp> for some discussion of why we do it the way we do. -- Mark Sapiro <msapiro(a)value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 3

My new job
by Barry Warsaw 04 Feb '07

04 Feb '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello everyone, I just wanted to drop a quick note to let you know that I have started working for Canonical, the folks who distribute Ubuntu Linux and develop the Launchpad service for distributed open source development. I'm quite excited to be working for Canonical. They are a Python and Zope shop, very open source friendly, and based on Linux, so I'm working in an environment that's both familiar and fun. They're also a distributed company, without traditional offices, but with many folks working all over the world. They're also hiring <wink, wink>. :) While this is good for Barry (and hopefully good for Canonical :), I know you're thinking, "what does this mean for Mailman?" Well, I'm glad you asked! Part of my official duties will be to integrate Mailman with Launchpad so as to enable more powerful communication patterns among members of a distributed development team. While I won't be spending /all/ of my time on Mailman, it will be getting some official love. I hope that this will help move the current development branch closer to a release some time later this year. Other than that, not much will really change. Mailman is and will always be released under the GPL, and it will continue to be a GNU project. If you have any questions or concerns, please send them directly to me. I will attempt to answer them as best I can, and if there are common themes, I'll post updates here. Thanks for your support and keep on delivering with Mailman! - -Barry Here are some related links: http://www.canonical.com http://www.ubuntu.com http://launchpad.net P.S. I'll be at PyCon this year so if you're coming too, please say hi! -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRcUTb3EjvBPtnXfVAQIQKgP/XFVeeDBn6QVXkE9oK1YJxyrLZZET5GxT TAvTJgfndkcWPuUpbC5D6hcpDNa6sfIgJnR3enoW7MgKpOAtIXTuXqPpNiFMBVT2 qUhlDHhwYzdJWWyzI/oXGRvt0lMqqA69Iu7A6DAKrgksBB128V/mYxTfv8BPmF4W uASb/9MkmAQ= =h+IQ -----END PGP SIGNATURE-----

2 1

dkim-signature headers
by Michael Thomas 03 Feb '07

03 Feb '07

Hi all, I'm one of the authors of the DKIM protocol and it recently came to my attention that you've recently changed mailman to remove DK and DKIM signature headers when you remail the message. This is incorrect behavior: in Section 4: Signers SHOULD NOT remove any DKIM-Signature header fields from messages they are signing, even if they know that the signatures cannot be verified. This actually applies to everybody. There are several reasons for this. First is that DKIM allows you to specify the length of a body so it is not the case a priori that mailman will destroy the signature. Second, other heuristics can be applied to make mailing list traversal even better such as using the z= tag to determine whether trivial subject modifications have been made. Third and probably most important is that removing the signature is actually harmful rather than helpful: a broken signature and a missing signature MUST be treated as equivalent to no signature at all (lest an attacker just add a fake DKIM-signature header to get preferential treatment), and as above the verifier loses the ability to recover the signature. Just as an FYI, we have deployed DKIM across all of Cisco and our successful mailing list traversal rate is about 99% -- a large percentage of which are through mailman lists. By making this change, you've taken the verify rate from 99% to 0% in one swell foop. Not good. Mike

7 15

Re: [Mailman-Developers] dkim-signature headers
by Michael Thomas 02 Feb '07

02 Feb '07

The very basic test I use is what's in the From: address. That's the thing that's pretty universally displayed and one that users are most likely to grok. Anything beyond From, and you've probably lost at least half of the user population at least. So mailing lists preserve the original From: and perhaps add a Sender:. If I had to chose a signature which was most relevant for _display_ purposes (note that display purposes is but _one_ reason for signatures), I'd like to be able to say something good about the From address that most users grok. It's unclear that I'd want to say something nice about the Sender or any other third party signature because it may confuse users that the mere presence of a signature imputes some trust, where it actually needs to be considered on a case by case basis. I for one would only render the sender: as "verified" both if it verified correctly as well as it being on some known whitelist of domains that I trust. So the bottom line is that a valid third party signature from, say, a mailing list is not safe a priori. It requires special handling by the ultimate receiver in the form of a trust relationship with that domain which needs be done out of band. The same is not of a first party signature because you're only vouching for yourself which is already as good you trust that domain (or not). Mike, is this making any sense? Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I'm not sure what the right answer is just yet, but I'll offer some of > my thoughts FWIW. > > I think the fundamental question is whether the mailing list is the > originator of the messages its members receive or whether the original > author is. This question has come up in other contexts before, and I > don't think it's ever been answered satisfactorily. A quick search > through DKIM archives seems to indicate that this question has come up > there too, and I think answering it will help us understand what we > should be doing here. > > On Feb 1, 2007, at 3:00 PM, Mark Sapiro wrote: > >> Consider that while Mailman doesn't do all of these things to every >> message, it can do any of the following: > [munge the original message] > > From the DKIM FAQ: > > - What is the purpose of DKIM? > > DKIM lets an organization take responsibility for a message. The > organization taking responsibility is a handler of the > message, either as its originator or as an intermediary. Their > reputation is the basis for evaluating whether to trust the > message for delivery. > > I think you can make a legitimate case that Mailman is the originator > of messages its members receive. The message is certainly different > than the one the original poster sent to the list, and Mailman is > clearly an intermediary. Perhaps the message has only been munged in > very trivial ways, but it's also possible to munge it in ways that > could potentially be viewed as spammy. For example, what if a site > decides to put some advertisements in the footer? > > If you take this view then it seems reasonable to say that it is the > mailing list's system that "take[s] responsibility for a message." > Sure, the mailing list system could verify the DKIM headers on the > message it receives, but ultimate, it is up to the mailing list system > to decide whether that message (or some derivative of that message) > gets transmitted to its recipients. > > Or looked at another way, if I send a message through a mailing list, > I wouldn't want to vouch for whatever comes out the other end because > I don't know what they're going to do to my original content. Maybe > then, it's correct for the DKIM signature on the copied message to be > broken because what recipients got was /not/ the message I sent, and I > don't know how it was munged. But that view implies that I am the > originator of the recipient's message. I am, sort of, but also sort > of not. > > I'm not convinced that DKIM is really designed to handle the mailing > list use case. It seems to me that it was designed to handle > point-to-point messages, not messages that flow through an > intermediary, because it's not an enveloping system. Contrast that > with S/MIME or OpenPGP. I can sign the message I send from my mailer > and that could be preserved through the transformations that Mailman > performs, with Mailman wrapping my original in its own signature if it > wanted to. > > Practically speaking, if we can't come up with a consensus on the > interpretation of which "organization [should] take responsibility > for" the actual message that recipients receive, then what would be > the right thing to do? (Note that this answer is different depending > on whether we're talking about Mailman 2.1 or some future version.) > > When this came up before I statement my preference not to make a > "strip DKIM headers" selectable by the list owner. I still prefer > this for Mailman 2.1 because doing so would clearly be a new feature. > Maybe a future version could treat the DKIM header the way it treats > the RFC 2369 headers, with a separate selector for List-Post. > Ideally, we'd have a more general way to decide which headers get > cleansed and which new ones get added. But that's for the future. > > One elaboration you /might/ be able to get away with in Mailman 2.1 > occurs to me as I look at Mark's list: > >> - Add text to the beginning of the message body (msg_header) >> - Add text to the end of the message body (msg_footer) >> - Remove text from the beginning of the message body (Approved: line) >> - Add additional MIME parts to a multipart message (msg_header, >> msg_footer) >> - Convert a single part message to multipart in order to add >> msg_header/msg_footer >> - Remove parts from a multipart message (content filtering) >> - Convert an HTML part to plain text (content filtering) >> - Decode a base64 or quoted-printable encoded part and perhaps >> re-encode it with a different encoding. >> - Change or delete various headers including Subject:, To:, From: >> - Replace some MIME parts with URLs of where they were stored and >> flatten the entire message into a single plain text message (scrubber). >> - Probably other things I'm overlooking. > > If you could identify the message transformations that break the > signature, then you could remove the signature. If the signature of > the outgoing message were still valid because Mailman didn't touch any > part of the message affecting the signature, then you could keep it. > The implementation of this would be fairly simple; the hard part is > writing the code to verify a DKIM signature and parse the selectors > (IIUC the specification) to figure out which of the above > transformations would break the signature. That might be enough to > not do it in Mailman 2.1. > > I'm not sure how much I like that anyway, so comments are definitely > welcome. After mulling over this post for an hour ;) I'm starting to > believe that it's the mailing list system that needs to vouch for the > messages its recipients receive. Of course, it could be Mailman doing > the DKIM signing, or it could be Mailman's outgoing MTA, etc. But, > ISTM Mailman is ultimately deciding what goes into the list copy, so > it is responsible for it. > > - -Barry > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (Darwin) > > iEYEARECAAYFAkXCt04ACgkQ2YZpQepbvXHsAACfVOu25u2Ps2MrC0qQE7i/W5sx > ZdYAoL0z9q0+zjt7gpI3JrFy62m4DkAq > =+a49 > -----END PGP SIGNATURE-----

1 0

Mailman-Developers February 2007