There is a bug report at
<https://sourceforge.net/tracker/index.php?func=detail&aid=1656289&group_id=…>.
The basic issue here is there is an extremely large message in the 'in'
queue. In followup emails with the submitter, he said "The file was
nearly half a terabyte in size".
Not surprisingly, IncomingRunner threw a MemoryError when SpamDetect
tried to flatten() the message, but then Runner tried to shunt the
message and Switchboard.enqueue() threw another MemoryError in the
attempt to pickle the new shunt queue entry.
The second MemoryError is uncaught in Runner._oneloop, so it causes the
main loop in Runner.run to exit.
Pre-Mailman 2.1.9, the master just restarts IncomingRunner - the
message is lost, but everything else is OK.
Because of the changes in 2.1.9 to prevent message loss in case of
disaster, there is now a .bak file left in the 'in' queue. When
IncomingRunner restarts, it recovers the .bak file and the whole
scenario repeats until the master reaches MAX_RESTARTS on
IncomingRunner and we are left with no IncomingRunner and the .bak
file still in the 'in' queue.
In order to fix this, I suggest we protect the shunt enqueue in a try.
I have worked up a patch for this which is attached. This patch also
adds a 'preserve' argument to Switchboard.finish such that if it is
called with preserve=True, instead of removing the .bak file, it just
renames it with a .psv extension. These changes ensure that
IncomingRunner doesn't exit, and no .bak file is left to cause a
subsequent problem while still preserving the original queue entry for
further analysis if possible.
I would like some feedback on whether or not this is the right approach.
--
Mark Sapiro <msapiro(a)value.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
Hello
I love and use python a lot. And I m trying to make mailman, into a yahoo
groups kind of interface,
In that I want to find out if any of you guys tried to do this and integrate
it into another webservice?
Does mailman uses coroutines to send email, because a coroutine based email
gateway can support much more email messages perday than a thread based one
Mark
Hello,
is there a reason why the roster page does not show full names?
Obvious concern is the privacy of the list members.
But on the other hand:
- the roster has its own access control,
- the mail address itself is usually more valuable and can be used to
get the real name.
- entering the real name on subscription is optional.
Thus IMHO showing real names does not disclose significant information,
but makes things more convenient for legitimate users.
I have attached a three-line patch against Mailman/HTMLFormatter.py
--
Martin Schütte
96a97,99
> realname = Utils.uncanonstr(self.getMemberName(person), lang)
> if realname:
> showing += " (%s)" % Utils.websafe(realname)
Anyone please correct me if this feature already exists, however I'm
wondering if it's already been developed so that if a user reply's to a
digest in their box then mailman will extract the relevant post they are
replying to and also warn the user.
Obviously there are a fair few metric with this, such as which post are they
replying to.
So is this already implemented somewhere or should I develop it?
Regards
Per http://www.list.org/bugs.html I am sending an email to this list
about the feature request I reported at:
https://sourceforge.net/tracker/index.php?func=detail&aid=1657458&group_id=…
It basically says that it would be nice to have another tickbox in the
admindb page that lets you "delete all pending *subscription*
*requests* marked /defer/" and not just a tickbox that does that for
*messages* since we're getting a fair amount of spam in the form of
subscription requests to lists that we manage.
Thanks for your attention,
--
Cristóbal M. Palmer
ibiblio.org system administrator
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I'm not sure what the right answer is just yet, but I'll offer some
of my thoughts FWIW.
I think the fundamental question is whether the mailing list is the
originator of the messages its members receive or whether the
original author is. This question has come up in other contexts
before, and I don't think it's ever been answered satisfactorily. A
quick search through DKIM archives seems to indicate that this
question has come up there too, and I think answering it will help us
understand what we should be doing here.
On Feb 1, 2007, at 3:00 PM, Mark Sapiro wrote:
> Consider that while Mailman doesn't do all of these things to every
> message, it can do any of the following:
[munge the original message]
From the DKIM FAQ:
- What is the purpose of DKIM?
DKIM lets an organization take responsibility for a message.
The organization taking responsibility is a handler of the
message, either as its originator or as an intermediary. Their
reputation is the basis for evaluating whether to trust the
message for delivery.
I think you can make a legitimate case that Mailman is the originator
of messages its members receive. The message is certainly different
than the one the original poster sent to the list, and Mailman is
clearly an intermediary. Perhaps the message has only been munged in
very trivial ways, but it's also possible to munge it in ways that
could potentially be viewed as spammy. For example, what if a site
decides to put some advertisements in the footer?
If you take this view then it seems reasonable to say that it is the
mailing list's system that "take[s] responsibility for a message."
Sure, the mailing list system could verify the DKIM headers on the
message it receives, but ultimate, it is up to the mailing list
system to decide whether that message (or some derivative of that
message) gets transmitted to its recipients.
Or looked at another way, if I send a message through a mailing list,
I wouldn't want to vouch for whatever comes out the other end because
I don't know what they're going to do to my original content. Maybe
then, it's correct for the DKIM signature on the copied message to be
broken because what recipients got was /not/ the message I sent, and
I don't know how it was munged. But that view implies that I am the
originator of the recipient's message. I am, sort of, but also sort
of not.
I'm not convinced that DKIM is really designed to handle the mailing
list use case. It seems to me that it was designed to handle point-
to-point messages, not messages that flow through an intermediary,
because it's not an enveloping system. Contrast that with S/MIME or
OpenPGP. I can sign the message I send from my mailer and that could
be preserved through the transformations that Mailman performs, with
Mailman wrapping my original in its own signature if it wanted to.
Practically speaking, if we can't come up with a consensus on the
interpretation of which "organization [should] take responsibility
for" the actual message that recipients receive, then what would be
the right thing to do? (Note that this answer is different depending
on whether we're talking about Mailman 2.1 or some future version.)
When this came up before I statement my preference not to make a
"strip DKIM headers" selectable by the list owner. I still prefer
this for Mailman 2.1 because doing so would clearly be a new
feature. Maybe a future version could treat the DKIM header the way
it treats the RFC 2369 headers, with a separate selector for List-
Post. Ideally, we'd have a more general way to decide which headers
get cleansed and which new ones get added. But that's for the future.
One elaboration you /might/ be able to get away with in Mailman 2.1
occurs to me as I look at Mark's list:
> - Add text to the beginning of the message body (msg_header)
> - Add text to the end of the message body (msg_footer)
> - Remove text from the beginning of the message body (Approved: line)
> - Add additional MIME parts to a multipart message (msg_header,
> msg_footer)
> - Convert a single part message to multipart in order to add
> msg_header/msg_footer
> - Remove parts from a multipart message (content filtering)
> - Convert an HTML part to plain text (content filtering)
> - Decode a base64 or quoted-printable encoded part and perhaps
> re-encode it with a different encoding.
> - Change or delete various headers including Subject:, To:, From:
> - Replace some MIME parts with URLs of where they were stored and
> flatten the entire message into a single plain text message
> (scrubber).
> - Probably other things I'm overlooking.
If you could identify the message transformations that break the
signature, then you could remove the signature. If the signature of
the outgoing message were still valid because Mailman didn't touch
any part of the message affecting the signature, then you could keep
it. The implementation of this would be fairly simple; the hard part
is writing the code to verify a DKIM signature and parse the
selectors (IIUC the specification) to figure out which of the above
transformations would break the signature. That might be enough to
not do it in Mailman 2.1.
I'm not sure how much I like that anyway, so comments are definitely
welcome. After mulling over this post for an hour ;) I'm starting to
believe that it's the mailing list system that needs to vouch for the
messages its recipients receive. Of course, it could be Mailman
doing the DKIM signing, or it could be Mailman's outgoing MTA, etc.
But, ISTM Mailman is ultimately deciding what goes into the list
copy, so it is responsible for it.
- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)
iEYEARECAAYFAkXCvIgACgkQ2YZpQepbvXFnVgCfeUkfQ0+h/bBAKiwznDTdrHJ6
7V0An2O9TcUYBJYlFhYFpLtYUUGalabq
=yCgT
-----END PGP SIGNATURE-----
Michael Thomas wrote:
>
>On Wed, 7 Feb 2007, Mark Sapiro wrote:
>
>> Mike talks about the l= parameter allowing adding trailing content, but
>> I don't see Y! and Gmail using it, and even if they did, how would we
>> (could we) add a footer without breaking either the signature or the
>> MIME structure of the message.
>
> l= is the number of canonical bytes added to the body hash.
> If l=5, for example, anything past the 5th canonical byte will
> not affect the verification of the signature. That's the reason
> we get such high verify rates through mailing lists.
My point is that for what I consider good reasons, Mailman will add the
msg_footer to such a message by wrapping additional MIME structure
around the original multipart/alternative message.
I.e., the original
multipart/alternative
text/plain
text/html
message will be recast as
multipart/mixed
multipart/alternative
text/plain
text/html
text/plain
with the final text/plain part containing the footer. Given that the
original content-type header is included in the signature, the
signature is now broken.
If we were to take a different approach with a signature containing l=,
either the l= includes all the text/plain and at least part of the
text/html, in which we can't add the footer to the text/plain
alternative without breaking the signature, or the l= includes none of
the text/html part in which case the signature is not very good at
verifying the validity of the text/html part. This further assumes we
even know how to add a footer to a text/html part.
See
<http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.039.htp>
for some discussion of why we do it the way we do.
--
Mark Sapiro <msapiro(a)value.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello everyone,
I just wanted to drop a quick note to let you know that I have
started working for Canonical, the folks who distribute Ubuntu Linux
and develop the Launchpad service for distributed open source
development.
I'm quite excited to be working for Canonical. They are a Python and
Zope shop, very open source friendly, and based on Linux, so I'm
working in an environment that's both familiar and fun. They're also
a distributed company, without traditional offices, but with many
folks working all over the world. They're also hiring <wink, wink>. :)
While this is good for Barry (and hopefully good for Canonical :), I
know you're thinking, "what does this mean for Mailman?" Well, I'm
glad you asked!
Part of my official duties will be to integrate Mailman with
Launchpad so as to enable more powerful communication patterns among
members of a distributed development team. While I won't be
spending /all/ of my time on Mailman, it will be getting some
official love. I hope that this will help move the current
development branch closer to a release some time later this year.
Other than that, not much will really change. Mailman is and will
always be released under the GPL, and it will continue to be a GNU
project.
If you have any questions or concerns, please send them directly to
me. I will attempt to answer them as best I can, and if there are
common themes, I'll post updates here.
Thanks for your support and keep on delivering with Mailman!
- -Barry
Here are some related links:
http://www.canonical.comhttp://www.ubuntu.comhttp://launchpad.net
P.S. I'll be at PyCon this year so if you're coming too, please say hi!
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
iQCVAwUBRcUTb3EjvBPtnXfVAQIQKgP/XFVeeDBn6QVXkE9oK1YJxyrLZZET5GxT
TAvTJgfndkcWPuUpbC5D6hcpDNa6sfIgJnR3enoW7MgKpOAtIXTuXqPpNiFMBVT2
qUhlDHhwYzdJWWyzI/oXGRvt0lMqqA69Iu7A6DAKrgksBB128V/mYxTfv8BPmF4W
uASb/9MkmAQ=
=h+IQ
-----END PGP SIGNATURE-----
Hi all,
I'm one of the authors of the DKIM protocol and it recently came to
my attention that you've recently changed mailman to remove DK and
DKIM signature headers when you remail the message. This is incorrect
behavior:
in Section 4:
Signers SHOULD NOT remove any DKIM-Signature header fields from
messages they are
signing, even if they know that the signatures cannot be verified.
This actually applies to everybody. There are several reasons for this.
First
is that DKIM allows you to specify the length of a body so it is not the
case
a priori that mailman will destroy the signature. Second, other
heuristics can
be applied to make mailing list traversal even better such as using the
z= tag
to determine whether trivial subject modifications have been made. Third and
probably most important is that removing the signature is actually
harmful rather
than helpful: a broken signature and a missing signature MUST be treated as
equivalent to no signature at all (lest an attacker just add a fake
DKIM-signature
header to get preferential treatment), and as above the verifier loses
the ability
to recover the signature.
Just as an FYI, we have deployed DKIM across all of Cisco and our successful
mailing list traversal rate is about 99% -- a large percentage of which
are through
mailman lists. By making this change, you've taken the verify rate from
99% to
0% in one swell foop. Not good.
Mike
The very basic test I use is what's in the From: address. That's the thing
that's pretty universally displayed and one that users are most likely to
grok. Anything beyond From, and you've probably lost at least half of
the user population at least.
So mailing lists preserve the original From: and perhaps add a Sender:.
If I had to chose a signature which was most relevant for _display_ purposes
(note that display purposes is but _one_ reason for signatures), I'd
like to
be able to say something good about the From address that most users
grok. It's unclear that I'd want to say something nice about the Sender or
any other third party signature because it may confuse users that the
mere presence of a signature imputes some trust, where it actually needs
to be considered on a case by case basis. I for one would only render the
sender: as "verified" both if it verified correctly as well as it being
on some
known whitelist of domains that I trust.
So the bottom line is that a valid third party signature from, say, a
mailing
list is not safe a priori. It requires special handling by the ultimate
receiver
in the form of a trust relationship with that domain which needs be done
out of band. The same is not of a first party signature because you're only
vouching for yourself which is already as good you trust that domain (or
not).
Mike, is this making any sense?
Barry Warsaw wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I'm not sure what the right answer is just yet, but I'll offer some of
> my thoughts FWIW.
>
> I think the fundamental question is whether the mailing list is the
> originator of the messages its members receive or whether the original
> author is. This question has come up in other contexts before, and I
> don't think it's ever been answered satisfactorily. A quick search
> through DKIM archives seems to indicate that this question has come up
> there too, and I think answering it will help us understand what we
> should be doing here.
>
> On Feb 1, 2007, at 3:00 PM, Mark Sapiro wrote:
>
>> Consider that while Mailman doesn't do all of these things to every
>> message, it can do any of the following:
> [munge the original message]
>
> From the DKIM FAQ:
>
> - What is the purpose of DKIM?
>
> DKIM lets an organization take responsibility for a message. The
> organization taking responsibility is a handler of the
> message, either as its originator or as an intermediary. Their
> reputation is the basis for evaluating whether to trust the
> message for delivery.
>
> I think you can make a legitimate case that Mailman is the originator
> of messages its members receive. The message is certainly different
> than the one the original poster sent to the list, and Mailman is
> clearly an intermediary. Perhaps the message has only been munged in
> very trivial ways, but it's also possible to munge it in ways that
> could potentially be viewed as spammy. For example, what if a site
> decides to put some advertisements in the footer?
>
> If you take this view then it seems reasonable to say that it is the
> mailing list's system that "take[s] responsibility for a message."
> Sure, the mailing list system could verify the DKIM headers on the
> message it receives, but ultimate, it is up to the mailing list system
> to decide whether that message (or some derivative of that message)
> gets transmitted to its recipients.
>
> Or looked at another way, if I send a message through a mailing list,
> I wouldn't want to vouch for whatever comes out the other end because
> I don't know what they're going to do to my original content. Maybe
> then, it's correct for the DKIM signature on the copied message to be
> broken because what recipients got was /not/ the message I sent, and I
> don't know how it was munged. But that view implies that I am the
> originator of the recipient's message. I am, sort of, but also sort
> of not.
>
> I'm not convinced that DKIM is really designed to handle the mailing
> list use case. It seems to me that it was designed to handle
> point-to-point messages, not messages that flow through an
> intermediary, because it's not an enveloping system. Contrast that
> with S/MIME or OpenPGP. I can sign the message I send from my mailer
> and that could be preserved through the transformations that Mailman
> performs, with Mailman wrapping my original in its own signature if it
> wanted to.
>
> Practically speaking, if we can't come up with a consensus on the
> interpretation of which "organization [should] take responsibility
> for" the actual message that recipients receive, then what would be
> the right thing to do? (Note that this answer is different depending
> on whether we're talking about Mailman 2.1 or some future version.)
>
> When this came up before I statement my preference not to make a
> "strip DKIM headers" selectable by the list owner. I still prefer
> this for Mailman 2.1 because doing so would clearly be a new feature.
> Maybe a future version could treat the DKIM header the way it treats
> the RFC 2369 headers, with a separate selector for List-Post.
> Ideally, we'd have a more general way to decide which headers get
> cleansed and which new ones get added. But that's for the future.
>
> One elaboration you /might/ be able to get away with in Mailman 2.1
> occurs to me as I look at Mark's list:
>
>> - Add text to the beginning of the message body (msg_header)
>> - Add text to the end of the message body (msg_footer)
>> - Remove text from the beginning of the message body (Approved: line)
>> - Add additional MIME parts to a multipart message (msg_header,
>> msg_footer)
>> - Convert a single part message to multipart in order to add
>> msg_header/msg_footer
>> - Remove parts from a multipart message (content filtering)
>> - Convert an HTML part to plain text (content filtering)
>> - Decode a base64 or quoted-printable encoded part and perhaps
>> re-encode it with a different encoding.
>> - Change or delete various headers including Subject:, To:, From:
>> - Replace some MIME parts with URLs of where they were stored and
>> flatten the entire message into a single plain text message (scrubber).
>> - Probably other things I'm overlooking.
>
> If you could identify the message transformations that break the
> signature, then you could remove the signature. If the signature of
> the outgoing message were still valid because Mailman didn't touch any
> part of the message affecting the signature, then you could keep it.
> The implementation of this would be fairly simple; the hard part is
> writing the code to verify a DKIM signature and parse the selectors
> (IIUC the specification) to figure out which of the above
> transformations would break the signature. That might be enough to
> not do it in Mailman 2.1.
>
> I'm not sure how much I like that anyway, so comments are definitely
> welcome. After mulling over this post for an hour ;) I'm starting to
> believe that it's the mailing list system that needs to vouch for the
> messages its recipients receive. Of course, it could be Mailman doing
> the DKIM signing, or it could be Mailman's outgoing MTA, etc. But,
> ISTM Mailman is ultimately deciding what goes into the list copy, so
> it is responsible for it.
>
> - -Barry
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (Darwin)
>
> iEYEARECAAYFAkXCt04ACgkQ2YZpQepbvXHsAACfVOu25u2Ps2MrC0qQE7i/W5sx
> ZdYAoL0z9q0+zjt7gpI3JrFy62m4DkAq
> =+a49
> -----END PGP SIGNATURE-----