[Mailman-Users] Re: [Mailman-Developers] Looping digest - mailman bug?

[Barry A. Warsaw]

>>>>> "KPC" == Kaja P Christiansen <kaja at daimi.au.dk> writes:

    KPC> Several of our lists suffered from Mailman's mail bombing and
    KPC> we turned the digest option off in hope it'll help. It did,
    KPC> for a while, until the same happened with a non-digest
    KPC> message being send over and over again (once per minute, by
    KPC> qrunner).

    KPC> We were able to locate where and how it happens, and to
    KPC> 'reproduce' the error (in laboratory conditions :-) both
    KPC> under mailman 2.0.3 and 2.0.6.

    KPC> The mail looping occurs when there is a MIME message with a
    KPC> single . (dot) in a line; Mailman sends it to sendmail 'as
    KPC> is'. But since sendmail/postfix interprets a line with single
    KPC> dot as the end of the message, it sends everything before the
    KPC> dot and exits with 'Broken pipe'. Mailman, however, still has
    KPC> the message in it's queue and sends it all over again...

    KPC> When I tried sending non-MIME message with single-dot-line in
    KPC> it to a Mailman list, there was no looping, but the message
    KPC> body after the single dot was missing.

    KPC> Until there is something better, I suggest adding to
    KPC> Sendmail.py a patch which perhaps is not pretty (it adds a
    KPC> space before the infamous dot), but it works:

Okay, several issues going on here.  First, let me ask: why are you
using Sendmail.py instead of SMTPDirect.py?  The former has well known
adverse security holes, including being able to trick the shell used
during the os.popen() to do evil things.  I include your mailbomb
example as another security hole in Sendmail.py.  I'm strongly
considering removing Sendmail.py from MM2.1, but I want to know why
some people seem to prefer to use it instead of SMTPDirect.py first.

I suspect the reason is because Sendmail tends to want to do recipient
domain verification when invoked through smtp, even if the connection
is through localhost, while connecting through "sendmail -bs" it does
not.  If that's the primary reason, then we have two choices: 1)
document what Sendmail users should do in order to fix this problem in
their MTA, or 2) dig up the patches to enhance smtplib.py to do
sendmail -bs connections (I know there have been at least one such
contribution, but I'd have to search around for it).

I really want everybody to use SMTPDirect.py from now on, so let's fix
things so even the Sendmail.py'ers can.

>>>>> "TW" == Thomas Wouters <thomas at xs4all.net> writes:

    TW> Sorry, this is the wrong fix. "\n." isn't the pattern that
    TW> breaks it; you want "\r?\n.\r?\n". And the proper escape of a
    TW> single dot on a line is doubling it (see the SMTP standard.)
    TW> Barry, do you have a clue whether msgtext is guaranteed (not)
    TW> to have \r's (CR) in them ? If either is guaranteed, we don't
    TW> even need to use that ugly 're' module :)

Now, as to line endings and single-dot termination lines.  Python's
smtplib does the right thing here, and it fortunately hides everything
from the module client.  That's a big reason why SMTPDirect.py users
never get into trouble; it uses smtplib.  smtplib properly line
terminates according to RFC 2821 regardless of the line termination of
the source text.  It also properly implements dot-line transparency
according to $4.5.2.

On the receiving end, it appears that all the major Unix MTAs properly
undo the dot-line escaping, and convert all line endings to Unix line
endings (LF-only) before handing the text off to a program or file.
I didn't find Postfix documentation on this, but testing and this
message

    http://archives.neohapsis.com/archives/postfix/2000-02/0686.html

confirm this to be the case.  Exim has a configuration option
"use_crlf" which controls this behavior, but since the default value
is false, I'm guessing that most Exim installations canonicalize the
line endings to Unix-style too.  I suspect Sendmail does the same
thing, although I haven't found a definitive description yet.

So I think it's fair to assume that by the time Mailman's wrapper gets
a message, the line endings have all been converted to Unix-style LF
termination.

So I think Thomas is right that we don't need re and can just check
for the first character being a `.' and if so, quote it by adding a
leading `.'.

But I'm still more inclined to 1) zap Sendmail.py /and/ 2) merge
support for sendmail -bs into smtplib.py

-Barry