[Mailman-Users] Problems with uuencoded attachments

[Mark Sapiro]

Dimitrios Gerasimatos wrote:
>
>If I send an attachment encoded with old HP-UX uuencode or
>Eudora 6.2.3 uuencode then mailman shunts the message.


This occurs during archiving (see below). The message is delivered - at
least to non-digest subscribers.


>The
>error is "Warning: Trailing garbage" which is actually coming
>from Python's uu.py module, which in turn gets it from the
>binascii library. I wrote a little script to decode the file
>in the shunt directory after running show_qfiles on it:
>
>
>#!/usr/bin/python
>
>
>import uu
>import os, sys
>
>
>infile = "myfile"
>uu.decode(infile, sys.stdout)
>
>
>This is where the above error message is coming from. Looking
>at the C source code to binascii, this message results because
>there are trailing characters in the message which are not
>blanks, newlines, or '`' (a special character which means ' '
>to some encoders). The character is above the 'end' tag and is
>actually the last character in the encoding. Here are the last
>few lines:
>
>
>M,3,P,3(S,B`P,#`P,"!N(`IT<F%I;&5R"CP\("]3:7IE(#DR("]2;V]T(#@Y
>M(#`@4B`O26YF;R`Y,"`P(%(@+TE$(#DQ(#`@4B`^/@IS=&%R='AR968*,3,P
>+,3,R, at HE)45/1 at HY
>`
>end
>
>
>The character Python complains about is the 'Y'. I actually
>tried two different files (a PDF and an Excel file) and both
>failed with the same error and complained about the last
>character. I thought this must be a Python bug, because when I
>take the same original file and encode it on an HP-UX command
>line I get the same as above, therefore absolving Eudora. However,
>if I uuencode the file using MS Outlook I get the following:
>
>
>M,3,P,3(S,B P,# P,"!N( IT<F%I;&5R"CP\("]3:7IE(#DR("]2;V]T(#@Y
>M(# @4B O26YF;R Y," P(%(@+TE$(#DQ(# @4B ^/@IS=&%R='AR968*,3,P
>+,3,R, at HE)45/1 at H`
>`
>end
>
>
>Mailman/python have no problems with this file sent via Outlook.


For one or both of two reasons.


>Notice the final character is a '`' and not a 'Y'.


That's one reason. The other is if Outlook is like OE, the uuencoded
'attachment' is not a separate MIME part so it will not be 'scrubbed'.


>I have a few questions:
>
>
>1. Why do Eudora and HP-UX both produce code different from
>   Outlook? Did two vendors make the same bug or is this a
>   specification issue?


The last line of your encoded file encodes 11 bytes or 88 bits - the
count is '+' which in Python means there are ord('+') - ord(' ') bytes
in the line. This requires 15 characters to encode, however the number
of characters in a line is required to be a multiple of 4, so there
are 16 characters in the encoding, the last of which is immaterial.
Most encoders will set these pad characters to '`', but it's not
required, however Python will complain with a warning message if
uu.decode() is called without a 'quiet=True' argument when these pad
bytes are not ' ' or '`'.

In my testing, Windows Eudora 5.2.0.9 encoded a file which required two
pad characters and they were both '`'. So this version of Eudora seems
to do what Python expects. Brief testing with HP-UX B.11.00 and
examination of your data indicates that in HP-UX uuencode, the pad
characters are residue from the same positions in the previous line.

Also, here are a few other things to consider.

MS Outlook Express and maybe Outlook does not actually 'attach'
uuencoded attachments. It just puts the uuencoded file in the
text/plain body of the message. Of course, when it receives a message
with a uuencoded file in the text/plain body, it calls it an
'attachment' even though it is not an attachment in the MIME sense.

Eudora however, will place the uuencoded file in a separate MIME part
with an appropriate 'Content-Type:' and 'Content-Transfer-Encoding:
x-uuencode'. Private x-* Content-Transfer-Encodings are strongly
discouraged by RFC 2045.



>2. What can I do to mailman such that it can process the
>   attachments sent by these rogue mailers like Eudora
>   under MacOS X (definitely modern software)? Right now
>   it chokes with:
>
>
>Jan 14 06:36:21 2006 (2768) SHUNTING:
>1137189869.4197209+f52dada65a850f1abfbdd0e
>2044e7a117a6828f9
>Jan 14 06:36:21 2006 (2768) uncaught archiver exception at filepos: 0
>Jan 14 06:36:21 2006 (2768) Uncaught runner exception: [Errno 32] Broken
>pipe
>Jan 14 06:36:21 2006 (2768) Traceback (most recent call last):
>  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 111, in _oneloop
>    self._onefile(msg, msgdata)
>  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 167, in _onefile
>    keepqueued = self._dispose(mlist, msg, msgdata)
>  File "/usr/lib/mailman/Mailman/Queue/ArchRunner.py", line 73, in _dispose
>    mlist.ArchiveMail(msg)
>  File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 216, in
>ArchiveMail
>    h.processUnixMailbox(f)
>  File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 569, in
>processUni
>xMailbox
>    m = mbox.next()
>  File "/usr/lib/python2.3/mailbox.py", line 35, in next
>    return self.factory(_Subfile(self.fp, start, stop))
>  File "/usr/lib/mailman/Mailman/Mailbox.py", line 89, in scrubber
>    return mailbox.scrub(msg)
>  File "/usr/lib/mailman/Mailman/Mailbox.py", line 109, in scrub
>    return self._scrubber(self._mlist, msg)
>  File "/usr/lib/mailman/Mailman/Handlers/Scrubber.py", line 290, in process
>    payload = part.get_payload(decode=True)
>  File "/usr/lib/mailman/pythonlib/email/Message.py", line 223, in
>get_payload
>    uu.decode(StringIO(payload+'\n'), sfp)
>  File "/usr/lib/python2.3/uu.py", line 139, in decode
>    sys.stderr.write("Warning: %s\n" % str(v))
>  File "/usr/lib/mailman/Mailman/Logging/MultiLogger.py", line 45, in write
>    _logexc(logger, msg)
>  File "/usr/lib/mailman/Mailman/Logging/Utils.py", line 22, in _logexc
>    sys.__stderr__.write('Logging error: %s\n' % logger)
>IOError: [Errno 32] Broken pipe


The scrubber is trying to remove the attachment from the original
message to store it separately in the archive. It calls the
get_payload(decode=True) method to get the decoded attachment to save.
This ultimately calls Pytnon's uu.decode function which trys to write
the Warning to sys.stderr. The real problem is that at this point
sys.stderr is not writable, so that throws the IOError exception which
causes the message to be shunted.


>3. Is this possibly a python bug? HP-UX uudecode can decode
>   these files no problem, but python gives me the
>   "Warning: Trailing garbage" error. What's more, if I
>   mail from Eudora to an IMAP account (not mailman) I have
>   no problem opening the Eudora attachment in MS Outlook.
>   Therefore, why does mailman dislike it?


I would say it's not a Python bug since Python is only trying to issue
a warning and provides a means to suppress the warning.

I think this could be fixed by changing
"/usr/lib/mailman/pythonlib/email/Message.py", line 223 from

                    uu.decode(StringIO(payload+'\n'), sfp)

to

                    uu.decode(StringIO(payload+'\n'), sfp, quiet=True)


-- 
Mark Sapiro <msapiro at value.net>       The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan