Bugs item #1518281, was opened at 2006-07-06 10:27 Message generated for change (Comment added) made by msapiro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=100103&aid=1518281&group_id=103 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: mail delivery Group: 2.1 (stable) Status: Open Resolution: None Priority: 5 Submitted By: Philip Crotwell (crotwell) Assigned to: Nobody/Anonymous (nobody) Summary: Bad Content-Disposition breaks senddigest Initial Comment: We are using the default mailman on mac osx server 10.3 which is mailman 2.1.2, so this may be old. But just in case, we got an email with attachement that caused mailman to croak in senddigest. Here is the output of running it by hand. I added 2 prints to see what the offending filename variable had in it. The Content-Disposition headers are below. Note that the two parts of the filename have been joined, but because of the bizarre endquoting in filename*0, there is an embedded \n\t. We fixed this locally by editing the mbox to change the offending filename to be shorter and not split into two parts, but it might be nice if mailman would check for illegal characters embedded in filenames? In addition to senddigest croaking, all emails after this bad one were not added to the archive. Luckily we had made a copy of digest.mbox before running senddigest manually and were able to manually add the lost emails to the archive mbox. roo:/usr/share/mailman root# python -S cron/senddigests -l chair-archive (None, None, 'Outside Professional Activities GS Policy_4-22-03 rev\n\t5-16-05.doc') (None, None, 'Outside Professional Activities GS Policy_4-22-03 rev\n\t5-16-05.doc') Traceback (most recent call last): File "cron/senddigests", line 94, in ? main() File "cron/senddigests", line 86, in main mlist.send_digest_now() File "/usr/share/mailman/Mailman/Digester.py", line 60, in send_digest_now ToDigest.send_digests(self, mboxfp) File "/usr/share/mailman/Mailman/Handlers/ToDigest.py", line 130, in send_digests send_i18n_digests(mlist, mboxfp) File "/usr/share/mailman/Mailman/Handlers/ToDigest.py", line 303, in send_i18n_digests msg = scrubber(mlist, msg) File "/usr/share/mailman/Mailman/Handlers/Scrubber.py", line 257, in process url = save_attachment(mlist, part, dir) File "/usr/share/mailman/Mailman/Handlers/Scrubber.py", line 348, in save_attachment fnext = os.path.splitext(msg.get_filename(''))[1] File "/usr/share/mailman/pythonlib/email/Message.py", line 709, in get_filename return unicode(newvalue[2], newvalue[0]) TypeError: unicode() argument 2 must be string, not None --Apple-Mail-5-348793981 Content-Transfer-Encoding: base64 Content-Type: application/msword; x-unix-mode=0666; name="Outside Professional Activities GS Policy_4-22-03 rev 5-16-05.doc" Content-Disposition: attachment; filename*0="Outside Professional Activities GS Policy_4-22-03 rev 5-16-05.do"; filename*1=c ----------------------------------------------------------------------

Comment By: Mark Sapiro (msapiro) Date: 2006-07-07 18:07

Message: Logged In: YES user_id=1123998

We fixed this locally by editing the mbox to change the offending filename to be shorter and not split into two parts, but it might be nice if mailman would check for illegal characters embedded in filenames?

I'm not sure if it is clear from my first comment, but the cause of this exception (bug) was the RFC2231 splitting, not the '\n\t' from header folding. Also, in later Mailman versions, Scrubber is much more robust against bad characters/encodings in filenames.

In addition to senddigest croaking, all emails after this bad one were not added to the archive. Luckily we had made a copy of digest.mbox before running senddigest manually and were able to manually add the lost emails to the archive mbox.

I suspect these messages were shunted and could have been reprocessed to the archive by bin/unshunt. At this point, since the messages are in the archive, you should check to see if they are still in qfiles/shunt/ and if so, delete them from there so the don't get inadvertently unshunted in the future. ---------------------------------------------------------------------- Comment By: Mark Sapiro (msapiro) Date: 2006-07-07 17:50 Message: Logged In: YES user_id=1123998 There are a couple of different issues here. The '\n\t' in the filename come about because the header is folded in the middle of the quoted value of the filename*0 parameter. The filename*0 and filename*1 parameters split the filename per RFC2231, the very purpose of which is to allow breaking of long parameter values into smaller pieces so that header folding/unfolding can be accomplished at syntactic breaks - e.g. at the semicolons. In this case, either the generating MUA (Apple Mail) folded the header in the middle of the filename*0 value or it made the value so long that a subsequent MTA folded it. Thus, with respect to the '\n\t' in the middle of the name, it results from a non-compliant message. The immediate cause of the exception in your case was fixed in the email library shipped with Mailman 2.1.3. The fix replaces return unicode(newvalue[2], newvalue[0]) with return unicode(newvalue[2], newvalue[0] or 'us-ascii') and makes a similar change in a few other places. The patch for this change is attached here. There is also a bug in the (in your case) /usr/share/mailman/pythonlib/email/Utils.py module. See the post at http://mail.python.org/pipermail/email-sig/2006-July/000293.html for more on this and a possible patch. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=100103&aid=1518281&group_id=103