Mailman 3 September 2001 - Mailman-Developers

[PATCH] mimelib base64 and q-p message decoding
by Ben Gertzfield 16 Sep '01

16 Sep '01

This patch is against mimelib CVS. It adds a few new extremely useful functions to mimelib.Message, which let users recursively get all parts of a message, decoded into viewable text format -- even non-text attachments will be replaced with a user-configurable message! It's required for the next patch I'm about to send, which ports pipermail/HyperArch and a few various other scripts from pythonlib/rfc822.py to mimelib. This is a huge help, because it was very difficult to follow the code when two slightly different Message classes (mimelib's and rfc822's) were being used. In addition, it adds a feature to Mailman that's been requested for some time: Base64-encoded gobbeldygook will no longer show up in the archives! Instead, non-viewable attachments will be displayed (by default) as: [Non-text (image/gif) part of message not included, filename foo.gif] In addition, users who attach multiple text files to a message will have them all show up in Pipermail, without the message delimiter gobbeldygook.. Both quoted-printable and base64 are dealt with in this patch, so Pine users who attach text files encoded in base64 will have them show up properly in the archives. This also lets us remove the quoted-printable code from Pipermail completely, as all the MIME-related stuff should be dealt with in mimelib from now on. This patch also includes my previous mimelib patch, which adds the getcharsets function. Here's a list of the functions this patch adds. It doesn't modify anything else besides including a few more modules. getfilename(self, failobj=None): Returns the filename associated with the payload if present. decode_body(self): Returns a string of the non-multipart message's body, decoded. If the message is encoded with quoted-printable or base64, will decode and return its payload. Otherwise, returns the payload as-is. Returns None if the message is multipart. get_decoded_payload(self, non_text_msg): Returns an array containing all decoded text parts of a MIME message. Will recurse through all payloads even if the message is multi-part. When a text part is seen, decodes the text if it's Base64 or Quoted-Printable encoded, and appends the (possibly decoded) text part to the end of the resulting array. When a non-text part is seen, replaces it with non_text_msg, which defaults to: [Non-text (%(type)s) part of message not included, filename %(filename)s]\n and appends it to the end of the resulting array. The following keywords will be expanded in non_text_msg: type: Full MIME type of the non-text part maintype: Main MIME type of the non-text part subtype: Sub MIME type of the non-text part filename: Filename of the non-text part description: Description associated with the non-text part transfer-encoding: Transfer encoding of the non-text part get_text_payload(self, non_text_msg): Return the decoded body of the message in a text format. If the message is not text, will return non_text_msg, formatted as described in get_decoded_payload(). getcharsets(self, default): Return an array containing the charset[s] used in a message. Returns an array containing one element for each part of the message; will return an array of one element if the message is not a multipart message. Each element will either be a string (the charset in the Content-Type of that part) or the value of the 'default' parameter (defaults to None), if the part is not a text part or the charset is not defined. Patch follows. Index: mimelib/Message.py =================================================================== RCS file: /cvsroot/mimelib/mimelib/mimelib/Message.py,v retrieving revision 1.13 diff -u -r1.13 Message.py --- mimelib/Message.py 2001/05/04 18:47:22 1.13 +++ mimelib/Message.py 2001/09/13 06:15:07 @@ -5,6 +5,11 @@ import re import address +import base64 +import quopri +import string +from StringIO import StringIO +from rfc822 import unquote from types import ListType SEMISPACE = '; ' @@ -272,3 +277,132 @@ if name.lower() == param: return address.unquote(val) return failobj + + def getcharsets(self, default=None): + """Return an array containing the charset[s] used in a message. + + Returns an array containing one element for each part of the + message; will return an array of one element if the message is not + a multipart message. + + Each element will either be a string (the charset in the + Content-Type of that part) or the value of the 'default' + parameter (defaults to None), if the part is not a text part + or the charset is not defined. + """ + result = [] + + if self.ismultipart(): + for p in self.get_payload(): + if p.getmaintype() == "text": + result.append(p.getparam("charset")) + else: + result.append(default) + else: + if self.getmaintype() == "text": + result.append(self.getparam("charset")) + else: + result.append(default) + + return result + + def getfilename(self, failobj=None): + """Return the filename associated with the payload if present.""" + + disp = self.get("Content-Disposition") + + if disp is None: + return failobj + + # Match up to and not including the next semicolon, if any + filename = re.search(r'filename\s*=\s*([^;]*)', disp, re.IGNORECASE) + + if filename is None: + return failobj + + # Trim the whitespace if there was any at the end of the filename, + # then remove quotes from around it. + return unquote(string.rstrip(filename.group(1))) + + def get_decoded_payload(self, non_text_msg="[Non-text (%(type)s) part of message not included, filename %(filename)s]\n"): + """Return an array containing all decoded text parts of a MIME message. + + Will recurse through all payloads even if the message is + multi-part. When a text part is seen, decodes the text if + it's Base64 or Quoted-Printable encoded, and appends the + (possibly decoded) text part to the end of the resulting + array. + + When a non-text part is seen, replaces it with non_text_msg, + which defaults to: + + [Non-text (%(type)s) part of message not included, filename %(filename)s]\n + + and appends it to the end of the resulting array. + + The following keywords will be expanded in non_text_msg: + + type: Full MIME type of the non-text part + maintype: Main MIME type of the non-text part + subtype: Sub MIME type of the non-text part + filename: Filename of the non-text part + description: Description associated with the non-text part + transfer-encoding: Transfer encoding of the non-text part + """ + result = [] + if self.ismultipart(): + for p in self.get_payload(): + # Can we even get multipart messages w/ multiparts inside? + if p.ismultipart(): + result.append(p.get_decoded_payload(non_text_msg)) + else: + result.append(p.get_text_payload(non_text_msg)) + else: + result.append(self.get_text_payload(non_text_msg)) + + return result + + def get_text_payload(self, non_text_msg): + """Return the decoded body of the message in a text format. + + If the message is not text, will return non_text_msg, formatted + as described in get_decoded_payload(). + """ + if self.getmaintype('text') == 'text': + return self.decode_body() + else: + return non_text_msg % { + 'type': self.gettype(), + 'maintype': self.getmaintype(), + 'subtype': self.getsubtype(), + 'filename': self.getfilename(), + 'description': self.get('Content-Description'), + 'transfer-encoding': self.get('Content-Transfer-Encoding') + } + + def decode_body(self): + """Return a string of the non-multipart message's body, decoded. + + If the message is encoded with quoted-printable or base64, will + decode and return its payload. Otherwise, returns the payload + as-is. + + Returns None if the message is multipart. + """ + if self.ismultipart(): + return None + + cte = self.get('Content-Transfer-Encoding') + # Assume no encoding if header not specified + if cte is None: + return self.get_payload() + elif string.lower(cte) == 'quoted-printable': + input = StringIO(self.get_payload()) + output = StringIO() + quopri.decode(input, output) + return output.getvalue() + elif string.lower(cte) == 'base64': + return base64.decodestring(self.get_payload()) + # Otherwise, could be 7bit, 8bit, or binary.. don't mess with it + else: + return self.get_payload()

3 4

Authentication errors in admin pages.
by Chuq Von Rospach 14 Sep '01

14 Sep '01

Someone brought this up a couple of days ago, and I mentioned I'd been seeing something similar. Here's the traceback I'm seeing. I see it intermittently, usually if I'm going through a number of lists, one after another. It never happens on the first list, but on one of the later ones. And if I hit the "refresh" button on the browser, it goes away and brings up the password page just fine (that's why I never mentioned it; I was sort of assuming it was a weird interaction between python, cookies and mod_layout, and hadn't had time to debug it. It might STILL be that, for all I know...) Any thoughts on what this traceback is telling us, I'd be happy to hear and test... Especially since the refresh of the page makes it leave again... Sep 14 20:30:47 2001 admin(24621): @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ admin(24621): [----- Mailman Version: 2.0.5 -----] admin(24621): [----- Traceback ------] admin(24621): Traceback (innermost last): admin(24621): File "/home/mailman/scripts/driver", line 96, in run_main admin(24621): main() admin(24621): File "/home/mailman/Mailman/Cgi/admindb.py", line 78, in main admin(24621): Auth.authenticate(mlist, cgidata) admin(24621): File "/home/mailman/Mailman/Cgi/Auth.py", line 65, in authenticate admin(24621): if cgidata.has_key('adminpw') and \ admin(24621): File "/home/mailman/Mailman/pythonlib/cgi.py", line 564, in has_key admin(24621): raise TypeError, "not indexable" admin(24621): TypeError: not indexable admin(24621): [----- Python Information -----] -- Chuq Von Rospach, Internet Gnome <http://www.chuqui.com> [<chuqui(a)plaidworks.com> = <me(a)chuqui.com> = <chuq(a)apple.com>] Yes, yes, I've finally finished my home page. Lucky you. 95% of being a net.god is sounding persuasive and convincing people you know what you're talking about, even when you're making it up as you go along. (chuq von rospach, 1992)

1 0

help! 2.0.6 just *stopped*
by Bill Bradford 13 Sep '01

13 Sep '01

Around 5pm this afternoon, my long-standing 2.0.6 installation, working great, just *stopped* working. here's all i can find, in ~mailman/logs/error: Sep 13 21:30:02 2001 qrunner(12488): Traceback (most recent call last): Sep 13 21:30:02 2001 qrunner(12488): File "/usr/local/mailman/cron/qrunner", line 282, in ? Sep 13 21:30:02 2001 qrunner(12488): kids = main(lock) Sep 13 21:30:02 2001 qrunner(12488): File "/usr/local/mailman/cron/qrunner", line 252, in main Sep 13 21:30:02 2001 qrunner(12488): keepqueued = dispose_message(mlist, msg, msgdata) Sep 13 21:30:02 2001 qrunner(12488): File "/usr/local/mailman/cron/qrunner", line 121, in dispose_message Sep 13 21:30:02 2001 qrunner(12488): if BouncerAPI.ScanMessages(mlist, mimemsg): Sep 13 21:30:02 2001 qrunner(12488): File "/usr/local/mailman/Mailman/Bouncers/BouncerAPI.py", line 59, in ScanMessages Sep 13 21:30:02 2001 qrunner(12488): addrs = func(msg) Sep 13 21:30:02 2001 qrunner(12488): File "/usr/local/mailman/Mailman/Bouncers/DSN.py", line 46, in process Sep 13 21:30:02 2001 qrunner(12488): if string.lower(msg.gettype()) <> 'multipart/report' or \ Sep 13 21:30:02 2001 qrunner(12488): File "/usr/local/lib/python2.1/string.py", line 51, in lower Sep 13 21:30:02 2001 qrunner(12488): return s.lower() Sep 13 21:30:02 2001 qrunner(12488): AttributeError : 'None' object has no attribute 'lower' Any suggestions? I literally havent changed a *thing*... and no mail is hitting the lists.... its not even showing up in logs/post. Bill -- Bill Bradford mrbill(a)mrbill.net Austin, TX

1 1

Re: Fw: [Mailman-Developers] Problem to gate_news REPEAT
by Rodolfo Pilas 12 Sep '01

12 Sep '01

El Wed, 12 Sep 2001 18:01:41 +0200 Luca Maranzano <liuk(a)publinet.it> ha escrito: > Try to restart mailmanctl and dig into the log files, expecially > the logs/error file. The problem is that qrunner is not running. Log files do not say anything. To restart the mailmanctl was the first choice, but see this interesting problem: guru:/usr/lib/mailman # bin/mailmanctl -n stop PID unreadable in: /var/spool/mailman/data/qrunner.pid [Errno 2] No such file or directory: '/var/spool/mailman/data/qrunner.pid' Is qrunner even running? guru:/usr/lib/mailman # bin/mailmanctl -n start guru:/usr/lib/mailman # Traceback (most recent call last): File "bin/mailmanctl", line 169, in ? main() File "bin/mailmanctl", line 162, in main Control.start(restart) File "/usr/lib/mailman/Mailman/Queue/Control.py", line 209, in start master(restart, lock) File "/usr/lib/mailman/Mailman/Queue/Control.py", line 127, in master pid = start_runner(qrclass, slice, count) File "/usr/lib/mailman/Mailman/Queue/Control.py", line 111, in start_runner qrunner.run() File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 58, in run filecnt = self.__oneloop() File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 85, in __oneloop msg, msgdata = self._switchboard.dequeue(filebase) File "/usr/lib/mailman/Mailman/Queue/Switchboard.py", line 149, in dequeue msg = cPickle.load(msgfp) cPickle.UnpicklingError: unpickling stack underflow -- ____________________________________________________ -=(( Rodolfo Pilas - rodolfo(a)linux.org.uy ))=- -==((( Are you ready for XtraLinuX - http://xtralinux.org )))==- -=((____________________________________________________))=- Public GnuPG key: http://www.keyserver.net 1024D/57153363 2001-06-02 key fingerprint = DAAE 3246 3F7D A420 B7A0 48A5 D120 C773 5715 3363

1 1

Re: Fw: [Mailman-Developers] Problem to gate_news REPEAT
by Rodolfo Pilas 12 Sep '01

12 Sep '01

El Wed, 12 Sep 2001 10:49:12 +0200 Luca Maranzano <liuk(a)publinet.it> ha escrito: > Have you tryed 2.1a3 ? You must get it via CVS. Let me know if > you need help. Ok, I can try it... but do you know a way to fix this problem? > > File "/usr/lib/mailman/cron/gate_news", line 147, in process_lists > > mlist = MailList.MailList(listname, lock=0) > > File "/usr/lib/mailman/Mailman/MailList.py", line 83, in __init__ > > self.Load() > > File "/usr/lib/mailman/Mailman/MailList.py", line 999, in Load > > raise Errors.MMCorruptListDatabaseError, e > > Mailman.Errors.MMCorruptListDatabaseError: bad marshal data you can go to http://www.espaciolibre.net/cgi-bin/listinfo to see this problem. If I need to delete databases I can do it, but I need to know how to do it. -- ____________________________________________________ -=(( Rodolfo Pilas - rodolfo(a)linux.org.uy ))=- -==((( Are you ready for XtraLinuX - http://xtralinux.org )))==- -=((____________________________________________________))=- Public GnuPG key: http://www.keyserver.net 1024D/57153363 2001-06-02 key fingerprint = DAAE 3246 3F7D A420 B7A0 48A5 D120 C773 5715 3363

1 0

Mailman CVS sends out Japanese template mails in EUC-JP
by Ben Gertzfield 12 Sep '01

12 Sep '01

Right now, Mailman 2.1 and Japanese are not working together terribly well. The web pages are fully internationalized, which works great. But there's a big problem with sending template mails out; they're stored in EUC-JP, when Japanese mail *MUST* be formatted in ISO-2022-JP (JIS). Kikuchi-san has developed some Handler modules to solve this problem, but I cannot get them to work with the latest Mailman CVS. His solution is to add a ja_To_EUC_JP module at the beginning of the handler pipeline, then use a ja_SMTPDirect module to convert to ISO-2022-JP at the end of the pipeline, instead of SMTPDirect. However, with the latest Mailman CVS, even following the instructions at http://mm.tkikuchi.net/mailman-2.1alpha.ja/README.ja (both English and Japanese are in there), and doing the following, welcome and administration mails in Japanese are sent verbatim in EUC-JP, which of course is not readable in any mail reader. 1) Install kconv.py from http://hito.eccosys.com/~mak/kconv/index_jp.html (python setup.py; cp -a kconv /usr/local/lib/python2.0/site-packages/) 2) Copy ja_SMTPDirect.py, ja_To_EUC_JP.py, and ja_To_ISO_2022_JP.py into Mailman/Handlers 3) Set DEFAULT_SERVER_LANGUAGE to 'ja', add ja_To_EUC_JP.py to the beginning of the pipeline, and set DELIVERY_MODULE to ja_SMTPDirect. (I tried both via mm_cfg.py and by making them the default in Defaults.py.in, then configuring and making fresh). I'm not getting any errors in the logs, but mail is not converted to ISO-2022-JP with this method -- in fact, the Content-Type: text/plain; encoding=iso-2022-jp header that ja_To_ISO-2022-JP.py is supposed to add is not even added. It seems like even this is a really roundabout way to get Japanese to work in Mailman. Without converting the emails to ISO-2022-JP, Mailman might as well not have any Japanese support at all; it's misleading to have the web pages support Japanese and have the welcome mails translated, but then sending them out in an 8-bit encoding that may wreck people's mail servers or crash their mail clients. Please let me know if there's anything I've missed. I would really like to get Japanese support working in mainstream Mailman 2.1. Ben -- Brought to you by the letters N and H and the number 17. "I'm with insurance." Debian GNU/Linux maintainer of Gimp and GTK+ -- http://www.debian.org/

3 14

Fw: [Mailman-Developers] Problem to gate_news REPEAT
by Rodolfo Pilas 11 Sep '01

11 Sep '01

Sorry to return about the same problem of my recent message, but I am with my MM without routing messages. Please, give me a bit help to fix this problem. Thank you, thank you. thank you. (I am using MM 2.1a2) Since some minutes ago I am receiving these reports form the cron daemon: Traceback (most recent call last): File "/usr/lib/mailman/cron/gate_news", line 232, in ? main() File "/usr/lib/mailman/cron/gate_news", line 213, in main process_lists(lock) File "/usr/lib/mailman/cron/gate_news", line 147, in process_lists mlist = MailList.MailList(listname, lock=0) File "/usr/lib/mailman/Mailman/MailList.py", line 83, in __init__ self.Load() File "/usr/lib/mailman/Mailman/MailList.py", line 999, in Load raise Errors.MMCorruptListDatabaseError, e Mailman.Errors.MMCorruptListDatabaseError: bad marshal data It is ease to understand that the server do not route any list. :( Can you give me some idea to fix this problem? -- ____________________________________________________ -=(( Rodolfo Pilas - rodolfo(a)linux.org.uy ))=- -==((( Are you ready for XtraLinuX - http://xtralinux.org )))==- -=((____________________________________________________))=- Guarde sus mail en Echelon: OLP, Greenpeace, ETA, Hamas, Hizbullah, FMI Public GnuPG key: http://www.keyserver.net 1024D/57153363 2001-06-02 key fingerprint = DAAE 3246 3F7D A420 B7A0 48A5 D120 C773 5715 3363 -- ____________________________________________________ -=(( Rodolfo Pilas - rodolfo(a)linux.org.uy ))=- -==((( Are you ready for XtraLinuX - http://xtralinux.org )))==- -=((____________________________________________________))=- Guarde sus mail en Echelon: OLP, Greenpeace, ETA, Hamas, Hizbullah, FMI Public GnuPG key: http://www.keyserver.net 1024D/57153363 2001-06-02 key fingerprint = DAAE 3246 3F7D A420 B7A0 48A5 D120 C773 5715 3363 _______________________________________________ Mailman-Developers mailing list Mailman-Developers(a)python.org http://mail.python.org/mailman/listinfo/mailman-developers -- ____________________________________________________ -=(( Rodolfo Pilas - rodolfo(a)linux.org.uy ))=- -==((( Are you ready for XtraLinuX - http://xtralinux.org )))==- -=((____________________________________________________))=- Guarde sus mail en Echelon: OLP, Greenpeace, ETA, Hamas, Hizbullah, FMI Public GnuPG key: http://www.keyserver.net 1024D/57153363 2001-06-02 key fingerprint = DAAE 3246 3F7D A420 B7A0 48A5 D120 C773 5715 3363

1 0

Problem to gate_news
by Rodolfo Pilas 11 Sep '01

11 Sep '01

Since some minutes ago I am receiving these reports form the cron daemon: Traceback (most recent call last): File "/usr/lib/mailman/cron/gate_news", line 232, in ? main() File "/usr/lib/mailman/cron/gate_news", line 213, in main process_lists(lock) File "/usr/lib/mailman/cron/gate_news", line 147, in process_lists mlist = MailList.MailList(listname, lock=0) File "/usr/lib/mailman/Mailman/MailList.py", line 83, in __init__ self.Load() File "/usr/lib/mailman/Mailman/MailList.py", line 999, in Load raise Errors.MMCorruptListDatabaseError, e Mailman.Errors.MMCorruptListDatabaseError: bad marshal data It is ease to understand that the server do not route any list. :( Can you give me some idea to fix this problem? -- ____________________________________________________ -=(( Rodolfo Pilas - rodolfo(a)linux.org.uy ))=- -==((( Are you ready for XtraLinuX - http://xtralinux.org )))==- -=((____________________________________________________))=- Guarde sus mail en Echelon: OLP, Greenpeace, ETA, Hamas, Hizbullah, FMI Public GnuPG key: http://www.keyserver.net 1024D/57153363 2001-06-02 key fingerprint = DAAE 3246 3F7D A420 B7A0 48A5 D120 C773 5715 3363 -- ____________________________________________________ -=(( Rodolfo Pilas - rodolfo(a)linux.org.uy ))=- -==((( Are you ready for XtraLinuX - http://xtralinux.org )))==- -=((____________________________________________________))=- Guarde sus mail en Echelon: OLP, Greenpeace, ETA, Hamas, Hizbullah, FMI Public GnuPG key: http://www.keyserver.net 1024D/57153363 2001-06-02 key fingerprint = DAAE 3246 3F7D A420 B7A0 48A5 D120 C773 5715 3363

1 0

2 Bouncer exceptions
by Luca Maranzano 11 Sep '01

11 Sep '01

Hi, here again with an error report, from logs/error 2 distinct errors: ========= Sep 11 13:10:48 2001 (8707) Bouncer exception: 'None' object has no attribute 'lower' Sep 11 13:10:48 2001 (8707) Traceback (most recent call last): File "/home/mailman/Mailman/Bouncers/BouncerAPI.py", line 71, in ScanMessages mlist.RegisterBounce(addr, msg) File "/home/mailman/Mailman/Bouncer.py", line 93, in RegisterBounce this_dude = Utils.FindMatchingAddresses(email, self.bounce_info) File "/home/mailman/Mailman/Utils.py", line 298, in FindMatchingAddresses p_matches = GetPossibleMatchingAddrs(name) File "/home/mailman/Mailman/Utils.py", line 262, in GetPossibleMatchingAddrs name = name.lower() AttributeError: 'None' object has no attribute 'lower' ========= Grepp'ing in the mail.info log file for the exact time I found an entry saying that the command it was executing was: "|/home/mailman/mail/wrapper mailowner listname" I'm not sure this may be correlated with the error, but I've no other clue :) =================== Sep 11 13:26:07 2001 (8707) Bouncer exception: not enough arguments; expected 1, got 0 Sep 11 13:26:07 2001 (8707) Traceback (most recent call last): File "/home/mailman/Mailman/Bouncers/BouncerAPI.py", line 71, in ScanMessages mlist.RegisterBounce(addr, msg) File "/home/mailman/Mailman/Bouncer.py", line 123, in RegisterBounce self.HandleBouncingAddress(addr, msg) File "/home/mailman/Mailman/Bouncer.py", line 214, in HandleBouncingAddress text = text + '\n\n--' + boundary + \ TypeError: not enough arguments; expected 1, got 0 =================== Instead for this error I've found something in the logs/bounce (listname and email address has been modified): Sep 11 13:26:07 2001 (8707) LISTNAME: XXXXXXX(a)tiscalinet.it - 2 more allowed over 345719 secs Sep 11 13:26:07 2001 (8707) LISTNAME: YYYYYYY(a)tiscalinet.it - exceeded limits Sep 11 13:26:07 2001 (8707) LISTNAME: disabled YYYYYYY(a)tiscalinet.it I'm using latest CVS, as always let me know if you need more infos! Hoping to be useful :) --Luca

1 0

Mailman 500 error when some users access web pages.
by Marc MERLIN 10 Sep '01

10 Sep '01

So, Sourceforge has been plagued by reports of users saying that they were getting sporadic 500 errors when accessing mailman admin pages. I think I just found one reason if not _the_ reason: --------------------------------------------------------------------------- > My Sourceforge.net Cookie was > > .sourceforge.net TRUE / FALSE 1000695393 username > gQdb8WNeEUc%3D Aaaaahhhhh, I think I get it. sourceforge.net (the web site) gives you a .sourceforge.net cookie. When you hit lists.sourceforge.net, your browser sends the cookie and mailman freaks out because it can't decode the cookie (it's not a mailman cookie, it's a SF cookie) At least, that's my guess. --------------------------------------------------------------------------- Does that make sense? (apparently, for whatever weird reason, it mostly happens with mozilla) If so, is it easy to fix mailman to be more robust and ignore bad cookies? Thanks, Marc -- Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f(a)merlins.org for PGP key

2 1