Bugs item #1908112, was opened at 2008-03-05 08:02 Message generated for change (Comment added) made by msapiro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=100103&aid=1908112&group_id=103 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: 2.1 (stable) Status: Open Resolution: None Priority: 5 Private: No Submitted By: Sebastian Hagedorn (shagedorn) Assigned to: Mark Sapiro (msapiro) Summary: Uncaught runner exception: iteration over non-sequence Initial Comment: I don't normally check /var/log/mailman/error as long as everything seems to be working, but today I noticed loads of errors like this one: Mar 04 07:50:44 2008 (4711) SHUNTING: 1167220351.925714+8af35bc9b49ca21d542d7fac99bdb87a688619ef Mar 04 07:50:44 2008 (4711) Uncaught runner exception: iteration over non-sequence Mar 04 07:50:44 2008 (4711) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 112, in _oneloop self._onefile(msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 170, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/usr/lib/mailman/Mailman/Handlers/SpamDetect.py", line 111, in process g.flatten(p) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 101, in flatten self._write(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 129, in _write self._dispatch(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 155, in _dispatch meth(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 226, in _handle_multipart g.flatten(part, unixfrom=False) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 101, in flatten self._write(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 129, in _write self._dispatch(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 155, in _dispatch meth(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 293, in _handle_message g.flatten(msg.get_payload(0), unixfrom=False) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 101, in flatten self._write(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 129, in _write self._dispatch(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 155, in _dispatch meth(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 226, in _handle_multipart g.flatten(part, unixfrom=False) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 101, in flatten self._write(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 129, in _write self._dispatch(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 155, in _dispatch meth(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 273, in _handle_message_delivery_status g.flatten(part, unixfrom=False) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 101, in flatten self._write(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 129, in _write self._dispatch(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 155, in _dispatch meth(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 270, in _handle_message_delivery_status for part in msg.get_payload(): TypeError: iteration over non-sequence I can provide more samples if necessary. I have searched the tracker for other reports like this, but haven't seen any ... ----------------------------------------------------------------------

Comment By: Mark Sapiro (msapiro) Date: 2008-03-06 10:09

Message: Logged In: YES user_id=1123998 Originator: NO Thanks for the followup information. I have attached a small script which will work to obtain information from a shunted queue entry when bin/show_qfiles and bin/dumpdb throw exceptions. This script is not very robust and has only been tested on the single queue entry attached to this report, but it shold report at least some information from the queue entry. The script should be put in Mailman's bin directory and then run via bin/dump_qfile path/to/shunted/.....pck And yes, running unshunt in a daily cron is a very bad idea. At best, this will just fill up your error log with errors relating to the unshunting. I suggest that instead of doing this, you first remove from the shunt queue, any messages that can't be dumped by bin/show_qfiles or bin/dumpdb, as these can never be successfully unshunted anyway. Then you look at the rest of the messages and remove any that aren't current. Then if there are any left, you can try to unshunt those once, and if the unshunting fails (i.e. if they just get shunted again), proceed as if they are new as I describe below. What you should run as a daily cron, is a script to look at your logs and queues and mail you a report. An excellent script is at <http://sourceforge.net/tracker/index.php?func=detail&aid=1123383&group_id=103&atid=300103>. If you don't want as much information as this produces, as a minimum you could just 'ls -l /path/to/qfiles/*'. This in itself will tell you if there are shunted messages and if any queues are backing up because their runner has died. Then if you find one or more shunted messages, you have to look at the error log and the message to figure out the problem. It could be something bad about the message itself, a Mailman bug, some list issue, some transient problem, or some system configuration issue. It is generally not helpful to try to unshunt the message without first addressing the cause. If you can't figure out the cause, you can post to mailman-users@python.org <http://mail.python.org/mailman/listinfo/mailman-users> for help. File Added: dump_qfile ---------------------------------------------------------------------- Comment By: Sebastian Hagedorn (shagedorn) Date: 2008-03-06 00:31 Message: Logged In: YES user_id=309132 Originator: YES Hi, thanks for looking into this. To address your questions: - we are running 2.1.9 - we have a cronjob in cron.daily that does an "unshunt" - the file I mentioned originally doesn't exist anymore (because it has been unshunted?) - the file I attached had this error message: Mar 05 07:56:19 2008 (4711) SHUNTING: 1167220351.925714+67431f92e260e4da420214c9a343ace677ca1acd Mar 05 07:56:19 2008 (4711) Uncaught runner exception: iteration over non-sequence Mar 05 07:56:19 2008 (4711) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 112, in _oneloop self._onefile(msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 170, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) ... So the error is different from what I see with show_qfiles. - We have not modified the charset of German to UTF-8 Now my question is whether the "unshunt" cron job is evil? I confess I never properly understood how it works. It seemed to get messages delivered that were shunted originally, so at some point I set up that cronjob ... I suppose I should just remove all files that are too old. But: how do I see the contents when show_qfiles does not work? Are there other tools? Thanks for all your help! ---------------------------------------------------------------------- Comment By: Mark Sapiro (msapiro) Date: 2008-03-05 21:55 Message: Logged In: YES user_id=1123998 Originator: NO I have looked further at the shunt queue entry 1167220351.925714+67431f92e260e4da420214c9a343ace677ca1acd.pck. Here's what I see. This message was a password reminder, originally generated by mailman on May 1, 2006. It was addressed to a user whose preferred language appears to be Korean as it is 'Content-Type: text/plain, charset="euc-kr"' with 'Subject: =?euc-kr?q?uni-koeln=2Ede_mailing_list_memberships_reminder?='. Apparently there was some error at that time which caused the message to be shunted. I am a bit confused about what happened next because the time stamp on the queue entry and the received_time in the message's metadata are 1167220351.925714 which is 11:52:31 Dec 27, 2006 +0000. However, the message metadata says the message was originally in the 'in' queue, so I think it must have been unshunted to the 'in' queue where it couldn't be properly processed, causing it to be shunted again in December, 2006. The question now is what is causing these messages to be reprocessed at this point, and why would a message that threw an exception in trying to flatten the message in SpamDetect be shunted with a 14+ month old time stamp? In any case, I think the current error with this particular message is due to it's being unshunted to the wrong queue. ---------------------------------------------------------------------- Comment By: Mark Sapiro (msapiro) Date: 2008-03-05 08:58 Message: Logged In: YES user_id=1123998 Originator: NO The file you attached appears to be a password reminder with a lot of garbled unicode. Have you changed the characterset for the german language to utf-8? There is some issue with the templates/de/cronpass.txt template or a list specific version of it. ---------------------------------------------------------------------- Comment By: Mark Sapiro (msapiro) Date: 2008-03-05 08:47 Message: Logged In: YES user_id=1123998 Originator: NO The queue entry you attached is not the one from the reported traceback and it seems to have a different problem. When I attempt to look at it with bin/dumpdb or bin/show_qfiles, I get UnicodeEncodeError: 'ascii' codec can't encode character u'\uc774' in position 0: ordinal not in range(128) in a different part of Generator.py. I see that you see the same. What is your Mailman version? Also, if I look at the contents of the queue entry, it appears to be a password reminder which is strange. I don't have time to look at this further right now, but if you can, can you find the 1167220351.925714+8af35bc9b49ca21d542d7fac99bdb87a688619ef.pck and attach that. Also, is the error in Mailman's error log for the 1167220351.925714+67431f92e260e4da420214c9a343ace677ca1acd.pck file the same UnicodeEncodeError that you get from show_qfiles and dumpdb, or is it the TypeError as in the original report? ---------------------------------------------------------------------- Comment By: Sebastian Hagedorn (shagedorn) Date: 2008-03-05 08:22 Message: Logged In: YES user_id=309132 Originator: YES OK, I have attached the file. Its name is different, but I guess it's the same(?): # grep 1167220351.925714 /var/log/mailman/error Mar 02 07:37:24 2008 (4711) SHUNTING: 1167220351.925714+55bb5ab698f528a8061d4ef867a634abbed6769e Mar 03 07:36:03 2008 (4711) SHUNTING: 1167220351.925714+d29bf7b17b742fca8b0b71c09c1e52ebcf08620e Mar 04 07:50:44 2008 (4711) SHUNTING: 1167220351.925714+8af35bc9b49ca21d542d7fac99bdb87a688619ef Mar 05 07:56:19 2008 (4711) SHUNTING: 1167220351.925714+67431f92e260e4da420214c9a343ace677ca1acd Neither show_qfiles nor dbdump works: /usr/lib/mailman/bin/show_qfiles /var/spool/mailman/shunt/1167220351.925714+67431f92e260e4da420214c9a343ace677ca1acd.pck ====================> /var/spool/mailman/shunt/1167220351.925714+67431f92e260e4da420214c9a343ace677ca1acd.pck Traceback (most recent call last): File "/usr/lib/mailman/bin/show_qfiles", line 95, in ? main() File "/usr/lib/mailman/bin/show_qfiles", line 88, in main sys.stdout.write(msg.as_string()) File "/usr/lib/mailman/pythonlib/email/Message.py", line 135, in as_string g.flatten(self, unixfrom=unixfrom) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 101, in flatten self._write(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 129, in _write self._dispatch(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 155, in _dispatch meth(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 198, in _handle_text self._fp.write(payload) UnicodeError: ASCII encoding error: ordinal not in range(128) [root@lvr13 Mailman]# /usr/lib/mailman/bin/dumpdb /var/spool/mailman/shunt/1167220351.925714+67431f92e260e4da420214c9a343ace677ca1acd.pck [----- start pickle file -----] <----- start object 1 -----> Traceback (most recent call last): File "/usr/lib/mailman/bin/dumpdb", line 159, in ? msg = main() File "/usr/lib/mailman/bin/dumpdb", line 149, in main pp.pprint(obj) File "/usr/lib/python2.2/pprint.py", line 110, in pprint self.__stream.write(self.pformat(object) + "\n") File "/usr/lib/python2.2/pprint.py", line 114, in pformat self.__format(object, sio, 0, 0, {}, 0) File "/usr/lib/python2.2/pprint.py", line 136, in __format rep = self.__repr(object, context, level - 1) File "/usr/lib/python2.2/pprint.py", line 200, in __repr self.__depth, level) File "/usr/lib/python2.2/pprint.py", line 287, in _safe_repr rep = `object` File "/usr/lib/mailman/Mailman/Message.py", line 51, in __repr__ return self.__str__() File "/usr/lib/mailman/pythonlib/email/Message.py", line 121, in __str__ return self.as_string(unixfrom=True) File "/usr/lib/mailman/pythonlib/email/Message.py", line 135, in as_string g.flatten(self, unixfrom=unixfrom) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 101, in flatten self._write(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 129, in _write self._dispatch(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 155, in _dispatch meth(msg) File "/usr/lib/mailman/pythonlib/email/Generator.py", line 198, in _handle_text self._fp.write(payload) UnicodeError: ASCII encoding error: ordinal not in range(128) ---------------------------------------------------------------------- Comment By: Sebastian Hagedorn (shagedorn) Date: 2008-03-05 08:19 Message: Logged In: YES user_id=309132 Originator: YES File Added: 1167220351.925714+67431f92e260e4da420214c9a343ace677ca1acd.pck ---------------------------------------------------------------------- Comment By: Mark Sapiro (msapiro) Date: 2008-03-05 08:14 Message: Logged In: YES user_id=1123998 Originator: NO There is a problem with the message. Try running bin/show_qfiles or if that doesn't work, bin/dumpdb on one or more of the shunted files (e.g. qfiles/shunt/1167220351.925714+8af35bc9b49ca21d542d7fac99bdb87a688619ef.pck). In any case, can you send me this file or attach it to this report. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=100103&aid=1908112&group_id=103