[Bug 1130957] [NEW] Unicode errors in mailman3
Public bug reported: This is with mailman3. I'm getting the following unicode errors: Feb 20 23:07:43 2013 (13496) Uncaught runner exception: Expected unicode, found <type 'str'>: 'foobar@example.com' Feb 20 23:07:43 2013 (13496) Traceback (most recent call last): File "/srv/dev/mailman/mailman/src/mailman/core/runner.py", line 145, in _one_iteration self._process_one_file(msg, msgdata) File "/srv/dev/mailman/mailman/src/mailman/core/runner.py", line 228, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/srv/dev/mailman/mailman/src/mailman/runners/outgoing.py", line 93, in _dispose self._func(mlist, msg, msgdata) File "/srv/dev/mailman/mailman/src/mailman/mta/deliver.py", line 93, in deliver refused = agent.deliver(mlist, msg, msgdata) File "/srv/dev/mailman/mailman/src/mailman/mta/base.py", line 167, in deliver member = mlist.members.get_member(recipient) File "/srv/dev/mailman/mailman/src/mailman/database/transaction.py", line 78, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/srv/dev/mailman/mailman/src/mailman/model/roster.py", line 111, in get_member Address.email == address, File "/srv/dev/mailman/venv/lib/python2.7/site-packages/storm-0.19-py2.7-linux-x86_64.egg/storm/expr.py", line 412, in __eq__ other = getattr(self, "variable_factory", Variable)(value=other) File "/srv/dev/mailman/venv/lib/python2.7/site-packages/storm-0.19-py2.7-linux-x86_64.egg/storm/variables.py", line 396, in parse_set % (type(value), value)) TypeError: Expected unicode, found <type 'str'>: 'foobar@example.com' Feb 20 23:07:43 2013 (13496) SHUNTING: 1361401663.441109+267302c50d71d512974fe5caa7aa0214eab699f6 ----------------- Feb 20 22:20:45 2013 (13337) Uncaught runner exception: decoding Unicode is not supported Feb 20 22:20:45 2013 (13337) Traceback (most recent call last): File "/srv/dev/mailman/mailman/src/mailman/core/runner.py", line 145, in _one_iteration self._process_one_file(msg, msgdata) File "/srv/dev/mailman/mailman/src/mailman/core/runner.py", line 228, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/srv/dev/mailman/mailman/src/mailman/runners/digest.py", line 344, in _dispose rfc1153_digest.add_message(message, count) File "/srv/dev/mailman/mailman/src/mailman/runners/digest.py", line 269, in add_message payload = unicode(payload, 'us-ascii', 'replace') TypeError: decoding Unicode is not supported Feb 20 22:20:45 2013 (13337) SHUNTING: 1361398845.847068+d12dc2ef44dcca60ade2fc94f6f830bd09a6b912 Feb 20 22:20:52 2013 (13337) Uncaught runner exception: decoding Unicode is not supported Feb 20 22:20:52 2013 (13337) Traceback (most recent call last): File "/srv/dev/mailman/mailman/src/mailman/core/runner.py", line 145, in _one_iteration self._process_one_file(msg, msgdata) File "/srv/dev/mailman/mailman/src/mailman/core/runner.py", line 228, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/srv/dev/mailman/mailman/src/mailman/runners/digest.py", line 344, in _dispose rfc1153_digest.add_message(message, count) File "/srv/dev/mailman/mailman/src/mailman/runners/digest.py", line 269, in add_message payload = unicode(payload, 'us-ascii', 'replace') TypeError: decoding Unicode is not supported Feb 20 22:20:52 2013 (13337) SHUNTING: 1361398852.901291+bc2f1ab7dad828020238e06815fd2b4adab0fe61 The attached patch fixes both tracebacks. ** Affects: mailman Importance: Undecided Status: New ** Tags: mailman3 ** Patch added: "mailman-unicode.patch" https://bugs.launchpad.net/bugs/1130957/+attachment/3539184/+files/mailman-u... -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1130957 Title: Unicode errors in mailman3 To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1130957/+subscriptions
Can you provide a reproducible recipe, or test case? Is this caused by a member with non-ASCII in their name? Or a message with non-ASCII in the From field? Any more information about how to reproduce this will really help (e.g. to create a test case). -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1130957 Title: Unicode errors in mailman3 To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1130957/+subscriptions
I'm trying to write a testcase for the second set of tracebacks, but I'm having a hard time understanding where the message that causes it comes from. When I open the pickle file I get: 'From nobody Sat Mar 23 09:42:34 2013\n\n' This goes through the digest runner and tracebacks at line 269 with "TypeError: decoding Unicode is not supported" Do you know where such a message can come from ? Are you OK with just writing a test case with this string as an entry message ? -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1130957 Title: Unicode errors in mailman3 To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1130957/+subscriptions
I'm still trying to figure out the unicode errors in the digest runner, but something happens that I don't understand. In the add_message method, around line 264, the msg.as_string() call returns an str instance, but the msg.as_string().split("\n\n") call returns a list of unicode instances. This makes no sense to me. Could it come from the unicode-related overrides in mailman.email.message ? Could it be a bug in the python interpreter ? I'm using python 2.7.3. I can't find a way to reproduce it outside mailman. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1130957 Title: Unicode errors in mailman3 To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1130957/+subscriptions
OK, thanks to Toshio, I *finally* know what's going on here. Because of the import of "unicode_literals" from __future__ at the top of the file, the call to .split("\n\n") is actually interpreted as .split(u"\n\n"), thus the result is a list of unicode strings, which of course can't be decoded further in the following lines. To keep string literals, the string should be prefixed with "b", here's the patch: === modified file 'src/mailman/runners/digest.py' --- src/mailman/runners/digest.py 2013-01-01 14:05:42 +0000 +++ src/mailman/runners/digest.py 2013-04-11 15:52:53 +0000 @@ -260,7 +260,7 @@ # Add the payload. If the decoded payload is empty, this may be a # multipart message. In that case, just stringify it. payload = msg.get_payload(decode=True) - payload = (payload if payload else msg.as_string().split('\n\n', 1)[1]) + payload = (payload if payload else msg.as_string().split(b'\n\n', 1)[1]) try: charset = msg.get_content_charset('us-ascii') payload = unicode(payload, charset, 'replace') That's one of the errors I reported in this but, I'll investigate the other one now. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1130957 Title: Unicode errors in mailman3 To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1130957/+subscriptions
OK, I found out where the second error comes from. When there is no administrator available, the message is sent to the site_owner from the configuration file, but this entry is not converted to unicode before being set in the msgdata dict. Thus, when it's added to the database, Storm refuses it. The following patch fixes this bug: === modified file 'src/mailman/handlers/owner_recipients.py' --- src/mailman/handlers/owner_recipients.py 2013-01-01 14:05:42 +0000 +++ src/mailman/handlers/owner_recipients.py 2013-04-12 16:47:29 +0000 @@ -55,7 +55,7 @@ # To prevent -owner messages from going into a black hole, if there # are no administrators available, the message goes to the site owner. if len(recipients) == 0: - msgdata['recipients'] = set((config.mailman.site_owner,)) + msgdata['recipients'] = set((unicode(config.mailman.site_owner),)) else: msgdata['recipients'] = recipients # Don't decorate these messages with the header/footers. Eventually -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1130957 Title: Unicode errors in mailman3 To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1130957/+subscriptions
I'm using mailman 3.0.0b4 (with mailman-bundler) and ran into the same problem as described above. Luckily I found this bug report and could manage to fix it with the patches above. May I ask anybody (with more bazar-knowledge than me) to merge this two patches into the mailman 3 branch? Thank you! -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1130957 Title: Unicode errors in mailman3 To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1130957/+subscriptions
On Apr 30, 2014, at 09:38 PM, Tim Marx wrote:
I'm using mailman 3.0.0b4 (with mailman-bundler) and ran into the same problem as described above. Luckily I found this bug report and could manage to fix it with the patches above.
May I ask anybody (with more bazar-knowledge than me) to merge this two patches into the mailman 3 branch? Thank you!
I think we just need some test cases. Tim, can you provide any details on how you it the bug?
Am 30.04.2014 23:53, schrieb Barry Warsaw:
On Apr 30, 2014, at 09:38 PM, Tim Marx wrote:
I'm using mailman 3.0.0b4 (with mailman-bundler) and ran into the same problem as described above. Luckily I found this bug report and could manage to fix it with the patches above.
May I ask anybody (with more bazar-knowledge than me) to merge this two patches into the mailman 3 branch? Thank you! I think we just need some test cases. Tim, can you provide any details on how you it the bug? I had exactly the same stacktraces and error messages as Aurélien described in his bug report. They appeared after I had sent mails from list-member email adresses to my mailing list server. I have used 'normal' email addresses without any special characters. I'm sorry if I could not really help you with this!
Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/tim%40xrammit.de
Security Policy: http://wiki.list.org/x/QIA9
OK, I've written the unit tests and pushed them with the fixes in my dedicated branch: lp:~abompard/mailman/bug-1130957. The merge proposal has just been sent. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1130957 Title: Unicode errors in mailman3 To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1130957/+subscriptions
** Branch linked: lp:~abompard/mailman/bug-1130957 -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1130957 Title: Unicode errors in mailman3 To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1130957/+subscriptions
** Changed in: mailman Milestone: None => 3.0.0b5 ** Changed in: mailman Assignee: (unassigned) => Barry Warsaw (barry) ** Changed in: mailman Importance: Undecided => High ** Changed in: mailman Status: New => Fix Committed -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1130957 Title: Unicode errors in mailman3 To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1130957/+subscriptions
** Changed in: mailman Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1130957 Title: Unicode errors in mailman3 To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1130957/+subscriptions
participants (5)
-
Aurélien Bompard -
Barry Warsaw -
Barry Warsaw -
Tim Marx -
Tim Marx