[Mailman-Developers] Try 2: URGENT BREAKAGE: New unicode-related error with CVS
Sent this last night; haven't seen it on the list yet
Updated to CVS today, and now I'm seeing these; anyone else? This seems to be stopping all traffic to the list
Feb 05 20:17:15 2003 (4633) Uncaught runner exception: ASCII decoding error: ordinal not in range(128) Feb 05 20:17:15 2003 (4633) Traceback (most recent call last): File "/export/home/mailman/Mailman/Queue/Runner.py", line 105, in _oneloop self._onefile(msg, msgdata) File "/export/home/mailman/Mailman/Queue/Runner.py", line 155, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/export/home/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/export/home/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 79, in process send_digests(mlist, mboxfp) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 120, in send_digests send_i18n_digests(mlist, mboxfp) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 213, in send_i18n_digests addresses = getaddresses([lheader(msg.get('from', ''), lcset)]) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 372, in lheader h = make_header(decode_header(s), **kws) File "/export/home/mailman/pythonlib/email/Header.py", line 123, in make_header h.append(s, charset) File "/export/home/mailman/pythonlib/email/Header.py", line 230, in append ustr = unicode(s, incodec) UnicodeError: ASCII decoding error: ordinal not in range(128)
Feb 05 20:17:15 2003 (4633) SHUNTING: 1044504218.645468+31ecfd687a04619284497495da50c740cbddf364
One of my users has 8bit characters in his "username" string (the 'comment' part of his From address); that's causing this problem. The Unicode stuff doesn't like that.
This hacks around the problem for now (just discarding those characters when manipulating the address). It's not a fix, but it let mail flow for me until a better one comes along.
*** Header.py Mon Oct 14 09:50:57 2002 --- /export/home/mailman/pythonlib/email/Header.py Thu Feb 6 12:15:55 2003
*** 183,189 **** def __unicode__(self): """Helper for the built-in unicode function.""" # charset item is a Charset instance so we need to stringify it. ! uchunks = [unicode(s, str(charset)) for s, charset in self._chunks] return u''.join(uchunks)
# Rich comparison operators for equality only. BAW: does it make sense to--- 183,189 ---- def __unicode__(self): """Helper for the built-in unicode function.""" # charset item is a Charset instance so we need to stringify it. ! uchunks = [unicode(s, str(charset), 'ignore') for s, charset in self._chunks] return u''.join(uchunks)
# Rich comparison operators for equality only. BAW: does it make sense to*** 227,233 **** # Possibly raise UnicodeError if the byte string can't be # converted to a unicode with the input codec of the charset. incodec = charset.input_codec or 'us-ascii' ! ustr = unicode(s, incodec) # Now make sure that the unicode could be converted back to a # byte string with the output codec, which may be different # than the iput coded. Still, use the original byte string. --- 227,233 ---- # Possibly raise UnicodeError if the byte string can't be # converted to a unicode with the input codec of the charset. incodec = charset.input_codec or 'us-ascii' ! ustr = unicode(s, incodec, 'ignore') # Now make sure that the unicode could be converted back to a # byte string with the output codec, which may be different # than the iput coded. Still, use the original byte string.
Dan Mick wrote:
Sent this last night; haven't seen it on the list yet
Updated to CVS today, and now I'm seeing these; anyone else? This seems to be stopping all traffic to the list
Feb 05 20:17:15 2003 (4633) Uncaught runner exception: ASCII decoding error: ordinal not in range(128) Feb 05 20:17:15 2003 (4633) Traceback (most recent call last): File "/export/home/mailman/Mailman/Queue/Runner.py", line 105, in _oneloop self._onefile(msg, msgdata) File "/export/home/mailman/Mailman/Queue/Runner.py", line 155, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/export/home/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/export/home/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 79, in process send_digests(mlist, mboxfp) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 120, in send_digests send_i18n_digests(mlist, mboxfp) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 213, in send_i18n_digests addresses = getaddresses([lheader(msg.get('from', ''), lcset)]) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 372, in lheader h = make_header(decode_header(s), **kws) File "/export/home/mailman/pythonlib/email/Header.py", line 123, in make_header h.append(s, charset) File "/export/home/mailman/pythonlib/email/Header.py", line 230, in append ustr = unicode(s, incodec) UnicodeError: ASCII decoding error: ordinal not in range(128)
Feb 05 20:17:15 2003 (4633) SHUNTING: 1044504218.645468+31ecfd687a04619284497495da50c740cbddf364
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
Dan,
It is not a good idea to hack email package. The error raised should be treated in higher coding level. Please try patch #674401, most recent one (ToDigest.py.diff2.txt). https://sourceforge.net/tracker/download.php?group_id=103&atid=300103&file_id=41727&aid=674401
Dan Mick wrote:
One of my users has 8bit characters in his "username" string (the 'comment' part of his From address); that's causing this problem. The Unicode stuff doesn't like that.
This hacks around the problem for now (just discarding those characters when manipulating the address). It's not a fix, but it let mail flow for me until a better one comes along.
*** Header.py Mon Oct 14 09:50:57 2002 --- /export/home/mailman/pythonlib/email/Header.py Thu Feb 6 12:15:55 2003
*** 183,189 **** def __unicode__(self): """Helper for the built-in unicode function.""" # charset item is a Charset instance so we need to stringify it. ! uchunks = [unicode(s, str(charset)) for s, charset in self._chunks] return u''.join(uchunks)
# Rich comparison operators for equality only. BAW: does it makesense to --- 183,189 ---- def __unicode__(self): """Helper for the built-in unicode function.""" # charset item is a Charset instance so we need to stringify it. ! uchunks = [unicode(s, str(charset), 'ignore') for s, charset in self._chunks] return u''.join(uchunks)
# Rich comparison operators for equality only. BAW: does it makesense to
*** 227,233 **** # Possibly raise UnicodeError if the byte string can't be # converted to a unicode with the input codec of the charset. incodec = charset.input_codec or 'us-ascii' ! ustr = unicode(s, incodec) # Now make sure that the unicode could be converted back to a # byte string with the output codec, which may be different # than the iput coded. Still, use the original byte string. --- 227,233 ---- # Possibly raise UnicodeError if the byte string can't be # converted to a unicode with the input codec of the charset. incodec = charset.input_codec or 'us-ascii' ! ustr = unicode(s, incodec, 'ignore') # Now make sure that the unicode could be converted back to a # byte string with the output codec, which may be different # than the iput coded. Still, use the original byte string.
Dan Mick wrote:
Sent this last night; haven't seen it on the list yet
Updated to CVS today, and now I'm seeing these; anyone else? This seems to be stopping all traffic to the list
Feb 05 20:17:15 2003 (4633) Uncaught runner exception: ASCII decoding error: ordinal not in range(128) Feb 05 20:17:15 2003 (4633) Traceback (most recent call last): File "/export/home/mailman/Mailman/Queue/Runner.py", line 105, in _oneloop self._onefile(msg, msgdata) File "/export/home/mailman/Mailman/Queue/Runner.py", line 155, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/export/home/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/export/home/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 79, in process send_digests(mlist, mboxfp) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 120, in send_digests send_i18n_digests(mlist, mboxfp) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 213, in send_i18n_digests addresses = getaddresses([lheader(msg.get('from', ''), lcset)]) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 372, in lheader h = make_header(decode_header(s), **kws) File "/export/home/mailman/pythonlib/email/Header.py", line 123, in make_header h.append(s, charset) File "/export/home/mailman/pythonlib/email/Header.py", line 230, in append ustr = unicode(s, incodec) UnicodeError: ASCII decoding error: ordinal not in range(128)
Feb 05 20:17:15 2003 (4633) SHUNTING: 1044504218.645468+31ecfd687a04619284497495da50c740cbddf364
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
-- Tokio Kikuchi tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
Tokio Kikuchi wrote:
Dan,
It is not a good idea to hack email package. The error raised should be treated in higher coding level.
heh. Desperate men have fewer options open to them. ;)
Please try patch #674401, most recent one (ToDigest.py.diff2.txt). https://sourceforge.net/tracker/download.php?group_id=103&atid=300103&file_id=41727&aid=674401
I'll have a look. Thanks.
Dan Mick wrote:
One of my users has 8bit characters in his "username" string (the 'comment' part of his From address); that's causing this problem. The Unicode stuff doesn't like that.
This hacks around the problem for now (just discarding those characters when manipulating the address). It's not a fix, but it let mail flow for me until a better one comes along.
*** Header.py Mon Oct 14 09:50:57 2002 --- /export/home/mailman/pythonlib/email/Header.py Thu Feb 6 12:15:55 2003
*** 183,189 **** def __unicode__(self): """Helper for the built-in unicode function.""" # charset item is a Charset instance so we need to stringify it. ! uchunks = [unicode(s, str(charset)) for s, charset in self._chunks] return u''.join(uchunks)
# Rich comparison operators for equality only. BAW: does itmake sense to --- 183,189 ---- def __unicode__(self): """Helper for the built-in unicode function.""" # charset item is a Charset instance so we need to stringify it. ! uchunks = [unicode(s, str(charset), 'ignore') for s, charset in self._chunks] return u''.join(uchunks)
# Rich comparison operators for equality only. BAW: does itmake sense to
*** 227,233 **** # Possibly raise UnicodeError if the byte string can't be # converted to a unicode with the input codec of the charset. incodec = charset.input_codec or 'us-ascii' ! ustr = unicode(s, incodec) # Now make sure that the unicode could be converted back to a # byte string with the output codec, which may be different # than the iput coded. Still, use the original byte string. --- 227,233 ---- # Possibly raise UnicodeError if the byte string can't be # converted to a unicode with the input codec of the charset. incodec = charset.input_codec or 'us-ascii' ! ustr = unicode(s, incodec, 'ignore') # Now make sure that the unicode could be converted back to a # byte string with the output codec, which may be different # than the iput coded. Still, use the original byte string.
Dan Mick wrote:
Sent this last night; haven't seen it on the list yet
Updated to CVS today, and now I'm seeing these; anyone else? This seems to be stopping all traffic to the list
Feb 05 20:17:15 2003 (4633) Uncaught runner exception: ASCII decoding error: ordinal not in range(128) Feb 05 20:17:15 2003 (4633) Traceback (most recent call last): File "/export/home/mailman/Mailman/Queue/Runner.py", line 105, in _oneloop self._onefile(msg, msgdata) File "/export/home/mailman/Mailman/Queue/Runner.py", line 155, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/export/home/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/export/home/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 79, in process send_digests(mlist, mboxfp) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 120, in send_digests send_i18n_digests(mlist, mboxfp) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 213, in send_i18n_digests addresses = getaddresses([lheader(msg.get('from', ''), lcset)]) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 372, in lheader h = make_header(decode_header(s), **kws) File "/export/home/mailman/pythonlib/email/Header.py", line 123, in make_header h.append(s, charset) File "/export/home/mailman/pythonlib/email/Header.py", line 230, in append ustr = unicode(s, incodec) UnicodeError: ASCII decoding error: ordinal not in range(128)
Feb 05 20:17:15 2003 (4633) SHUNTING: 1044504218.645468+31ecfd687a04619284497495da50c740cbddf364
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
Tokio:
I haven't tried it yet, but I guess I don't understand how this is going to help. The problem is that unicode() inside Header.__unicode__() fails, because the input string has 8-bit characters in it; your patch still calls __unicode__() outside a try/except.
Tokio Kikuchi wrote:
Dan,
It is not a good idea to hack email package. The error raised should be treated in higher coding level. Please try patch #674401, most recent one (ToDigest.py.diff2.txt). https://sourceforge.net/tracker/download.php?group_id=103&atid=300103&file_id=41727&aid=674401
Dan Mick wrote:
One of my users has 8bit characters in his "username" string (the 'comment' part of his From address); that's causing this problem. The Unicode stuff doesn't like that.
This hacks around the problem for now (just discarding those characters when manipulating the address). It's not a fix, but it let mail flow for me until a better one comes along.
*** Header.py Mon Oct 14 09:50:57 2002 --- /export/home/mailman/pythonlib/email/Header.py Thu Feb 6 12:15:55 2003
*** 183,189 **** def __unicode__(self): """Helper for the built-in unicode function.""" # charset item is a Charset instance so we need to stringify it. ! uchunks = [unicode(s, str(charset)) for s, charset in self._chunks] return u''.join(uchunks)
# Rich comparison operators for equality only. BAW: does itmake sense to --- 183,189 ---- def __unicode__(self): """Helper for the built-in unicode function.""" # charset item is a Charset instance so we need to stringify it. ! uchunks = [unicode(s, str(charset), 'ignore') for s, charset in self._chunks] return u''.join(uchunks)
# Rich comparison operators for equality only. BAW: does itmake sense to
*** 227,233 **** # Possibly raise UnicodeError if the byte string can't be # converted to a unicode with the input codec of the charset. incodec = charset.input_codec or 'us-ascii' ! ustr = unicode(s, incodec) # Now make sure that the unicode could be converted back to a # byte string with the output codec, which may be different # than the iput coded. Still, use the original byte string. --- 227,233 ---- # Possibly raise UnicodeError if the byte string can't be # converted to a unicode with the input codec of the charset. incodec = charset.input_codec or 'us-ascii' ! ustr = unicode(s, incodec, 'ignore') # Now make sure that the unicode could be converted back to a # byte string with the output codec, which may be different # than the iput coded. Still, use the original byte string.
Dan Mick wrote:
Sent this last night; haven't seen it on the list yet
Updated to CVS today, and now I'm seeing these; anyone else? This seems to be stopping all traffic to the list
Feb 05 20:17:15 2003 (4633) Uncaught runner exception: ASCII decoding error: ordinal not in range(128) Feb 05 20:17:15 2003 (4633) Traceback (most recent call last): File "/export/home/mailman/Mailman/Queue/Runner.py", line 105, in _oneloop self._onefile(msg, msgdata) File "/export/home/mailman/Mailman/Queue/Runner.py", line 155, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/export/home/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/export/home/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 79, in process send_digests(mlist, mboxfp) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 120, in send_digests send_i18n_digests(mlist, mboxfp) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 213, in send_i18n_digests addresses = getaddresses([lheader(msg.get('from', ''), lcset)]) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 372, in lheader h = make_header(decode_header(s), **kws) File "/export/home/mailman/pythonlib/email/Header.py", line 123, in make_header h.append(s, charset) File "/export/home/mailman/pythonlib/email/Header.py", line 230, in append ustr = unicode(s, incodec) UnicodeError: ASCII decoding error: ordinal not in range(128)
Feb 05 20:17:15 2003 (4633) SHUNTING: 1044504218.645468+31ecfd687a04619284497495da50c740cbddf364
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
Sorry, Dan.
I have just updated the patch. I uploaded an intermediate version. https://sourceforge.net/tracker/download.php?group_id=103&atid=300103&file_id=41736&aid=674401
Dan Mick wrote:
Tokio:
I haven't tried it yet, but I guess I don't understand how this is going to help. The problem is that unicode() inside Header.__unicode__() fails, because the input string has 8-bit characters in it; your patch still calls __unicode__() outside a try/except.
Tokio Kikuchi wrote:
Dan,
It is not a good idea to hack email package. The error raised should be treated in higher coding level. Please try patch #674401, most recent one (ToDigest.py.diff2.txt). https://sourceforge.net/tracker/download.php?group_id=103&atid=300103&file_id=41727&aid=674401
Dan Mick wrote:
One of my users has 8bit characters in his "username" string (the 'comment' part of his From address); that's causing this problem. The Unicode stuff doesn't like that.
This hacks around the problem for now (just discarding those characters when manipulating the address). It's not a fix, but it let mail flow for me until a better one comes along.
*** Header.py Mon Oct 14 09:50:57 2002 --- /export/home/mailman/pythonlib/email/Header.py Thu Feb 6 12:15:55 2003
*** 183,189 **** def __unicode__(self): """Helper for the built-in unicode function.""" # charset item is a Charset instance so we need to stringify it. ! uchunks = [unicode(s, str(charset)) for s, charset in self._chunks] return u''.join(uchunks)
# Rich comparison operators for equality only. BAW: does itmake sense to --- 183,189 ---- def __unicode__(self): """Helper for the built-in unicode function.""" # charset item is a Charset instance so we need to stringify it. ! uchunks = [unicode(s, str(charset), 'ignore') for s, charset in self._chunks] return u''.join(uchunks)
# Rich comparison operators for equality only. BAW: does itmake sense to
*** 227,233 **** # Possibly raise UnicodeError if the byte string can't be # converted to a unicode with the input codec of the charset. incodec = charset.input_codec or 'us-ascii' ! ustr = unicode(s, incodec) # Now make sure that the unicode could be converted back to a # byte string with the output codec, which may be different # than the iput coded. Still, use the original byte string. --- 227,233 ---- # Possibly raise UnicodeError if the byte string can't be # converted to a unicode with the input codec of the charset. incodec = charset.input_codec or 'us-ascii' ! ustr = unicode(s, incodec, 'ignore') # Now make sure that the unicode could be converted back to a # byte string with the output codec, which may be different # than the iput coded. Still, use the original byte string.
Dan Mick wrote:
Sent this last night; haven't seen it on the list yet
Updated to CVS today, and now I'm seeing these; anyone else? This seems to be stopping all traffic to the list
Feb 05 20:17:15 2003 (4633) Uncaught runner exception: ASCII decoding error: ordinal not in range(128) Feb 05 20:17:15 2003 (4633) Traceback (most recent call last): File "/export/home/mailman/Mailman/Queue/Runner.py", line 105, in _oneloop self._onefile(msg, msgdata) File "/export/home/mailman/Mailman/Queue/Runner.py", line 155, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/export/home/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/export/home/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 79, in process send_digests(mlist, mboxfp) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 120, in send_digests send_i18n_digests(mlist, mboxfp) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 213, in send_i18n_digests addresses = getaddresses([lheader(msg.get('from', ''), lcset)]) File "/export/home/mailman/Mailman/Handlers/ToDigest.py", line 372, in lheader h = make_header(decode_header(s), **kws) File "/export/home/mailman/pythonlib/email/Header.py", line 123, in make_header h.append(s, charset) File "/export/home/mailman/pythonlib/email/Header.py", line 230, in append ustr = unicode(s, incodec) UnicodeError: ASCII decoding error: ordinal not in range(128)
Feb 05 20:17:15 2003 (4633) SHUNTING: 1044504218.645468+31ecfd687a04619284497495da50c740cbddf364
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
TK> I have just updated the patch.Applied! Thanks, it seems to have fixed the problems we were having on python.org.
-Barry
participants (3)
-
barry＠python.org -
Dan Mick -
Tokio Kikuchi