
Hi, I encountered a subtil problem when a component logs unicode strings (present case was the hyperkitty archiver) : (py27)mailman@mail:~$ Traceback (most recent call last): File "/home/mailman/mailman/src/mailman/core/logging.py", line 76, in emit msg = self.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 723, in format return fmt.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 464, in format record.message = record.getMessage() File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage msg = msg % self.args UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 21: ordinal not in range(128) Logged from file runner.py, line 261 Traceback (most recent call last): File "/home/mailman/mailman/src/mailman/core/logging.py", line 76, in emit msg = self.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 723, in format return fmt.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 464, in format record.message = record.getMessage() File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage msg = msg % self.args UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 2982: ordinal not in range(128) Logged from file runner.py, line 264 then found how to fix it : === modified file 'src/mailman/core/runner.py' --- src/mailman/core/runner.py 2013-12-01 13:28:49 +0000 +++ src/mailman/core/runner.py 2013-12-03 10:47:56 +0000 @@ -258,10 +258,10 @@ self.switchboard.enqueue(msg, msgdata) def _log(self, exc): - elog.error('Uncaught runner exception: %s', exc) + elog.error(b'Uncaught runner exception: %s', exc) s = StringIO() traceback.print_exc(file=s) - elog.error('%s', s.getvalue()) + elog.error(b'%s', s.getvalue()) def _clean_up(self): """See `IRunner`.""" unless it can be considered as a problem in the component itself... But (personal thought) mailman's stability may not depend of third party components... I can propose a merge for it if it's revelant (just have to go deeper into lp/bzr) Cheers Nico

Basicaly it was not hyperkitty archiver, but mailman itself (or in fact Storm localized error message strings), cause here is the output : Dec 03 12:37:49 2013 (23760) ACCEPT: <CAMYTGwj+v6q+Utc2=wK_7nLOeOKc+R5CtB7pyC95fdDMQ0LNgw@mail.gmail.com> Dec 03 12:37:50 2013 (23762) conf loaded Dec 03 12:37:50 2013 (23762) Uncaught runner exception: ERREUR: la relation \xc2\xab listarchiver \xc2\xbb n\'existe pas\nLINE 1: ...tarchiver.mailing_list_id, listarchiver.name FROM listarchiv...\n ^\nDec 03 12:37:50 2013 (23762) Traceback (most recent call last):\n File "/home/mailman/mailman/src/mailman/core/runner.py", line 169, in _one_iteration\n self._process_one_file(msg, msgdata)\n File "/home/mailman/mailman/src/mailman/core/runner.py", line 252, in _process_one_file\n keepqueued = self._dispose(mlist, msg, msgdata)\n File "/home/mailman/mailman/src/mailman/runners/pipeline.py", line 36, in _dispose\n process(mlist, msg, msgdata, pipeline)\n File "/home/mailman/mailman/src/mailman/core/pipelines.py", line 66, in process\n handler.process(mlist, msg, msgdata)\n File "/home/mailman/mailman/src/mailman/handlers/rfc_2369.py", line 120, in process\n process(mlist, msg, msgdata)\n File "/home/mailman/mailman/src/mailman/handlers/rfc_2369.py", line 87, in process\n archiver_set = IListArchiverSet(mlist)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/zope.component-4.1.0-py2.7.egg/zope/component/hookable.py", line 33, in __call__\n return self.__implementation(*args, **kw)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/zope.component-4.1.0-py2.7.egg/zope/component/_api.py", line 156, in adapter_hook\n return sitemanager.queryAdapter(object, interface, name, default)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/zope.interface-4.0.5-py2.7-linux-x86_64.egg/zope/interface/registry.py", line 229, in queryAdapter\n return self.adapters.queryAdapter(object, interface, name, default)\n File "/home/mailman/mailman/src/mailman/model/mailinglist.py", line 590, in __init__\n ListArchiver.name == archiver_name)).one()\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/store.py", line 1158, in one\n result = self._store._connection.execute(select)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/databases/postgres.py", line 266, in execute\n return Connection.execute(self, statement, params, noresult)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/database.py", line 241, in execute\n raw_cursor = self.raw_execute(statement, params)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/databases/postgres.py", line 276, in raw_execute\n return Connection.raw_execute(self, statement, params)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/database.py", line 374, in raw_execute\n self._run_execution(raw_cursor, args, params, statement)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/database.py", line 388, in _run_execution\n self._check_disconnect(raw_cursor.execute, *args)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/database.py", line 454, in _check_disconnect\n return function(*args, **kwargs)\nProgrammingError: ERREUR: la relation \xc2\xab listarchiver \xc2\xbb n\'existe pas\nLINE 1: ...tarchiver.mailing_list_id, listarchiver.name FROM listarchiv...\n ^\n\nDec 03 12:37:50 2013 (23762) SHUNTING: 1386070670.221361+6120e5d3eaedee1815abe8fc6bf7dd306961f1dd and the real problem origin is missing schema upgrades for postgres Cheers Le 2013-12-03 12:09, nicolas a écrit :

So, if I understand correctly, Storm add to its exception the error message it gets from PostgreSQL, which is a bytestream (encoded in UTF-8 on your system). But Mailman sends its logs to the logger as unicode, thus the exception must be converted to unicode, and that's where the UnicodeDecodeError is raised. What can we do? Should we try to decode the Storm exception using the default system encoding before sending it to the logger?
Aurélien

Basicaly it was not hyperkitty archiver, but mailman itself (or in fact Storm localized error message strings), cause here is the output : Dec 03 12:37:49 2013 (23760) ACCEPT: <CAMYTGwj+v6q+Utc2=wK_7nLOeOKc+R5CtB7pyC95fdDMQ0LNgw@mail.gmail.com> Dec 03 12:37:50 2013 (23762) conf loaded Dec 03 12:37:50 2013 (23762) Uncaught runner exception: ERREUR: la relation \xc2\xab listarchiver \xc2\xbb n\'existe pas\nLINE 1: ...tarchiver.mailing_list_id, listarchiver.name FROM listarchiv...\n ^\nDec 03 12:37:50 2013 (23762) Traceback (most recent call last):\n File "/home/mailman/mailman/src/mailman/core/runner.py", line 169, in _one_iteration\n self._process_one_file(msg, msgdata)\n File "/home/mailman/mailman/src/mailman/core/runner.py", line 252, in _process_one_file\n keepqueued = self._dispose(mlist, msg, msgdata)\n File "/home/mailman/mailman/src/mailman/runners/pipeline.py", line 36, in _dispose\n process(mlist, msg, msgdata, pipeline)\n File "/home/mailman/mailman/src/mailman/core/pipelines.py", line 66, in process\n handler.process(mlist, msg, msgdata)\n File "/home/mailman/mailman/src/mailman/handlers/rfc_2369.py", line 120, in process\n process(mlist, msg, msgdata)\n File "/home/mailman/mailman/src/mailman/handlers/rfc_2369.py", line 87, in process\n archiver_set = IListArchiverSet(mlist)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/zope.component-4.1.0-py2.7.egg/zope/component/hookable.py", line 33, in __call__\n return self.__implementation(*args, **kw)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/zope.component-4.1.0-py2.7.egg/zope/component/_api.py", line 156, in adapter_hook\n return sitemanager.queryAdapter(object, interface, name, default)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/zope.interface-4.0.5-py2.7-linux-x86_64.egg/zope/interface/registry.py", line 229, in queryAdapter\n return self.adapters.queryAdapter(object, interface, name, default)\n File "/home/mailman/mailman/src/mailman/model/mailinglist.py", line 590, in __init__\n ListArchiver.name == archiver_name)).one()\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/store.py", line 1158, in one\n result = self._store._connection.execute(select)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/databases/postgres.py", line 266, in execute\n return Connection.execute(self, statement, params, noresult)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/database.py", line 241, in execute\n raw_cursor = self.raw_execute(statement, params)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/databases/postgres.py", line 276, in raw_execute\n return Connection.raw_execute(self, statement, params)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/database.py", line 374, in raw_execute\n self._run_execution(raw_cursor, args, params, statement)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/database.py", line 388, in _run_execution\n self._check_disconnect(raw_cursor.execute, *args)\n File "/home/mailman/py27/local/lib/python2.7/site-packages/storm-0.20-py2.7-linux-x86_64.egg/storm/database.py", line 454, in _check_disconnect\n return function(*args, **kwargs)\nProgrammingError: ERREUR: la relation \xc2\xab listarchiver \xc2\xbb n\'existe pas\nLINE 1: ...tarchiver.mailing_list_id, listarchiver.name FROM listarchiv...\n ^\n\nDec 03 12:37:50 2013 (23762) SHUNTING: 1386070670.221361+6120e5d3eaedee1815abe8fc6bf7dd306961f1dd and the real problem origin is missing schema upgrades for postgres Cheers Le 2013-12-03 12:09, nicolas a écrit :

So, if I understand correctly, Storm add to its exception the error message it gets from PostgreSQL, which is a bytestream (encoded in UTF-8 on your system). But Mailman sends its logs to the logger as unicode, thus the exception must be converted to unicode, and that's where the UnicodeDecodeError is raised. What can we do? Should we try to decode the Storm exception using the default system encoding before sending it to the logger?
Aurélien
participants (2)
-
Aurelien Bompard
-
nicolas