Fwd: Uncaught runner exception
Am 12.03.2019 um 23:47 schrieb Mark Sapiro:
An example of a complete error message:
Mar 11 17:02:22 2019 (15400) SHUNTING: 1552320069.504329+15fb4f71a5404fd1af3dafbe2780486d5b474cc7 Mar 11 17:22:37 2019 (15400) Uncaught runner exception: 'ascii' codec can't decode byte 0xb5 in position 26: ordinal not in range(128) Mar 11 17:22:37 2019 (15400) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 119, in _oneloop self._onefile(msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 190, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/ArchRunner.py", line 77, in _dispose mlist.ArchiveMail(msg) File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 216, in ArchiveMail h.processUnixMailbox(f) File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 602, in processUnixMailbox self.add_article(a) File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 646, in add_article author = fixAuthor(article.decoded['author']) File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 63, in fixAuthor while i>0 and (L[i-1][0] in lowercase or UnicodeDecodeError: 'ascii' codec can't decode byte 0xb5 in position 26: ordinal not in range(128) Again, this is the code trying to "Canonicalize a name into Last, First
On 3/12/19 2:37 AM, Lothar Schilling wrote: format". The complete statement is
while i>0 and (L[i-1][0] in lowercase or L[i-1].lower() in smallNameParts): i = i - 1smallNameParts is all lower case ascii, i.e.
= ['van', 'von', 'der', 'de']
and L is the list returned by
L = author.split()and none of the shunted .pck files you provided me have any name pieces longer than 14 bytes and only 2 of the 5 even have any non-ascii, but the decode error refers to byte 0xb5 in position 26 (B5 in iso-8859-1 is µ).
I know you said in your original post
I am using Python 2.6.6 on a Centos 6 system. The output of /usr/lib64/python >>> import string >>> string.lowercase is 'abcdefghijklmnopqrstuvwxyz' (as it is supposed to be). but see below.
The problem is site-wide, not just for one or two lists. Anyway, there are no altered templates or messages for a specific list. The only files deviating from the default configuration are: Defaults.py mm_cfg.py
Of course Defaults.py has DEFAULT_SERVER_LANGUAGE = 'en'.
So in mm_cfg.py I added DEFAULT_SERVER_LANGUAGE = 'de' add_language('en', 'English (USA)', 'iso-8859-1', 'ltr')
But I cannot see how this would cause the problem. Neither can I, but I tried again with those settings just to be sure, and got no issues.
All of this leads me to conclude that the issue must be non-ascii following the 'z' in string.lowercase.
What is the path to the Python interpreter that Mailman is using?
/usr/bin/python /usr/bin/python -V: Python 2.6.6
What do you get if in that Python you do
import string lowercase string
For example, I get
import string lowercase 'abcdefghijklmnopqrstuvwxyz' string <module 'string' from '/usr/lib/python2.7/string.pyc'>
import string
lowercase Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'lowercase' is not defined
string <module 'string' from '/usr/lib64/python2.6/string.pyc'>
Lothar Schilling writes:
What do you get if in that Python you do
import string lowercase
I think this should be 'string.lowercase'.
import string
lowercase Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'lowercase' is not defined
This is what I would expect. I'm not sure why Mark got something different (he may have previously done 'from string import *'). We apologize for the inaccuracy.
string <module 'string' from '/usr/lib64/python2.6/string.pyc'>
It occurred to me that that you're probably in a German locale, in which case umlauts and sharp S might be added to the string of lowercase letters, but that doesn't seem to be the case:
$ LC_ALL=de_DE python Python 2.7.10 (default, Oct 6 2017, 22:29:07) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)] on darwin Type "help", "copyright", "credits" or "license" for more information.
import string string.lowercase 'abcdefghijklmnopqrstuvwxyz' len(string.lowercase) 26 string <module 'string' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/string.pyc'>
I added 'len(string.lowercase)' to the experiment as a quick consistency check. I also tried the same environment setting on Linux with the same result, except that the module was loaded from /usr/lib/python2.7/string.pyc and Python's version was 2.7.16rc1. So the locale doesn't seem to be at issue.
Regards, Steve
-- Associate Professor Division of Policy and Planning Science http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull@sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
On 3/14/19 8:14 AM, Stephen J. Turnbull wrote:
Lothar Schilling writes:
What do you get if in that Python you do
import string lowercase
I think this should be 'string.lowercase'.
import string
lowercase Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'lowercase' is not defined
This is what I would expect. I'm not sure why Mark got something different (he may have previously done 'from string import *'). We apologize for the inaccuracy.
Yes, this was definitely an error on my part. I had initially done
from string import lowercase
but I wanted to see where 'string' came from so I did import string so
I could get that and thel when I did 'lowercase' instead of
'string.lowercase', it worked.
I apologize for the confusion.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 3/14/19 2:39 AM, Lothar Schilling wrote:
Am 12.03.2019 um 23:47 schrieb Mark Sapiro:
What is the path to the Python interpreter that Mailman is using?
/usr/bin/python /usr/bin/python -V: Python 2.6.6
What is the python command in the result of
ps -fwwA|grep ArchRunner
What do you get if in that Python you do
import string lowercase
As noted in other replies, this should have been 'string.lowercase'.
string ... string <module 'string' from '/usr/lib64/python2.6/string.pyc'>
Get the python command from the 'ps' result above. run that full path and then do
import string string.lowercase string
I am expecting (hoping actually because it's the only explanation I can see) that the result of 'string.lowercase' will be longer than 26 characters and will contain non-ascii. Then the result of 'string' will point to the string.py(c) that this comes from.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Am 14.03.2019 um 18:55 schrieb Mark Sapiro:
On 3/14/19 2:39 AM, Lothar Schilling wrote:
Am 12.03.2019 um 23:47 schrieb Mark Sapiro:
What is the path to the Python interpreter that Mailman is using? /usr/bin/python /usr/bin/python -V: Python 2.6.6
What is the python command in the result of
ps -fwwA|grep ArchRunner /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=ArchRunner:0:1 -s
What do you get if in that Python you do
import string lowercase As noted in other replies, this should have been 'string.lowercase'.
string ... string <module 'string' from '/usr/lib64/python2.6/string.pyc'>
Get the python command from the 'ps' result above. run that full path and then do
import string string.lowercase string I am expecting (hoping actually because it's the only explanation I can see) that the result of 'string.lowercase' will be longer than 26 characters and will contain non-ascii. Then the result of 'string' will point to the string.py(c) that this comes from.
Sadly not:
import string string.lowercase 'abcdefghijklmnopqrstuvwxyz' string <module 'string' from 'string.pyc'>
On 3/15/19 4:27 AM, Lothar Schilling wrote:
Am 14.03.2019 um 18:55 schrieb Mark Sapiro:
On 3/14/19 2:39 AM, Lothar Schilling wrote: ...
string <module 'string' from '/usr/lib64/python2.6/string.pyc'> ... Sadly not:
import string string.lowercase 'abcdefghijklmnopqrstuvwxyz' string <module 'string' from 'string.pyc'>
This is interesting. Here you get <module 'string' from 'string.pyc'> and before (above) you got <module 'string' from '/usr/lib64/python2.6/string.pyc'>. I wonder why the difference. In the message at <https://mail.python.org/pipermail/mailman-users/2019-March/084243.html> I suggested this patch === modified file 'Mailman/Archiver/pipermail.py' --- Mailman/Archiver/pipermail.py 2018-05-03 21:23:47 +0000 +++ Mailman/Archiver/pipermail.py 2019-03-02 04:51:23 +0000 @@ -60,9 +60,12 @@ else: # Mixed case; assume that small parts of the last name will be # in lowercase, and check them against the list. - while i>0 and (L[i-1][0] in lowercase or - L[i-1].lower() in smallNameParts): - i = i - 1 + try: + while i>0 and (L[i-1][0] in lowercase or + L[i-1].lower() in smallNameParts): + i = i - 1 + except: + syslog('error', 'Exception in fixAuthor: %s', author) author = SPACE.join(L[-1:] + L[i:-1]) + ', ' + SPACE.join(L[:i]) return author I assume you still have that installed. In any case, if you add a second syslog line after the first syslog('error', 'lowercase value: %r', lowercase) What do you then get in Mailman's error log -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mit herzlichem Dank und freundlichen Grüßen! Lothar Schilling Förderverein Pro Asyl e.V. Postfach 160624 60069 Frankfurt Tel. 069 242314-40 Am 15.03.2019 um 17:25 schrieb Mark Sapiro:
On 3/15/19 4:27 AM, Lothar Schilling wrote:
Am 14.03.2019 um 18:55 schrieb Mark Sapiro:
On 3/14/19 2:39 AM, Lothar Schilling wrote: ...
string <module 'string' from '/usr/lib64/python2.6/string.pyc'> ... Sadly not:
import string string.lowercase 'abcdefghijklmnopqrstuvwxyz' string <module 'string' from 'string.pyc'>
This is interesting. Here you get <module 'string' from 'string.pyc'> and before (above) you got <module 'string' from '/usr/lib64/python2.6/string.pyc'>.
I wonder why the difference.
In the message at <https://mail.python.org/pipermail/mailman-users/2019-March/084243.html> I suggested this patch
=== modified file 'Mailman/Archiver/pipermail.py' --- Mailman/Archiver/pipermail.py 2018-05-03 21:23:47 +0000 +++ Mailman/Archiver/pipermail.py 2019-03-02 04:51:23 +0000 @@ -60,9 +60,12 @@ else: # Mixed case; assume that small parts of the last name will be # in lowercase, and check them against the list. - while i>0 and (L[i-1][0] in lowercase or - L[i-1].lower() in smallNameParts): - i = i - 1 + try: + while i>0 and (L[i-1][0] in lowercase or + L[i-1].lower() in smallNameParts): + i = i - 1 + except: + syslog('error', 'Exception in fixAuthor: %s', author) author = SPACE.join(L[-1:] + L[i:-1]) + ', ' + SPACE.join(L[:i]) return author
I assume you still have that installed. In any case, if you add a second syslog line after the first
syslog('error', 'lowercase value: %r', lowercase)
What do you then get in Mailman's error log
Exception in fixAuthor: D▒rthe Hinz - Fl▒chtlingsrat Nds. Mar 18 08:33:51 2019 (23745) lowercase value: 'abcdefghijklmnopqrstuvwxyz\xb5\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff' Mar 18 10:06:50 2019 (23745) Exception in fixAuthor: Philipp Millius (Fl▒chtlingsrat THR e.V.) Mar 18 10:06:50 2019 (23745) lowercase value: 'abcdefghijklmnopqrstuvwxyz\xb5\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff'
On 3/18/19 2:29 AM, Lothar Schilling wrote:
Exception in fixAuthor: D▒rthe Hinz - Fl▒chtlingsrat Nds. Mar 18 08:33:51 2019 (23745) lowercase value: 'abcdefghijklmnopqrstuvwxyz\xb5\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff' Mar 18 10:06:50 2019 (23745) Exception in fixAuthor: Philipp Millius (Fl▒chtlingsrat THR e.V.) Mar 18 10:06:50 2019 (23745) lowercase value: 'abcdefghijklmnopqrstuvwxyz\xb5\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff'
which is iso-8859-1 encoding of 'abcdefghijklmnopqrstuvwxyzµßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ' and which is the problem.
I don't know why in testing string.lowercase does not return the above string, but in the Python that's running ArchRunner, and presumably the rest of mailman, string.lowercase has that value. This is what needs to be fixed. The first step is possibly to do
locate string.py
And then look at all the ones you find to see which defines lowercase that way and then figure out why the Python that runs Mailman uses that one rather that the ones you see in testing.
Also,
which -a python
may help find things.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Am 18.03.2019 um 17:06 schrieb Mark Sapiro:
On 3/18/19 2:29 AM, Lothar Schilling wrote:
Exception in fixAuthor: D▒rthe Hinz - Fl▒chtlingsrat Nds. Mar 18 08:33:51 2019 (23745) lowercase value: 'abcdefghijklmnopqrstuvwxyz\xb5\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff' Mar 18 10:06:50 2019 (23745) Exception in fixAuthor: Philipp Millius (Fl▒chtlingsrat THR e.V.) Mar 18 10:06:50 2019 (23745) lowercase value: 'abcdefghijklmnopqrstuvwxyz\xb5\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff' which is iso-8859-1 encoding of 'abcdefghijklmnopqrstuvwxyzµßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ' and which is the problem.
I don't know why in testing string.lowercase does not return the above string, but in the Python that's running ArchRunner, and presumably the rest of mailman, string.lowercase has that value. This is what needs to be fixed. The first step is possibly to do
locate string.py
There is just one
/usr/lib64/python2.6/string.py
yielding
lowercase = 'abcdefghijklmnopqrstuvwxyz'.
And then look at all the ones you find to see which defines lowercase that way and then figure out why the Python that runs Mailman uses that one rather that the ones you see in testing.
Also,
which -a python
points to /usr/bin/python as it is supposed to.
may help find things.
As your fix makes for the mails being archived maybe we should leave it at that. Thank you very much anyway for your patient helping me.
On 3/20/19 5:31 AM, Lothar Schilling wrote:
There is just one
/usr/lib64/python2.6/string.py
yielding
lowercase = 'abcdefghijklmnopqrstuvwxyz'.
You could try one more thing. Run Mailman's
bin/withlist -i ...
from string import lowercase lowercase (response) import string string (response)
and see if that returns the extended lowercase and where it says lowercase comes from.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Am 20.03.2019 um 18:54 schrieb Mark Sapiro:
On 3/20/19 5:31 AM, Lothar Schilling wrote:
There is just one
/usr/lib64/python2.6/string.py
yielding
lowercase = 'abcdefghijklmnopqrstuvwxyz'.
You could try one more thing. Run Mailman's
bin/withlist -i ...
from string import lowercase lowercase (response) import string string (response)
and see if that returns the extended lowercase and where it says lowercase comes from.
It doesn't return the extended lowercase:
Python 2.6.6 (r266:84292, Aug 18 2016, 15:13:37) [GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2 (InteractiveConsole)
from string import lowercase lowercase 'abcdefghijklmnopqrstuvwxyz' import string string <module 'string' from '/usr/lib64/python2.6/string.pyc'>
On 3/21/19 9:20 AM, Lothar Schilling wrote:
Python 2.6.6 (r266:84292, Aug 18 2016, 15:13:37) [GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2 (InteractiveConsole)
from string import lowercase lowercase 'abcdefghijklmnopqrstuvwxyz' import string string <module 'string' from '/usr/lib64/python2.6/string.pyc'>
I'm out of ideas. I have no clue as to where the extended string.lowercase is coming from and why only Mailman is seeing it, but to work around that, I suggest you change my suggested patch to === modified file 'Mailman/Archiver/pipermail.py' --- Mailman/Archiver/pipermail.py 2018-05-03 21:23:47 +0000 +++ Mailman/Archiver/pipermail.py 2019-03-02 04:51:23 +0000 @@ -60,9 +60,12 @@ else: # Mixed case; assume that small parts of the last name will be # in lowercase, and check them against the list. - while i>0 and (L[i-1][0] in lowercase or - L[i-1].lower() in smallNameParts): - i = i - 1 + try: + while i>0 and (L[i-1][0] in lowercase or + L[i-1].lower() in smallNameParts): + i = i - 1 + except UnicodeDecodeError: + pass author = SPACE.join(L[-1:] + L[i:-1]) + ', ' + SPACE.join(L[:i]) return author This will catch the exception and ignore it without the logging. Also, I have no idea why when you run bin/unshunt, the .pck files remain in qfiles/shunt without errors being reported to the terminal or logged in mailman's error log. Something is strange about that, but I don't know what could cause it. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Am 21.03.2019 um 17:48 schrieb Mark Sapiro:
On 3/21/19 9:20 AM, Lothar Schilling wrote:
Python 2.6.6 (r266:84292, Aug 18 2016, 15:13:37) [GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2 (InteractiveConsole)
from string import lowercase lowercase 'abcdefghijklmnopqrstuvwxyz' import string string <module 'string' from '/usr/lib64/python2.6/string.pyc'>
I'm out of ideas. I have no clue as to where the extended string.lowercase is coming from and why only Mailman is seeing it, but to work around that, I suggest you change my suggested patch to
=== modified file 'Mailman/Archiver/pipermail.py' --- Mailman/Archiver/pipermail.py 2018-05-03 21:23:47 +0000 +++ Mailman/Archiver/pipermail.py 2019-03-02 04:51:23 +0000 @@ -60,9 +60,12 @@ else: # Mixed case; assume that small parts of the last name will be # in lowercase, and check them against the list. - while i>0 and (L[i-1][0] in lowercase or - L[i-1].lower() in smallNameParts): - i = i - 1 + try: + while i>0 and (L[i-1][0] in lowercase or + L[i-1].lower() in smallNameParts): + i = i - 1 + except UnicodeDecodeError: + pass author = SPACE.join(L[-1:] + L[i:-1]) + ', ' + SPACE.join(L[:i]) return author
This will catch the exception and ignore it without the logging.
Also, I have no idea why when you run bin/unshunt, the .pck files remain in qfiles/shunt without errors being reported to the terminal or logged in mailman's error log. Something is strange about that, but I don't know what could cause it.
Applied the patch. Thanks again for your kind advice, Mark!
participants (3)
-
Lothar Schilling -
Mark Sapiro -
Stephen J. Turnbull