Re: [Mailman-Users] utf-8 error with all lists
Realizing I didn't respond to the list. So scroll down for my response. Then come back here to continue reading.
Now that I have my Python-fu working I'm retesting Mark's suggestion:
webserv:~# python Python 2.7.13 (default, Sep 26 2018, 18:42:22) [GCC 6.3.0 20170516] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import string string.lowercase 'abcdefghijklmnopqrstuvwxyz'
It appears that lowercase does return an apparent properly lowercased string of letters. So, I'm uncertain why the hand-edited code would function differently.
Thanks,
Jason Morrill
IT Director
Child and Family Agency
860-443-2896 x1422
http://www.childandfamilyagency.org/
From: Jason Morrill Sent: Monday, May 13, 2019 10:29 AM To: Mark Sapiro Subject: RE: [Mailman-Users] utf-8 error with all lists
Mark, Thanks for the suggestions and link to the remainder of that prior message thread.
I too get the same results related to Python and lowercase.
Now, I'm not a Python programmer but can certainly stumble through to sort things out. Here's what I'm seeing:
webserv:~# python Python 2.7.13 (default, Sep 26 2018, 18:42:22) [GCC 6.3.0 20170516] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import string lowercase Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'lowercase' is not defined string <module 'string' from '/usr/lib/python2.7/string.pyc'>
Per your suggestion I modified /var/lib/mailman/Mailman/Archiver/pipermail.py to set lowercase = 'abcdefghijklmnopqrstuvwxyz'
And indeed the issue worked itself out. No more errors and I was able to clear the qfiles/shunt folder too.
Per the Python docs, and your info, the string returned by lowercase should be a-z. And it depends upon the locale of the server. Something I've never purposefully changed.
In Debian it looks at /etc/locale.gen
In that file I see my locale set to be en_US ISO-8859-1
Could that somehow be related to this issue?
Thanks,
Jason Morrill
IT Director
Child and Family Agency
860-443-2896 x1422
http://www.childandfamilyagency.org/
From: Mailman-Users [mailman-users-bounces+morrillj=childandfamilyagency.org@python.org] on behalf of Mark Sapiro [mark@msapiro.net] Sent: Saturday, May 11, 2019 2:13 PM To: mailman-users@python.org Subject: Re: [Mailman-Users] utf-8 error with all lists
On 5/10/19 12:37 PM, Jason Morrill wrote:
Since the beginning of May I've had a problem with all messages going through Mailman. They are being delivered to the lists properly but copies are staying in qfiles/shunt
The errors appearing in the Mailman error.1 log file indicate " May 05 10:16:29 2019 (1167) Uncaught runner exception: 'utf8' codec can't decode byte 0xaa in position 26: invalid start byte ... File "/var/lib/mailman/Mailman/Archiver/pipermail.py", line 63, in fixAuthor while i>0 and (L[i-1][0] in lowercase or UnicodeDecodeError: 'utf8' codec can't decode byte 0xaa in position 26: invalid start byte
May 05 10:16:29 2019 (1167) SHUNTING: 1557065787.914554+25c37ef6e662cc98d41ac9d45855fe04f151e11a
"
This is on Debian 9.6 with Python 2.7.13 and Mailman 2.1.23 , all installed from the Debian package manager.
I've spent several hours hunting around for a potential solution. Then stumbled upon this: https://mail.python.org/pipermail/mailman-users/2019-March/084243.html
If you read the entire thread (which is split in the archive with the second part beginning at <https://mail.python.org/pipermail/mailman-users/2019-March/084266.html>), you will see that I think the issue is that whatever changed in your system at the beginning of May is causing the Python
from string import lowercase
in /var/lib/mailman/Mailman/Archiver/pipermail.py to import a lowercase object which should be the string 'abcdefghijklmnopqrstuvwxyz', but in fact is longer and has non-ascii accented characters following 'z'.
The OP in that thread couldn't verify that that was the case, but I am convinced that it is.
There doesn't seem to be a Debian patch to mailman 1:2.1.23-1+deb9u4 that would affect this so it is apparently something in Python or the OS.
I am certain that if you start with the base /var/lib/mailman/Mailman/Archiver/pipermail.py and change
from string import lowercase
at line 13 to
lowercase = 'abcdefghijklmnopqrstuvwxyz'
the problem will be fixed.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/morrillj%40childandfam...
On 5/13/19 7:46 AM, Jason Morrill wrote:
Now that I have my Python-fu working I'm retesting Mark's suggestion:
webserv:~# python Python 2.7.13 (default, Sep 26 2018, 18:42:22) [GCC 6.3.0 20170516] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import string string.lowercase 'abcdefghijklmnopqrstuvwxyz'
It appears that lowercase does return an apparent properly lowercased string of letters. So, I'm uncertain why the hand-edited code would function differently.
This is what we were unable to solve in the original thread starting at <https://mail.python.org/pipermail/mailman-users/2019-March/084241.html> for 10 posts and then continuing at <https://mail.python.org/pipermail/mailman-users/2019-March/084266.html> for another 13 posts.
When the OP put in the logging suggested at <https://mail.python.org/pipermail/mailman-users/2019-March/084273.html>, we discovered that in this Mailman/Archiver/pipermail.py at least, string .lowercase returns the iso-8859-1 encoding of 'abcdefghijklmnopqrstuvwxyzµßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ' (see <https://mail.python.org/pipermail/mailman-users/2019-March/084280.html>), but we were never able to determine where this came from.
If you are curious, you could modify your current /var/lib/mailman/Mailman/Archiver/pipermail.py which i think will contain
lowercase = 'abcdefghijklmnopqrstuvwxyz' at line 13.
If you add following that, the two lines
import string syslog('error', 'string: %r', string)
You should then see (after at least one post) messages in Mailman's error log like
<timestamp> string: <module 'string' from ...
where ... is the location of the Python string module. This will probable be a .pyc file, but you can look in the corresponding .py file for the definition of lowercase. in my /usr/lib/python2.7/string.py, it is
lowercase = 'abcdefghijklmnopqrstuvwxyz'
There is also
ascii_lowercase = lowercase
I'm curious as to what these are in yours.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 5/13/19 10:19 AM, Mark Sapiro wrote:
On 5/13/19 7:46 AM, Jason Morrill wrote:
It appears that lowercase does return an apparent properly lowercased string of letters. So, I'm uncertain why the hand-edited code would function differently.
This is what we were unable to solve in the original thread starting at <https://mail.python.org/pipermail/mailman-users/2019-March/084241.html> for 10 posts and then continuing at <https://mail.python.org/pipermail/mailman-users/2019-March/084266.html> for another 13 posts.
Just pipe it through
lambda fixstr s : return unicodedata.normalize('NFKD', s.decode('utf-8')).encode('ascii','ignore'))
or something along those lines first and be done with it.
-- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu
participants (3)
-
Dimitri Maziuk -
Jason Morrill -
Mark Sapiro