[Mailman-Users] diagnosing messages missing from archives

[Mark Sapiro]

On 05/09/2017 07:54 AM, Matt Morgan wrote:
> 
> I do see this in the error log:
> 
> Apr 27 20:58:43 2017 (517) Uncaught runner exception: 'ascii' codec can't
> decode
>  byte 0xb5 in position 26: ordinal not in range(128)
> Apr 27 20:58:43 2017 (517) Traceback (most recent call last):
>   File "/usr/local/mailman/Mailman/Queue/Runner.py", line 119, in _oneloop
>     self._onefile(msg, msgdata)
>   File "/usr/local/mailman/Mailman/Queue/Runner.py", line 190, in _onefile
>     keepqueued = self._dispose(mlist, msg, msgdata)
>   File "/usr/local/mailman/Mailman/Queue/ArchRunner.py", line 77, in
> _dispose
>     mlist.ArchiveMail(msg)
>   File "/usr/local/mailman/Mailman/Archiver/Archiver.py", line 216, in
> ArchiveMail
>     h.processUnixMailbox(f)
>   File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 596, in
> processUnixMailbox
>     self.add_article(a)
>   File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 640, in
> add_article
>     author = fixAuthor(article.decoded['author'])
>   File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 63, in
> fixAuthor
>     while i>0 and (L[i-1][0] in lowercase or
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xb5 in position 26:
> ordinal not in range(128)
> 
> Apr 27 20:58:43 2017 (517) SHUNTING:
> 1493351921.551243+8e185be2446d37cbda2f99d888a5a7b7e0643d85
> 
> Is that the error I'm looking for?


Yes.


> That ArchRunner line looks relevant.
> It's a reasonably common series of entries in the log (10ish times a day),
> though the timing doesn't line up with the date/time on the messages that
> are missing. Is that normal? Is there a way to verify that the error is
> specific to any given missing message?


Every one of those messages is in Mailman's shunt queue with names like
1493351921.551243+8e185be2446d37cbda2f99d888a5a7b7e0643d85.pck. You can
see the messages in a couple of ways.

In Mailman's directory (/usr/local/mailman in your case)

bin/show_qfiles qfiles/shunt/*

will show them all or you can view individual ones by name, or you can run

bin/dumpdb qfiles/shunt/xxx.pck

where xxx.pck is an individual name. The latter will also show the
message metadata which will indicate, among other things, the queue the
message came from which in the case of these should be 'archive'.

The issue appears to be a bug in Mailman. but I need more info to be
sure including the headers from one of the shunted messages.

The exception is thrown when the archiver is trying to canonicalize the
author's display name into "Last, First" form and I think the actual
exception in this case is in checking if a "word" of the name begins
with a lowercase letter. Since the word is already unidode in this case,
it is apparently Python's string.lowercase that contains \xb5 in
position 26.

What do you get if you invoke Python interactively on this server and do

import string
string.lowercase

I get 'abcdefghijklmnopqrstuvwxyz'

This is hard coded in /usr/lib/python2.7/string.py and not locale
dependent (and if it were, I think we'd have seen many bug reports
before now), but it looks like yours has extra non-ascii following the
26 ascii lowercase letters.

In any case, after you fix the underlying issue and are satisfied that
you want to reprocess all the shunted messages (move ones you don't want
out of the shunt queue), you can run bin/unshunt to reprocess them.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan