[Mailman-Users] diagnosing messages missing from archives

[Matt Morgan]

On Thu, May 11, 2017 at 1:57 PM, Mark Sapiro <mark at msapiro.net> wrote:

> On 05/11/2017 09:00 AM, Matt Morgan wrote:
> > On Tue, May 9, 2017 at 12:58 PM, Mark Sapiro <mark at msapiro.net> wrote:
> >>
> >> In Mailman's directory (/usr/local/mailman in your case)
> >>
> >> bin/show_qfiles qfiles/shunt/*
> >>
> >
> > Just FYI, in case anyone's reading this in the list archives in future,
> you
> > may need a "../" in front of "qfiles/shunt/*" there.
>
>
> Are you saying in your case Mailman's qfiles directory is
> /usr/local/qfiles? That seems unusual? Or did you do
>
> cd /usr/local/mailman/bin
> show_qfiles ../qfiles/shunt/*
>

You're right, sorry! I misread your instruction.


>
> ...
> >   File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 63, in
> > fixAuthor
> >     while i>0 and (L[i-1][0] in lowercase or
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xb5 in position 26:
> > ordinal
> > not in range(128)
>
>
> The full statement throwing the exception is
>
>         while i>0 and (L[i-1][0] in lowercase or
>                        L[i-1].lower() in smallNameParts):
>
> lowercase is string.lowercase which from below appears to be all ascii.
> smallNameParts is defined in the module as a list of short ascii strings
>
> smallNameParts = ['van', 'von', 'der', 'de']
>
> That leaves L as the only possible source of non-ascii. L is a list of
> the 'words' in the From: display name, but at that point in the code, it
> should be unicode. I suppose this may be involved in the issue, but then
> what is the XXXXXXXXXX in the anonymized address below? Does it contain
> non-ascii?
>

in this example it is all regular letters. The list is a very international
so on the whole we would definitely be dealing with every possible alphabet
(but not this time).



>
>
> > Here is the output from the dumpdb command (anonymized a little):
> >
> ...
> > From: XXXXXXXXXX <xxxxxxxx at me.com>
>
>
> ...
> >> What do you get if you invoke Python interactively on this server and do
> >>
> >> import string
> >> string.lowercase
> >>
> >> I get 'abcdefghijklmnopqrstuvwxyz'
> >>
> >
> > I get the same thing! Does that make any sense?
> >
> > xxxx at yyyyy:/usr/local/mailman/logs# python
> > Python 2.7.5 (default, May 29 2013, 02:28:51)
> > [GCC 4.8.0] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> >>>> import string
> >>>> string.lowercase
> > 'abcdefghijklmnopqrstuvwxyz'
>
>
> I'm still leaning towards non-ascii in string.lowercase, because I think
> that's the only thing it could be and because the \xb5 is in 0-based
> position 26 which would be the caharacter after 'z'.
>
> Is Python 2.7.5 (default, May 29 2013, 02:28:51) the python that Mailman
> is using? Look at whatever script starts/stops/restarts Mailman and the
> Python it uses and do the above test in that Python.
>

It looks like it's 2.7.5 but this is an ancient server (it's been kept
alive and upgraded since at least 2009) I inherited and I'm never 100%
confident I know how it works.

It's a customized Slack (which I'm not familiar with in the first place).
Init scripts are in /etc/rc.d and /etc/rc.d/rc.mailman has a line

PYTHON=/usr/bin/python

and I get

root at cool:/etc/rc.d# /usr/bin/python --version
Python 2.7.5

Nothing in /usr/local/mailman/bin/mailmanctl seems to suggest it uses any
other python that what the init script uses? But again, not being a
Slack-jockey in the first place, if there's somewhere else to look I would
miss it.

Thanks again!
Matt