[Mailman-Users] diagnosing messages missing from archives

Matt Morgan minxmertzmomo at gmail.com
Thu May 11 12:00:51 EDT 2017 

On Tue, May 9, 2017 at 12:58 PM, Mark Sapiro <mark at msapiro.net> wrote:

> On 05/09/2017 07:54 AM, Matt Morgan wrote:
> >
> > I do see this in the error log:
> >
> > Apr 27 20:58:43 2017 (517) Uncaught runner exception: 'ascii' codec can't
> > decode
> >  byte 0xb5 in position 26: ordinal not in range(128)
> > Apr 27 20:58:43 2017 (517) Traceback (most recent call last):
> >   File "/usr/local/mailman/Mailman/Queue/Runner.py", line 119, in
> _oneloop
> >     self._onefile(msg, msgdata)
> >   File "/usr/local/mailman/Mailman/Queue/Runner.py", line 190, in
> _onefile
> >     keepqueued = self._dispose(mlist, msg, msgdata)
> >   File "/usr/local/mailman/Mailman/Queue/ArchRunner.py", line 77, in
> > _dispose
> >     mlist.ArchiveMail(msg)
> >   File "/usr/local/mailman/Mailman/Archiver/Archiver.py", line 216, in
> > ArchiveMail
> >     h.processUnixMailbox(f)
> >   File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 596, in
> > processUnixMailbox
> >     self.add_article(a)
> >   File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 640, in
> > add_article
> >     author = fixAuthor(article.decoded['author'])
> >   File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 63, in
> > fixAuthor
> >     while i>0 and (L[i-1][0] in lowercase or
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xb5 in position 26:
> > ordinal not in range(128)
> >
> > Apr 27 20:58:43 2017 (517) SHUNTING:
> > 1493351921.551243+8e185be2446d37cbda2f99d888a5a7b7e0643d85
> >
> > Is that the error I'm looking for?
>
>
> Yes.
>
>
> > That ArchRunner line looks relevant.
> > It's a reasonably common series of entries in the log (10ish times a
> day),
> > though the timing doesn't line up with the date/time on the messages that
> > are missing. Is that normal? Is there a way to verify that the error is
> > specific to any given missing message?
>
>
> Every one of those messages is in Mailman's shunt queue with names like
> 1493351921.551243+8e185be2446d37cbda2f99d888a5a7b7e0643d85.pck. You can
> see the messages in a couple of ways.
>
> In Mailman's directory (/usr/local/mailman in your case)
>
> bin/show_qfiles qfiles/shunt/*
>

Just FYI, in case anyone's reading this in the list archives in future, you
may need a "../" in front of "qfiles/shunt/*" there.


>
> will show them all or you can view individual ones by name, or you can run
>
> bin/dumpdb qfiles/shunt/xxx.pck
>
> where xxx.pck is an individual name. The latter will also show the
> message metadata which will indicate, among other things, the queue the
> message came from which in the case of these should be 'archive'.
>

I subscribed to the list so that I could track this more easily. A message
from yesterday was kept out of the archives in the same way.

Here are the headers (anonymized a little):

Delivered-To:     mattcncrt at gmail.com
Received:     by 10.25.153.69 with SMTP id b66csp370037lfe; Wed, 10 May
2017 12:55:08 -0700 (PDT)
X-Received:     by 10.84.128.65 with SMTP id
59mr10521582pla.121.1494446108067; Wed, 10 May 2017 12:55:08 -0700 (PDT)
ARC-Seal:     i=1; a=rsa-sha256; t=1494446108; cv=none; d=google.com;
s=arc-20160816;
b=FRn0ZWGg2oTxTU3+KArFayyfI4JJZKZiZsezc25C6hsUgjDqdAn5CAj4aBKIlfkR7Z
TQYcmBDk8KoNPaLphRrgPXmCg6rV1OLnU6EzcFdJoMxW3kLuG5tbhJFJSl4ypxh7TQDe
zJ3GY1KRWu7AWdaESXevJLyuMwwjJ3VTBwQUwMsXdNju3F3DfekHHuDJdlNWWVrFvLXM
96rGZi9T1Cihg2GW5bWvaNQAxkbmlOIlcsTB3SSx0Yq9P+aOFvQwDBq+U8KDYRCZcoi9
LI5WiMmivYJ0zEToSnd+EJZ+bn4Yd9xFt7KWJubjdc55lSRzsIrN1hjnaQeCD8Eq3RAP w12g==
ARC-Message-Signature:     i=1; a=rsa-sha256; c=relaxed/relaxed; d=
google.com; s=arc-20160816;
h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe
:list-help:list-post:list-archive:list-unsubscribe:list-id
:precedence:subject:message-id:in-reply-to:to:references:date
:mime-version:from:dkim-signature:arc-authentication-results;
bh=cizuuK18iQqPuWzF8CUKeju438HlKLJbPg8FiN9HnfU=;
b=kmeTsmGWUerbMCwqenno45KatPObxGKdigUWDNtufBWJKM2eNTK23S1qRIK2ele8Gh
tdJayjTyXIS432QUS40Ruc0/SCk6RI/MYns3khWDZNxfFYSCYppHBOwtM84i4sX6zILk
CQwSj3uVFdj6QbBM4TiUots9hgA3piHzwUINJn7SYatHcrJWeNSBDgAbRZDWJNsOTl0m
lOcQVDRZ+xSMb8Xe8QFnSQmSy2xCaI5ixYytUSIL5KO06dwGurnajE4D1I+vwfCgZGcb
nmUwNUwF6eWzvkWBL0M8UNbrcsdRq6f//ezkIrLr5Gmn4uTwON7dm/HL9gcYnbehS0Sp 0SeA==
ARC-Authentication-Results:     i=1; mx.google.com; dkim=neutral (body hash
did not verify) header.i=@me.com; spf=neutral (google.com: 206.180.233.162
is neither permitted nor denied by best guess record for domain of
cippnews-bounces at cool.conservation-us.org) smtp.mailfrom=
cippnews-bounces at cool.conservation-us.org; dmarc=fail (p=NONE sp=NONE
dis=NONE) header.from=me.com
Return-Path:     <cippnews-bounces at cool.conservation-us.org>
Received:     from cool.conservation-us.org (cool.tdl.com.
[206.180.233.162]) by mx.google.com with ESMTPS id
q185si4018820pga.417.2017.05.10.12.55.06 (version=TLS1 cipher=AES128-SHA
bits=128/128); Wed, 10 May 2017 12:55:08 -0700 (PDT)
Received-SPF:     neutral (google.com: 206.180.233.162 is neither permitted
nor denied by best guess record for domain of
cippnews-bounces at cool.conservation-us.org) client-ip=206.180.233.162;
Authentication-Results:     mx.google.com; dkim=neutral (body hash did not
verify) header.i=@me.com; spf=neutral (google.com: 206.180.233.162 is
neither permitted nor denied by best guess record for domain of
cippnews-bounces at cool.conservation-us.org) smtp.mailfrom=
cippnews-bounces at cool.conservation-us.org; dmarc=fail (p=NONE sp=NONE
dis=NONE) header.from=me.com
Received:     from localhost ([127.0.0.1] helo=cool.tdl.com) by
cool.conservation-us.org with esmtp (Exim 4.69) (envelope-from <
cippnews-bounces at cool.conservation-us.org>) id 1d8XhD-0000aU-GX; Wed, 10
May 2017 12:54:51 -0700
Received:     from pv35p22im-ztdg05131101.me.com ([17.133.189.43]) by
cool.conservation-us.org with esmtp (Exim 4.69) (envelope-from <
annshaftel at me.com>) id 1d8XhC-0000aM-7S for
cippnews at cool.conservation-us.org; Wed, 10 May 2017 12:54:50 -0700
Received:     from process-dkim-sign-daemon.pv35p22im-ztdg05131101.me.com
by pv35p22im-ztdg05131101.me.com (Oracle Communications Messaging Server
7.0.5.38.0 64bit (built Feb 26 2016)) id <
0OPR007005WLLE00 at pv35p22im-ztdg05131101.me.com> for
cippnews at cool.conservation-us.org; Wed, 10 May 2017 19:54:49 +0000 (GMT)
DKIM-Signature:     v=1; a=rsa-sha256; c=relaxed/relaxed; d=me.com;
s=4d515a; t=1494446089; bh=iMRS0rztCQMUAhGFAT7khfPTdH9Fy/59g3wJDcQzaxM=;
h=From:Content-type:MIME-version:Subject:Date:To:Message-id;
b=P994EyGUDRT5hcWBUpFuaXhGcIwIv5y+2Fc2iyA9I1vQsqyq4rg57CptHkrr/t3A7
yf4tU+IIbYu0Jq/F9xtVnPT3u/B5s2MfZvIORNOvKxxx1bm0BwXpGqwwLTAE73gfxO
y8akmoAfaoUJlbQ+Wptu8Y7uNLPBaMyuUoomX8Bcm7UiNW9lWBLtIgCAR0pAJ0StqP
Fm1N9Rqn4omlJ5z7uQ1V/wdMgO5QRAY0NT2punAB5sGs3eEJJavfWZsiGDbNBIq56v
9mYCxgyId1spAvb4HgVbKNd28V7C3y5OreWQsPdTp0HS218dODJQekkAv6dTd5RxSc
FMEP1hpTefwhg==
Received:     from icloud.com ([127.0.0.1]) by pv35p22im-ztdg05131101.me.com
(Oracle Communications Messaging Server 7.0.5.38.0 64bit (built Feb 26
2016)) with ESMTPSA id <0OPR00NJ85Z8KO40 at pv35p22im-ztdg05131101.me.com> for
cippnews at cool.conservation-us.org; Wed, 10 May 2017 19:54:46 +0000 (GMT)
X-Proofpoint-Virus-Version:     vendor=fsecure engine=2.50.10432:,,
definitions=2017-05-10_16:,, signatures=0
X-Proofpoint-Spam-Details:     rule=notspam policy=default score=0
spamscore=0 clxscore=1034 suspectscore=0 malwarescore=0 phishscore=0
adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
engine=8.0.1-1701120000 definitions=main-1705100136
From:     XXXXXXXX <xxxxxxxxxx at me.com>
MIME-version:     1.0 (Mac OS X Mail 10.3 \(3273\))
Date:     Wed, 10 May 2017 16:54:44 -0300
References:     <CACJwyVrjx-m1VZ=
S68r5cV-diHV3VchfxQ2A6qhwDDK5JZDGHA at mail.gmail.com>
To:     Conservators in Private Practice Discussion <
cippnews at cool.conservation-us.org>
In-reply-to:     <CACJwyVrjx-m1VZ=
S68r5cV-diHV3VchfxQ2A6qhwDDK5JZDGHA at mail.gmail.com>
Message-id:     <1442A6F7-ED43-4899-A0C9-403898F52F80 at me.com>
X-Mailer:     Apple Mail (2.3273)
Subject:     Re: [CIPPNEWS] closing private conservation studio
X-BeenThere:     cippnews at cool.conservation-us.org
X-Mailman-Version:     2.1.23
Precedence:     list
List-Id:     Conservators in Private Practice Discussion <
cippnews.cool.conservation-us.org>
List-Unsubscribe:     <
http://cool.conservation-us.org/mailman/options/cippnews>, <mailto:
cippnews-request at cool.conservation-us.org?subject=unsubscribe>
List-Archive:     <http://cool.conservation-us.org/mailman/private/cippnews/
>
List-Post:     <mailto:cippnews at cool.conservation-us.org>
List-Help:     <mailto:cippnews-request at cool.conservation-us.org
?subject=help>
List-Subscribe:     <
http://cool.conservation-us.org/mailman/listinfo/cippnews>, <mailto:
cippnews-request at cool.conservation-us.org?subject=subscribe>
Reply-To:     Conservators in Private Practice Discussion <
cippnews at cool.conservation-us.org>
Content-Type:     text/plain; charset="utf-8"
Content-Transfer-Encoding:     base64
Errors-To:     cippnews-bounces at cool.conservation-us.org
Sender:     "cippnews" <cippnews-bounces at cool.conservation-us.org>

Here is the section from the error log (same issue, I think):

May 10 12:54:51 2017 (517) SHUNTING:
1494446090.4214+cdb2b1813d6f36d682d0f74cfc581e6046735197
May 10 12:58:26 2017 (517) Uncaught runner exception: 'ascii' codec can't
decode byte 0xb5 in position 26: ordinal not in range(128)
May 10 12:58:26 2017 (517) Traceback (most recent call last):
  File "/usr/local/mailman/Mailman/Queue/Runner.py", line 119, in _oneloop
    self._onefile(msg, msgdata)
  File "/usr/local/mailman/Mailman/Queue/Runner.py", line 190, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/usr/local/mailman/Mailman/Queue/ArchRunner.py", line 77, in
_dispose
    mlist.ArchiveMail(msg)
  File "/usr/local/mailman/Mailman/Archiver/Archiver.py", line 216, in
ArchiveMail
    h.processUnixMailbox(f)
  File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 596, in
processUnixMailbox
    self.add_article(a)
  File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 640, in
add_article
    author = fixAuthor(article.decoded['author'])
  File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 63, in
fixAuthor
    while i>0 and (L[i-1][0] in lowercase or
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb5 in position 26:
ordinal
not in range(128)

Here is the output from the dumpdb command (anonymized a little):

[----- start pickle file -----]
<----- start object 1 ----->
>From xxxxxxxxxx at me.com Wed May 10 12:54:50 2017
Received: from pv35p22im-ztdg05131101.me.com ([17.133.189.43])
 by cool.conservation-us.org with esmtp (Exim 4.69)
 (envelope-from <xxxxxxxxxxx at me.com>) id 1d8XhC-0000aM-7S
 for cippnews at cool.conservation-us.org; Wed, 10 May 2017 12:54:50 -0700
Received: from process-dkim-sign-daemon.pv35p22im-ztdg05131101.me.com by
 pv35p22im-ztdg05131101.me.com
 (Oracle Communications Messaging Server 7.0.5.38.0 64bit (built Feb 26 2016))
 id <0OPR007005WLLE00 at pv35p22im-ztdg05131101.me.com> for
 cippnews at cool.conservation-us.org; Wed, 10 May 2017 19:54:49 +0000 (GMT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=me.com; s=4d515a;
 t=1494446089; bh=iMRS0rztCQMUAhGFAT7khfPTdH9Fy/59g3wJDcQzaxM=;
 h=From:Content-type:MIME-version:Subject:Date:To:Message-id;
 b=P994EyGUDRT5hcWBUpFuaXhGcIwIv5y+2Fc2iyA9I1vQsqyq4rg57CptHkrr/t3A7
 yf4tU+IIbYu0Jq/F9xtVnPT3u/B5s2MfZvIORNOvKxxx1bm0BwXpGqwwLTAE73gfxO
 y8akmoAfaoUJlbQ+Wptu8Y7uNLPBaMyuUoomX8Bcm7UiNW9lWBLtIgCAR0pAJ0StqP
 Fm1N9Rqn4omlJ5z7uQ1V/wdMgO5QRAY0NT2punAB5sGs3eEJJavfWZsiGDbNBIq56v
 9mYCxgyId1spAvb4HgVbKNd28V7C3y5OreWQsPdTp0HS218dODJQekkAv6dTd5RxSc
 FMEP1hpTefwhg==
Received: from icloud.com ([127.0.0.1]) by pv35p22im-ztdg05131101.me.com
 (Oracle Communications Messaging Server 7.0.5.38.0 64bit (built Feb 26 2016))
 with ESMTPSA id <0OPR00NJ85Z8KO40 at pv35p22im-ztdg05131101.me.com> for
 cippnews at cool.conservation-us.org; Wed, 10 May 2017 19:54:46 +0000 (GMT)
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,,
 definitions=2017-05-10_16:,, signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 clxscore=1034 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.0.1-1701120000 definitions=main-1705100136
From: XXXXXXXXXX <xxxxxxxx at me.com>
Content-type: text/plain; charset=utf-8
Content-transfer-encoding: quoted-printable
MIME-version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Date: Wed, 10 May 2017 16:54:44 -0300
References: <CACJwyVrjx-m1VZ=S68r5cV-diHV3VchfxQ2A6qhwDDK5JZDGHA at mail.gmail.com>
To: Conservators in Private Practice Discussion
 <cippnews at cool.conservation-us.org>
In-reply-to: <CACJwyVrjx-m1VZ=S68r5cV-diHV3VchfxQ2A6qhwDDK5JZDGHA at mail.gmail.com>
Message-id: <1442A6F7-ED43-4899-A0C9-403898F52F80 at me.com>
X-Mailer: Apple Mail (2.3273)
Subject: Re: [CIPPNEWS] closing private conservation studio
X-BeenThere: cippnews at cool.conservation-us.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Conservators in Private Practice Discussion
 <cippnews.cool.conservation-us.org>
List-Unsubscribe: <http://cool.conservation-us.org/mailman/options/cippnews>,
 <mailto:cippnews-request at cool.conservation-us.org?subject=unsubscribe>
List-Archive: <http://cool.conservation-us.org/mailman/private/cippnews/>
List-Post: <mailto:cippnews at cool.conservation-us.org>
List-Help: <mailto:cippnews-request at cool.conservation-us.org?subject=help>
List-Subscribe: <http://cool.conservation-us.org/mailman/listinfo/cippnews>,
 <mailto:cippnews-request at cool.conservation-us.org?subject=subscribe>
X-List-Received-Date: Wed, 10 May 2017 19:54:50 -0000

Send me list of what=E2=80=99s still available!
love
XXX



<----- start object 2 ----->
{   '_parsemsg': False,
    'add_header': {   'Reply-To': 'Conservators in Private Practice
Discussion <cippnews at cool.conservation-us.org>'},
    'from_is_list': 0,
    'lang': 'en',
    'listname': 'cippnews',
    'original_sender': 'xxxxxxxxx at me.com',
    'origsubj': 'Re: [CIPPNEWS] closing private conservation studio',
    'pipeline': [   'ToUsenet',
                    'AfterDelivery',
                    'Acknowledge',
                    'WrapMessage',
                    'ToOutgoing'],
    'received_time': 1494446090.4214,
    'recips': [  hundreds of recipients follow ],
'stripped_subject': <email.header.Header instance at 0x1ed6f80>,
    'tolist': 1,
    'version': 3,
    'whichq': '/usr/local/mailman/qfiles/archive'}
[----- end pickle file -----]



> The issue appears to be a bug in Mailman. but I need more info to be
> sure including the headers from one of the shunted messages.
>
> The exception is thrown when the archiver is trying to canonicalize the
> author's display name into "Last, First" form and I think the actual
> exception in this case is in checking if a "word" of the name begins
> with a lowercase letter. Since the word is already unidode in this case,
> it is apparently Python's string.lowercase that contains \xb5 in
> position 26.
>
> What do you get if you invoke Python interactively on this server and do
>
> import string
> string.lowercase
>
> I get 'abcdefghijklmnopqrstuvwxyz'
>

I get the same thing! Does that make any sense?

xxxx at yyyyy:/usr/local/mailman/logs# python
Python 2.7.5 (default, May 29 2013, 02:28:51)
[GCC 4.8.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import string
>>> string.lowercase
'abcdefghijklmnopqrstuvwxyz'


Thanks as always,
Matt

More information about the Mailman-Users mailing list