[Mailman-Users] Counting messages that went to postfix queue

[D G Teed]

Hi,

Thanks for the great replies and help.

I found a perl script "findsender.pl" with google and it seems to be
useful. I seem to have missing mail list members from my maillog,
and one python error from mailman.

Here is what I did to verify non-mailing...

list_members maillist > /tmp/listnames
for person in `cat  /tmp/listnames`; do  ./findsender.pl $person maillog |
grep 'Found 0'; done
{returns 28 email addresses}

I didn't trust that findsender.pl actually parses postfix logs 100% so I
verified
with plain grep for any mention of the user addresses in /var/log/maillog or
/var/log/mailman/*    I found one reference of the 28 which findsender.plmissed.
This list has 1890 members, and 27 are not showing up at all in postfix
logs.

Unless postfix has a bug where some emails are not being logged, there seems
to be a problem with mailman.  I'm not looking for delivery - just any
reference
to the expected attempts to deliver.

I do however see an error in /var/log/mailman which matches the time of
this mass mailing:

Apr 30 15:43:40 2007 (1877) Uncaught runner exception: No terminating
boundary and no trailing empty line
Apr 30 15:43:40 2007 (1877) Traceback (most recent call last):
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 99, in _oneloop
    msg, msgdata = self._switchboard.dequeue(filebase)
  File "/usr/lib/mailman/Mailman/Queue/Switchboard.py", line 151, in dequeue
    msg = email.message_from_string(msg, Message.Message)
  File "/usr/lib/mailman/pythonlib/email/__init__.py", line 52, in
message_from_string
    return Parser(_class, strict=strict).parsestr(s)
  File "/usr/lib/mailman/pythonlib/email/Parser.py", line 75, in parsestr
    return self.parse(StringIO(text), headersonly=headersonly)
  File "/usr/lib/mailman/pythonlib/email/Parser.py", line 64, in parse
    self._parsebody(root, fp, firstbodyline)
  File "/usr/lib/mailman/pythonlib/email/Parser.py", line 240, in _parsebody
    msgobj = self.parsestr(part)
  File "/usr/lib/mailman/pythonlib/email/Parser.py", line 75, in parsestr
    return self.parse(StringIO(text), headersonly=headersonly)
  File "/usr/lib/mailman/pythonlib/email/Parser.py", line 64, in parse
    self._parsebody(root, fp, firstbodyline)
  File "/usr/lib/mailman/pythonlib/email/Parser.py", line 265, in _parsebody
    msg = self.parse(fp)
  File "/usr/lib/mailman/pythonlib/email/Parser.py", line 64, in parse
    self._parsebody(root, fp, firstbodyline)
  File "/usr/lib/mailman/pythonlib/email/Parser.py", line 206, in _parsebody
    raise Errors.BoundaryError(
BoundaryError: No terminating boundary and no trailing empty line

Apr 30 15:43:40 2007 (1877) Ignoring unparseable message:
1177958619.246433+b72521b62f4a6aa7dc74c339613c4ef7e9c3eb
a7

There are no errors like it in our previous months of using mailman
on other smaller departmental mailing lists.  The mailman smtp log does not
show
the entry for the 1800 something messages which were delivered OK.

My hunch, is that there is some bad data in our mailing list subscription
which wasn't caught anywhere and has created this issue.  We are given
data from the Alumni Affairs department to inject into the mailing list.
It may contain odd things.  I've seen a '#' and single quote appear in
the mail subscriptions.

Regards,

--Donald


On 5/2/07, Brad Knowles <brad at shub-internet.org> wrote:
>
> On 5/1/07, D G Teed wrote:
>
> >  I've tried finding the message queue IDs which were used
> >  and grep QueueID maillog | wc but this seems difficult
> >  to do since some reappear again in the logs, which were initially
> >  bounces.
>
> Mailman will give you the message-id, the list name, and the number
> of recipients it was sent to.  However, this won't help you for
> digest recipients, since Mailman will have to generate a different
> and totally unique message-id for that.  In addition, it will use the
> same message-id on outbound as was used on inbound, so unless you
> have different copies of your MTA running on different ports and
> sending their logs to different locations, you're going to get a mix
> of both inbound and outbound activity in your log files.  This will
> make it more difficult to track down what happened when.
>
> Now, postfix will give you a one-to-many mapping of the message-id
> and the queue-ids, and then you can track the status of the
> individual queue-ids.
>
> >  I also tried pflogsumm and looked at those stats.  At first I thought
> >  the message sizes and multiples of that message size going to
> >  different domains would be a key, but it seems to be inaccurate.
>
> Speaking as the author of the "mmdsr" script, and as someone who has
> hacked around with the pflogsumm script, I can tell you that this is
> a process that is harder to do than most people would give credit for.
>
> Beyond that, I'm not sure I can give you any specific advice.  I've
> been looking at installing Splunk on the python.org machines so that
> we can more easily track down what happened when to which mail
> messages, but right now it doesn't appear to know anything about
> Mailman log format.  I don't mind teaching it about that kind of
> stuff, but that does mean that we're that much further away from
> having any kind of short-term solution for this problem.
>
> >  I don't have any indications that anyone in our pilot list was lost,
> >  but we just need a method to account for the reliability.
>
> Mailman doesn't currently have much for you in this area.  We know
> that this is an area of weakness, and are planning on fixing this in
> upcoming versions.  However, we do not otherwise have a whole lot of
> details as to what will be going on when.
>
> If you have specific requirements, I would urge you to join the
> discussions on this topic in the Mailman Wiki pages.
>
> --
> Brad Knowles <brad at shub-internet.org>, Consultant & Author
> LinkedIn Profile: <http://tinyurl.com/y8kpxu>
> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
>