[ mailman-Patches-1123383 ] Daily Status Report script...

SourceForge.net noreply at sourceforge.net
Sun Nov 6 19:25:21 CET 2005


Patches item #1123383, was opened at 2005-02-15 12:14
Message generated for change (Comment added) made by msapiro
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=300103&aid=1123383&group_id=103

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: list administration
Group: Mailman 2.1
Status: Open
Resolution: None
Priority: 5
Submitted By: Brad Knowles (shub)
Assigned to: Nobody/Anonymous (nobody)
Summary: Daily Status Report script...

Initial Comment:
Folks,

I quickly whacked together a Daily Status Report script for 
Mailman (using Bourne shell, not Python ;), and thought that other 
folks might be interested in seeing it.

The basic concept is a program that gets fired off at 23:59 every 
night, and goes through a variety of log files looking for entries 
specific to that date, and indicating problems or certain types of 
activity that might be of interest to someone trying to administer 
the server.  It also does an "ls -la" of /usr/local/mailman/qfiles/*, 
so that you can see what is in the queue at the time of the running 
of the script.

My concept was that this daily report would get e-mailed to the 
admin, or posted to a "reports" mailing list, where they could be 
archived and kept for future reference.

The script does not (yet) do any statistics calculations, although it 
should be relatively easy to hack together some basic stats using 
awk, sort, etc....

Anyway, I thought I'd share it and let folks take a look at it, and if 
anyone has any recommended improvements, we can incorporate 
those and share them back out with everyone.

The code is written under a BSD-style license, so if you don't want 
to contribute any changes back to me, that's okay.  Of course, I 
would prefer that you did, but I leave the choice up to you.

----------------------------------------------------------------------

>Comment By: Mark Sapiro (msapiro)
Date: 2005-11-06 10:25

Message:
Logged In: YES 
user_id=1123998

The vette log summary lists posts held for moderation
individually under "Other Errors:". The following patch
(watch out for wrapped lines) summarizes them by list instead.

--- mmdsr	2005-09-29 21:31:33.000000000 -0700
+++ mmdsrx	2005-11-06 08:20:58.835895418 -0800
@@ -415,10 +415,15 @@
 	    echo "------------------------------" >> $TMP
 	    $GREP -i 'Posting to a moderated newsgroup' $TMPLOG |
$AWK '{ print $6 }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
 
+            echo "" >> $TMP
+            echo "Post to moderated list (by list):" >> $TMP
+            echo "------------------------------" >> $TMP
+            $GREP -i 'Post to moderated list' $TMPLOG |
$AWK '{ print $6 }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
 	    echo "" >> $TMP
 	    echo "Other Errors:" >> $TMP
 	    echo "------------------------------" >> $TMP
-	    $EGREP -vi '(Post by non-member|suspicious
header|message approved|Discarded posting|bulk message
discarded|junk message discarded|Message has implicit
destination|Posting to a moderated newsgroup|Message
discarded, msgid)' $TMPLOG | $SED 's/^.* ([0-9]*) //' |
$SORT | $UNIQ -c | $SORT -nr >> $TMP 
+	    $EGREP -vi '(Post by non-member|suspicious
header|message approved|Discarded posting|bulk message
discarded|junk message discarded|Message has implicit
destination|Posting to a moderated newsgroup|Post to
moderated list|Message discarded, msgid)' $TMPLOG | $SED
's/^.* ([0-9]*) //' | $SORT | $UNIQ -c | $SORT -nr >> $TMP 
 
 	else
 


----------------------------------------------------------------------

Comment By: Tom G. Christensen (tgc99)
Date: 2005-10-19 01:23

Message:
Logged In: YES 
user_id=1159458

ps output on solaris is full of whitespace but a further
echo get's rid of it.
The lines in the smtp log are sometimes broken up by a
newline (right before the msgid) which throws of the
summary. Piping it through sed first will rejoin the broken
lines.
Use $AWK instead of awk.

Patch inserted below:
--- mmdsr.orig  2005-10-19 09:42:30.000000000 +0200
+++ mmdsr       2005-10-19 09:44:23.000000000 +0200
@@ -203,7 +203,8 @@
 #      there is an easier cross-platform way to do it,
please let me know.
 ###############################################################################
 
-MYUID=`$PS -o user -p $$ | $TAIL -1`
+GRABUID=`$PS -o user -p $$ | $TAIL -1`
+MYUID=`echo $GRABUID`
 RUNAS="mailman"
 
 ###############################################################################
@@ -254,7 +255,7 @@
     $TOUCH $TMPLOG
     echo "Log file: $LOG" >> $TMP
     echo "==============================" >> $TMP
-    $GREP -si "^$DAY [0-9][0-9:]* $YEAR" $LOGDIR/$LOG >>
$TMPLOG
+    $SED -e :a -e '$!N;s/\n //;ta' -e 'P;D' $LOGDIR/$LOG |
$GREP -si "^$DAY [0-9][0-9:]* $YEAR" >> $TMPLOG
 
     if [ -f "$LOGDIR/${LOG}" ] ; then
 
@@ -264,7 +265,7 @@
            echo "Hourly Summary of Posts" >> $TMP
            echo "-----------------------" >> $TMP
 
-           $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e
's/:.*$//' $TMPLOG | $UNIQ -c | $SORT -n +1 | awk '{ printf(
"%8d  %02d:00-%02d
:59\n", $1, $2, $2 ) }' >> $TMP
+           $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e
's/:.*$//' $TMPLOG | $UNIQ -c | $SORT -n +1 | $AWK '{
printf( "%8d  %02d:00-%02
d:59\n", $1, $2, $2 ) }' >> $TMP
 
            echo "" >> $TMP
            echo "Post Count by List" >> $TMP
@@ -295,7 +296,7 @@
            echo "" >> $TMP
            echo "Hourly Summary of Messages Sent" >> $TMP
            echo "-------------------------------" >> $TMP
-           $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e
's/:.* for / /' -e 's/ recips,.*$//' $TMPLOG | awk '{
val=int($1); sum[val]+=$2
 } END { for (i=0; i<24; i++) { printf "%8d
%02d:00-%02d:59\n", sum[i], i, i } }' >> $TMP
+           $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e
's/:.* for / /' -e 's/ recips,.*$//' $TMPLOG | $AWK '{
val=int($1); sum[val]+=$
2 } END { for (i=0; i<24; i++) { printf "%8d
%02d:00-%02d:59\n", sum[i], i, i } }' >> $TMP
 
        else


----------------------------------------------------------------------

Comment By: Brad Knowles (shub)
Date: 2005-09-22 16:17

Message:
Logged In: YES 
user_id=18417

Okay, I took Mark's comments and incorporated them.  Adrian Wells also 
pointed out a log file difference between Mailman 2.1.5 and 2.1.6 that 
caused the summary of the "smtp" log to be munged.

I've now fixed these bugs, deleted the old file, and uploaded the new one 
(version 0.0.12).

Thanks!

----------------------------------------------------------------------

Comment By: Mark Sapiro (msapiro)
Date: 2005-09-10 19:22

Message:
Logged In: YES 
user_id=1123998

Oooops!  I mistakenly thought I could add the mmdsr.patch as
a downloadable file, but I can't so here it is. Watch out
for wrapped lines ...

--- mmdsr.orig	2005-09-06 20:37:53.000000000 -0700
+++ mmdsr	2005-09-10 18:33:14.532393572 -0700
@@ -132,6 +132,7 @@
 
 MMDIR="/usr/local/mailman"
 TMPDIR="/tmp"
+LOGDIR="/var/log/mailman"
 
 ###############################################################################
 # Maximum number of subdirectory entries to display in report
@@ -234,9 +235,9 @@
     $TOUCH $TMPLOG
     echo "Log file: $LOG" >> $TMP
     echo "==============================" >> $TMP
-    $GREP -si "^$DAY [0-9][0-9:]* $YEAR" logs/$LOG >> $TMPLOG
+    $GREP -si "^$DAY [0-9][0-9:]* $YEAR" $LOGDIR/$LOG >>
$TMPLOG
 
-    if [ -f "logs/${LOG}" ] ; then
+    if [ -f "$LOGDIR/${LOG}" ] ; then
 
 	if [ "${LOG}" = "post" ] ; then
 
@@ -304,9 +305,9 @@
     $TOUCH $TMPLOG
     echo "Log file: $LOG" >> $TMP
     echo "==============================" >> $TMP
-    $GREP -si "^$DAY [0-9][0-9:]* $YEAR" logs/$LOG >> $TMPLOG
+    $GREP -si "^$DAY [0-9][0-9:]* $YEAR" $LOGDIR/$LOG >>
$TMPLOG
 
-    if [ -f "logs/${LOG}" ] ; then
+    if [ -f "$LOGDIR/${LOG}" ] ; then
 
 	if [ "${LOG}" = "error" ] ; then
 
@@ -453,4 +454,4 @@
     $CAT $TMP
 fi
 
-$RM $TMP
+$RM -f $TMP $TMPLOG


----------------------------------------------------------------------

Comment By: Mark Sapiro (msapiro)
Date: 2005-09-10 19:16

Message:
Logged In: YES 
user_id=1123998

I tried the script and other than the expected site specific
changes, I found two issues which are both addressed by the
patch in the mmdsr.patch file.

1) Mailman's log files are not necessarily in
$var_prefix/logs/ - they can be in any directory. The patch
adds a LOGDIR directory independant of MMDIR.

2) The $TMPLOG file is not removed at completion. The patch
removes it.

----------------------------------------------------------------------

Comment By: Brad Knowles (shub)
Date: 2005-09-06 07:06

Message:
Logged In: YES 
user_id=18417

Okay, I've deleted the old version of the mmdsr script that was attached, 
and uploaded the latest version.  This is what we're currently using to 
monitor the lists on python.org, and we have found it very useful.

Any comments you may have will be appreciated.

----------------------------------------------------------------------

Comment By: Brad Knowles (shub)
Date: 2005-02-22 12:10

Message:
Logged In: YES 
user_id=18417

The UID variable in the current code was already replaced by MYUID, 
because I got complaints on other platforms.  But UID wasn't available to 
me as a useful constant, so I had to use something else to obtain the 
value.

The recommended patch from tgc99 does work, and I will be uploading a 
new version of the code soon.

----------------------------------------------------------------------

Comment By: adrianwi (adrianwi)
Date: 2005-02-22 07:22

Message:
Logged In: YES 
user_id=1175103

Use of variable named UID does work well with OS X (version
10.2.8).  Apparently the variable UID is a constant already
in use.  When trying to the run the script without
modification, I was receiving the following error message:

UID: readonly variable

This issue was resolved by changing the name of variable,
UID, to something else, such as MMUID.  Works fine with this
change.

As an aside (& for what it is worth), the UID grab command
suggested by tgc99 on 2005-02-16 03:15 works on this system
(OS X - version 10.2.8)

----------------------------------------------------------------------

Comment By: Tom G. Christensen (tgc99)
Date: 2005-02-16 00:15

Message:
Logged In: YES 
user_id=1159458

The current UID grab command doesn't work on Solaris (2.6 &
8 tested).

I'd recommend this instead:
ps -o user -p $$|tail -1

This is tested and works on RH 6.2, RH 7.3, RHEL 2.1, RHEL3,
FC3, FreeBSD 4.9, Solaris 2.6, 8.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=300103&aid=1123383&group_id=103


More information about the Mailman-coders mailing list