Patches item #1123383, was opened at 2005-02-15 20:14 Message generated for change (Comment added) made by shub You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=1123383&group_id=103 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: list administration Group: Mailman 2.1 Status: Open Resolution: None Priority: 5 Submitted By: Brad Knowles (shub) Assigned to: Nobody/Anonymous (nobody) Summary: Daily Status Report script... Initial Comment: Folks, I quickly whacked together a Daily Status Report script for Mailman (using Bourne shell, not Python ;), and thought that other folks might be interested in seeing it. The basic concept is a program that gets fired off at 23:59 every night, and goes through a variety of log files looking for entries specific to that date, and indicating problems or certain types of activity that might be of interest to someone trying to administer the server. It also does an "ls -la" of /usr/local/mailman/qfiles/*, so that you can see what is in the queue at the time of the running of the script. My concept was that this daily report would get e-mailed to the admin, or posted to a "reports" mailing list, where they could be archived and kept for future reference. The script does not (yet) do any statistics calculations, although it should be relatively easy to hack together some basic stats using awk, sort, etc.... Anyway, I thought I'd share it and let folks take a look at it, and if anyone has any recommended improvements, we can incorporate those and share them back out with everyone. The code is written under a BSD-style license, so if you don't want to contribute any changes back to me, that's okay. Of course, I would prefer that you did, but I leave the choice up to you. ----------------------------------------------------------------------

Comment By: Brad Knowles (shub) Date: 2006-01-26 01:55

Message: Logged In: YES user_id=18417 Folks, Okay, I just uploaded version 0.0.15, which addresses the most recent comments here from Mark and Tom, and makes a few additional tweaks as well. See the version comments at the top of the script. ---------------------------------------------------------------------- Comment By: Mark Sapiro (msapiro) Date: 2005-11-06 18:25 Message: Logged In: YES user_id=1123998 The vette log summary lists posts held for moderation individually under "Other Errors:". The following patch (watch out for wrapped lines) summarizes them by list instead. --- mmdsr 2005-09-29 21:31:33.000000000 -0700 +++ mmdsrx 2005-11-06 08:20:58.835895418 -0800 @@ -415,10 +415,15 @@ echo "------------------------------" >> $TMP $GREP -i 'Posting to a moderated newsgroup' $TMPLOG | $AWK '{ print $6 }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP + echo "" >> $TMP + echo "Post to moderated list (by list):" >> $TMP + echo "------------------------------" >> $TMP + $GREP -i 'Post to moderated list' $TMPLOG | $AWK '{ print $6 }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP + echo "" >> $TMP echo "Other Errors:" >> $TMP echo "------------------------------" >> $TMP - $EGREP -vi '(Post by non-member|suspicious header|message approved|Discarded posting|bulk message discarded|junk message discarded|Message has implicit destination|Posting to a moderated newsgroup|Message discarded, msgid)' $TMPLOG | $SED 's/^.* ([0-9]*) //' | $SORT | $UNIQ -c | $SORT -nr >> $TMP + $EGREP -vi '(Post by non-member|suspicious header|message approved|Discarded posting|bulk message discarded|junk message discarded|Message has implicit destination|Posting to a moderated newsgroup|Post to moderated list|Message discarded, msgid)' $TMPLOG | $SED 's/^.* ([0-9]*) //' | $SORT | $UNIQ -c | $SORT -nr >> $TMP else ---------------------------------------------------------------------- Comment By: Tom G. Christensen (tgc99) Date: 2005-10-19 08:23 Message: Logged In: YES user_id=1159458 ps output on solaris is full of whitespace but a further echo get's rid of it. The lines in the smtp log are sometimes broken up by a newline (right before the msgid) which throws of the summary. Piping it through sed first will rejoin the broken lines. Use $AWK instead of awk. Patch inserted below: --- mmdsr.orig 2005-10-19 09:42:30.000000000 +0200 +++ mmdsr 2005-10-19 09:44:23.000000000 +0200 @@ -203,7 +203,8 @@ # there is an easier cross-platform way to do it, please let me know. ############################################################################### -MYUID=`$PS -o user -p $$ | $TAIL -1` +GRABUID=`$PS -o user -p $$ | $TAIL -1` +MYUID=`echo $GRABUID` RUNAS="mailman" ############################################################################### @@ -254,7 +255,7 @@ $TOUCH $TMPLOG echo "Log file: $LOG" >> $TMP echo "==============================" >> $TMP - $GREP -si "^$DAY [0-9][0-9:]* $YEAR" $LOGDIR/$LOG >> $TMPLOG + $SED -e :a -e '$!N;s/\n //;ta' -e 'P;D' $LOGDIR/$LOG | $GREP -si "^$DAY [0-9][0-9:]* $YEAR" >> $TMPLOG if [ -f "$LOGDIR/${LOG}" ] ; then @@ -264,7 +265,7 @@ echo "Hourly Summary of Posts" >> $TMP echo "-----------------------" >> $TMP - $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.*$//' $TMPLOG | $UNIQ -c | $SORT -n +1 | awk '{ printf( "%8d %02d:00-%02d :59\n", $1, $2, $2 ) }' >> $TMP + $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.*$//' $TMPLOG | $UNIQ -c | $SORT -n +1 | $AWK '{ printf( "%8d %02d:00-%02 d:59\n", $1, $2, $2 ) }' >> $TMP echo "" >> $TMP echo "Post Count by List" >> $TMP @@ -295,7 +296,7 @@ echo "" >> $TMP echo "Hourly Summary of Messages Sent" >> $TMP echo "-------------------------------" >> $TMP - $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.* for / /' -e 's/ recips,.*$//' $TMPLOG | awk '{ val=int($1); sum[val]+=$2 } END { for (i=0; i<24; i++) { printf "%8d %02d:00-%02d:59\n", sum[i], i, i } }' >> $TMP + $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.* for / /' -e 's/ recips,.*$//' $TMPLOG | $AWK '{ val=int($1); sum[val]+=$ 2 } END { for (i=0; i<24; i++) { printf "%8d %02d:00-%02d:59\n", sum[i], i, i } }' >> $TMP else ---------------------------------------------------------------------- Comment By: Brad Knowles (shub) Date: 2005-09-22 23:17 Message: Logged In: YES user_id=18417 Okay, I took Mark's comments and incorporated them. Adrian Wells also pointed out a log file difference between Mailman 2.1.5 and 2.1.6 that caused the summary of the "smtp" log to be munged. I've now fixed these bugs, deleted the old file, and uploaded the new one (version 0.0.12). Thanks! ---------------------------------------------------------------------- Comment By: Mark Sapiro (msapiro) Date: 2005-09-11 02:22 Message: Logged In: YES user_id=1123998 Oooops! I mistakenly thought I could add the mmdsr.patch as a downloadable file, but I can't so here it is. Watch out for wrapped lines ... --- mmdsr.orig 2005-09-06 20:37:53.000000000 -0700 +++ mmdsr 2005-09-10 18:33:14.532393572 -0700 @@ -132,6 +132,7 @@ MMDIR="/usr/local/mailman" TMPDIR="/tmp" +LOGDIR="/var/log/mailman" ############################################################################### # Maximum number of subdirectory entries to display in report @@ -234,9 +235,9 @@ $TOUCH $TMPLOG echo "Log file: $LOG" >> $TMP echo "==============================" >> $TMP - $GREP -si "^$DAY [0-9][0-9:]* $YEAR" logs/$LOG >> $TMPLOG + $GREP -si "^$DAY [0-9][0-9:]* $YEAR" $LOGDIR/$LOG >> $TMPLOG - if [ -f "logs/${LOG}" ] ; then + if [ -f "$LOGDIR/${LOG}" ] ; then if [ "${LOG}" = "post" ] ; then @@ -304,9 +305,9 @@ $TOUCH $TMPLOG echo "Log file: $LOG" >> $TMP echo "==============================" >> $TMP - $GREP -si "^$DAY [0-9][0-9:]* $YEAR" logs/$LOG >> $TMPLOG + $GREP -si "^$DAY [0-9][0-9:]* $YEAR" $LOGDIR/$LOG >> $TMPLOG - if [ -f "logs/${LOG}" ] ; then + if [ -f "$LOGDIR/${LOG}" ] ; then if [ "${LOG}" = "error" ] ; then @@ -453,4 +454,4 @@ $CAT $TMP fi -$RM $TMP +$RM -f $TMP $TMPLOG ---------------------------------------------------------------------- Comment By: Mark Sapiro (msapiro) Date: 2005-09-11 02:16 Message: Logged In: YES user_id=1123998 I tried the script and other than the expected site specific changes, I found two issues which are both addressed by the patch in the mmdsr.patch file. 1) Mailman's log files are not necessarily in $var_prefix/logs/ - they can be in any directory. The patch adds a LOGDIR directory independant of MMDIR. 2) The $TMPLOG file is not removed at completion. The patch removes it. ---------------------------------------------------------------------- Comment By: Brad Knowles (shub) Date: 2005-09-06 14:06 Message: Logged In: YES user_id=18417 Okay, I've deleted the old version of the mmdsr script that was attached, and uploaded the latest version. This is what we're currently using to monitor the lists on python.org, and we have found it very useful. Any comments you may have will be appreciated. ---------------------------------------------------------------------- Comment By: Brad Knowles (shub) Date: 2005-02-22 20:10 Message: Logged In: YES user_id=18417 The UID variable in the current code was already replaced by MYUID, because I got complaints on other platforms. But UID wasn't available to me as a useful constant, so I had to use something else to obtain the value. The recommended patch from tgc99 does work, and I will be uploading a new version of the code soon. ---------------------------------------------------------------------- Comment By: adrianwi (adrianwi) Date: 2005-02-22 15:22 Message: Logged In: YES user_id=1175103 Use of variable named UID does work well with OS X (version 10.2.8). Apparently the variable UID is a constant already in use. When trying to the run the script without modification, I was receiving the following error message: UID: readonly variable This issue was resolved by changing the name of variable, UID, to something else, such as MMUID. Works fine with this change. As an aside (& for what it is worth), the UID grab command suggested by tgc99 on 2005-02-16 03:15 works on this system (OS X - version 10.2.8) ---------------------------------------------------------------------- Comment By: Tom G. Christensen (tgc99) Date: 2005-02-16 08:15 Message: Logged In: YES user_id=1159458 The current UID grab command doesn't work on Solaris (2.6 & 8 tested). I'd recommend this instead: ps -o user -p $$|tail -1 This is tested and works on RH 6.2, RH 7.3, RHEL 2.1, RHEL3, FC3, FreeBSD 4.9, Solaris 2.6, 8. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=1123383&group_id=103