[Tutor] My first real python project: LOLLERSKATES

[Tracy R Reed]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have been a sysadmin for quite a while now and used to do a lot of
perl. I haven't done much programming at all in the last couple of years
but have been meaning to dump perl for python anyhow (for the usual
reasons) and have finally gotten around to doing it.

The first itch that I wanted to scratch was something to help me find
anomalies in our system logs. We have a lot of servers and need some
automation for watching over them including logfiles. I have used
logwatch and logcheck but they are both too complicated for what I
really need not to mention practically unmaintained these days. So I
decided to write my own.

Features I wanted were:

1. A basic grep -v sort of functionality on the logfile where I have a
file full of regexes/lines to be ignored that are applied to the
logfiles to filter everything uninteresting out and whatever is left
gets emailed to the admin.

2. A sort of macro system where I can substitute nasty long commonly
used regexes in my filter/ignore file with something more easily read
and typed.

3. Simplicity. Logcheck had several different levels of logfile events
which it could report back on and if you wanted something ignored you
had to put it in the right config file for whatever level it was popping
up in. This drove me nuts because I often got the wrong one. Logwatch
tries to analyze your logs and provide various summaries and statistics
and other things I don't care about. I just want to see the interesting
lines from the logfile.

My code seems to have all of the above and a silly name as well:

http://ultraviolet.org/Members/treed/lollerskates

It has a list of logfiles to look at, a list of regexes to compare to
each line to know what is uninteresting and to filter out/ignore, and
emails anything left to the admin, and a simple macro facility.

I would appreciate it if anyone interested would download and critique
my code. Am I doing anything terribly un-pythonic?

Questions/problems/TODO's:

This is a fairly simple structured programming implementation. No OO.
Should I be using some classes somewhere?

The config file is just a module which I import which causes all of my
configs to become globals. Globals are bad. Is there a better way or
would just about anything else be overkill? A singleton config class or
something? Overkill?

I have several loops in this code for processing the logfiles. I once
tried to convert these for loops to list comprehensions and totally
confused myself and backed out the change (yeay svn!). Is there any
benefit to list comprehensions in this case?

I would kinda like to play with unit tests. Not sure how I would
construct unit tests for this. And again, perhaps overkill. But some
people tell me you should write the unit tests before you even begin
coding and code until the tests are satisfied. So even on a small
project you would have tests. I run into a couple nasty bugs I created
which caused the script to never return anything from the logfile (so
you don't immediately realize something is broken) where I thought "It
sure would be nice to have a test to catch that if it ever happens again."

Stuff tends to build up in the ignore file. If a line is typo'd and
never gets used/matched you will never know. If a service is no longer
running that we had lines in the ignore file for they will be there
forever unused. I don't like this sort of clutter. I am really tempted
to somehow make it keep track of when a rule is matched and if it
doesn't match in a whole month email the admin because it doesn't need
to be there. I am wondering if this is needless complexity or something
useful. I think it could be useful. If so, how to save the info on when
each line was last matched? I am thinking maybe of just having a
dictionary keyed on the compiled regex object and the value being a
tuple of the uncompiled regex string from the ignore file and a date
string containing when it was last matched and pickling this data
structure to disk to be read in each start of the program. Comments?

I am currently pushing this code to around 20 systems with cfengine
using just one ignore file for them all. If I use the scheme mentioned
in the previous paragraph to alert me of unused rules in the ignore file
 I will get tons of such messages from machines not running a service
that other machines of mine are. I guess that means perhaps I should
split my ignore file out into ignore.hostname.conf or something like
that so there is a separate one for each host.

Anyhow, I appreciate any feedback on this potentially very handy
sysadmin tool. Thanks in advance!

- --
Tracy R Reed
http://ultraviolet.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFFObPK9PIYKZYVAq0RAiflAJ9jprJgGnNRXkB+nKsljFUsAUGnFwCgiJg/
M3U24dNDtH+hAgSh7kZ40hQ=
=nxQS
-----END PGP SIGNATURE-----