What are the Python tools to mine data from log files ?

[Barak, Ron]

Hi All,

I need to create log files (data) mining tool.
Are there any Python classes/modules/functions that are usually used in these cases, apart from "playing" with regular expressions ?

The logs comprise of lines like so:

Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x8:fcs:FcsGetVpdListCompletion[669] FCS Get VPD List: lport 3,  rport 10c00,  LunIdx 1 Unknown Cmpl [ 0 0 88 ]
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x8:fcs:FcsGetVpdListCompletion[672] Data: 88888888 88888888 88888888 88888888 88888888 88888888 88888888 88888888
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x8:fcs:FcsGetVpdListCompletion[669] FCS Get VPD List: lport 1,  rport 10400,  LunIdx 1 Unknown Cmpl [ 0 0 88 ]
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x8:fcs:FcsGetVpdListCompletion[672] Data: 88888888 88888888 88888888 88888888 88888888 88888888 88888888 88888888
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x10:fcs:FcsCallDiscoveryClientCallback[507] Client: fps   event: RportStChg  lport: 1  rport: 45a5abc8
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x10:fcs:FcsCallDiscoveryClientCallback[507] Client: fcpi  event: RportStChg  lport: 1  rport: 45a5abc8
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x10:fcs:FcsCallDiscoveryClientCallback[507] Client: css   event: RportStChg  lport: 1  rport: 45a5abc8
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x10:fcs:FcsCallDiscoveryClientCallback[507] Client: fcpt  event: RportStChg  lport: 1  rport: 45a5abc8
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x8:fcs:FcsGetVpdListCompletion[669] FCS Get VPD List: lport 1,  rport 10d00,  LunIdx 1 Unknown Cmpl [ 0 0 88 ]
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x8:fcs:FcsGetVpdListCompletion[672] Data: 88888888 88888888 88888888 88888888 88888888 88888888 88888888 88888888
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x10:fcs:FcsCallDiscoveryClientCallback[507] Client: fps   event: RportStChg  lport: 1  rport: 45a5a488
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x10:fcs:FcsCallDiscoveryClientCallback[507] Client: fcpi  event: RportStChg  lport: 1  rport: 45a5a488
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x10:fcs:FcsCallDiscoveryClientCallback[507] Client: css   event: RportStChg  lport: 1  rport: 45a5a488
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x10:fcs:FcsCallDiscoveryClientCallback[507] Client: fcpt  event: RportStChg  lport: 1  rport: 45a5a488
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x10:fcs:dscvr_dpc[471] Call Client [fps ]: event RportStChg,  rport: 22001111550002bf
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x10:fcs:dscvr_dpc[471] Call Client [fcpi]: event RportStChg,  rport: 22001111550002bf
Jul 31 09:53:13 iSW-00-091 kernel: T3=0x3a52b1e:3:0x0:fcpi:fcpi_handle_rport_event[1074] HBT:1/0/2. lport:01/010900 rpevt:RportStChg rport:22001111550002bf/010400 type:7 lucnt:1

And I need to be able to zero in on lines like:

Jul 31 09:56:13 ssp8400 kernel: Dump for [IB_FCP SEST]  xxx=0x1592
And then go forward/backwards in the log and get data related to the error message.

Thanks,
Ron.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20081109/35ed8e33/attachment.html>