python program to watch a file - example.

[Bengt Richter]

On Sun, 8 Sep 2002 07:52:56 -0400, "Wayne R" <wayne at tbnets.com> wrote:

>Sorry for the html,  I forgot to turn it off (as I usually turn it off
>as soon as I install a new system, forgot this time).
>
>The platform is Gentoo Linux,  I am writing a piece of code for the
>Twiggi project to patch in adduser support.
>
>
>|-> > >I am trying to write a python program that will run as a deamon
>and
>|-> > >watch a file for data to arrive and then wake up and perform
>duties
>|-> on
>|-> > >that data.   I found ways to make a python program run as a
>deamon
>|-> but I
>|-> > >am at a loss to find anything related to watching a file for data
>so
>|-> > >that it can then process the data.  I didn't really want to do a
>|-> sleep
>|-> > >loop or something like, I am sure there is a better way.
>|-> > >
>|-> > >Any ideas or links to more info?
>|-> > >
>|-> > Don't annoy people you want help from, for starters.
>|-> > What system are you targeting? (Outlook in the headers says you're
>on
>|-> windows
>|-> > generating this abominable mix of actual message and useless
>cruft).
>|-> >
>[Wayne R] 
>
Sorry 'bout that. I don't very often express myself so testily, I hope.

Anyway, if you are going to have to read new data periodically, you could just
poll by doing it, I suppose. Unless you can control the writer or have access to
something like Isaac describes, you might almost as well poll, ugly as that is.

E.g., this seems to work, and I made it back off if there's errors due to
permissions etc. The question would be if open/seek/tell/seek/read/close
could cause an unacceptable error for some writer trying to get exclusive
access or such. But it should be more robust than that, probably.

If you control the source, it's a different ballgame, of course.
Anyway, try this on Linux (mine is slackware 2.2.16 on an old p90 with Python 1.5.2 ;-)

--< monf.py >--
#!/usr/bin/python
def mon(fname, firstpos, period):
    import os, time, string
    last_len = firstpos
    next_errp = 1
    ioerr_ctr = 0
    while 1:
        f = None; data = ''
        try:
            f = open(fname)
            f.seek(0,2) # to EOF
            told = f.tell()
            if told < last_len:
                print 'File shortened from %s to %s. Starting from 0.'%(
                    last_len, told
                )
                last_len = 0
            f.seek(last_len)
            data = f.read()
            ioerr_ctr = 0    # fresh sequence
            next_errp = 1
        except IOError, e:
            ioerr_ctr = ioerr_ctr + 1
            if ioerr_ctr >= next_errp:
                next_errp = next_errp*2
                print '%s IOErrors: %s' % (ioerr_ctr, e)
        if f is not None: f.close()
        if len(data):
            print '----< %s bytes of new data >----\n%s' % (
                    len(data), `data`
            )
            last_len = last_len + len(data)  # use actually read length
            print '----< end, now totaling %s bytes >---' % last_len
        time.sleep(period)

if __name__ == '__main__':
    import sys, string
    try:
        fname, firstpos, period = sys.argv[1:4]
        firstpos = int(firstpos)
        period = float(period)
        mon(fname, firstpos, period)
    except Exception, e:
        print '%s: %s' % (string.split(str(e.__class__),'.')[-1], e)
        print 'Usage monf.py filename firstpos(byte offset) period(seconds)'
---------------

Running it locally via telnet from windows NT, and messing with
the file directly on the Linux local terminal on the desk at my left:

bokr at springbok:~/junk$ monf.py junk.txt 0 5
1 IOErrors: [Errno 2] No such file or directory: 'junk.txt'
2 IOErrors: [Errno 2] No such file or directory: 'junk.txt'
4 IOErrors: [Errno 2] No such file or directory: 'junk.txt'
----< 16 bytes of new data >----
'A line of data.\012'
----< end, now totaling 16 bytes >---
----< 22 bytes of new data >----
'Another line of data.\012'
----< end, now totaling 38 bytes >---
File shortened from 38 to 18. Starting from 0.
----< 18 bytes of new data >----
'Replaced totally.\012'
----< end, now totaling 18 bytes >---
KeyboardInterrupt:
Usage monf.py filename firstpos(byte offset) period(seconds)
bokr at springbok:~/junk$

Here I'll start monitoring at position 10, and then chmod it -+r
from the other computer.

bokr at springbok:~/junk$ monf.py junk.txt 10 5
----< 8 bytes of new data >----
'otally.\012'
----< end, now totaling 18 bytes >---
----< 35 bytes of new data >----
'Will now chmod it -r then +r again\012'
----< end, now totaling 53 bytes >---
1 IOErrors: [Errno 13] Permission denied: 'junk.txt'
2 IOErrors: [Errno 13] Permission denied: 'junk.txt'
4 IOErrors: [Errno 13] Permission denied: 'junk.txt'
----< 38 bytes of new data >----
'Ok now add some data to make it read.\012'
----< end, now totaling 91 bytes >---
KeyboardInterrupt:
Usage monf.py filename firstpos(byte offset) period(seconds)
bokr at springbok:~/junk$

(I hit Ctrl-C via telnet to kill it).

This is just simple a hack, not tested beyond what you see. YMMV with
your environment. You can mod to suit your task, hopefully. It
would have been better to make a directory keyed by error strings
so unique info wouldn't get supressed in the total. Plus a time-tagged
log file would be better than a scrolling screen. Etc., etc., but
it was an experiment. You could also wath the file size with

    int(string.split(os.popen('ls -ls' % fname).read())[4]) # or something like that ;-)

but I imagine that's a fair bit more overhead than open/seek/tell.
Of course it would also give you a chance to monitor other aspects
of the file, or a directory. I guess if you had the privileges and
the time, you could possibly create a special file system that you
could symlink the file into and that would provide side effect hooks
to block on and read event-driven special data. Maybe that's what sgi did?

Regards,
Bengt Richter