Reading a file that is changing and getting the new lines

[Dave Angel]

Ouray Viney wrote:
> Hi:
>
> Problem:
> =========
> I want to read a ASCII text file that can have data appended to it.  I
> have hacked some code together that handles the basics, but it falls
> short.  My code doesn't read in the new lines that could have been
> added to the end of the file.  Not python's fault, more that I don't
> know how to do it in python.
>
> Example scenario:  As the python script is running a user/application
> adds new entries to the end of the test case file, example, adds the
> following to the file.
>
> test4
> test5
> test6
>
>   
I need clarification.  Is this an independent process, that appends data 
onto the same file, after the python script has opened it, but while 
it's still running?  Can you give me an example point in your source 
code that your script might be running when the other application does 
its append?
> The python code below, won't pick these changes up.
>
> Analysis:
> ========
> I understand why my code doesn't handle the addition of new data.
> This is simply because the os.open reads all the lines in the
>   
os.open() doesn't read the lines, the readlines() call does.  Once 
you've got the data in a list, of course changing the file has no impact 
on your code.  And by the way, you didn't call os.open(), you called 
open().  They are not interchangeable.
> beginning and since the data is added after that, the for loop doesn't
> iterate through it since it doesn't know about it.  Please note, my
> code is not pretty and isn't at all advanced.
>
> Code:
> ======
> import os,string,sys,time
>
> # define locals
> tcListFile="testcase_list.txt"
> fh=open(tcListFile,"r")
> tcList=fh.readlines()
> fh.close()
>
> for tc in tcList:
> 	print tc.strip()
> 	print "sleeping for 2 seconds"
> 	time.sleep(2)
>
> # close the file handle
> fh.close()
>
> text file:
> ======
> test1
> test2
> test3
>
> Question:
> =========
> What is the best way to handle this type of scenario using python?
>   
You can solve your first level of problem by simply skipping the 
readlines() call, and eliminating the first close() call.  You close it 
again later, let that be the only close.  You want your loop to actually 
get a line from the file, not from a list which is frozen in time.  
Naturally, you'd want a better name than tcList.

Next problem is likely to be one of sharing the file.  You open it 
read-only, the other application simultaneously tries to append.  Does 
the OS permit that?  You find out.  You might actually have to close the 
file during those sleep() calls, and then do an open/seek to get to the 
earlier position.  You most likely will have to switch from open() to 
os.open().  The calls are very different.

Final problem that I note is how to decide when to quit your loop.  Just 
because you hit eof (end of file), doesn't mean the other application 
isn't about to append another 50 lines.  So you need a different 
termination condition.

I suspect the sharing problems aren't going to be portable between 
operating systems, so you might want to answer the usual "what version 
of Python, on what OS" question that we seem to have for most new questions.

DaveA