Executing a python script while it is running

[Dave Angel]

Zach Hobesh wrote:
>> A lot more information would be useful.  What version of Python, and what
>> operating system environment?  Exactly what would you like to happen when
>> the batch file is invoked a second time?
>>     
>
> I'm running Python 2.6.2 on Windows.  I'm passing filenames to the
> batch files and I need all filenames to be processed.  I can't have
> any fails.  I'm working on logging any fails I do have so that I can
> maybe batch process at the end of the day.
>
>   
>>  2) let them both run as separate processes
>>     
>
> This sounds like a good option, but I'm not totally sure on how to go
> about this?
>
>   
>>  4) queue something to be processed when the first run finishes
>>     
>
> I had the same idea, but I believe it would involve having another
> python script run all day long, which wouldn't necessarily be a bad
> thing, but I'd like to explore other options as well.
>
>   
>> What provisions does this existing application have for long-running batch
>> files?  Seems the synchronization ought to happen there.  Do you have any
>> constraints on how long your script might take, worst case?  What if the
>> application finishes its tasks at a faster average rate than your script can
>> process them?
>>     
>
> The batch file is moving large video files.  Duration probably ranges
> from 10 sec to 45 mins.  On average, the application takes longer to
> process the files than it does the batch file/python script takes to
> copy them, but I'm concerned about the occasional time that the
> application finishes a small file right after finishing a large file.
>
> Thanks for your response!
>
> -Zach
>
>   
Option 2 is what you get by default.  Naturally it depends on what the 
application  is using to launch the batch file, but the most common 
cases will launch a separate process.  That means most of python will 
work fine;  the only likely conflict you'll have is if one script is 
trying to move or copy the same file the other one is doing.  So I'd 
suggest you do a rename (which I believe is atomic on all the platforms) 
of the source file, then do the move.  Notice that if the destination is 
on the same physical volume, you can just "rename" it directly to the 
final location, at least on Windows.  Such a rename is essentially 
instantaneous, regardless of file size.

For option 4, you could just have the batch file launch a script that 
moves the file locally into a standard place, then the all-day-long 
(background) script will actually do the slow move on any file it spots 
in the special directory.  If you need more information, you could add 
also create a text file that identifies the other parameters to go with 
the file.  In any case, the background script would poll the directory 
for work to do, do it, then check again.  Any time the poll fails, just 
sleep for 30 seconds or so.