Help replacing os.system call with subprocess call

[David Pratt]

Hi David and Matt. I appreciate your help which has got me moving 
forward again so many thanks for your reply. I have been using 
subprocess.Popen a fair bit but this was the first time I had to use 
subprocess to capture large file output. The trouble I was having was 
with the process would just hang. Chunking was the solution. I guess I 
assumed this would be taken care of in the internals.

Overall, I wish subprocess had some better documentation since it is 
definitely not a drop in replacement for os.system. In other 
circumstances I am using subprocess.call() for simple calls which works 
fine.

The speed of this solution is slower than os.system. Would a queue of 
some kind be needed to speed this up? Has anyone implemented something 
like this? Many thanks.

Regards,
David

Matt Nordhoff wrote:
> David Pratt wrote:
>> Hi. I am trying to replace a system call with a subprocess call. I have 
>> tried subprocess.Popen and subprocess.call with but have not been 
>> successful. The command line would be:
>>
>> svnadmin dump /my/repository > svndump.db
>>
>> This is what I am using currently:
>>
>> os.system('svnadmin dump %s > %s' % (svn_dir,
>>      os.path.join(backup_dir, 'svndump.db')))
>>
>> Many thanks.
> 
> Try this:
> 
> import os.path
> import subprocess
> 
> p = subprocess.Popen(
>     ['svnadmin', 'dump', svndir],
>     stdout=subprocess.PIPE,
> )
> 
> fh = open(os.path.join(backup_dir, 'svndump.db'), 'wb')
> 
> while True:
>     chunk = p.stdout.read(2**20) # 1 MB
>     if not chunk:
>         break
>     fh.write(chunk)
> 
> fh.close()
> 
> It reads svnadmin's stdout in 1 MB chunks, in case it's large enough
> that reading the whole thing into RAM at once would be a bad idea.
> 
> No error handling. For one, you might want to add a try...finally to
> ensure that fh will get closed. (Or if you have Python 2.5, use a with
> statement! :-) ) Also, Popen will raise an OSError if svnadmin can't be
> found or something. And this isn't even considering svnadmin erroring out...
> 
> svnadmin's stderr will go to your stderr.
> 
> I didn't test it, but I'm pretty sure it will work. (I spotted a syntax
> error while writing that though.) I don't have much experience with
> Popen's stdio objects, so it's possible you'd need to do something like
> call p.wait() to wait for it to exit before being able to read its stdout.
> 
> It could be slower than the os.system version, since now Python is doing
> all of the I/O, instead of your shell, but I doubt that'll be a big problem.
> 
> (Also, insert suggestion about using a good VCS. ;-) )