Python deadlock using subprocess.popen and communicate

[Roy Smith]

In article 
<098f3d78-85f5-44e7-ba72-f2270a24d1b0 at o9g2000vbo.googlegroups.com>,
 Atherun <atherun at gmail.com> wrote:

> This is on windows with python 2.6.
> I can't seem to remove a possibility of a  deadlock in one of my
> scripts at the moment.  Its not a constant deadlock but it appears
> from time to time.  The code is below:
> 
> try:
> 
>         process =
> subprocess.Popen(cmd,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
> 
>         NotDone = True
>         data = []
>         while NotDone:
>             NotDone = False
> 
>             result = process.poll()
>             if result == None:
>                 NotDone = True
>             if NotDone:
>                 out, err = process.communicate()
>                 Log(out)
> 
> I was able to get the program to write out the stack trace for my
> threads and it is usually deadlocked on sys.stdout.read or
> _internal_poll of subproceess.
> 
> I've even tried the above, with using "with
> tempfile.NamedTemporaryFiles() as buff:" and writing to the file, but
> it still deadlocks.  In the past I work around this by running fewer
> processes asynchronously but I would really like to get this solved so
> I don't have to wait to see if it'll be caused with any changes I
> make.
> 
> Any tips would be appreciated.

My reading of the docs 
(http://docs.python.org/release/2.6.7/library/subprocess.html#popen-objec
ts) says that Popen.poll() doesn't return a value, it sets the object's 
return code attribute, which you can then interrogate.

More than that, your loop logic looks amazingly complicated for what's 
basically a simple thing.  I suggest something along the lines of:

   # assuming process.returncode is initially None 
   while process.returncode is None:
      out, err = process.communicate()
      Log(out)

I haven't tested that, but I think (from reading the docs) that's the 
right idea.