Python deadlock using subprocess.popen and communicate

[Chris Rebert]

On Wed, Sep 21, 2011 at 8:09 PM, Atherun <atherun at gmail.com> wrote:
> This is on windows with python 2.6.
> I can't seem to remove a possibility of a  deadlock in one of my
> scripts at the moment.  Its not a constant deadlock but it appears
> from time to time.  The code is below:
>
> try:
>
>        process =
> subprocess.Popen(cmd,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
>
>        NotDone = True
>        data = []
>        while NotDone:
>            NotDone = False
>
>            result = process.poll()
>            if result == None:
>                NotDone = True
>            if NotDone:
>                out, err = process.communicate()
>                Log(out)
>
> I was able to get the program to write out the stack trace for my
> threads and it is usually deadlocked on sys.stdout.read or
> _internal_poll of subproceess.
>
> I've even tried the above, with using "with
> tempfile.NamedTemporaryFiles() as buff:" and writing to the file, but
> it still deadlocks.  In the past I work around this by running fewer
> processes asynchronously but I would really like to get this solved so
> I don't have to wait to see if it'll be caused with any changes I
> make.
>
> Any tips would be appreciated.

Your polling loop is completely pointless. The code can be simplified to:

process = subprocess.Popen(cmd,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
out, err = process.communicate()
log(out)

Depending on the particular program you're running in a subprocess,
you may need to take heed of the Note in the communicate() docs:
"Note: The data read is buffered in memory, so *do not use this
method* if the data size is large or unlimited." [Emphasis added]

Cheers,
Chris
--
http://rebertia.com