Python deadlock using subprocess.popen and communicate
atherun at gmail.com
Thu Sep 22 14:19:28 EDT 2011
On Sep 22, 10:44 am, Nobody <nob... at nowhere.com> wrote:
> On Thu, 22 Sep 2011 08:55:40 -0700, Atherun wrote:
> >> Just handle process.stdout/stderr by yourself - read it out until EOF
> >> and then wait() for the process.
> > Thats what confuses me though, the documentation says
> > process.stdout.read()/stderr.read() can deadlock and apparently so can
> > communicate, how do you read the stdout/stderr on yourself if its
> > documented using them can cause a deadlock?
> If you try to read/write two or more of stdin/stdout/stderr via the
> "naive" approach, you run the risk of the child process writing more than
> a pipe's worth of data to one stream (and thus blocking) while the
> parent is performing a blocking read/write on another stream, resulting in
> The .communicate() method avoids the deadlock by either:
> 1. On Unix, using non-blocking I/O and select(), or
> 2. On Windows, creating a separate thread for each stream.
> Either way, the result is that it can always read/write whichever
> streams are ready, so the child will never block indefinitely while
> waiting for the parent.
> If .communicate() is blocking indefinitely, it suggests that the child
> process never terminates. There are many reasons why this might happen,
> and most of them depend upon exactly what the child process is doing.
> I suggest obtaining a copy of Process Explorer, and using it to
> investigate the state of both processes (but especially the child) at the
> point that the "deadlock" seems to occur.
In the one case I can easily reproduce, its in a p4.exe call that I'm
making both python and p4.exe have nearly the same stack for their
To me it looks like they're both waiting on each other.
More information about the Python-list