popen hangs until closed

[David Bolen]

"Mike Müller" <mmueller at dgfz.de> writes:

> As long as don't call popen.close() it hangs.

By "hangs" do you just mean that you wait to receive output that never
arrives?

> Is there any way to fix this from Python or is it a problem of the
> application not supporting pipes properly?

The latter, although it's not necessarily true that what the
application is doing is "improper".

What is most likely happening is that your application realizes (when
running beneath a pipe) that it is not directly connected to a console
session.  In many cases (for example, the C Run Time Library), that
implements buffered I/O on standard streams such as stdout for better
performance.  If the RTL detects a console on the output, it unbuffers
(or line buffers) it by default for better interactive behavior.

It's difficult to say without knowing the application involved as to
whether or not you can fix this without changing the application
source itself.  If, for example, the application is another Python
script, then you can use the "-u" option to Python to explicitly set
it to use unbuffered I/O regardless of whether or not it thinks there
is an interactive console.  If the application is a Perl script, then
there are several methods shown in the Perl docs to disable buffering
(such as adding the statement "select((select(STDOUT), $| = 1)[0]);"
to the top of the script).  For C RTL applications, something such as
"setbuf(stdout,NULL)" will do the trick.

Or, you can often explicitly flush your output at points in the
application just before input is required rather than disabling
buffering entirely.

But if you're working with an externally supplied executable that
doesn't provide a way to control buffering as an option and which you
can't rebuild, then you may be in trouble.  Unfortunately, since the
buffering choice is something being made by the application itself
internally (such as in the RTL), there's no standard way to impose
buffering instructions from a source external to the application (such
as your Python code).

Under Unix, ptys provide a simulation of a normal tty and can fool
applications into working as if they had a real interactive session
even though it is through a pipe (this is what 'expect' uses for
example), but there really isn't any equivalent under Windows.  I
believe I heard in the past that the Windows expect version does
somehow manage some of this, but I'm not sure.

--
-- David
-- 
/-----------------------------------------------------------------------\
 \               David Bolen            \   E-mail: db3l at fitlinxx.com  /
  |             FitLinxx, Inc.            \  Phone: (203) 708-5192    |
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150     \
\-----------------------------------------------------------------------/