[Python-Dev] Re: PEP 324: popen5 - New POSIX process module

[Josiah Carlson]

[Barry]
> When it does support windows please make it work the same on all platforms.
> The existing popen code for unix is buggy and not compatible with the
> windows version or the docs.

Agreed.

> The win32all extension is a thin wrapper over the windows API. The
> proposed popen5 code would simply be some windows specific code that
> calls windows API directly. There is no code in win32all that would
> be needed to be duplicated as far as I can see. Did I miss something?

I was not familliar with the structure of win32all.

My vote goes for 'add in the required C portions for windows support'.
And 'make all versions work the same'.

[Peter in regards to asyncore]
> Probably not. The description says:
> 
> "This module provides the basic infrastructure for writing asynchronous
> socket service clients and servers."
> 
> It's not obvious to me how this module could be use as a "shell backquote"  
> replacement (which is what communicate() is about). It's probably possible
> though; I haven't tried. Even if this is possible I guess we need some
> kind of "entry" or "wrapper" method in the popen module to simplify things
> for the user. My guess is that an communicate() method that uses asyncore
> would be as long/complicated as the current implementation. The current 
> implementation is only 68 lines, including comments. 

Using asyncore in *nix to do file IO doesn't seem to be that bad. I
tried to do the same thing on Windows (to get stdin, stdout, and stderr
to be non-blocking), but select doesn't work for file handles or pipes
in Windows.  If my memory serves me correctly, it was fairly trivial.

I've actually got a bit of other code for buffered IO on sockets that is
an asynchat work-alike, is around 70 lines, and uses asyncore.  Using
asynchat itself and setting a proper terminator would be easy, less than
20 lines (by my estimation).  You include all the logic of asyncore.poll
in popen5.Popen.communicate.  Using asyncore.poll may be to your benefit.

One thing to note is that string concatenation is not necessarily fast. 
That is: STR += DATA can be slow for initially large STR. Using a list
with list.append and ''.join(list) is significantly faster for large
buffers.  I have been contemplating submitting a patch for asynchat that
makes it behave better for large buffers.

The only issue that I can see in using asyncore or asynchat is that each
of {stdin, stdout, stderr} may need to have their own instance of
asyncore.dispatcher or asynchat.async_chat.  That would be pretty ugly.


I still like the idea of popen5, but I think you may want to at least
take a look at asyncore and friends.

 - Josiah