[Python-Dev] Addition of "pyprocessing" module to standard lib.

[r.m.oudkerk]

2008/5/14 "Martin v. Löwis" <martin at v.loewis.de>:
>> I really do feel that inclusion of this library offers us the best of
>> both worlds - it gives us (as a community) an easy answer to those
>> people who would dismiss python due to the GIL and it also allows
>> users to easily implement their applications.

I am the author of processing -- sorry for arriving late in this thread.

> For inclusion into Python, a number of things need to be changed,
> in particular with respect to the C code. It duplicates a lot of
> code from the socket and os modules, and should (IMO) be merged into
> these, rather than being separate - or perhaps merged into the
> subprocess module.

Now that socket.fromfd() and socket._dup() is available on Windows in
2.6 and 3.0 (after a patch from me) some of the code could be removed.
I would also like to see recvfd() and sendfd() get added to the socket
module -- these functions allow the transfer of file descriptors
between processes over a Unix domain socket.

But I am not sure that there is much duplication with the socket
module.  The Connection type is basically a message oriented version
of a socket with builtin pickling/unpickling -- it could be
implemented in pure Python but that would *much* slower.  I cannot see
any opportunity to reuse code from socketmodule.c in implementing the
Connection type, particularly given that the code also allows the use
of Windows named pipes which are much faster than sockets on Windows.

As far as I know there is no duplication with the os modules.

Basically the C code separates as follows

(1) a Connection type (plus PipeConnection on windows)
(2) a "process shared" lock/semaphore type
(3) Win32 functions/constants to allow use of named pipes
(4) A few other Win32 functions/constants
(5) A wrapper making PyObject_AsWriteBuffer() available from Python

I suppose (4) and perhaps (3) could sensibly be added/merged with
_subprocess.c.  (5) could also be moved somewhere else.

> Why isn't it possible to implement all this in pure Python, on
> top of the subprocess module?

I am not sure how much you include when you say "all this" -- (2)
certainly cannot be done in pure python, and I certainly believe that
(1) needs to be done in C for the sake of performance.

On Windows I did use subprocess.py in earlier versions, but I had bug
reports saying that in non-console programs it would choke because it
was failing to duplicate GetStdHandle(STD_OUTPUT_HANDLE) etc.  Using
_subprocess.CreateProcess() directly avoided the problem and did not
add noticeably to code length.  I also found that the Popen.__del__()
methods kept raising errors on process shutdown because of unavailable
globals, so I was happy to get shot of subprocess.py.

(Off-topic but I think that the way that subprocess.Popen.__del__()
keeps the object alive if the process has not finished is a little
distasteful.  The __del__() method and the _cleanup() function are
completely unecessary on Windows since Windows does not have zombie
processes.  On Unix __del__() could simply save the pid to the _active
list and _cleanup() could be rewritten to use os.waitpid().)

> If there are limitations in Python that make it impossible to
> implement such functionality in pure Python - then *those*
> limitations should be overcome, rather than including the code
> wholesale.

Apart from (1) and (2) which I think are unavoidable, nearly all the C
code is Windows specific.  I assume you don't want to bring
in the whole of pywin32 into stdlib...

> Regards,
> Martin


Richard

PS. I am more than willing to maintain the code if it goes in.