I'm bumping this PEP again in hopes of getting some feedback. Thanks, Eric On Tue, Sep 8, 2009 at 23:52, Eric Pruitt wrote:

PEP: 3145 Title: Asynchronous I/O For subprocess.Popen Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson Type: Standards Track Content-Type: text/plain Created: 04-Aug-2009 Python-Version: 3.2

Abstract:

   In its present form, the subprocess.Popen implementation is prone to    dead-locking and blocking of the parent Python script while waiting on data    from the child process.

Motivation:

   A search for "python asynchronous subprocess" will turn up numerous    accounts of people wanting to execute a child process and communicate with    it from time to time reading only the data that is available instead of    blocking to wait for the program to produce data [1] [2] [3].  The current    behavior of the subprocess module is that when a user sends or receives    data via the stdin, stderr and stdout file objects, dead locks are common    and documented [4] [5].  While communicate can be used to alleviate some of    the buffering issues, it will still cause the parent process to block while    attempting to read data when none is available to be read from the child    process.

Rationale:

   There is a documented need for asynchronous, non-blocking functionality in    subprocess.Popen [6] [7] [2] [3].  Inclusion of the code would improve the    utility of the Python standard library that can be used on Unix based and    Windows builds of Python.  Practically every I/O object in Python has a    file-like wrapper of some sort.  Sockets already act as such and for    strings there is StringIO.  Popen can be made to act like a file by simply    using the methods attached the the subprocess.Popen.stderr, stdout and    stdin file-like objects.  But when using the read and write methods of    those options, you do not have the benefit of asynchronous I/O.  In the    proposed solution the wrapper wraps the asynchronous methods to mimic a    file object.

Reference Implementation:

   I have been maintaining a Google Code repository that contains all of my    changes including tests and documentation [9] as well as blog detailing    the problems I have come across in the development process [10].

   I have been working on implementing non-blocking asynchronous I/O in the    subprocess.Popen module as well as a wrapper class for subprocess.Popen    that makes it so that an executed process can take the place of a file by    duplicating all of the methods and attributes that file objects have.

   There are two base functions that have been added to the subprocess.Popen    class: Popen.send and Popen._recv, each with two separate implementations,    one for Windows and one for Unix based systems.  The Windows    implementation uses ctypes to access the functions needed to control pipes    in the kernel 32 DLL in an asynchronous manner.  On Unix based systems,    the Python interface for file control serves the same purpose.  The    different implementations of Popen.send and Popen._recv have identical    arguments to make code that uses these functions work across multiple    platforms.

   When calling the Popen._recv function, it requires the pipe name be    passed as an argument so there exists the Popen.recv function that passes    selects stdout as the pipe for Popen._recv by default.  Popen.recv_err    selects stderr as the pipe by default. "Popen.recv" and "Popen.recv_err"    are much easier to read and understand than "Popen._recv('stdout' ..." and    "Popen._recv('stderr' ..." respectively.

   Since the Popen._recv function does not wait on data to be produced    before returning a value, it may return empty bytes.  Popen.asyncread    handles this issue by returning all data read over a given time    interval.

   The ProcessIOWrapper class uses the asyncread and asyncwrite functions to    allow a process to act like a file so that there are no blocking issues    that can arise from using the stdout and stdin file objects produced from    a subprocess.Popen call.

References:

   [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess        http://mail.python.org/pipermail/python-bugs-list/2006-December/          036524.html

   [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess        http://ivory.idyll.org/blog/feb-07/problems-with-subprocess

   [3] How can I run an external command asynchronously from Python? - Stack        Overflow        http://stackoverflow.com/questions/636561/how-can-i-run-an-external-          command-asynchronously-from-python

   [4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation        http://docs.python.org/library/subprocess.html#subprocess.Popen.wait

   [5] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation        http://docs.python.org/library/subprocess.html#subprocess.Popen.kill

   [6] Issue 1191964: asynchronous Subprocess - Python tracker        http://bugs.python.org/issue1191964

   [7] Module to allow Asynchronous subprocess use on Windows and Posix        platforms - ActiveState Code        http://code.activestate.com/recipes/440554/

   [8] subprocess.rst - subprocdev - Project Hosting on Google Code        http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=svn2c925e935cad0166d5da85e37c742d8e7f609de5&r=2c925e935cad0166d5da85e37c742d8e7f609de5#437

   [9] subprocdev - Project Hosting on Google Code        http://code.google.com/p/subprocdev

   [10] Python Subprocess Dev         http://subdev.blogspot.com/

Copyright:

   This P.E.P. is licensed under the Open Publication License;    http://www.opencontent.org/openpub/.

Hello, I would like to know if your approach is based on Python 2.x or 3.x. Python 3.x has new API provisions, in the I/O layer, for non-blocking I/O and it would be nice if your work could fit in that framework.

...

Popen can be made to act like a file by simply using the methods attached the the subprocess.Popen.stderr, stdout and stdin file-like objects. But when using the read and write methods of those options, you do not have the benefit of asynchronous I/O.

I'm not sure I understand the latter sentence. Do you imply that, with your work, read() and write() do allow you to benefit from async I/O? If so, how? Another question: what mechanism does it use internally? Is this mechanism accessible from the outside, so that people can e.g. integrate this inside a third-party event loop (Twisted, asyncore or whatever else)? The PEP should probably outline the additional APIs a bit more precisely and formally than it currently does.

...

The Windows implementation uses ctypes to access the functions needed to control pipes in the kernel 32 DLL in an asynchronous manner.

Sorry for the naive question (I'm not a Windows specialist), but does the allusion to "kernel32.dll" mean that it doesn't function on 64-bit variants? Thanks for your work, Regards Antoine.

On Tue, Sep 15, 2009 at 9:24 PM, wrote:

On 04:25 pm, eric.pruitt@gmail.com wrote:

...

I'm bumping this PEP again in hopes of getting some feedback.

This is useful, indeed. ActiveState recipe for this has 10 votes, which is high for ActiveState (and such hardcore topic FWIW).

On Tue, Sep 8, 2009 at 23:52, Eric Pruitt wrote:

...

...

PEP: 3145 Title: Asynchronous I/O For subprocess.Popen Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson Type: Standards Track Content-Type: text/plain Created: 04-Aug-2009 Python-Version: 3.2

Abstract:

In its present form, the subprocess.Popen implementation is prone to dead-locking and blocking of the parent Python script while waiting on data from the child process.

Motivation:

A search for "python asynchronous subprocess" will turn up numerous accounts of people wanting to execute a child process and communicate with it from time to time reading only the data that is available instead of blocking to wait for the program to produce data [1] [2] [3]. The current behavior of the subprocess module is that when a user sends or receives data via the stdin, stderr and stdout file objects, dead locks are common and documented [4] [5]. While communicate can be used to alleviate some of the buffering issues, it will still cause the parent process to block while attempting to read data when none is available to be read from the child process.

Rationale:

There is a documented need for asynchronous, non-blocking functionality in subprocess.Popen [6] [7] [2] [3]. Inclusion of the code would improve the utility of the Python standard library that can be used on Unix based and Windows builds of Python. Practically every I/O object in Python has a file-like wrapper of some sort. Sockets already act as such and for strings there is StringIO. Popen can be made to act like a file by simply using the methods attached the the subprocess.Popen.stderr, stdout and stdin file-like objects. But when using the read and write methods of those options, you do not have the benefit of asynchronous I/O. In the proposed solution the wrapper wraps the asynchronous methods to mimic a file object.

Reference Implementation:

I have been maintaining a Google Code repository that contains all of my changes including tests and documentation [9] as well as blog detailing the problems I have come across in the development process [10].

I have been working on implementing non-blocking asynchronous I/O in the subprocess.Popen module as well as a wrapper class for subprocess.Popen that makes it so that an executed process can take the place of a file by duplicating all of the methods and attributes that file objects have.

"Non-blocking" and "asynchronous" are actually two different things. From the rest of this PEP, I think only a non-blocking API is being introduced. I haven't looked beyond the PEP, though, so I might be missing something.

I suggest renaming http://www.python.org/dev/peps/pep-3145/ to 'Non-blocking I/O for subprocess' and continue. IMHO on this stage is where examples with deadlocks that occur with current subprocess implementation are badly needed. There are two base functions that have been added to the

...

...

subprocess.Popen class: Popen.send and Popen._recv, each with two separate implementations, one for Windows and one for Unix based systems. The Windows implementation uses ctypes to access the functions needed to control pipes in the kernel 32 DLL in an asynchronous manner. On Unix based systems, the Python interface for file control serves the same purpose. The different implementations of Popen.send and Popen._recv have identical arguments to make code that uses these functions work across multiple platforms.

Why does the method for non-blocking read from a pipe start with an "_"? This is the convention (widely used) for a private API. The name also doesn't suggest that this is the non-blocking version of reading. Similarly, the name "send" doesn't suggest that this is the non-blocking version of writing.

The implementation is based on http://code.activestate.com/recipes/440554/which is more clearly illustrates integrated functionality. _recv() is a private base function, which is takes stdout or stderr as parameter. Corresponding user-level functions to read from stdout and stderr are .recv() and .recv_err() I thought about renaming API to .asyncread() and .asyncwrite(), but that may mean that you call method and then result asynchronously start to fill some buffer, which is not the case here. Then I thought about .check_read() and .check_write(), literally meaning 'check and read' or 'check and return' for non-blocking calls if there is nothing. But then again, poor naming convention of subprocess uses .check_output() for blocking read until command completes. Currently, subversion doesn't have .read and .write methods. It may be the best option: .write(what) to pipe more stuff into input buffer of child process. .read(from) where `from` is either subprocess.STDOUT or STDERR Both functions should be marked as non-blocking in docs and returning None if pipe is closed. When calling the Popen._recv function, it requires the pipe name be

...

...

passed as an argument so there exists the Popen.recv function that passes selects stdout as the pipe for Popen._recv by default. Popen.recv_err selects stderr as the pipe by default. "Popen.recv" and "Popen.recv_err" are much easier to read and understand than "Popen._recv('stdout' ..." and "Popen._recv('stderr' ..." respectively.

What about reading from other file descriptors? subprocess.Popen allows arbitrary file descriptors to be used. Is there any provision here for reading and writing non-blocking from or to those?

On Windows it is WriteFile/ReadFile and PeekNamedPipe. On Linux it is select. Of course a test is needed, but why it should not just work?

Since the Popen._recv function does not wait on data to be produced

...

...

before returning a value, it may return empty bytes. Popen.asyncread handles this issue by returning all data read over a given time interval.

Oh. Popen.asyncread? What's that? This is the first time the PEP mentions it.

I guess that's for blocking read with timeout. Among the most popular questions about Python it is the question number ~500. http://stackoverflow.com/questions/1191374/subprocess-with-timeout

The ProcessIOWrapper class uses the asyncread and asyncwrite

...

...

functions to allow a process to act like a file so that there are no blocking issues that can arise from using the stdout and stdin file objects produced from a subprocess.Popen call.

What's the ProcessIOWrapper class? And what's the asyncwrite function? Again, this is the first time it's mentioned.

Oh. That's a wrapper to access subprocess pipes with familiar file API. It is interesting: http://code.google.com/p/subprocdev/source/browse/subprocess.py?name=python3...

So, to sum up, I think my main comment is that the PEP seems to be missing a significant portion of the details of what it's actually proposing. I suspect that this information is present in the implementation, which I have not looked at, but it probably belongs in the PEP.

Jean-Paul

Writing PEPs is definitely a job, and a hard one for developers. Too bad a good idea *and* implementation (tests needed) is put on hold, because there is nobody, who can help with that part. IMHO PEP needs to expand on user stories even if there is significant amount of cited sources, a practical summary and problem illustration by examples are missing.

I'm really not sure what this PEP is trying to get at given that it contains no examples and sounds from the descriptions to be adding a complicated api on top of something that already, IMNSHO, has too much it (subprocess.Popen). Regardless, any user can use the stdout/err/in file objects with their own code that handles them asynchronously (yes that can be painful but that is what is required for _any_ socket or pipe I/O you don't want to block on). It *sounds* to me like this entire PEP could be written and released as a third party module on PyPI that offers a subprocess.Popen subclass adding some more convenient non-blocking APIs. That's where I'd start if I were interested in this as a future feature. -gps On Fri, Dec 7, 2012 at 5:10 PM, anatoly techtonik wrote:

On Tue, Sep 15, 2009 at 9:24 PM, wrote:

...

On 04:25 pm, eric.pruitt@gmail.com wrote:

...

I'm bumping this PEP again in hopes of getting some feedback.

This is useful, indeed. ActiveState recipe for this has 10 votes, which is high for ActiveState (and such hardcore topic FWIW).

...

On Tue, Sep 8, 2009 at 23:52, Eric Pruitt wrote:

...

...

PEP: 3145 Title: Asynchronous I/O For subprocess.Popen Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson Type: Standards Track Content-Type: text/plain Created: 04-Aug-2009 Python-Version: 3.2

Abstract:

In its present form, the subprocess.Popen implementation is prone to dead-locking and blocking of the parent Python script while waiting on data from the child process.

Motivation:

A search for "python asynchronous subprocess" will turn up numerous accounts of people wanting to execute a child process and communicate with it from time to time reading only the data that is available instead of blocking to wait for the program to produce data [1] [2] [3]. The current behavior of the subprocess module is that when a user sends or receives data via the stdin, stderr and stdout file objects, dead locks are common and documented [4] [5]. While communicate can be used to alleviate some of the buffering issues, it will still cause the parent process to block while attempting to read data when none is available to be read from the child process.

Rationale:

There is a documented need for asynchronous, non-blocking functionality in subprocess.Popen [6] [7] [2] [3]. Inclusion of the code would improve the utility of the Python standard library that can be used on Unix based and Windows builds of Python. Practically every I/O object in Python has a file-like wrapper of some sort. Sockets already act as such and for strings there is StringIO. Popen can be made to act like a file by simply using the methods attached the the subprocess.Popen.stderr, stdout and stdin file-like objects. But when using the read and write methods of those options, you do not have the benefit of asynchronous I/O. In the proposed solution the wrapper wraps the asynchronous methods to mimic a file object.

Reference Implementation:

I have been maintaining a Google Code repository that contains all of my changes including tests and documentation [9] as well as blog detailing the problems I have come across in the development process [10].

I have been working on implementing non-blocking asynchronous I/O in the subprocess.Popen module as well as a wrapper class for subprocess.Popen that makes it so that an executed process can take the place of a file by duplicating all of the methods and attributes that file objects have.

"Non-blocking" and "asynchronous" are actually two different things. From the rest of this PEP, I think only a non-blocking API is being introduced. I haven't looked beyond the PEP, though, so I might be missing something.

I suggest renaming http://www.python.org/dev/peps/pep-3145/ to 'Non-blocking I/O for subprocess' and continue. IMHO on this stage is where examples with deadlocks that occur with current subprocess implementation are badly needed.

There are two base functions that have been added to the

...

...

...

subprocess.Popen class: Popen.send and Popen._recv, each with two separate implementations, one for Windows and one for Unix based systems. The Windows implementation uses ctypes to access the functions needed to control pipes in the kernel 32 DLL in an asynchronous manner. On Unix based systems, the Python interface for file control serves the same purpose. The different implementations of Popen.send and Popen._recv have identical arguments to make code that uses these functions work across multiple platforms.

Why does the method for non-blocking read from a pipe start with an "_"? This is the convention (widely used) for a private API. The name also doesn't suggest that this is the non-blocking version of reading. Similarly, the name "send" doesn't suggest that this is the non-blocking version of writing.

The implementation is based on http://code.activestate.com/recipes/440554/which is more clearly illustrates integrated functionality.

_recv() is a private base function, which is takes stdout or stderr as parameter. Corresponding user-level functions to read from stdout and stderr are .recv() and .recv_err()

I thought about renaming API to .asyncread() and .asyncwrite(), but that may mean that you call method and then result asynchronously start to fill some buffer, which is not the case here.

Then I thought about .check_read() and .check_write(), literally meaning 'check and read' or 'check and return' for non-blocking calls if there is nothing. But then again, poor naming convention of subprocess uses .check_output() for blocking read until command completes.

Currently, subversion doesn't have .read and .write methods. It may be the best option: .write(what) to pipe more stuff into input buffer of child process. .read(from) where `from` is either subprocess.STDOUT or STDERR Both functions should be marked as non-blocking in docs and returning None if pipe is closed.

When calling the Popen._recv function, it requires the pipe name be

...

...

...

passed as an argument so there exists the Popen.recv function that passes selects stdout as the pipe for Popen._recv by default. Popen.recv_err selects stderr as the pipe by default. "Popen.recv" and "Popen.recv_err" are much easier to read and understand than "Popen._recv('stdout' ..." and "Popen._recv('stderr' ..." respectively.

What about reading from other file descriptors? subprocess.Popen allows arbitrary file descriptors to be used. Is there any provision here for reading and writing non-blocking from or to those?

On Windows it is WriteFile/ReadFile and PeekNamedPipe. On Linux it is select. Of course a test is needed, but why it should not just work?

...

Since the Popen._recv function does not wait on data to be produced

...

...

before returning a value, it may return empty bytes. Popen.asyncread handles this issue by returning all data read over a given time interval.

Oh. Popen.asyncread? What's that? This is the first time the PEP mentions it.

I guess that's for blocking read with timeout. Among the most popular questions about Python it is the question number ~500. http://stackoverflow.com/questions/1191374/subprocess-with-timeout

...

The ProcessIOWrapper class uses the asyncread and asyncwrite

...

...

functions to allow a process to act like a file so that there are no blocking issues that can arise from using the stdout and stdin file objects produced from a subprocess.Popen call.

What's the ProcessIOWrapper class? And what's the asyncwrite function? Again, this is the first time it's mentioned.

Oh. That's a wrapper to access subprocess pipes with familiar file API. It is interesting:

http://code.google.com/p/subprocdev/source/browse/subprocess.py?name=python3...

...

So, to sum up, I think my main comment is that the PEP seems to be missing a significant portion of the details of what it's actually proposing. I suspect that this information is present in the implementation, which I have not looked at, but it probably belongs in the PEP.

Jean-Paul

Writing PEPs is definitely a job, and a hard one for developers. Too bad a good idea *and* implementation (tests needed) is put on hold, because there is nobody, who can help with that part.

IMHO PEP needs to expand on user stories even if there is significant amount of cited sources, a practical summary and problem illustration by examples are missing.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org

On Dec 8, 2012, at 8:37 PM, Gregory P. Smith wrote:

Is twisted's spawnProcess thread safe and async signal safe by using restricted C code for everything between the fork() and exec()? I'm not familiar enough with the twisted codebase to find things easily in it but I'm not seeing such an extension module within twisted and the code in http://twistedmatrix.com/trac/browser/trunk/twisted/internet/process.py certainly is not safe. Just sayin'. :)

It's on the agenda: http://twistedmatrix.com/trac/ticket/5710. -glyph

On Sun, Dec 9, 2012 at 7:17 AM, Gregory P. Smith wrote:

I'm really not sure what this PEP is trying to get at given that it contains no examples and sounds from the descriptions to be adding a complicated api on top of something that already, IMNSHO, has too much it (subprocess.Popen).

Regardless, any user can use the stdout/err/in file objects with their own code that handles them asynchronously (yes that can be painful but that is what is required for _any_ socket or pipe I/O you don't want to block on).

And how to use stdout/stderr/in asynchronously in cross-platform manner? IIUC the problem is that every read is blocking.

It *sounds* to me like this entire PEP could be written and released as a third party module on PyPI that offers a subprocess.Popen subclass adding some more convenient non-blocking APIs. That's where I'd start if I were interested in this as a future feature.

I've rewritten the PEP based on how do I understand the code. I don't know how to update it and how to comply with open documentation license, so I just attach it and add PEPs list to CC. Me too has a feeling that the PEP should be stripped of additional high level API until low level functionality is well understood and accepted. -- anatoly t.

On Dec 19, 2012, at 2:14 PM, anatoly techtonik wrote:

On Sun, Dec 9, 2012 at 7:14 AM, Glyph wrote: On Dec 7, 2012, at 5:10 PM, anatoly techtonik wrote:

...

What about reading from other file descriptors? subprocess.Popen allows arbitrary file descriptors to be used. Is there any provision here for reading and writing non-blocking from or to those?

On Windows it is WriteFile/ReadFile and PeekNamedPipe. On Linux it is select. Of course a test is needed, but why it should not just work?

This is exactly why the provision needs to be made explicitly.

On Windows it is WriteFile and ReadFile and PeekNamedPipe - unless the handle is a socket in which case it needs to be WSARecv. Or maybe it's some other weird thing - like, maybe a mailslot - and you need to call a different API.

IIRC on Windows there is no socket descriptor that can be used as a file descriptor. Seems reasonable to limit the implementation to standard file descriptors in this platform.

Via the documentation of ReadFile: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365467(v=vs.85).as... hFile [in] A handle to the device (for example, a file, file stream, physical disk, volume, console buffer, tape drive, socket, communications resource, mailslot, or pipe). (...) For asynchronous read operations, hFile can be any handle that is opened with the FILE_FLAG_OVERLAPPED flag by the CreateFilefunction, or a socket handle returned by the socket or accept function. (emphasis mine). So, you can treat sockets as regular files in some contexts, and not in others. Of course there are other reasons to use WSARecv instead of ReadFile sometimes, which is why there are multiple functions.

On *nix it really shouldn't be select. select cannot wait upon a file descriptor whose value is greater than FD_SETSIZE, which means it sets a hard (and small) limit on the number of things that a process which wants to use this facility can be doing.

I didn't know that. Should a note be added to http://docs.python.org/2/library/select ?

The note that should be added there is simply "you should know how the select system call works in C if you want to use this module".

I also thought that poll acts like, well, a polling function - eating 100% CPU while looping over inputs over and over checking if there is something to react to.

Nope. Admittedly, the naming is slightly misleading.

On the other hand, if you hard-code another arbitrary limit like this into the stdlib subprocess module, it will just be another great reason why Twisted's spawnProcess is the best and everyone should use it instead, so be my guest ;-).

spawnProcess requires a reactor. This PEP is an alternative for the proponents of green energy. =)

Do you know what happens when you take something that is supposed to be happening inside a reactor, and then move it outside a reactor? It's not called "green energy", it's called "a bomb" ;-). -glyph

On Dec 19, 2012, at 7:46 PM, anatoly techtonik wrote:

...

On *nix it really shouldn't be select. select cannot wait upon a file descriptor whose value is greater than FD_SETSIZE, which means it sets a hard (and small) limit on the number of things that a process which wants to use this facility can be doing.

I didn't know that. Should a note be added to http://docs.python.org/2/library/select ?

The note that should be added there is simply "you should know how the select system call works in C if you want to use this module".

Why spreading FUD if it is possible to define a good entrypoint for those who want to learn, but don't have enough time? Why not to say directly that select interface is outdated?

It's not FUD. If you know how select() works in C, you may well want to call it. It's the most portable multiplexing API, although it has a number of limitations. Really, what most users in this situation ought to be using is Twisted, but it seems there is not sufficient interest to bundle Twisted's core in the stdlib. However, the thing Guido is working on lately may be interoperable enough with Twisted that you can upgrade to it more easily in future versions of Python, so one day it may be reasonable to say select is outdated. (Maybe not though. It's a good thing nobody told me that select was deprecated in favor of asyncore.)

...

On the other hand, if you hard-code another arbitrary limit like this into the stdlib subprocess module, it will just be another great reason why Twisted's spawnProcess is the best and everyone should use it instead, so be my guest ;-).

spawnProcess requires a reactor. This PEP is an alternative for the proponents of green energy. =)

Do you know what happens when you take something that is supposed to be happening inside a reactor, and then move it outside a reactor? It's not called "green energy", it's called "a bomb" ;-).

The biggest complain about nuclear physics is that to understand what's going on it should have been gone 3D long ago. =) I think Twisted needs to organize competition on the best visualization of underlying concepts. It will help people grasp the concepts behind and different problems much faster (as well as gain an ability to compare different reactors).

I would love for someone to do this, of course, but now we're _really_ off topic. -glyph

On Thu, Dec 20, 2012 at 12:18 PM, Brett Cannon wrote:

And please do not CC the peps mailing list on discussions. It should only be used to mail in new PEPs or acceptable patches to PEPs.

PEP 1 should perhaps be clarified if the above is the case. Currently, PEP 1 says all PEP-related e-mail should be sent there: "The PEP editors assign PEP numbers and change their status. Please send all PEP-related email to (no cross-posting please). Also see PEP Editor Responsibilities & Workflow below." as well as: "A PEP editor must subscribe to the list. All PEP-related correspondence should be sent (or CC'd) to (but please do not cross-post!)." (Incidentally, the statement not to cross-post seems contradictory if a PEP-related e-mail is also sent to python-dev, for example.) --Chris

On Wed, Dec 19, 2012 at 5:20 PM, anatoly techtonik wrote:

...

On Sun, Dec 9, 2012 at 7:17 AM, Gregory P. Smith wrote:

...

I'm really not sure what this PEP is trying to get at given that it contains no examples and sounds from the descriptions to be adding a complicated api on top of something that already, IMNSHO, has too much it (subprocess.Popen).

Regardless, any user can use the stdout/err/in file objects with their own code that handles them asynchronously (yes that can be painful but that is what is required for _any_ socket or pipe I/O you don't want to block on).

And how to use stdout/stderr/in asynchronously in cross-platform manner? IIUC the problem is that every read is blocking.

...

It sounds to me like this entire PEP could be written and released as a third party module on PyPI that offers a subprocess.Popen subclass adding some more convenient non-blocking APIs. That's where I'd start if I were interested in this as a future feature.

I've rewritten the PEP based on how do I understand the code. I don't know how to update it and how to comply with open documentation license, so I just attach it and add PEPs list to CC. Me too has a feeling that the PEP should be stripped of additional high level API until low level functionality is well understood and accepted.

-- anatoly t.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com

On Thu, Dec 20, 2012 at 1:12 PM, Brett Cannon wrote:

On Thu, Dec 20, 2012 at 3:55 PM, Chris Jerdonek wrote:

...

On Thu, Dec 20, 2012 at 12:18 PM, Brett Cannon wrote:

...

And please do not CC the peps mailing list on discussions. It should only be used to mail in new PEPs or acceptable patches to PEPs.

PEP 1 should perhaps be clarified if the above is the case. Currently, PEP 1 says all PEP-related e-mail should be sent there:

"The PEP editors assign PEP numbers and change their status. Please send all PEP-related email to (no cross-posting please). Also see PEP Editor Responsibilities & Workflow below."

as well as:

"A PEP editor must subscribe to the list. All PEP-related correspondence should be sent (or CC'd) to (but please do not cross-post!)."

(Incidentally, the statement not to cross-post seems contradictory if a PEP-related e-mail is also sent to python-dev, for example.)

But it very clearly states to NOT cross-post which is exactly what Anatoly did and that is what I take issue with the most. I personally don't see any confusion with the wording. It clearly states that if you are a PEP author you should mail the peps editors and NOT cross-post. If you are an editor, make sure any emailing you do with an individual CCs the list but do NOT cross-post.

I don't disagree that he shouldn't have cross-posted. I was just pointing out that the language should be clarified. What's confusing is that the current language implies that one shouldn't send any PEP-related e-mails to any mailing list other than peps@. In particular, how can one discuss PEPs on python-dev or python-ideas without violating that language (e.g. this e-mail which is related to PEP 1)? It is probably just a matter of clarifying what "PEP-related" means. --Chris

Please stop copying me on this thread. Thanks, Jean-Paul

What should I do in case Eric lost interest after his GSoC project for PSF appeared as useless for python-dev community? Should I rewrite the proposal from scratch? On Thu, Dec 20, 2012 at 11:18 PM, Brett Cannon wrote:

You cannot rewrite an existing PEP if you are not one of the original owners, nor can you add yourself as an author to a PEP without permission from the original authors.

And please do not CC the peps mailing list on discussions. It should only be used to mail in new PEPs or acceptable patches to PEPs.

On Wed, Dec 19, 2012 at 5:20 PM, anatoly techtonik wrote:

...

On Sun, Dec 9, 2012 at 7:17 AM, Gregory P. Smith wrote:

...

I'm really not sure what this PEP is trying to get at given that it contains no examples and sounds from the descriptions to be adding a complicated api on top of something that already, IMNSHO, has too much it (subprocess.Popen).

Regardless, any user can use the stdout/err/in file objects with their own code that handles them asynchronously (yes that can be painful but that is what is required for _any_ socket or pipe I/O you don't want to block on).

And how to use stdout/stderr/in asynchronously in cross-platform manner? IIUC the problem is that every read is blocking.

...

It *sounds* to me like this entire PEP could be written and released as a third party module on PyPI that offers a subprocess.Popen subclass adding some more convenient non-blocking APIs. That's where I'd start if I were interested in this as a future feature.

I've rewritten the PEP based on how do I understand the code. I don't know how to update it and how to comply with open documentation license, so I just attach it and add PEPs list to CC. Me too has a feeling that the PEP should be stripped of additional high level API until low level functionality is well understood and accepted.

-- anatoly t.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org

On Mon, Dec 24, 2012 at 7:42 AM, anatoly techtonik wrote:

What should I do in case Eric lost interest after his GSoC project for PSF appeared as useless for python-dev community? Should I rewrite the proposal from scratch?

Before you attempt that, start by trying to have a better attitude towards people's contributions around here.