[Python-Dev] Addition of "pyprocessing" module to standard lib.

[Jesse Noller]

On Wed, May 14, 2008 at 4:45 AM, "Martin v. Löwis" <martin at v.loewis.de> wrote:
> Jesse Noller wrote:
>  > I am looking for any questions, concerns or benchmarks python-dev has
>  > regarding the possible inclusion of the pyprocessing module to the
>  > standard library - preferably in the 2.6 timeline.
>
>  I think for inclusion in 2.6 it's to late. For 3.0, it's definitely
>  too late - the PEP acceptance deadline was a year ago (IIRC).
>

That's a fair point - however, as this is a proposed inclusion of a
standalone library that includes tests, documentation and benchmarks -
I was hoping it would be possible to attempt to get it in. A target
for 3.1 or 2.7 is also possible.

>
>  > As I am trying to finish up the PEP, I want to see if I can address
>  > any questions or include any other useful data (including benchmarks)
>  > in the PEP prior to publishing it. I am also intending to include
>  > basic benchmarks for both the processing module against the threading
>  > module as a comparison.
>
>  I'm worried whether it's stable, what user base it has, whether users
>  (other than the authors) are lobbying for inclusion. Statistically,
>  it seems to be not ready yet: it is not even a year old, and has not
>  reached version 1.0 yet.

Just as a point of clarification: I am not the module author. I am
simply a relatively passionate user using it to solve real problems. I
am working on the PEP after speaking with the author about it.

One of the outstanding issues I have listed is the "lack" of a 1.0
version - however given the fairly random way that most open source
projects pick numbering schemes - I don't see this as much more than
an administrative issue.

As for user base - at PyCon this year, I was involved in several open
space discussion about the concurrency/parallelism "problem" as it
relates to Python, as well as giving a lightning talk on this precise
subject including mentioning this module and alternatives. The sheer
number of people working on "big" problems who approached me to talk
about this was astounding. The presence online of the same like-minded
people is also quite large.

Overwhelmingly, many of the python programmers I spoke to are looking
for "a solution" that does not require the alteration of a known
programming paradigm (i.e: threads) to allow them to take advantage of
systems which are not getting "faster" - instead, they are getting
wider. Simply put due to the GIL - pure python applications can not
take advantage of these machines which are now more common than not
without switching to an alternative interpreter - which for many -
myself included - is not an attractive option.

Guido, and others called this module out specifically as a
"work-around" to the GIL for concurrency within cPython applications.
The API matches a known one and in most (if not all) cases, it is an
easy drop in to allow programmers to achieve what they may not be able
to do with the standard library as it is today.

Obviously, there are other solutions that aim to solve this problem
domain - Twisted, ParallelPython, etc all "work around" the GIL
through various methods - but the fundamental problem with those is
that they ask a user to "re-learn" how to construct their
applications. While the concept of shared memory and "threads" may not
be the "final answer" in the concurrent programming world -
fundamentally, it's wrong for us to say that in order for people to
really use python to solve concurrent problems they have to:

1: Re-architect their applications
2: Re-learn how to program (i.e: Async programming, Actor models,
shared-nothing)

I really do feel that inclusion of this library offers us the best of
both worlds - it gives us (as a community) an easy answer to those
people who would dismiss python due to the GIL and it also allows
users to easily implement their applications.

The GIL is currently advantageous to the cPython code base for simple
ease of implementation and maintenance. Barring the one-day adoption
of the bulk of Adam Olsen's safe-threading work or a massive trend in
programming to "rid" ourselves of the threaded mindset - this gives
cPython a leg-up in the concurrency world and gives users a tool to
solve hard problems.

-Jesse