[Python-Dev] Addition of "pyprocessing" module to standard lib.

[M.-A. Lemburg]

On 2008-05-14 14:15, Jesse Noller wrote:
> On Wed, May 14, 2008 at 5:45 AM, Christian Heimes <lists at cheimes.de> wrote:
>> Martin v. Löwis schrieb:
>>
>>> I'm worried whether it's stable, what user base it has, whether users
>>  > (other than the authors) are lobbying for inclusion. Statistically,
>>  > it seems to be not ready yet: it is not even a year old, and has not
>>  > reached version 1.0 yet.
>>
>>  I'm on Martin's side here. Although I like to see some sort of multi
>>  processing mechanism in Python 'cause I need it for lots of projects I'm
>>  against the inclusion of pyprocessing in 2.6 and 3.0. The project isn't
>>  old and mature enough and it has some competitors like pp (parallel
>>  processing).
>>
>>  On the one hand the inclusion of a package gives it an unfair advantage
>>  over similar packages. On the other hand it slows down future
>>  development because a new feature release must be synced with Python
>>  releases about every 1.5 years.
>>
>>  -0.5 from me
>>
>>  Christian
>>
> 
> I said this in reply to Martin - but the competitors (in my mind) are
> not as compelling due to the "alternative" paradigm for application
> construction they propose. The processing module is an "easy win" for
> us if included.
> 
> Personally - I don't see how inclusion in the stdlib would slow down
> development - yes, you have to stick with the same release cycle as
> python-core, but if the module is "feature complete" and provides a
> stable API as it stands I don't see following python-core timelines as
> overly onerous.
> 
> The module itself doesn't change that frequently - the last release in
> April was a bugfix release and API consistency change (the API would
> have to be locked for inclusion obviously - targeting a 2.7/3.1
> release may be advantageous to achieve this).

Why don't you start a parallel-sig and then hash this out with other
distributed computing users ?

You could then reach a decision by the time 2.7 is scheduled for release
and then add the chosen module to the stdlib.

The API of the processing module does look simple and nice, but
parallel processing is a minefield - esp. when it comes to handling
error situations (e.g. a worker failing, network going down, fail-over,
etc.).

What I'm missing with the processing module is a way to spawn processes
on clusters (rather than just on a single machine).

In the scientific world, MPI is the standard API of choice for doing
parallel processing, so if we're after standards, supporting MPI
would seem to be more attractive than the processing module.

     http://pypi.python.org/pypi/mpi4py

In the enterprise world, you often find CORBA based solutions.

     http://omniorb.sourceforge.net/

And then, of course, you have a gazillion specialized solutions
such as PyRO:

     http://pyro.sourceforge.net/

OTOH, perhaps the stdlib should just include entry-level support
for some form of parallel processing, in which case processing
does look attractive.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 14 2008)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
            Registered at Amtsgericht Duesseldorf: HRB 46611