multiprocessing Greg Ewing wrote:

Jesse Noller wrote:

...

I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library

Sounds good, but I'd suggest giving a more specific name than "processing", which is so generic as to be meaningless.

Nick Coghlan wrote:

Talin wrote:

...

multiprocessing

multiprocessing would work for me

Me, too. -- Greg

The inclusion of the processing module does not exclude the potential to also use or one day include MPI bindings. The goal is to add a module with a "known" API and semantics which allows programmer using cPython to easily take advantage of multple cores (and as a side benefit, machines). In theory - one could also use MPI within an application using the processing module as a communications/message passing system. -Jesse On May 13, 2008, at 9:23 PM, Tom Pinckney wrote:

Why not use MPI? It's cross platform, cross language and very widely supported already. And there're Python bindings already.

On May 13, 2008, at 8:52 PM, Jesse Noller wrote:

...

I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library - preferably in the 2.6 timeline. In March, I began working on the PEP for the inclusion of the pyprocessing (processing) module into the python standard library[1]. The original email to the stdlib-sig can be found here, it includes a basic overview of the module:

http://mail.python.org/pipermail/stdlib-sig/2008-March/000129.html

The processing module mirrors/mimics the API of the threading module - and with simple import/subclassing changes depending on the code, allows you to leverage multi core machines via an underlying forking mechanism. The module also supports the sharing of data across groups of networked machines - a feature obviously not part of the core threading module, but useful in a distributed environment.

As I am trying to finish up the PEP, I want to see if I can address any questions or include any other useful data (including benchmarks) in the PEP prior to publishing it. I am also intending to include basic benchmarks for both the processing module against the threading module as a comparison.

-Jesse

[1] Processing page: http://pyprocessing.berlios.de/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/thomaspinckney3%40gmail.co...

Tom Pinckney wrote:

Why not use MPI? It's cross platform, cross language and very widely supported already. And there're Python bindings already.

The point of pyprocessing is that fact that the API is the same as that of the threading module - making it very easy to convert a multithreaded program to a multiprocessing one, and hence making it easy to exploit multiple processors in a CPU bound Python program, regardless of the restrictions imposed by Python's GIL. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

Martin v. Löwis schrieb:

I'm worried whether it's stable, what user base it has, whether users (other than the authors) are lobbying for inclusion. Statistically, it seems to be not ready yet: it is not even a year old, and has not reached version 1.0 yet.

I'm on Martin's side here. Although I like to see some sort of multi processing mechanism in Python 'cause I need it for lots of projects I'm against the inclusion of pyprocessing in 2.6 and 3.0. The project isn't old and mature enough and it has some competitors like pp (parallel processing). On the one hand the inclusion of a package gives it an unfair advantage over similar packages. On the other hand it slows down future development because a new feature release must be synced with Python releases about every 1.5 years. -0.5 from me Christian

On Wed, May 14, 2008 at 5:45 AM, Christian Heimes wrote:

Martin v. Löwis schrieb:

...

I'm worried whether it's stable, what user base it has, whether users (other than the authors) are lobbying for inclusion. Statistically, it seems to be not ready yet: it is not even a year old, and has not reached version 1.0 yet.

I'm on Martin's side here. Although I like to see some sort of multi processing mechanism in Python 'cause I need it for lots of projects I'm against the inclusion of pyprocessing in 2.6 and 3.0. The project isn't old and mature enough and it has some competitors like pp (parallel processing).

On the one hand the inclusion of a package gives it an unfair advantage over similar packages. On the other hand it slows down future development because a new feature release must be synced with Python releases about every 1.5 years.

-0.5 from me

Christian

I said this in reply to Martin - but the competitors (in my mind) are not as compelling due to the "alternative" paradigm for application construction they propose. The processing module is an "easy win" for us if included. Personally - I don't see how inclusion in the stdlib would slow down development - yes, you have to stick with the same release cycle as python-core, but if the module is "feature complete" and provides a stable API as it stands I don't see following python-core timelines as overly onerous. The module itself doesn't change that frequently - the last release in April was a bugfix release and API consistency change (the API would have to be locked for inclusion obviously - targeting a 2.7/3.1 release may be advantageous to achieve this). -jesse

On Wed, May 14, 2008 at 11:46 AM, M.-A. Lemburg wrote:

On 2008-05-14 14:15, Jesse Noller wrote:

...

On Wed, May 14, 2008 at 5:45 AM, Christian Heimes wrote:

...

Martin v. Löwis schrieb:

...

I'm worried whether it's stable, what user base it has, whether users

...

(other than the authors) are lobbying for inclusion. Statistically, it seems to be not ready yet: it is not even a year old, and has not reached version 1.0 yet.

I'm on Martin's side here. Although I like to see some sort of multi processing mechanism in Python 'cause I need it for lots of projects I'm against the inclusion of pyprocessing in 2.6 and 3.0. The project isn't old and mature enough and it has some competitors like pp (parallel processing).

On the one hand the inclusion of a package gives it an unfair advantage over similar packages. On the other hand it slows down future development because a new feature release must be synced with Python releases about every 1.5 years.

-0.5 from me

Christian

I said this in reply to Martin - but the competitors (in my mind) are not as compelling due to the "alternative" paradigm for application construction they propose. The processing module is an "easy win" for us if included.

Personally - I don't see how inclusion in the stdlib would slow down development - yes, you have to stick with the same release cycle as python-core, but if the module is "feature complete" and provides a stable API as it stands I don't see following python-core timelines as overly onerous.

The module itself doesn't change that frequently - the last release in April was a bugfix release and API consistency change (the API would have to be locked for inclusion obviously - targeting a 2.7/3.1 release may be advantageous to achieve this).

Why don't you start a parallel-sig and then hash this out with other distributed computing users ?

You could then reach a decision by the time 2.7 is scheduled for release and then add the chosen module to the stdlib.

The API of the processing module does look simple and nice, but parallel processing is a minefield - esp. when it comes to handling error situations (e.g. a worker failing, network going down, fail-over, etc.).

What I'm missing with the processing module is a way to spawn processes on clusters (rather than just on a single machine).

In the scientific world, MPI is the standard API of choice for doing parallel processing, so if we're after standards, supporting MPI would seem to be more attractive than the processing module.

http://pypi.python.org/pypi/mpi4py

In the enterprise world, you often find CORBA based solutions.

http://omniorb.sourceforge.net/

And then, of course, you have a gazillion specialized solutions such as PyRO:

http://pyro.sourceforge.net/

OTOH, perhaps the stdlib should just include entry-level support for some form of parallel processing, in which case processing does look attractive.

-- Marc-Andre Lemburg eGenix.com

Thanks for bringing up something I was going to mention - I am not attempting to "solve" the distributed computing problem with this proposal - you are right in mentioning there's a variety of technologies out there for achieving "true" loosely-coupled distributed computing, including all of those which you pointed out. I am proposing exactly what you mentioned: Entry level parallel processing. The fact that the processing module does have remote capabilities is a bonus: Not core to the proposal. While in a "perfect" world - a system might exist which truly insulates programmers from the difference between local concurrency and distributed systems - the two are really different problems. My concern is the taking advantage of the 8 core machine sitting under my desk (or the 10 or so I have in the lab)- the processing module allows me to do that - easily. The module is basic enough to be flexible in other technologies to use with it for highly distributed systems but it is also simple enough to act as an entry point for those people just starting out in the domain. Think of it like the difference between asyncore and Twisted. I could easily see more loosely-coupled-highly-distributed tools being built on top of the basics it has provided. -jesse

On May 14, 2008, at 12:32 PM, Andrew McNabb wrote:

...

Think of the processing module as an alternative to the threading module, not as an alternative to MPI. In Python, multi-threading can be extremely slow. The processing module gives you a way to convert from using multiple threads to using multiple processes.

If it made people feel better, maybe it should be called threading2 instead of multiprocessing. The word "processing" seems to make people think of parallel processing and clusters, which is missing the point.

Anyway, I would love to see the processing module included in the standard library.

Is the goal of the pyprocessing module to be exactly drop in compatible with threading, Queue and friends? I guess the idea would be that if my problem is compute bound I'd use pyprocessing and if it was I/O bound I might just use the existing threading library? Can I write a program using only threading and Queue interfaces for inter-thread communication and just change my import statements and have my program work? Currently, it looks like the pyprocessing.Queue interface is slightly different than than Queue.Queue for example (no task_done() etc). Perhaps a stdlib version of pyprocessing could be simplified down to not try to be a cross-machine computing environment and just be a same- machine threading replacement? This would make the maintenance easier and reduce confusion with users about how they should do cross-machine multiprocessing. By the way, a thread-safe simple dict in the style of Queue would be extremely helpful in writing multi-threaded programs, whether using the threading or pyprocessing modules.

Charles Cazabon wrote:

"threading" is to threads as "processing" is to processes; that's why it was named processing.

Unfortunately, the word "processing" is already used in the field of computing with a very general meaning -- any kind of data transfomation at all can be, and is, referred to as "processing". So the intended meaning fails to come across. -- Greg

Greg> Maybe "multicore" would help give the right impression? "multiproc", "multiprocess"? Skip

On Wed, May 14, 2008 at 6:48 PM, Phillip J. Eby wrote:

At 12:19 PM 5/15/2008 +1200, Greg Ewing wrote:

...

Andrew McNabb wrote:

...

If it made people feel better, maybe it should be called threading2 instead of multiprocessing.

I think that errs in the other direction, making it sound like just another way of doing single-process threading, which it's not.

Maybe "multicore" would help give the right impression?

Sounds like a marketing win to me, since it directly addresses the "python doesn't do multicore" meme.

-1 on "multicore" - multiprocess or multiprocessing are a fine names. cores are irrelevant. systems have multiple cpus real or virtual regardless of how many dies, sockets and cores there are. +0.5 on inclusion. that means i am happy if it does but don't think it needs to make it into 2.6/3.0. leave inclusion for 2.7/3.1. its easy for people to install from an external source for now if they want it. -gps

Gregory P. Smith wrote:

-1 on "multicore" - multiprocess or multiprocessing are a fine names. cores are irrelevant. systems have multiple cpus real or virtual regardless of how many dies, sockets and cores there are.

+0.5 on inclusion. that means i am happy if it does but don't think it needs to make it into 2.6/3.0. leave inclusion for 2.7/3.1. its easy for people to install from an external source for now if they want it.

This is a nice summary of my opinion as well. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

In the scientific world, MPI is the standard API of choice for doing parallel processing, so if we're after standards, supporting MPI would seem to be more attractive than the processing module.

http://pypi.python.org/pypi/mpi4py

Of course, for MPI, pyprocessing's main functionality (starting new activities) isn't needed - you use the vendor's mpirun binary, which will create as many processes as you wish, following a policy that was set up by the cluster administration, or that you chose in a product-specific manner (e.g. what nodes to involve in the job). If my task was high-performance computing, I would indeed use MPI and ignore pyprocessing.

In the enterprise world, you often find CORBA based solutions.

http://omniorb.sourceforge.net/

Same here: I would prefer CORBA of pyprocessing when I want a componentized, distributed application.

And then, of course, you have a gazillion specialized solutions such as PyRO:

http://pyro.sourceforge.net/

I personally would not use that library, although I know others are very fond of it. Regards, Martin

On Wed, May 14, 2008 at 4:45 AM, "Martin v. Löwis" wrote:

Jesse Noller wrote:

...

I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library - preferably in the 2.6 timeline.

I think for inclusion in 2.6 it's to late. For 3.0, it's definitely too late - the PEP acceptance deadline was a year ago (IIRC).

That's a fair point - however, as this is a proposed inclusion of a standalone library that includes tests, documentation and benchmarks - I was hoping it would be possible to attempt to get it in. A target for 3.1 or 2.7 is also possible.

...

As I am trying to finish up the PEP, I want to see if I can address any questions or include any other useful data (including benchmarks) in the PEP prior to publishing it. I am also intending to include basic benchmarks for both the processing module against the threading module as a comparison.

I'm worried whether it's stable, what user base it has, whether users (other than the authors) are lobbying for inclusion. Statistically, it seems to be not ready yet: it is not even a year old, and has not reached version 1.0 yet.

Just as a point of clarification: I am not the module author. I am simply a relatively passionate user using it to solve real problems. I am working on the PEP after speaking with the author about it. One of the outstanding issues I have listed is the "lack" of a 1.0 version - however given the fairly random way that most open source projects pick numbering schemes - I don't see this as much more than an administrative issue. As for user base - at PyCon this year, I was involved in several open space discussion about the concurrency/parallelism "problem" as it relates to Python, as well as giving a lightning talk on this precise subject including mentioning this module and alternatives. The sheer number of people working on "big" problems who approached me to talk about this was astounding. The presence online of the same like-minded people is also quite large. Overwhelmingly, many of the python programmers I spoke to are looking for "a solution" that does not require the alteration of a known programming paradigm (i.e: threads) to allow them to take advantage of systems which are not getting "faster" - instead, they are getting wider. Simply put due to the GIL - pure python applications can not take advantage of these machines which are now more common than not without switching to an alternative interpreter - which for many - myself included - is not an attractive option. Guido, and others called this module out specifically as a "work-around" to the GIL for concurrency within cPython applications. The API matches a known one and in most (if not all) cases, it is an easy drop in to allow programmers to achieve what they may not be able to do with the standard library as it is today. Obviously, there are other solutions that aim to solve this problem domain - Twisted, ParallelPython, etc all "work around" the GIL through various methods - but the fundamental problem with those is that they ask a user to "re-learn" how to construct their applications. While the concept of shared memory and "threads" may not be the "final answer" in the concurrent programming world - fundamentally, it's wrong for us to say that in order for people to really use python to solve concurrent problems they have to: 1: Re-architect their applications 2: Re-learn how to program (i.e: Async programming, Actor models, shared-nothing) I really do feel that inclusion of this library offers us the best of both worlds - it gives us (as a community) an easy answer to those people who would dismiss python due to the GIL and it also allows users to easily implement their applications. The GIL is currently advantageous to the cPython code base for simple ease of implementation and maintenance. Barring the one-day adoption of the bulk of Adam Olsen's safe-threading work or a massive trend in programming to "rid" ourselves of the threaded mindset - this gives cPython a leg-up in the concurrency world and gives users a tool to solve hard problems. -Jesse

I really do feel that inclusion of this library offers us the best of both worlds - it gives us (as a community) an easy answer to those people who would dismiss python due to the GIL and it also allows users to easily implement their applications.

I really feel that you can get the best of both worlds even without inclusion of the module in the standard library. It should be fairly easy to install, assuming pre-compiled packages are available. For inclusion into Python, a number of things need to be changed, in particular with respect to the C code. It duplicates a lot of code from the socket and os modules, and should (IMO) be merged into these, rather than being separate - or perhaps merged into the subprocess module. Why isn't it possible to implement all this in pure Python, on top of the subprocess module? If there are limitations in Python that make it impossible to implement such functionality in pure Python - then *those* limitations should be overcome, rather than including the code wholesale. Regards, Martin

Apart from (1) and (2) which I think are unavoidable, nearly all the C code is Windows specific. I assume you don't want to bring in the whole of pywin32 into stdlib...

I wouldn't structure it the same way as pywin32, but yes, I have pondered exposing all of Win32 in a module, as part of the standard library. This has somewhat lost relevance with ctypes. Regards, Martin

Christian Heimes wrote:

I assume recvfd and sendfd aren't syscalls but the proposed names for the functions.

Yes, the functionality in question is accessed through the sendmsg() and recvmsg() system calls. -- Greg

Martin v. Löwis schrieb:

...

I can neither find recvfd in my man pages nor in my header files in /usr/include on Linux (Ubuntu 8.04 i686). I assume recvfd and sendfd aren't syscalls but the proposed names for the functions.

Yes (and no). The system call is sendmsg, with a cmsg_type of SCM_RIGHTS. I don't think there is a plan to add library functions with that name.

There is at least a patch: http://bugs.python.org/issue1194378 Georg

On Wed, May 14, 2008 at 6:58 AM, Nick Craig-Wood wrote:

Jesse Noller wrote:

...

I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library - preferably in the 2.6 timeline. In March, I began working on the PEP for the inclusion of the pyprocessing (processing) module into the python standard library[1]. The original email to the stdlib-sig can be found here, it includes a basic overview of the module:

http://mail.python.org/pipermail/stdlib-sig/2008-March/000129.html

The processing module mirrors/mimics the API of the threading module - and with simple import/subclassing changes depending on the code, allows you to leverage multi core machines via an underlying forking mechanism. The module also supports the sharing of data across groups of networked machines - a feature obviously not part of the core threading module, but useful in a distributed environment.

I think processing looks interesting and useful, especially since it works on Windows as well as Un*x.

However I'd like to see a review of the security - anything which can run across networks of machines has security implications and I didn't see these spelt out in the documentation.

Networked running should certainly be disabled by default and need explicitly enabling by the user - I'd hate for a new version of python to come with a remote exploit by default...

-- Nick Craig-Wood -- http://www.craig-wood.com/nick

See the Manager documentation: http://pyprocessing.berlios.de/doc/manager-objects.html And the Listener/Client documentation: http://pyprocessing.berlios.de/doc/connection-ref.html Remote access is not implicit - it is explicit - you must spawn a Manager/Client instance and tell it to use the network instead of it being "always networked". I'll add a security audit to the list of open issues though - that's a good point. -jesse

On Fri, May 16, 2008 at 10:32:30AM +0200, Ulrich Berning wrote:

As long as the ctypes extension doesn't build on major Un*x platforms (AIX, HP-UX), I don't like to see ctypes dependend modules included into the stdlib. Please keep the stdlib as portable as possible. More and more people tend to say "Runs on Un*x" when they really mean "Tested on Linux". Un*x is not Linux.

Python development doesn't seem to have any volunteers who use AIX or HP-UX and can fix bugs on these platforms. Searching bugs.python.org, I find about 20 open AIX issues, 4 or 5 HP-UX issues, 6 IRIX bugs, and about 15 Solaris bugs; but I don't know if any of the developers here use these platforms. (There is a Solaris buildbot, at least.) This means that if people didn't use ctypes and wrote C extension modules instead, those extension modules probably wouldn't work on AIX and HP-UX either. I'd rather see a standard library that's as featureful and useful as possible, and not constrain it in order to support platforms that we don't really seem to be supporting anyway. There's a bug report about ctypes on AIX (1637120) and an old one about ctypes on Solaris that looks like it's fixed, but I can't find a report for HP-UX. Part of the problem seems to be that libffi is written for GCC, so using the vendor compiler often causes difficulties. --amk

Ulrich Berning wrote:

More and more people tend to say "Runs on Un*x" when they really mean "Tested on Linux". Un*x is not Linux.

Hmm, perhaps that would be why there are Solaris, FreeBSD and Tru64 machines amongst the main Python buildbots, to go along with the assorted OS X, Windows and Linux boxes - and as far as I know test_ctypes runs quite happily on all of them. On the specific problems with AIX, HP-UX and ctypes, was it ctypes itself that was failing to build, or the underlying libffi? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

Ulrich Berning wrote:

I'm trying to use the vendor specific compilers whenever possible, because using gcc puts in additional dependencies (libgcc), I want to avoid, and even if I could live with these dependencies, it's not easy to get/build the 'right' gcc version, if your software also depends on other big packages like Qt and PyQt.

I'm not using these platforms for my own pleasure (in fact, I would be happy if these platforms would disappear from the market), but as long as our customers use these platforms, we want to promise our software runs on those platforms.

I have no problem with the fact that ctypes doesn't build on those platforms because I don't use it, but if more and more essential packages depend on ctypes, I'm running into trouble. PyOpenGL is an example of an extension, that moved completely from C-Source (SWIG generated) to ctypes usage.

Hmm, perhaps the ctypes documentation could use a more prominent warning that it may not be available on some Unix platforms (HP-UX, AIX, IRIX), and that it may require the use of GCC rather than the vendor compiler on others (Solaris). At the moment, I suspect some projects may be switching to using it without realising the implications for cross-platform portability. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

Bill Janssen schrieb:

...

Hmm, perhaps the ctypes documentation could use a more prominent warning that it may not be available on some Unix platforms (HP-UX, AIX, IRIX), and that it may require the use of GCC rather than the vendor compiler on others (Solaris).

At the moment, I suspect some projects may be switching to using it without realising the implications for cross-platform portability.

I think it's a tad more problematic. As other modules, both in and out of the standard library, move to use ctypes, it implies that *Python* isn't supported on those combinations. Personally, that's fine with me (as long as there's a workaround for Solaris!), but I think that Ulrich is right in saying this should be more prominent.

I won't object if anyone adds this notice to the Python docs, so please go ahead. A table of platforms (on the wiki?) where ctypes builds/works or does not may also be helpful. Myself I would rather spend my energy to make ctypes more portable, within my skills and the platforms I have access to. Thomas

A.M. Kuchling wrote:

On Mon, May 19, 2008 at 08:50:39PM +0200, Thomas Heller wrote:

...

Myself I would rather spend my energy to make ctypes more portable, within my skills and the platforms I have access to.

Someone could run Solaris x86 inside a hosted virtual machine and make it available to the Python developers. Is it possible to find similar hosting for HP-UX and AIX? Or might IBM or HP be willing to donate a low-end machine to the PSF for porting use?

You can test HPUX and VMS with a (free) HP Testdrive account (along with a bunch of Linuxes and *BSDs) on various hardware platforms. I used it extensively when doing portability testing of another project. http://www.testdrive.hp.com/ Charles -- ----------------------------------------------------------------------- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ -----------------------------------------------------------------------

A.M. Kuchling schrieb:

On Mon, May 19, 2008 at 08:50:39PM +0200, Thomas Heller wrote:

...

Myself I would rather spend my energy to make ctypes more portable, within my skills and the platforms I have access to.

Someone could run Solaris x86 inside a hosted virtual machine and make it available to the Python developers. Is it possible to find similar hosting for HP-UX and AIX? Or might IBM or HP be willing to donate a low-end machine to the PSF for porting use?

I have a vmware appliance installed that runs "solaris 10 update 2", gcc and the sun compiler are installed. As expected (?), ctypes works fine when compiled with gcc, but fails to build with the siun compiler. There is also a solaris buildbot running on a sparc machine. Thomas

So maybe Python just doesn't run on Solaris with the Sun C compiler. Certainly doesn't build out of the box.

Hmmm, when I look at http://www.sun.com/software/solaris/freeware/, I see that Python is listed as "included with Solaris 10" as "Sun-supported software". But the version installed on my Solaris 11 system is 2.4.4. Still, this gives me hope that Sun is looking at Python 2.5, including ctypes, and will eventually make it work. Bill

Bill Janssen schrieb:

...

...

Hmm, perhaps the ctypes documentation could use a more prominent warning that it may not be available on some Unix platforms (HP-UX, AIX, IRIX), and that it may require the use of GCC rather than the vendor compiler on others (Solaris).

At the moment, I suspect some projects may be switching to using it without realising the implications for cross-platform portability.

I think it's a tad more problematic. As other modules, both in and out of the standard library, move to use ctypes, it implies that *Python* isn't supported on those combinations. Personally, that's fine with me (as long as there's a workaround for Solaris!), but I think that Ulrich is right in saying this should be more prominent.

I won't object if anyone adds this notice to the Python docs, so please go ahead. A table of platforms (on the wiki?) where ctypes builds/works or does not may also be helpful.

What happens on those platforms where ctypes doesn't work? Does the module fail to import, either because it isn't present, or because it can't load the libffi library? Or does it fail silently in some way? Bill

Ulrich Berning schrieb:

Nick Craig-Wood wrote:

...

Jesse Noller wrote:

...

I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library - preferably in the 2.6 timeline. In March, I began working on the PEP for the inclusion of the pyprocessing (processing) module into the python standard library[1]. The original email to the stdlib-sig can be found here, it includes a basic overview of the module:

http://mail.python.org/pipermail/stdlib-sig/2008-March/000129.html

The processing module mirrors/mimics the API of the threading module - and with simple import/subclassing changes depending on the code, allows you to leverage multi core machines via an underlying forking mechanism. The module also supports the sharing of data across groups of networked machines - a feature obviously not part of the core threading module, but useful in a distributed environment.

I think processing looks interesting and useful, especially since it works on Windows as well as Un*x.

However I'd like to see a review of the security - anything which can run across networks of machines has security implications and I didn't see these spelt out in the documentation.

Networked running should certainly be disabled by default and need explicitly enabling by the user - I'd hate for a new version of python to come with a remote exploit by default...

As long as the ctypes extension doesn't build on major Un*x platforms (AIX, HP-UX), I don't like to see ctypes dependend modules included into the stdlib. Please keep the stdlib as portable as possible. More and more people tend to say "Runs on Un*x" when they really mean "Tested on Linux". Un*x is not Linux.

Hm, I've never looked at the processing module. Does it depend on ctypes? Anyway, the trunk version of ctypes is a lot more portable than the release25-maint version. I have once tried to build the trunk on HP-UX machines, and, IIRC, it did build on IA64 and PA machines. Of course only with GCC, not with the vendor compilers. Thomas

On Fri, May 16, 2008 at 1:32 AM, Ulrich Berning wrote:

As long as the ctypes extension doesn't build on major Un*x platforms (AIX, HP-UX), I don't like to see ctypes dependend modules included into the stdlib. Please keep the stdlib as portable as possible.

Nice in theory but ctypes already works on at least the top 3 popular platforms. Lets not hold Python's stdlib back because nobody who uses IBM and HP proprietary stuff has contributed the necessary support. Making nice libraries available for other platforms is a good way to encourage people to either pitch in and add support or consider their platform choices in the future. -gps

On Mon, May 19, 2008 at 2:03 AM, Ulrich Berning wrote:

Gregory P. Smith wrote:

...

On Fri, May 16, 2008 at 1:32 AM, Ulrich Berning wrote:

...

As long as the ctypes extension doesn't build on major Un*x platforms (AIX, HP-UX), I don't like to see ctypes dependend modules included into the stdlib. Please keep the stdlib as portable as possible.

Nice in theory but ctypes already works on at least the top 3 popular platforms. Lets not hold Python's stdlib back because nobody who uses IBM and HP proprietary stuff has contributed the necessary support. Making nice libraries available for other platforms is a good way to encourage people to either pitch in and add support or consider their platform choices in the future.

-gps

It's not my platform choice, it's the choice of our customers. I'm not using these platforms just for fun (in fact it isn't fun compared to Linux or Windows).

If porting libffi to AIX, HP-UX, IRIX, Solaris... (especially using vendor compilers) would be an easy job, I'm sure it would have been done already.

Well, ctypes isn't simple. =)

If more and more essential packages depend on ctypes, we should make a clear statement, that Python isn't supported any longer on platform/compiler combinations where libffi/ctypes doesn't build. This would give me arguments to drop support of our software on those platforms.

You are mixing the stdlib in with the language in terms of what is required for Python to work, which I think is unfair. Just because some part of the stdlib isn't portable to some OS does not mean Python is not supported on that platform. If you can run a pure Python module that does not depend on any C extension, then that platform has the support needed to run Python. Everything else is extra (which is why we have modules in the stdlib only available on specific platforms). -Brett

Brett Cannon wrote:

On Mon, May 19, 2008 at 2:03 AM, Ulrich Berning wrote:

...

Gregory P. Smith wrote:

...

On Fri, May 16, 2008 at 1:32 AM, Ulrich Berning wrote:

...

As long as the ctypes extension doesn't build on major Un*x platforms (AIX, HP-UX), I don't like to see ctypes dependend modules included into the stdlib. Please keep the stdlib as portable as possible.

Nice in theory but ctypes already works on at least the top 3 popular platforms. Lets not hold Python's stdlib back because nobody who uses IBM and HP proprietary stuff has contributed the necessary support. Making nice libraries available for other platforms is a good way to encourage people to either pitch in and add support or consider their platform choices in the future.

-gps

It's not my platform choice, it's the choice of our customers. I'm not using these platforms just for fun (in fact it isn't fun compared to Linux or Windows).

If porting libffi to AIX, HP-UX, IRIX, Solaris... (especially using vendor compilers) would be an easy job, I'm sure it would have been done already.

Well, ctypes isn't simple. =)

...

If more and more essential packages depend on ctypes, we should make a clear statement, that Python isn't supported any longer on platform/compiler combinations where libffi/ctypes doesn't build. This would give me arguments to drop support of our software on those platforms.

You are mixing the stdlib in with the language in terms of what is required for Python to work, which I think is unfair. Just because some part of the stdlib isn't portable to some OS does not mean Python is not supported on that platform. If you can run a pure Python module that does not depend on any C extension, then that platform has the support needed to run Python. Everything else is extra (which is why we have modules in the stdlib only available on specific platforms).

-Brett

I don't think it is unfair. If the development team decides one day to reimplement essential extensions like math, _socket, select, _ssl, pwd, grp, time, _locale, zlib... based on ctypes because it may be much easier to maintain python modules instead of dealing with complicated C code, Python will become pretty useless. It's like a cool radio without the chance to get any batteries for it, pretty useless. Platform specific modules are documented as such and nobody would expect a _winreg module on AIX or HP-UX. As said before, PyOpenGL is an example of an extension that moved from C code to Python/ctypes, luckily we don't use it, but what if the maintainers of MySQL-Python or cx_Oracle decide to move to ctypes. Having the ctypes extension in the stdlib doesn't imply it runs on any platform where python runs. Extension writers should keep this in mind when they decide to use ctypes. They should document, that their extension depends on ctypes and therefore doesn't run on platforms where ctypes doesn't work. Ulli

skip@pobox.com wrote:

Ulrich> Having the ctypes extension in the stdlib doesn't imply it runs Ulrich> on any platform where python runs.

Maybe the presence of a functioning ctypes (can|might|should|will) become the operational definition of "Python runs on platform X".

And what about platforms like the JVM or CLR? Incidentally there were a small but vocal group of Pythonistas who were (are?) certain that IronPython is not Python because it doesn't have [all of...] the C extensions. Michael Foord

Skip _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

James Y Knight wrote:

On May 21, 2008, at 11:26 AM, Michael Foord wrote:

...

And what about platforms like the JVM or CLR?

Incidentally there were a small but vocal group of Pythonistas who were (are?) certain that IronPython is not Python because it doesn't have [all of...] the C extensions.

It seems likely to be easier to port ctypes to Jython/IronPython than it is to port 500 C extensions to Jython. So more software using ctype seems like an improvement for those communities.

Yes, both of them have their own FFIs. Seo Sanghyeon started a port of ctypes for IronPython that should be revived by 'someone'. :-) Michael Foord

James

From my perspective IronPython isn't Python because it doesn't run on my

Skip> Maybe the presence of a functioning ctypes (can|might|should|will) Skip> become the operational definition of "Python runs on platform X". Michael> And what about platforms like the JVM or CLR? Sorry, allow me to rephrase: Maybe the presence of a functioning ctypes (can|might|should|will) become the operational definition of "CPython runs on platform X". Michael> Incidentally there were a small but vocal group of Pythonistas Michael> who were (are?) certain that IronPython is not Python because Michael> it doesn't have [all of...] the C extensions. platforms (Solaris & Mac). ;-) (At least not easily enough for an old codger like me to install.) Skip

On Wed, May 21, 2008 at 10:11:05AM -0500, skip@pobox.com wrote:

Maybe the presence of a functioning ctypes (can|might|should|will) become the operational definition of "Python runs on platform X".

It is not black-or-white, runs or doesn't. PythonD, e.g., runs on DOS, can use sockets (from WatTCP library), but certainly cannot do multithreading or multitasking. So the wording should be "Python supports platform X with the following limitations: ..." Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.

Martin v. Löwis schrieb:

...

As said before, PyOpenGL is an example of an extension that moved from C code to Python/ctypes, luckily we don't use it, but what if the maintainers of MySQL-Python or cx_Oracle decide to move to ctypes. Having the ctypes extension in the stdlib doesn't imply it runs on any platform where python runs. Extension writers should keep this in mind when they decide to use ctypes. They should document, that their extension depends on ctypes and therefore doesn't run on platforms where ctypes doesn't work.

Plus, even if ctypes works, the code might be incorrect, because they had been assuming structure layouts and symbolic constants that have just a different definition on some other platform, causing the extension module to crash.

Writing portable ctypes modules is really hard - significantly harder than writing portable C code (although writing non-portable ctypes code is apparently easier than writing non-portable C code).

I would say that writing portable C code is hard as well, aren't there just more tools that help? Thomas

On Thu, 22 May 2008, "Martin v. Löwis" wrote:

...

I would say that writing portable C code is hard as well, aren't there just more tools that help?

The C compiler in particular. It already gets symbolic constants and struct layouts right, something that ctypes can't do (because it doesn't use header files).

I don't think it makes much sense to talk about "The C Compiler" when you are discussing matters of portability. Just my $0.02. -- Cheers, Leif

On May 21, 2008, at 3:58 PM, Jean-Paul Calderone wrote:

...

Plus, even if ctypes works, the code might be incorrect, because they had been assuming structure layouts and symbolic constants that have just a different definition on some other platform, causing the extension module to crash.

Writing portable ctypes modules is really hard - significantly harder than writing portable C code (although writing non-portable ctypes code is apparently easier than writing non-portable C code).

True. There's some room for improvement in ctypes here, fortunately.

For example, PyPy has some tools which resolve the particular problem you're talking about; the library is even available separately and can (and probably should) be used by anyone writing a ctypes module.

Sample usage and installation instructions available here:

http://codespeak.net/~fijal/configure.html

The "csvn" subversion bindings use "ctypesgen" to grovel header files: http://code.google.com/p/ctypesgen/ There's also ctypeslib, which uses gcc-xml to parse the headers instead of a pure python C parser as ctypesgen does: http://svn.python.org/projects/ctypes/trunk/ctypeslib/ I don't really know how all these tools compare to eachother. It sure would be nice if someone could compare them, and figure out if one can be "blessed" and included with python so that doing the right thing is easy. James

On Wed, May 21, 2008 at 1:53 PM, Raymond Hettinger wrote:

This thread has diverged a bit from the original topic.

I suggest going ahead and adding pyprocessing to the library. IMO, its functionality is going to be an essential capability as more and more computers ship with multiple processors. At this point, the basic API for pyprocessing seems well thought-out and somewhat stable. Over time, I expect the implementation will get tweaked in a number of ways including support more platforms as developers figure-out that they like the idea enough to write some platform dependent patches.

Putting this functionality in 2.6/3.0 would provide a really nice incentive to update from Py2.5. It would be a sad lost opportunity if this module had to wait another couple years.

I feel essentially the same way: it WOULD be sad to waste this excellent opportunity, so I second the suggestion to put pyprocessing in the library right now. Alex

Nick Coghlan schrieb:

Alex Martelli wrote:

...

On Wed, May 21, 2008 at 1:53 PM, Raymond Hettinger wrote:

...

Putting this functionality in 2.6/3.0 would provide a really nice incentive to update from Py2.5. It would be a sad lost opportunity if this module had to wait another couple years.

I feel essentially the same way: it WOULD be sad to waste this excellent opportunity, so I second the suggestion to put pyprocessing in the library right now.

Thirded (I think I'm contradicting what I posted earlier in the thread, but I've had more of a chance to think about it and 18 months really is quite a long time to wait for this functionality...)

This seems to be enough support to put an entry into PEP 361, which I've just done. Personally, I'd also find it useful to have such a thing in the stdlib, since at the moment multiple processes are really the way to go for parallelism in Python, and Python needs to stress this fact. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

Ulrich Berning schrieb:

If porting libffi to AIX, HP-UX, IRIX, Solaris... (especially using vendor compilers) would be an easy job, I'm sure it would have been done already. If more and more essential packages depend on ctypes, we should make a clear statement, that Python isn't supported any longer on platform/compiler combinations where libffi/ctypes doesn't build. This would give me arguments to drop support of our software on those platforms.

Ulrich, if *you* have access to these platforms and want to help a good start which hopefully wouldn't take too much time would be to compile the current libffi release [1], run the test suite, and report the results. Compile with gcc, for a start, and note that the testsuite requires dejagnu (the PyObjC folks have written a dejagnu replacement in Python for testing libffi, but I haven't tried that). [1] http://sourceware.org/libffi/ -- Thanks, Thomas

[Facundo]

I think that including in the stdlib a threading-like-API module is a clear win.

I also think this is vital and should not wait for Py2.7. Raymond