On Thu, May 29, 2008 at 6:22 AM, Michael Foord wrote:

Nick Coghlan wrote:

...

Jesse Noller wrote:

...

Georg kindly published the PEP I submitted last night to the PEP site:

http://www.python.org/dev/peps/pep-0371/

This PEP includes some of the previous discussion on the processing module's inclusion, and I hope clears up/clarifies some of the goals/non goals and issues. I also included benchmark data and a link to the code used for said benchmarks.

I would like to renew the discussion now that "there is a PEP" to see if there are any outstanding things people would like to get resolved. I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback both here and elsewhere that people would rather see this in sooner in some form, rather than later (i.e.: 2.7/3.1).

+1 from me (under the 'multiprocessing' name, with the understanding that some code duplication with other parts of the standard library may still remain in 2.6/3.0).

+1 from me as well.

I think multiple-processes is over played as a concurrency solution in Python (where you need to marshal lots of data in and out, the overheads of multiple processes can be very expensive) - but it is a very good solution for some problems.

Michael Foord

Agreed - this is a "step" rather than the final solution. As I pointed out in the PEP this is a method to side-step GIL limitations rather than to address the larger "GIL issue", I am implicitly assuming that no movement will be made on that front until the bulk of Adam Olsen's safethreading work is rolled into future versions. -jesse

On Thu, May 29, 2008 at 9:39 AM, Jesus Cea wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Could you possibly extend the PEP to also document performance when, for instance, passing "big" objects via a queue, or sending "Events" back, testing "thread.isAlive()", and stuff like that?. What about mutexes? (not to protect shared objects, but files, for example).

A share-nothing without data-passing doesn't need a new module :). I'm interesting in an almost direct conversion from thread module, and so I'm interested in knowing performance data outside "pyprocessing" sweet point (that is, "fire and forget" code, with little communication).

How is implemented "thread.setDaemon()"?.

Alec Thomas sent me a bit of code to benchmark Queue-based object passing performance which I will incorporate when I get a chance. As for the provided examples/benchmarks - I can work on adding more, or if you want - as linked in the PEP, Oudkerk already has some of those outlined in a benchmark script here: http://pyprocessing.berlios.de/examples/benchmarks.py I chose not to recreate his tests directly, rather I chose to link to them. I will work on adding Queue-based numbers. I also wouldn't say I picked the "sweet spot" for the module - rather I picked the poor-spot for the threading module (parallel, python-based crunching). I do again want to point out that the goal is not to pick on threading, but to offer an API which mimics the existing threading API that allows for actual multi-processor/core usage. -jesse

...

...

I would like to renew the discussion now that "there is a PEP" to see if there are any outstanding things people would like to get resolved. I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback both here and elsewhere that people would rather see this in sooner in some form, rather than later (i.e.: 2.7/3.1).

+1 from me (under the 'multiprocessing' name, with the understanding that some code duplication with other parts of the standard library may still remain in 2.6/3.0).

+1 from me as well, and I support the rename to multiprocessing ("processing" is just too much of a common word).

This will provide a good short-term answer to those who still clamor for dropping the GIL.

I'm very happy this is going forward. It's a big win. Raymond

On Fri, May 30, 2008 at 2:19 AM, Farshid Lashkari wrote:

On Wed, May 28, 2008 at 10:03 AM, Jesse Noller wrote:

...

I would like to renew the discussion now that "there is a PEP" to see if there are any outstanding things people would like to get resolved. I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback both here and elsewhere that people would rather see this in sooner in some form, rather than later (i.e.: 2.7/3.1).

I'm not sure if this is a big issue, but I just tested the module with an application that embeds Python and it doesn't work properly. Instead of spawning worker threads using python.exe, it attempts to use the application. Does the processing module allow specifying the exe to use for spawning worker threads? I would definitely like to see this included in the next release, and having support for embedded Python interpreters would be a big plus.

-Farshid

Let's take this off-list Farshid - but I would like to know more about your application (I'm not up to speed with embedding python in other applications) so I can either add it to the PEP or discuss how best to address this within the module itself. -jesse

Thomas Heller wrote:

Jesse Noller schrieb:

...

On Fri, May 30, 2008 at 2:19 AM, Farshid Lashkari wrote:

...

...

I would like to renew the discussion now that "there is a PEP" to see if there are any outstanding things people would like to get resolved. I chose to continue to push it for 2.6 / 3.0 inclusion due to feedback both here and elsewhere that people would rather see this in sooner in some form, rather than later (i.e.: 2.7/3.1). I'm not sure if this is a big issue, but I just tested the module with an application that embeds Python and it doesn't work properly. Instead of spawning worker threads using python.exe, it attempts to use the application. Does the processing module allow specifying the exe to use for spawning worker threads? I would definitely like to see

On Wed, May 28, 2008 at 10:03 AM, Jesse Noller wrote: this included in the next release, and having support for embedded Python interpreters would be a big plus.

-Farshid

Let's take this off-list Farshid - but I would like to know more about your application (I'm not up to speed with embedding python in other applications) so I can either add it to the PEP or discuss how best to address this within the module itself.

This may be an issue for py2exe users as well. Please keep me informed (or is there a pyprocessing mailing list where this is discussed?).

py2exe is explicitly supported (via the freezeSupport() call). That may work for the embedded case as well, or it may be something that can be addressed by modifying sys.executable. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

2008/5/30 Farshid Lashkari :

I'm not sure if there will be any side affects to modifying sys.executable though. Should this be the official way of supporting embedded interpreters or should there be a multiprocessing.setExecutable() method?

+1 for setExecutable (I'd prefer set_executable, to be PEP 8 compliant). Make it explicit, rather than fiddling with stuff in sys manually. Paul.

2008/5/31 r. m. oudkerk :

I am certainly open to using lowercase/lower_case_with_underscores for all functions/methods except for Process's methods and possibly currentProcess(), but I would like some feed back on that.

I dislike mixedCase, but consistency with the rest of the library is more important - and as processing is matching the API of threading, which used mixedCase, it should follow that convention. Wasn't there some talk of changing modules to use PEP 8 conventions (lowercase_with_underscore) as part of the Python 3.0 conversion? Did that ever happen? Paul.

Nick> We fixed the module names that used mixed case - the amount of Nick> work that turned out to be involved in just doing that much for Nick> PEP 3108 makes me shudder at the thought of trying to fix all of Nick> the standard library APIs that currently don't follow the style Nick> guide... If the 3.0 API of a module is going to involve breakage which requires authors to update their applications wouldn't this be a good time to PEP-8-ify the module? (Not suggesting that threading would fall into this category.) Skip

>> If the 3.0 API of a module is going to involve breakage which >> requires authors to update their applications wouldn't this be a good >> time to PEP-8-ify the module? (Not suggesting that threading would >> fall into this category.) Nick> Updating application code to deal with a module name change is Nick> easy - just use an "import x as y" statement. Catching the Nick> ImportError for the 3.0 name and falling back to the 2.6 name (or Nick> vice-versa) even makes it possible to support both names fairly Nick> easily. Nick> Changing the names of actual objects within the modules is tougher Nick> though - there are many more ways to access those. I think you misunderstood what I wrote. Suppose you decided that the API for the threading modules needs significant rework, changes which will break much, if not all current usage. I'm only suggesting that if you decide the API change is worthwhile that you take the opportunity to make the class, method and data names PEP8-compliant. I'm not talking about gratuitous change to the identifier names, but situations where the user is going to have to rework their code anyway. Skip

2008/5/30 Farshid Lashkari :

...

I'm not sure if there will be any side affects to modifying sys.executable though. Should this be the official way of supporting embedded interpreters or should there be a multiprocessing.setExecutable() method?

+1 for setExecutable (I'd prefer set_executable, to be PEP 8 compliant). Make it explicit, rather than fiddling with stuff in sys manually.

sys.executable typically means the "current" executable, and py2exe etc already fiddles with that. The question in such a context seems to be "what executable should I use for this processing functionality?". In a py2exe like environment, it might not be unreasonable to assume that if a custom executable is to be used, that custom executable may have a different command-line or other special requirements. Further, I could imagine a system that uses an alternative way of starting processes (eg, 'distributed COM') where the concept of 'executable' (or even 'command-line') don't make much sense. So it seems that maybe simply "setExecutable()" isn't the correct abstraction here, but maybe a "factory" approach, so the entire process creation mechanism can be replaced rather than just the name of the executable to spawn? Cheers, Mark