Moving forward with the concurrent package
Now that we have concurrent.futures, is there any plan for multiprocessing to follow suit? PEP 3148 mentions a hope to add or move things in the future [0], which would be now. [0] http://www.python.org/dev/peps/pep-3148/#naming
2011/8/10 Brian Curtin <brian.curtin@gmail.com>:
Now that we have concurrent.futures, is there any plan for multiprocessing to follow suit? PEP 3148 mentions a hope to add or move things in the future
Is there some sort of concrete proposal? The PEP just seems to mention it as an idea. In general, -1. I think we don't need to be moving things around more to little advantage. -- Regards, Benjamin
Le Wed, 10 Aug 2011 14:54:33 -0500, Benjamin Peterson <benjamin@python.org> a écrit :
2011/8/10 Brian Curtin <brian.curtin@gmail.com>:
Now that we have concurrent.futures, is there any plan for multiprocessing to follow suit? PEP 3148 mentions a hope to add or move things in the future
Is there some sort of concrete proposal? The PEP just seems to mention it as an idea.
In general, -1. I think we don't need to be moving things around more to little advantage.
Agreed. Also, flat is better than nested. Whoever wants to populate the concurrent package should work on new features to be added to it, rather than plans to rename things around. Regards Antoine.
On Wed, Aug 10, 2011 at 15:36, Antoine Pitrou <solipsis@pitrou.net> wrote:
Le Wed, 10 Aug 2011 14:54:33 -0500, Benjamin Peterson <benjamin@python.org> a écrit :
2011/8/10 Brian Curtin <brian.curtin@gmail.com>:
Now that we have concurrent.futures, is there any plan for multiprocessing to follow suit? PEP 3148 mentions a hope to add or move things in the future
Is there some sort of concrete proposal? The PEP just seems to mention it as an idea.
In general, -1. I think we don't need to be moving things around more to little advantage.
Agreed. Also, flat is better than nested. Whoever wants to populate the concurrent package should work on new features to be added to it, rather than plans to rename things around.
I agree with flat being better than nested and won't be pushing to move things around, but the creation of the concurrent package seemed like a place to put those things. I just found myself typing "concurrent.multiprocessing" a minute ago, so I figured I'd put it out there.
On Aug 10, 2011, at 1:36 PM, Antoine Pitrou wrote:
Le Wed, 10 Aug 2011 14:54:33 -0500, Benjamin Peterson <benjamin@python.org> a écrit :
2011/8/10 Brian Curtin <brian.curtin@gmail.com>:
Now that we have concurrent.futures, is there any plan for multiprocessing to follow suit? PEP 3148 mentions a hope to add or move things in the future
Is there some sort of concrete proposal? The PEP just seems to mention it as an idea.
In general, -1. I think we don't need to be moving things around more to little advantage.
Agreed. Also, flat is better than nested. Whoever wants to populate the concurrent package should work on new features to be added to it, rather than plans to rename things around.
I concur. Raymond
On Wed, Aug 10, 2011 at 4:45 PM, Brian Curtin <brian.curtin@gmail.com> wrote:
On Wed, Aug 10, 2011 at 15:36, Antoine Pitrou <solipsis@pitrou.net> wrote:
Le Wed, 10 Aug 2011 14:54:33 -0500, Benjamin Peterson <benjamin@python.org> a écrit :
2011/8/10 Brian Curtin <brian.curtin@gmail.com>:
Now that we have concurrent.futures, is there any plan for multiprocessing to follow suit? PEP 3148 mentions a hope to add or move things in the future
Is there some sort of concrete proposal? The PEP just seems to mention it as an idea.
In general, -1. I think we don't need to be moving things around more to little advantage.
Agreed. Also, flat is better than nested. Whoever wants to populate the concurrent package should work on new features to be added to it, rather than plans to rename things around.
I agree with flat being better than nested and won't be pushing to move things around, but the creation of the concurrent package seemed like a place to put those things. I just found myself typing "concurrent.multiprocessing" a minute ago, so I figured I'd put it out there.
I would like to move certain *features* of multiprocessing into that namespace - some things like map and others don't belong in the multiprocessing namespace, and should have been put into concurrent.* a long time ago. As for my plans: I had intended on making multiprocessing a closer corollary to threading, and moving the bigger features that should have been broken out into a different package (such as http://bugs.python.org/issue12708) and the managers. Those plans are obviously stalled as my time is being spent elsewhere. I disagree on the "flat is better than nested" point - multiprocessing's namespace is flat - but bloated, and many of it's features could work just as well in a threaded context (e.g, they are generally useful outside of multiprocessing alone). Regardless; currently I can't lead this, and multiprocessing-sig is silent. Jesse
2011/8/10 Raymond Hettinger <raymond.hettinger@gmail.com>:
On Aug 10, 2011, at 1:36 PM, Antoine Pitrou wrote:
Le Wed, 10 Aug 2011 14:54:33 -0500, Benjamin Peterson <benjamin@python.org> a écrit :
2011/8/10 Brian Curtin <brian.curtin@gmail.com>:
Now that we have concurrent.futures, is there any plan for multiprocessing to follow suit? PEP 3148 mentions a hope to add or move things in the future
Is there some sort of concrete proposal? The PEP just seems to mention it as an idea.
In general, -1. I think we don't need to be moving things around more to little advantage.
Agreed. Also, flat is better than nested. Whoever wants to populate the concurrent package should work on new features to be added to it, rather than plans to rename things around.
I concur.
So we could put yourself, Antoine, and me in the concurrent package. :) Sorry, Benjamin
On Thu, Aug 11, 2011 at 4:55 AM, Brian Curtin <brian.curtin@gmail.com> wrote:
Now that we have concurrent.futures, is there any plan for multiprocessing to follow suit? PEP 3148 mentions a hope to add or move things in the future [0], which would be now.
As Jesse said, moving multiprocessing or threading wholesale was never part of the plan. The main motivator of that comment in PEP 3148 was the idea of creating 'concurrent.pool', which would provide a concurrent worker pool API modelled on multiprocessing.Pool that supported either threads or processes as the back end, just like the executor model in concurrent.futures. The basic approach is to look at a feature in threading or multiprocessing that is only available in one of them and ask the question: Does it make sense to allow a project to switch easily between a threading strategy and a multiprocessing strategy when using this feature? If the answer to that question is yes (as it was for concurrent.futures itself, and as I believe it to be for multiprocessing.Pool), then a feature request (and probably a PEP) proposing the definition of a common API in the concurrent namespace would be appropriate. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Aug 10, 2011 at 7:03 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Thu, Aug 11, 2011 at 4:55 AM, Brian Curtin <brian.curtin@gmail.com> wrote:
Now that we have concurrent.futures, is there any plan for multiprocessing to follow suit? PEP 3148 mentions a hope to add or move things in the future [0], which would be now.
As Jesse said, moving multiprocessing or threading wholesale was never part of the plan. The main motivator of that comment in PEP 3148 was the idea of creating 'concurrent.pool', which would provide a concurrent worker pool API modelled on multiprocessing.Pool that supported either threads or processes as the back end, just like the executor model in concurrent.futures.
The basic approach is to look at a feature in threading or multiprocessing that is only available in one of them and ask the question: Does it make sense to allow a project to switch easily between a threading strategy and a multiprocessing strategy when using this feature?
If the answer to that question is yes (as it was for concurrent.futures itself, and as I believe it to be for multiprocessing.Pool), then a feature request (and probably a PEP) proposing the definition of a common API in the concurrent namespace would be appropriate.
Precisely. Thank you Nick, want a job working for PyCon? ;)
Le Thu, 11 Aug 2011 09:03:35 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
On Thu, Aug 11, 2011 at 4:55 AM, Brian Curtin <brian.curtin@gmail.com> wrote:
Now that we have concurrent.futures, is there any plan for multiprocessing to follow suit? PEP 3148 mentions a hope to add or move things in the future [0], which would be now.
As Jesse said, moving multiprocessing or threading wholesale was never part of the plan. The main motivator of that comment in PEP 3148 was the idea of creating 'concurrent.pool', which would provide a concurrent worker pool API modelled on multiprocessing.Pool that supported either threads or processes as the back end, just like the executor model in concurrent.futures.
Executors *are* pools, so I don't know what you're talking about. Besides, multiprocessing.Pool is quite bloated and therefore difficult to improve. It should be slowly phased out in favour of concurrent.futures. In general, it would be nice if people wanting to improve the concurrent primitives made actual, concrete propositions. We've had lots of hand-waving in that area for years, to no effect. Regards Antoine.
On Thu, Aug 11, 2011 at 5:07 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Le Thu, 11 Aug 2011 09:03:35 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
On Thu, Aug 11, 2011 at 4:55 AM, Brian Curtin <brian.curtin@gmail.com> wrote:
Now that we have concurrent.futures, is there any plan for multiprocessing to follow suit? PEP 3148 mentions a hope to add or move things in the future [0], which would be now.
As Jesse said, moving multiprocessing or threading wholesale was never part of the plan. The main motivator of that comment in PEP 3148 was the idea of creating 'concurrent.pool', which would provide a concurrent worker pool API modelled on multiprocessing.Pool that supported either threads or processes as the back end, just like the executor model in concurrent.futures.
Executors *are* pools, so I don't know what you're talking about.
Yes, that's the point. A developer shouldn't be forced into using a particular invocation model (i.e. futures) just to get thread or process pool functionality - the pool should be a lower layer building block that's provided separately. As you say, though, nobody has stepped up for the task of actually defining that common, lower level interface. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Aug 11, 2011 at 10:56 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Thu, Aug 11, 2011 at 5:07 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Le Thu, 11 Aug 2011 09:03:35 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
On Thu, Aug 11, 2011 at 4:55 AM, Brian Curtin <brian.curtin@gmail.com> wrote:
Now that we have concurrent.futures, is there any plan for multiprocessing to follow suit? PEP 3148 mentions a hope to add or move things in the future [0], which would be now.
As Jesse said, moving multiprocessing or threading wholesale was never part of the plan. The main motivator of that comment in PEP 3148 was the idea of creating 'concurrent.pool', which would provide a concurrent worker pool API modelled on multiprocessing.Pool that supported either threads or processes as the back end, just like the executor model in concurrent.futures.
Executors *are* pools, so I don't know what you're talking about.
Also the Pool from multiprocessing "works" for threads and process: from multiprocessing.pool import Pool as ProcessPool from multiprocessing.dummy import Pool as ThreadPool
participants (7)
-
Antoine Pitrou -
Benjamin Peterson -
Brian Curtin -
Jesse Noller -
Nick Coghlan -
Raymond Hettinger -
yoav glazner