Re: [Python-ideas] A concurrency survey of sorts
Is this a serious proposal? Without threads, is there any reasonable cross-platform way to communicate interactively with multiple subprocesses over pipes? Windows doesn't have select. I don't *like* using threads, but as far as I can tell they're the only available way to do this.
2011/11/4 Weeble <clockworksaint@gmail.com>
Windows does have a select(), even if this only works for sockets, (you can use WaitForSingleObject for any win32 handle) and the Twisted reactor has a spawnProcess() function that works without any additional thread. -- Amaury Forgeot d'Arc
On Fri, Nov 4, 2011 at 6:13 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
select: Yes. WaitForSingleObject: Really? The documentation seems to say only on events, mutexes, semaphores, processes and threads. No mention of pipes or files. http://msdn.microsoft.com/en-us/library/bb202783.aspx Also, presumably it would be WaitForMultipleObjects? Regardless, I'm sure there's some way to do this. I just don't think it's obvious or easy enough to be considered a reasonable alternative to using threads. Twisted: That actually looks quite nice, but pulling in Twisted isn't always going to be a reasonable option. I just want to be sure - the idea of removing the threading module sounds ludicrous. It's not getting serious consideration, is it?
On Fri, 4 Nov 2011 18:42:25 +0000 Weeble <clockworksaint@gmail.com> wrote:
It works with pipes for sure, it's used in multiprocessing (in 3.3). However, the pipes have to be set non-blocking, which implies using overlapped I/O (or another form of non-blocking Windows I/O) which is quite a pain.
I just want to be sure - the idea of removing the threading module sounds ludicrous. It's not getting serious consideration, is it?
No :-) Regards Antoine.
On Fri, Nov 4, 2011 at 11:42 AM, Weeble <clockworksaint@gmail.com> wrote:
I just want to be sure - the idea of removing the threading module sounds ludicrous. It's not getting serious consideration, is it?
Removing the threading module - as part of a change that includes providing some other concurrent processing facility - seems like a reasonable enough idea that I'll be including it in the sequel. Rewriting it to use those facilities might be reasonable for backwards compatibility, but the new facilities should have enough advantages that new code would just use them. <mike
On Fri, Nov 04, 2011 at 12:02:06PM -0700, Mike Meyer wrote:
I obviously haven't been paying enough attention to this conversation, because this strikes me as a horrible idea. In particular, I write C extensions that release the GIL and scale just fine across multiple processors. Is the idea to force those kind of applications to use a new communications model? cheers, --titus -- C. Titus Brown, ctb@msu.edu
On Fri, Nov 4, 2011 at 12:09 PM, C. Titus Brown <ctb@msu.edu> wrote:
That's one of the ideas. The goal is to get the Python processor to help locate concurrency bugs. Since the threading module communications model is "everything is shared by default", I'm pretty sure it's going to break code that depends on it. But that's why it's a proposal for Python 4 or something even further away. <mike
On Sat, Nov 5, 2011 at 5:20 AM, Mike Meyer <mwm@mired.org> wrote:
I'd suggest it's actually a proposal for a language that *isn't Python*. There's no way an idea like this would ever be accepted into Python itself without real world experimentation in another context. It's going to be hard to get anyone interested enough to seriously try it out, but I'd suggest going the "Python-based language" route (ala Boo and Cython) rather than claiming the idea is Python as such. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, 5 Nov 2011 16:34:06 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
That's been obvious from the start. But this does raise an interesting question: assuming that this change could be made in a way that the affect on most single-threaded code is minor (which is assuming a *lot*), is the fact that it will break almost all concurrent code enough to mean that it's really for a language other than Python? Thanks, <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
2011/11/4 Weeble <clockworksaint@gmail.com>
Windows does have a select(), even if this only works for sockets, (you can use WaitForSingleObject for any win32 handle) and the Twisted reactor has a spawnProcess() function that works without any additional thread. -- Amaury Forgeot d'Arc
On Fri, Nov 4, 2011 at 6:13 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
select: Yes. WaitForSingleObject: Really? The documentation seems to say only on events, mutexes, semaphores, processes and threads. No mention of pipes or files. http://msdn.microsoft.com/en-us/library/bb202783.aspx Also, presumably it would be WaitForMultipleObjects? Regardless, I'm sure there's some way to do this. I just don't think it's obvious or easy enough to be considered a reasonable alternative to using threads. Twisted: That actually looks quite nice, but pulling in Twisted isn't always going to be a reasonable option. I just want to be sure - the idea of removing the threading module sounds ludicrous. It's not getting serious consideration, is it?
On Fri, 4 Nov 2011 18:42:25 +0000 Weeble <clockworksaint@gmail.com> wrote:
It works with pipes for sure, it's used in multiprocessing (in 3.3). However, the pipes have to be set non-blocking, which implies using overlapped I/O (or another form of non-blocking Windows I/O) which is quite a pain.
I just want to be sure - the idea of removing the threading module sounds ludicrous. It's not getting serious consideration, is it?
No :-) Regards Antoine.
On Fri, Nov 4, 2011 at 11:42 AM, Weeble <clockworksaint@gmail.com> wrote:
I just want to be sure - the idea of removing the threading module sounds ludicrous. It's not getting serious consideration, is it?
Removing the threading module - as part of a change that includes providing some other concurrent processing facility - seems like a reasonable enough idea that I'll be including it in the sequel. Rewriting it to use those facilities might be reasonable for backwards compatibility, but the new facilities should have enough advantages that new code would just use them. <mike
On Fri, Nov 04, 2011 at 12:02:06PM -0700, Mike Meyer wrote:
I obviously haven't been paying enough attention to this conversation, because this strikes me as a horrible idea. In particular, I write C extensions that release the GIL and scale just fine across multiple processors. Is the idea to force those kind of applications to use a new communications model? cheers, --titus -- C. Titus Brown, ctb@msu.edu
On Fri, Nov 4, 2011 at 12:09 PM, C. Titus Brown <ctb@msu.edu> wrote:
That's one of the ideas. The goal is to get the Python processor to help locate concurrency bugs. Since the threading module communications model is "everything is shared by default", I'm pretty sure it's going to break code that depends on it. But that's why it's a proposal for Python 4 or something even further away. <mike
On Sat, Nov 5, 2011 at 5:20 AM, Mike Meyer <mwm@mired.org> wrote:
I'd suggest it's actually a proposal for a language that *isn't Python*. There's no way an idea like this would ever be accepted into Python itself without real world experimentation in another context. It's going to be hard to get anyone interested enough to seriously try it out, but I'd suggest going the "Python-based language" route (ala Boo and Cython) rather than claiming the idea is Python as such. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, 5 Nov 2011 16:34:06 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
That's been obvious from the start. But this does raise an interesting question: assuming that this change could be made in a way that the affect on most single-threaded code is minor (which is assuming a *lot*), is the fact that it will break almost all concurrent code enough to mean that it's really for a language other than Python? Thanks, <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
participants (6)
-
Amaury Forgeot d'Arc -
Antoine Pitrou -
C. Titus Brown -
Mike Meyer -
Nick Coghlan -
Weeble