Exploration PEP : Concurrency for moderately massive (4 to 32 cores) multi-core architectures
Thinking about how Python can better support parallelism and concurrency is an important topic. Here is how I see it: if we don't address the issue, the Python interpreter 5 or 10 years from now will run at roughly the same speed as it does today. This is because single CPU cores are not getting much faster (power consumption is too high). Instead, most of the performance gains in hardware will be due to increased hardware parallelism, which means multi/many core CPUs. What to do about this pending crisis is a complicated issue. There are (at least) two levels that are important: 1. Language level features that make it possible to build higher-level libraries/tools for parallelism. 2. The high-level libraries/tools that most users and developers would use to express parallelism. I think it is absolutely critical that we worry about (1) before jumping to (2). So, some thoughts about (1). Does Python itself need to be changed to better enable people to write libraries for expressing parallelism? My answer to this is no. The dominant languages for parallel computing (C/C++/Fortran) don't really have any additional constructs or features above Python in this respect. Java has a more sophisticated support for threads. Erlang has concurrency built into its core. But, Python is not Erlang or Java. As Twisted demonstrates, Python as a language is plenty powerful enough to express concurrency in an elegant way. I am not saying that parallelism and concurrency is easy or wonderful today in Python, just that the language itself is not the problem. We don't necessarily need new language features, we simply need bright people to sit down and think about the right way to express parallelism in Python and then write libraries (maybe in the stdlib) that implement those ideas. But, there is a critical problem in CPython's implementation that prevents people from really breaking new ground in this area with Python. It is the GIL and here is why: * For the platforms on which Python runs, threads are what the hardware+OS people have given to us as the most fine grained way of mapping parallelism onto hardware. This is true, even if you have philosophical or existential problems with threads. With the limitations of the GIL, we can't take advantage of what hardware gives to us. * A process based solution using message passing is simply not suitable for many parallel algorithms that are communications bound. The shared state of threads is needed in many cases, not because sharing state is a "fantastic idea", but rather because it is fast. This will only become more true as multicore CPUs gain more sophisticated memory architectures with higher bandwidths. Also, the overhead of managing processes is much greater than with threads. Many exellent fine grained parallel approaches like Cilk would not be possible with processes only. * There are a number of powerful, high-level Python packages that already exist (these have been named in the various threads) that allow parallelism to be expressed. All of these suffer from a GIL related problem even though they are process based and use message passing. Regardless of whether you are using blocking/non-blocking sockets/IPC, you can't run long running CPU bound code, because all the network related stuff will stop. You then think, "OK, I will run the CPU intensive stuff in a different thread." If the CPU intensive code is just regular Python, you are fine, the Python interpreter will switch between the network thread and the CPU intensive thread every so often. But the second you run extension code that doesn't release the GIL, you are screwed. The network thread will die until the extension code is done. When it comes to implementing robust process based parallelism using sockets, the last thing you can afford is to have your networking black out like this, and in CPython it can't be avoided. <disclaimer> I am not saying that threads are what everyone should be using to express parallelism. I am only saying that they are needed to implement robust higher-level forms of parallelism on multicore systems, regardless of whether the solution is using process+ threads or threads alone. </disclaimer> Of the dozen or so "parallel Python" packages that currently exist, they _all_ suffer from this problem (some hide it better than others though using clever tricks). We can run but we can't hide. Because of these things, I think the current "Exploratory PEP" is entirely premature. Let's figure out exactly what to do with the GIL and _then_ think about the fun stuff. Brian
Brian, Good points.
What we need to do is not to line them (i.e. GIL removal, parallelism) serially, but work on them simultaneously; that way they will leverage each other. Also, what ever paradigm(s) we zero-in on, can be implemented in other implementations, anyway. Moreover, IMHO, we need not force the GIL issue just yet. I firmly believe that it will find a solution in it's own time frame ... </KS> Cheers <k/> Brian Granger wrote:
While I too love to think about parallelism, until the limitations of the GIL in CPython are fixed, all our grand thoughts will be dead ends at some level.
Also GIL is a challenge only in one implementation (I know, it is an important implementation!) My assumption is that the GIL restriction will be removed one way or another, soon. (same reason, quoting Joe Lewis)
I am not that optimistic I guess. Hopeful though.
What we need to do is not to line them (i.e. GIL removal, parallelism) serially, but work on them simultaneously; that way they will leverage each other. Also, what ever paradigm(s) we zero-in on, can be implemented in other implementations, anyway.
If there were infinitely many people willing for work on this stuff, then I agree, but I don't see even a dozen people hacking on the GIL. And in my mind, once the limitations of the GIL are relaxed, the other parallel stuff won't be very difficult given the work that people have already done in this area.
Moreover, IMHO, we need not force the GIL issue just yet. I firmly believe that it will find a solution in it's own time frame ...
On 9/18/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
If there were infinitely many people willing for work on this stuff, then I agree, but I don't see even a dozen people hacking on the GIL.
In part because many people don't believe it would be productive. For threading to be useful in terms of parallel processing, most memory access has to be read-only. That isn't true today, largely because of reference counts. There are ways around that, by using indirection, or delayed counts, or multiple refcount buckets per object, or even just switching to a tracing GC. So far, no one has been able to make these changes without seriously mangling the C API and/or slowing things down a lot. The current refcount mechanism is so lightweight that it isn't clear this would even be possible. (With 4 or more cores dedicated to just python, it might be worth it anyhow -- but it isn't yet.) So if you want the GIL removed, you need to provide an existence proof that (CPython) memory management can be handled efficiently without it. -jJ
On 9/19/07, Jim Jewett <jimjjewett@gmail.com> wrote:
Is 60-65% of normal CPython "a lot"? (I really should clean things up and post a patch...) -- Adam Olsen, aka Rhamphoryncus
On 9/19/07, Adam Olsen <rhamph@gmail.com> wrote:
On 9/19/07, Jim Jewett <jimjjewett@gmail.com> wrote:
On 9/18/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
Is 60-65% of normal CPython "a lot"?
Yes, but I think it is still better than the last serious attempt, so it would be worth posting patches anyhow. -jJ
On 9/19/07, Jim Jewett <jimjjewett@gmail.com> wrote:
It's not even comparable to the last serious attempt. Even mere atomic refcounting has *negative* scalability at two threads when running pystones. My approach has 95-100% scalability. -- Adam Olsen, aka Rhamphoryncus
Is 60-65% of normal CPython "a lot"?
(I really should clean things up and post a patch...)
Is your implementation based on Python 3000/3.0? or Python 2.x? Regardless, I'd love to see it. How would you describe the changes to the C-api? Radical departure? Somewhat compatible? mostly compatible? No changes? - Matt
On 9/20/07, Matt Knox <mattknox_ca@hotmail.com> wrote:
It's based of py3k. I will post eventually (heh). The C API is mostly compatible (source wise only), notably excepting the various threading things. -- Adam Olsen, aka Rhamphoryncus
On Tue, Sep 18, 2007, Brian Granger wrote:
What to do about this pending crisis is a complicated issue.
Oh, please. Calling this a crisis only causes people to ignore you. Take a look at this URL that Guido posted to the baypiggies list: http://marknelson.us/2007/07/30/multicore-panic/ -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ The best way to get information on Usenet is not to ask a question, but to post the wrong information.
What to do about this pending crisis is a complicated issue.
Oh, please. Calling this a crisis only causes people to ignore you.
I apologize if this statement is a little exaggerated. But, I do think this is a really critical problem that is going to affect certain groups of Python users and developers in adverse ways. Perhaps I have not made a very strong case that it is a true "crisis" though.
Take a look at this URL that Guido posted to the baypiggies list:
I actually agree with many of the comments made by the author. The quotes from John Dvorak and Wired are over the top. The author make good points about operating systems' abilities to handle threading on modest numbers of cores. It does work, even today, and will continue to work (for a while) as the number of cores increases. But the main point of the author is that operating systems DO have threading support that works reasonably well. Python is another story. The author even makes comments that provide a string critique of Python's (and Ruby's) threading capabilities: <quote> Modern programs tend to be moderately multithreaded, with individual threads dedicated to the GUI, to user I/O, to socket I/O, and often to computation. Multicore CPUs take advantage of this quite well. And we don't need any new technology to make sure multi-threaded programs are well-behaved - these techniques are pretty well understood, and in use in most software you use today. Modern languages like Java support threads and various concurrency issues right out of the box. C++ requires non-standard libraries, but all modern C++ environments worth their salt deal with multithreading in a fairly sane way. </quote> Modern programs tend to be moderately multithreaded.....except for Python. Modern languages like Java and C++ support threads and concurrency, even if those capabilities aren't built in at a low level (C++). I don't think the same thing can be said about Python. The GIL in CPython does in fact prevent threading from being a general solution for CPU bound parallelism. The author is wrong in one important respect: <quote> In this future view, by 2010 we should have the first eight-core systems. In 2014, we're up to 32 cores. By 2017, we've reached an incredible 128 core CPU on a desktop machine. </quote> I don't know where the author got this information, but it is way off. Here are some currently available examples: * Appro now offers a workstation with up to 4 quad core Opterons: http://www.appro.com/product/workstationxtreme_opteron.asp That is a 16 core system. It is 2007, not 2012. * Tilera offers a 64 core CPU that runs SMP Linux. * SiCortex offers a low power 648 processor Linux system that is organized into 6 core SMPs, each of which runs Linux. This week I am at a Department of Defense conference on multicore computing: http://www.ll.mit.edu/HPEC/2007/index.html There is broad agreement from all corners that multi/manycore CPUs will require new ways of expressing parallelism. But in all of the discussions about new approaches, people do assume that threads are a low-level building block that can and should be used for building the higher-level stuff. Intel's Threaded Building Blocks is a perfect example. While it is implemented using threads, it provides much higher level abstractions for developers to use in building applications that will scale well on multicore systems. I would _love_ to have such constructs in Python, but that is simply not possible. And process based solutions don't provide a solution for the many algorithms that require fine grained partitioning and fast data movement. For those of us who do use Python for high performance computing, these issues are critical. In fact, anytime I deal with fine grained parallel algorithms, I use C/C++. I do end up wrapping my low level C/C++ threaded code into Python, but this doesn't always map onto the problem well. The other group of people for whom this is a big issue, are general Python users that don't think they need parallelism. Eventually even these people will become frustrated that their Python codes are the same speed on a 1 core system as a 128 core system. To me this is a significant problem. Brian
Brian Granger writes:
No, you're missing the point. I don't see anybody denying that you understand your own needs. *You* may face a (true) crisis. *The Python community* does not perceive your crisis as its own. Personally, I don't see why it should. And I think you'd be much more successful at selling this with a two-pronged approach of evangelizing just how utterly cool it would be to have a totally-threading GIL-less Python on the one hand, and recruiting some gung-ho grad students with Google SoC projects (or *gasp* some of your DoE grant or VC money) on the other. Note that nobody has said anything to discourage this as a research project. Nothing like it's impossible, stupid, or YAGNI. But Guido, and other senior developers, are saying they're not going to devote their resources to it as things currently stand (and one of those resources the attention of the folks who review PEPs).
Stephen,
<KS> I am not sure that is true. I think if we have a well thought out PEP that addresses parallelism, it would be looked into by the folks. It is just that there are too many different ways of doing it and of course, the GIL doesn't help either. There is also a school of thought that we should save the GIL and as a result we will find another better way of leveraging the multi-core architectures. </KS> CHeers <k/> Stephen J. Turnbull wrote:
Krishna Sankar writes:
... and provides a plausible proof-of-concept implementation [Guido said that to you explicitly] ...
it would be looked into by the folks.
True. But AIUI without the implementation, it won't become a PEP. And at present you don't have a plausible implementation with the GIL. Now, it looks like Adam has a plausible implementation of removing the GIL. If that holds up under at least some common use-cases, I think you'll see enthusiasm from some of the top developers, and acceptance from most. But I really think that has to come first.
Stephen, Now I see where you are coming from and you are right. Plausible is the operative word. It is not that Guido and Sr.Developers will not look at it at all, but will not look at it in the absence of a plausible implementation. This was evident in the blog discussions with Bruce as well. I was encouraged by Guido's reply. He is doing the right thing. Anyway, the point is, now we have a good understanding of the dynamics. There is a small community who would like to see this happen and it is up to us to show it can be done. A few key things need to happen: a) A good set of benchmarks on multi-core systems, extending excellent work (for example http://blogs.warwick.ac.uk/dwatkins/entry/benchmarking_parallel_python_1_2/) I have an 8 core machine (2 X 4 core) and plan to run this benchmark as well as create others like the Red Black Tree (Guido had suggested that, even before that I was thinking of it) b.1) Plausible POC with GIL - pp, actor pattern/agent paradigm et al. b.2) Plausible one, as and when GIL is removed c) Figure out what support is needed from PVM et al (b.1 and/or b.2) d) PEP and onwards ... Most probably I am stating the obvious. My take is that lots of the work is already there. We need to converge and do the rest to work towards a PEP.I know that Guido and Sr Developers will involve themselves at the appropriate time. Cheers <k/> Stephen J. Turnbull wrote:
I do agree that there is a diversity of needs in the greater Python community. But the current discussion is oriented towards a subset of Python users who *do* or care about parallelism. For this subset of people, I do feel the issues are important. I don't expect people who don't need high performance and thus parallelism to feel the same way that I do.
Now I am really regretting using the word "crisis," as your statement implies that I am so negative about all this that I have lost sight positive side of this discussion. I do think it would be fantastic to have a more threading capable Python.
*gasp*, I wasn't aware that I had grad students, DOE grants or VC money. A lot of things must have changed while I have been out of town this week :) While the company at which I work does have DOE grants, work on the GIL is _far_ outside their scope of work. One of the difficulties with the actual work of removing the GIL is that it is difficult, grungy work that not many people are interested in funding. Most of the funding sources that are throwing money at parallel computing and multicore are focused on languages other than Python. But, I could imagine that someone like IBM that seems to have an interest in both Python and multicore CPUs would be interested in sponsoring such an effort. You would think that Google would also an interest in this. To me it seems that the situation with the GIL will remain the same until 1) someone with lots of free time and desire steps up to the plate or 2) someone ponies up the $ to pay someone to work on it. Currently, I am not in the first of these situations.
That has been made clear. Brian
Can you all stop the meta-discussion and start discussing ideas for parallel APIs please? -- --Guido van Rossum (home page: http://www.python.org/~guido/)
"Brian Granger" <ellisonbg.net@gmail.com> Brief responses to this and previous posts and threads: You withdraw 'crisis'. Fine. Let us move on. Hobbling Python on the billion (1000 million) or so current and future single-core CPUs is not acceptible. Let us move on. Multiple cores can be used with separate processes. I expect this is where Google is, but that is another discussion, except 'Do not hold your breath" waiting for Google to publicly support multicore threading. Improving multi-thread use of multiple cores -- without hobbling single core use -- would be good. Obviously. No need to argue the point. Let us move on. What is needed is persuasion of practicality by the means Guido has requested: concrete proposals and code. Terry Jan Reedy
Brian Granger wrote:
It's not just a matter of hard work -- so far, nobody knows *how* to remove the GIL without a big loss in efficiency. Somebody is going to have to have a flash of insight, and money can't buy that. -- Greg
Would like pointers to any work going on, on transactional memory constructs (like atomic et al) and python. I also have a few questions to any who is working on this topic. Cheers <k/>
Brian, Good points.
What we need to do is not to line them (i.e. GIL removal, parallelism) serially, but work on them simultaneously; that way they will leverage each other. Also, what ever paradigm(s) we zero-in on, can be implemented in other implementations, anyway. Moreover, IMHO, we need not force the GIL issue just yet. I firmly believe that it will find a solution in it's own time frame ... </KS> Cheers <k/> Brian Granger wrote:
While I too love to think about parallelism, until the limitations of the GIL in CPython are fixed, all our grand thoughts will be dead ends at some level.
Also GIL is a challenge only in one implementation (I know, it is an important implementation!) My assumption is that the GIL restriction will be removed one way or another, soon. (same reason, quoting Joe Lewis)
I am not that optimistic I guess. Hopeful though.
What we need to do is not to line them (i.e. GIL removal, parallelism) serially, but work on them simultaneously; that way they will leverage each other. Also, what ever paradigm(s) we zero-in on, can be implemented in other implementations, anyway.
If there were infinitely many people willing for work on this stuff, then I agree, but I don't see even a dozen people hacking on the GIL. And in my mind, once the limitations of the GIL are relaxed, the other parallel stuff won't be very difficult given the work that people have already done in this area.
Moreover, IMHO, we need not force the GIL issue just yet. I firmly believe that it will find a solution in it's own time frame ...
On 9/18/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
If there were infinitely many people willing for work on this stuff, then I agree, but I don't see even a dozen people hacking on the GIL.
In part because many people don't believe it would be productive. For threading to be useful in terms of parallel processing, most memory access has to be read-only. That isn't true today, largely because of reference counts. There are ways around that, by using indirection, or delayed counts, or multiple refcount buckets per object, or even just switching to a tracing GC. So far, no one has been able to make these changes without seriously mangling the C API and/or slowing things down a lot. The current refcount mechanism is so lightweight that it isn't clear this would even be possible. (With 4 or more cores dedicated to just python, it might be worth it anyhow -- but it isn't yet.) So if you want the GIL removed, you need to provide an existence proof that (CPython) memory management can be handled efficiently without it. -jJ
On 9/19/07, Jim Jewett <jimjjewett@gmail.com> wrote:
Is 60-65% of normal CPython "a lot"? (I really should clean things up and post a patch...) -- Adam Olsen, aka Rhamphoryncus
On 9/19/07, Adam Olsen <rhamph@gmail.com> wrote:
On 9/19/07, Jim Jewett <jimjjewett@gmail.com> wrote:
On 9/18/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
Is 60-65% of normal CPython "a lot"?
Yes, but I think it is still better than the last serious attempt, so it would be worth posting patches anyhow. -jJ
On 9/19/07, Jim Jewett <jimjjewett@gmail.com> wrote:
It's not even comparable to the last serious attempt. Even mere atomic refcounting has *negative* scalability at two threads when running pystones. My approach has 95-100% scalability. -- Adam Olsen, aka Rhamphoryncus
Is 60-65% of normal CPython "a lot"?
(I really should clean things up and post a patch...)
Is your implementation based on Python 3000/3.0? or Python 2.x? Regardless, I'd love to see it. How would you describe the changes to the C-api? Radical departure? Somewhat compatible? mostly compatible? No changes? - Matt
On 9/20/07, Matt Knox <mattknox_ca@hotmail.com> wrote:
It's based of py3k. I will post eventually (heh). The C API is mostly compatible (source wise only), notably excepting the various threading things. -- Adam Olsen, aka Rhamphoryncus
On Tue, Sep 18, 2007, Brian Granger wrote:
What to do about this pending crisis is a complicated issue.
Oh, please. Calling this a crisis only causes people to ignore you. Take a look at this URL that Guido posted to the baypiggies list: http://marknelson.us/2007/07/30/multicore-panic/ -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ The best way to get information on Usenet is not to ask a question, but to post the wrong information.
What to do about this pending crisis is a complicated issue.
Oh, please. Calling this a crisis only causes people to ignore you.
I apologize if this statement is a little exaggerated. But, I do think this is a really critical problem that is going to affect certain groups of Python users and developers in adverse ways. Perhaps I have not made a very strong case that it is a true "crisis" though.
Take a look at this URL that Guido posted to the baypiggies list:
I actually agree with many of the comments made by the author. The quotes from John Dvorak and Wired are over the top. The author make good points about operating systems' abilities to handle threading on modest numbers of cores. It does work, even today, and will continue to work (for a while) as the number of cores increases. But the main point of the author is that operating systems DO have threading support that works reasonably well. Python is another story. The author even makes comments that provide a string critique of Python's (and Ruby's) threading capabilities: <quote> Modern programs tend to be moderately multithreaded, with individual threads dedicated to the GUI, to user I/O, to socket I/O, and often to computation. Multicore CPUs take advantage of this quite well. And we don't need any new technology to make sure multi-threaded programs are well-behaved - these techniques are pretty well understood, and in use in most software you use today. Modern languages like Java support threads and various concurrency issues right out of the box. C++ requires non-standard libraries, but all modern C++ environments worth their salt deal with multithreading in a fairly sane way. </quote> Modern programs tend to be moderately multithreaded.....except for Python. Modern languages like Java and C++ support threads and concurrency, even if those capabilities aren't built in at a low level (C++). I don't think the same thing can be said about Python. The GIL in CPython does in fact prevent threading from being a general solution for CPU bound parallelism. The author is wrong in one important respect: <quote> In this future view, by 2010 we should have the first eight-core systems. In 2014, we're up to 32 cores. By 2017, we've reached an incredible 128 core CPU on a desktop machine. </quote> I don't know where the author got this information, but it is way off. Here are some currently available examples: * Appro now offers a workstation with up to 4 quad core Opterons: http://www.appro.com/product/workstationxtreme_opteron.asp That is a 16 core system. It is 2007, not 2012. * Tilera offers a 64 core CPU that runs SMP Linux. * SiCortex offers a low power 648 processor Linux system that is organized into 6 core SMPs, each of which runs Linux. This week I am at a Department of Defense conference on multicore computing: http://www.ll.mit.edu/HPEC/2007/index.html There is broad agreement from all corners that multi/manycore CPUs will require new ways of expressing parallelism. But in all of the discussions about new approaches, people do assume that threads are a low-level building block that can and should be used for building the higher-level stuff. Intel's Threaded Building Blocks is a perfect example. While it is implemented using threads, it provides much higher level abstractions for developers to use in building applications that will scale well on multicore systems. I would _love_ to have such constructs in Python, but that is simply not possible. And process based solutions don't provide a solution for the many algorithms that require fine grained partitioning and fast data movement. For those of us who do use Python for high performance computing, these issues are critical. In fact, anytime I deal with fine grained parallel algorithms, I use C/C++. I do end up wrapping my low level C/C++ threaded code into Python, but this doesn't always map onto the problem well. The other group of people for whom this is a big issue, are general Python users that don't think they need parallelism. Eventually even these people will become frustrated that their Python codes are the same speed on a 1 core system as a 128 core system. To me this is a significant problem. Brian
Brian Granger writes:
No, you're missing the point. I don't see anybody denying that you understand your own needs. *You* may face a (true) crisis. *The Python community* does not perceive your crisis as its own. Personally, I don't see why it should. And I think you'd be much more successful at selling this with a two-pronged approach of evangelizing just how utterly cool it would be to have a totally-threading GIL-less Python on the one hand, and recruiting some gung-ho grad students with Google SoC projects (or *gasp* some of your DoE grant or VC money) on the other. Note that nobody has said anything to discourage this as a research project. Nothing like it's impossible, stupid, or YAGNI. But Guido, and other senior developers, are saying they're not going to devote their resources to it as things currently stand (and one of those resources the attention of the folks who review PEPs).
Stephen,
<KS> I am not sure that is true. I think if we have a well thought out PEP that addresses parallelism, it would be looked into by the folks. It is just that there are too many different ways of doing it and of course, the GIL doesn't help either. There is also a school of thought that we should save the GIL and as a result we will find another better way of leveraging the multi-core architectures. </KS> CHeers <k/> Stephen J. Turnbull wrote:
Krishna Sankar writes:
... and provides a plausible proof-of-concept implementation [Guido said that to you explicitly] ...
it would be looked into by the folks.
True. But AIUI without the implementation, it won't become a PEP. And at present you don't have a plausible implementation with the GIL. Now, it looks like Adam has a plausible implementation of removing the GIL. If that holds up under at least some common use-cases, I think you'll see enthusiasm from some of the top developers, and acceptance from most. But I really think that has to come first.
Stephen, Now I see where you are coming from and you are right. Plausible is the operative word. It is not that Guido and Sr.Developers will not look at it at all, but will not look at it in the absence of a plausible implementation. This was evident in the blog discussions with Bruce as well. I was encouraged by Guido's reply. He is doing the right thing. Anyway, the point is, now we have a good understanding of the dynamics. There is a small community who would like to see this happen and it is up to us to show it can be done. A few key things need to happen: a) A good set of benchmarks on multi-core systems, extending excellent work (for example http://blogs.warwick.ac.uk/dwatkins/entry/benchmarking_parallel_python_1_2/) I have an 8 core machine (2 X 4 core) and plan to run this benchmark as well as create others like the Red Black Tree (Guido had suggested that, even before that I was thinking of it) b.1) Plausible POC with GIL - pp, actor pattern/agent paradigm et al. b.2) Plausible one, as and when GIL is removed c) Figure out what support is needed from PVM et al (b.1 and/or b.2) d) PEP and onwards ... Most probably I am stating the obvious. My take is that lots of the work is already there. We need to converge and do the rest to work towards a PEP.I know that Guido and Sr Developers will involve themselves at the appropriate time. Cheers <k/> Stephen J. Turnbull wrote:
I do agree that there is a diversity of needs in the greater Python community. But the current discussion is oriented towards a subset of Python users who *do* or care about parallelism. For this subset of people, I do feel the issues are important. I don't expect people who don't need high performance and thus parallelism to feel the same way that I do.
Now I am really regretting using the word "crisis," as your statement implies that I am so negative about all this that I have lost sight positive side of this discussion. I do think it would be fantastic to have a more threading capable Python.
*gasp*, I wasn't aware that I had grad students, DOE grants or VC money. A lot of things must have changed while I have been out of town this week :) While the company at which I work does have DOE grants, work on the GIL is _far_ outside their scope of work. One of the difficulties with the actual work of removing the GIL is that it is difficult, grungy work that not many people are interested in funding. Most of the funding sources that are throwing money at parallel computing and multicore are focused on languages other than Python. But, I could imagine that someone like IBM that seems to have an interest in both Python and multicore CPUs would be interested in sponsoring such an effort. You would think that Google would also an interest in this. To me it seems that the situation with the GIL will remain the same until 1) someone with lots of free time and desire steps up to the plate or 2) someone ponies up the $ to pay someone to work on it. Currently, I am not in the first of these situations.
That has been made clear. Brian
Can you all stop the meta-discussion and start discussing ideas for parallel APIs please? -- --Guido van Rossum (home page: http://www.python.org/~guido/)
"Brian Granger" <ellisonbg.net@gmail.com> Brief responses to this and previous posts and threads: You withdraw 'crisis'. Fine. Let us move on. Hobbling Python on the billion (1000 million) or so current and future single-core CPUs is not acceptible. Let us move on. Multiple cores can be used with separate processes. I expect this is where Google is, but that is another discussion, except 'Do not hold your breath" waiting for Google to publicly support multicore threading. Improving multi-thread use of multiple cores -- without hobbling single core use -- would be good. Obviously. No need to argue the point. Let us move on. What is needed is persuasion of practicality by the means Guido has requested: concrete proposals and code. Terry Jan Reedy
Brian Granger wrote:
It's not just a matter of hard work -- so far, nobody knows *how* to remove the GIL without a big loss in efficiency. Somebody is going to have to have a flash of insight, and money can't buy that. -- Greg
Would like pointers to any work going on, on transactional memory constructs (like atomic et al) and python. I also have a few questions to any who is working on this topic. Cheers <k/>
participants (10)
-
Aahz -
Adam Olsen -
Brian Granger -
Greg Ewing -
Guido van Rossum -
Jim Jewett -
Krishna Sankar -
Matt Knox -
Stephen J. Turnbull -
Terry Reedy