Re: [Python-ideas] [Python-Dev] [ANN]: "newthreading" - an approach to simplified thread usage, and a path to getting rid of the GIL

I'm moving this thread to python-ideas, where it belongs. I've looked at the implementation code (even stepped through it with pdb!), read the sample/test code, and read the two papers on animats.com fairly closely (they have a lot of overlap, and the memory model described below seems copied verbatim from http://www.animats.com/papers/languages/pythonconcurrency.html version 0.8). Some reactions (trying to hide my responses to the details of the code): - First of all, I'm very happy to see radical ideas proposed, even if they are at present unrealistic. We need a big brainstorm to come up with ideas from which an eventual solution to the multicore problem might be chosen. (Jesse Noller's multiprocessing is another; Adam Olsen's work yet another, at a different end of the spectrum.) - The proposed new semantics (frozen objects, memory model, auto-freezing of globals, enforcement of naming conventions) are radically different from Python's current semantics. They will break every 3rd party library in many more ways than Python 3. This is not surprising given the goals of the proposal (and its roots in Adam Olsen's work) but places a huge roadblock for acceptance. I see no choice but to keep trying to come up with a compromise that is more palatable and compatible without throwing away all the advantages. As it now stands, the proposal might as well be a new and different language. - SynchronizedObject looks like a mixture of a Java synchronized class (a non-standard concept in Java but easily understood as a class all whose public methods are synchronized) and a condition variable (which has the same semantics of releasing the lock while waiting but without crawling the stack for other locks to release). It looks like the examples showing off SynchronizedObject could be implemented just as elegantly using a condition variable (and voluntary abstention from using shared mutable objects). - If the goal is to experiment with new control structures, I recommend decoupling them from the memory model and frozen objects, instead relying (as is traditional in Python) on programmer caution to avoid races. This would make it much easier to see how programmers respond to the new control structures. - You could add the freeze() function for voluntary use, and you could even add automatic wrapping of arguments and return values for certain classes using a class decorator or a metaclass, but the performance overhead makes this unlikely to win over many converts. I don't see much use for the "whole program freezing" done by the current prototype -- there are way too many backdoors in Python for the prototype approach to be anywhere near foolproof, and if we want a non-foolproof approach, voluntary constraint (and, in some cases, voluntary, i.e. explicit, wrapping of modules or classes) would work just as well. - For a larger-scale experiment with the new memory model and semantic restrictions (or would it be better to call them syntactic restrictions? -- after all they are about statically detectable properties like naming conventions) I recommend looking at PyPy, which has as one of its explicitly stated project goals easy experimentation with different object models. - I'm sure I've forgotten something, but I wanted to keep my impressions fresh. - Again, John, thanks for taking the time to come up with an implementation of your idea! --Guido On Sat, Jun 26, 2010 at 9:39 AM, John Nagle <nagle@animats.com> wrote:
-- --Guido van Rossum (python.org/~guido)

On Mon, 28 Jun 2010 16:09:55 -0700 Guido van Rossum <guido@python.org> wrote:
I'm moving this thread to python-ideas, where it belongs. [...]
For the record, I really think the solution to the "try to remove the GIL" problem is to... try to remove it. I believe it implies several preparatory steps: - take full control of memory allocation - on top of that, devise a full garbage collector (probably including a notion of external references such that existing ways of writing C extensions are still correct) - then, do the the tedious, delicate grunt work of adding locking to critical structures without slowing them down (too much) Trying to invent schemes to make multithreading easier to program with is a nice endeavour in itself, but quite orthogonal IMO. Regards Antoine.

On 29 Jun 2010, at 00:40, Antoine Pitrou <solipsis@pitrou.net> wrote:
Full agreement. Ironclad, a project to enable the use of Python C extensions with IronPython - which has a generational moving GC, uses a hybrid approach. It allows C extensions to use reference counting but manipulates the reference count so that it can only drop to zero once there are no references left on the IronPython side. There are complications with this approach, which Ironclad handles, but that would be much easier when we have control over the implementation (Ironclad doesn't change the IronPython implementation). No link I'm afraid, sending from a mobile device. Incidentally, Ironclad also 'fakes' the GIL as IronPython has no GIL. In theory this could cause problems for C extensions that aren't thread safe but that hasn't yet been a problem in production (Ironclad is mainly used with numpy). Michael Foord

On Tue, Jun 29, 2010 at 9:54 AM, Michael Foord <fuzzyman@gmail.com> wrote:
How much do you know about Resolver's licensing setup for Ironclad? Combining Ironclad with a Boehm GC enabled PyMalloc mechanism might be a fruitful avenue of research on the way to a free-threading CPython implementation. Losing deterministic refcounting for pure Python code is no longer as big an issue as it once was, as many of the tricks it used to enable are now better covered by context managers. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 29 June 2010 13:05, Nick Coghlan <ncoghlan@gmail.com> wrote:
http://www.resolversystems.com/products/ironclad/ Ironclad is MIT licensed, but it is *very* tightly coupled to IronPython and .NET (it works primarily through the .NET FFI and uses a fair bit of C#). It may well be useful for inspiration, but I don't know how re-usable it is likely to be.
Combining Ironclad with a Boehm GC enabled PyMalloc mechanism might be
Boehm is a conservative collector, so whilst it may well be a "good first step" it can leak memory like a sieve... Mono has always had this problem and is finally getting rid of its conservative collector. The PyPy guys have experience in this area.
Right, and *most* of the alternative implementations are not reference counted - so relying on reference counting semantics has been discouraged by users of these platforms for a while now. All the best, Michael

On Mon, Jun 28, 2010 at 17:40, Antoine Pitrou <solipsis@pitrou.net> wrote:
+1. Designing an API in C for a precise GC is tedious, and would probably be very ugly, but it's entirely doable. We simply need the will to go through with it. I can't say what the overhead would look like, but so long as it scales well and it's a compile-time option it should find plenty of users.

On Mon, 28 Jun 2010 16:09:55 -0700 Guido van Rossum <guido@python.org> wrote:
I'm moving this thread to python-ideas, where it belongs. [...]
For the record, I really think the solution to the "try to remove the GIL" problem is to... try to remove it. I believe it implies several preparatory steps: - take full control of memory allocation - on top of that, devise a full garbage collector (probably including a notion of external references such that existing ways of writing C extensions are still correct) - then, do the the tedious, delicate grunt work of adding locking to critical structures without slowing them down (too much) Trying to invent schemes to make multithreading easier to program with is a nice endeavour in itself, but quite orthogonal IMO. Regards Antoine.

On 29 Jun 2010, at 00:40, Antoine Pitrou <solipsis@pitrou.net> wrote:
Full agreement. Ironclad, a project to enable the use of Python C extensions with IronPython - which has a generational moving GC, uses a hybrid approach. It allows C extensions to use reference counting but manipulates the reference count so that it can only drop to zero once there are no references left on the IronPython side. There are complications with this approach, which Ironclad handles, but that would be much easier when we have control over the implementation (Ironclad doesn't change the IronPython implementation). No link I'm afraid, sending from a mobile device. Incidentally, Ironclad also 'fakes' the GIL as IronPython has no GIL. In theory this could cause problems for C extensions that aren't thread safe but that hasn't yet been a problem in production (Ironclad is mainly used with numpy). Michael Foord

On Tue, Jun 29, 2010 at 9:54 AM, Michael Foord <fuzzyman@gmail.com> wrote:
How much do you know about Resolver's licensing setup for Ironclad? Combining Ironclad with a Boehm GC enabled PyMalloc mechanism might be a fruitful avenue of research on the way to a free-threading CPython implementation. Losing deterministic refcounting for pure Python code is no longer as big an issue as it once was, as many of the tricks it used to enable are now better covered by context managers. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 29 June 2010 13:05, Nick Coghlan <ncoghlan@gmail.com> wrote:
http://www.resolversystems.com/products/ironclad/ Ironclad is MIT licensed, but it is *very* tightly coupled to IronPython and .NET (it works primarily through the .NET FFI and uses a fair bit of C#). It may well be useful for inspiration, but I don't know how re-usable it is likely to be.
Combining Ironclad with a Boehm GC enabled PyMalloc mechanism might be
Boehm is a conservative collector, so whilst it may well be a "good first step" it can leak memory like a sieve... Mono has always had this problem and is finally getting rid of its conservative collector. The PyPy guys have experience in this area.
Right, and *most* of the alternative implementations are not reference counted - so relying on reference counting semantics has been discouraged by users of these platforms for a while now. All the best, Michael

On Mon, Jun 28, 2010 at 17:40, Antoine Pitrou <solipsis@pitrou.net> wrote:
+1. Designing an API in C for a precise GC is tedious, and would probably be very ugly, but it's entirely doable. We simply need the will to go through with it. I can't say what the overhead would look like, but so long as it scales well and it's a compile-time option it should find plenty of users.
participants (5)
-
Adam Olsen
-
Antoine Pitrou
-
Guido van Rossum
-
Michael Foord
-
Nick Coghlan