Dear pypyers, I've resumed work on WP9, with the intent to finish it ASAP. Right now I face a little difficulty. I'd like to have your input (essentially Armin's input I guess) on this. Back when I was thinking about the requirements of WP9, esp. the non-deterministic aspects, I thought clonable coroutines/threads would do, fuzzily. Yet obviously these features have been provided and could still be used to achieve my goals, but that is not what I have decided to do since then... What I'd like to know if I have painted myself in a corner with the current design of the logic objectspace. Right now, a computation space is defined as a set of concurrent activities (thread group); in the case of constraint solving, there is one 'main' thread doing the choices, and an army of propagator threads trying to empty the domains. The idea is to associate a garbage collector pool to such a set. I wrote the pool handling code and the clone stuff following as closely as possible what's in modules/_stackless/interp_clonable and friends (most recent versions I think); without a clear understanding of how it works ... (see in objspace/cclp/types.py the Pool class, some client code in space.py and scheduler.py) Then, what happens now is that the cloning routine returns the original thread group. My question is the following : * is there a chance that a peer review and eventual correction/advice on that non-working code can yield a working solution in a reasonnable amount of time ? or * is it time for me to simplify considerably the computation space down to one single thread, so as to directly peruse the (working) coroutine cloning facility ? The later option would entail throwing away ten days of work, and spending a more days to reorganize and rewrite stuff, but can be done while retaining a reasonnable amount of functionnality. (Well, in truth I expect a quick code review by Armin to expose some blunders of mine... Just don't hit too hard my poor head.) Regards, Aurélien.
Hi Aurelien, On Tue, Dec 19, 2006 at 06:50:46PM +0100, Aur?lien Camp?as wrote:
* is there a chance that a peer review and eventual correction/advice on that non-working code can yield a working solution in a reasonnable amount of time ?
No. Getting the original coroutine cloning code to really work correctly took us quite some time, and certainly more than 10 days, including a final night following data structures at the C level with ddd. (Btw, that's a gdb front-end that I really recommend for Debugging and Displaying Data :-) It looks innocent enough but it is really subtle code, so you should try to build on top of that and not reinvent it from scratch. Note also that if you need sets of several coroutines that are all cloned together, it's possible to put each set in a single InterpClonableCoroutine that acts as a "big" thread; and it would itself contain several regular Coroutines for its own local purposes. This works nicely if in each InterpClonableCoroutine you first create a new CoState() instance, and use it as the 'state' argument of all the local Coroutines. This local CoState has a current and a main that are local too. If you clone one of the "big" InterpClonableCoroutine, everything will be cloned - the local CoState and its Coroutines. Hum, actually for now that's theory only. The local Coroutines have a __del__ and objects with a __del__ are never cloned. I guess that my point is that we could make this work with minimal efforts (e.g. with a variety of Coroutines that doesn't need a __del__).
The later option would entail throwing away ten days of work, and spending a more days to reorganize and rewrite stuff, but can be done while retaining a reasonnable amount of functionnality.
Well, I would have given you the same answer 10 days ago, or even a few months ago, if only I had known. I should point out that we ("the technical board") tried to ask Logilab in general to tell us what plans they had, and we tried quite a few times now, so I fail to feel bad about telling you now to throw away just 10 days of work, sorry. A bientot, Armin
On Thu, Dec 21, 2006 at 05:44:00PM +0100, Armin Rigo wrote:
Hi Aurelien,
On Tue, Dec 19, 2006 at 06:50:46PM +0100, Aur?lien Camp?as wrote:
* is there a chance that a peer review and eventual correction/advice on that non-working code can yield a working solution in a reasonnable amount of time ?
No. Getting the original coroutine cloning code to really work correctly took us quite some time, and certainly more than 10 days, including a final night following data structures at the C level with ddd. (Btw, that's a gdb front-end that I really recommend for Debugging and Displaying Data :-) It looks innocent enough but it is really subtle code, so you should try to build on top of that and not reinvent it from scratch.
I never meant to reinvent this from scratch. You also said to me recently that the local pooling was not especially tied to coroutines, which matched the intuition I had from reading the interp_clonable stuff.
Note also that if you need sets of several coroutines that are all cloned together, it's possible to put each set in a single InterpClonableCoroutine that acts as a "big" thread; and it would itself contain several regular Coroutines for its own local purposes. This works nicely if in each InterpClonableCoroutine you first create a new CoState() instance, and use it as the 'state' argument of all the local Coroutines. This local CoState has a current and a main that are local too. If you clone one of the "big" InterpClonableCoroutine, everything will be cloned - the local CoState and its Coroutines.
Some time ago I tried that road but abandonned it because it yielded non-working builds. But at that time I missed the "this works nicely if ..." part. Thanks for taking the time to talk about it.
Hum, actually for now that's theory only. The local Coroutines have a __del__ and objects with a __del__ are never cloned. I guess that my point is that we could make this work with minimal efforts (e.g. with a variety of Coroutines that doesn't need a __del__).
I don't think in my usage of Coroutines I need a __del__. So maybe there is a tiny hope.
The later option would entail throwing away ten days of work, and spending a more days to reorganize and rewrite stuff, but can be done while retaining a reasonnable amount of functionnality.
Well, I would have given you the same answer 10 days ago, or even a few months ago, if only I had known. I should point out that we ("the technical board") tried to ask Logilab in general to tell us what plans they had, and we tried quite a few times now,
Logilab's plan wrt WP9 have always been clear I think, but whatever. Talking about logilab relationships with the PyPy group is for another day.
so I fail to feel bad about telling you now to throw away just 10 days of work, sorry.
Well, sorry if you interpreted my poor wording about days of work being thrown away as an attempt to culpabilize you. That was not my intent. Moreover I highly respect what you have done for the cloning facility and everything in general the core PyPy developpers have achieved so far. Thanks, Aurélien.
participants (2)
-
Armin Rigo -
Aurélien Campéas