How about introducing internal macros for explicit ob_refcnt accesses in the core? Actually, there are a number of places where one can see "op->ob_refcnt" logic, which could be replaced with _Py_GETREF(op), _Py_SETREF(op, n) thus decoupling completely the low level refcount management defined in object.h: #define _Py_GETREF(op) (((PyObject *)op)->ob_refcnt) #define _Py_SETREF(op, n) (((PyObject *)op)->ob_refcnt = (n)) Comments? I've contributed myself to the mess in intobject.c & floatobject.c, so I thought that such macros would make the code cleaner. Here's the current state of affairs: python/dist/src>find . -name "*.[c]" -exec grep ob_refcnt {} \; -print (void *) v, ((PyObject *) v)->ob_refcnt)) ./Modules/_tkinter.c if (self->arg->ob_refcnt > 1) { \ if (ob->ob_refcnt < 2 || self->fast) if (args->ob_refcnt > 1) { ./Modules/cPickle.c if (--inst->ob_refcnt > 0) { ./Objects/classobject.c if (result->ob_refcnt == 1) ./Objects/fileobject.c if (PyFloat_Check(p) && p->ob_refcnt != 0) if (!PyFloat_Check(p) || p->ob_refcnt == 0) { if (PyFloat_Check(p) && p->ob_refcnt != 0) { p, p->ob_refcnt, buf); ./Objects/floatobject.c if (PyInt_Check(p) && p->ob_refcnt != 0) if (!PyInt_Check(p) || p->ob_refcnt == 0) { if (PyInt_Check(p) && p->ob_refcnt != 0) p, p->ob_refcnt, p->ob_ival); ./Objects/intobject.c assert(v->ob_refcnt == 1); /* Since v will be used as accumulator! */ ./Objects/longobject.c if (op->ob_refcnt <= 0) op->ob_refcnt, (long)op); op->ob_refcnt = 1; if (op->ob_refcnt < 0) fprintf(fp, "[%d] ", op->ob_refcnt); ./Objects/object.c if (!PyString_Check(v) || v->ob_refcnt != 1) { if (key->ob_refcnt == 2 && key == value) { ./Objects/stringobject.c if (!PyTuple_Check(op) || op->ob_refcnt != 1) { if (v == NULL || !PyTuple_Check(v) || v->ob_refcnt != 1) { ./Objects/tupleobject.c if (PyList_Check(seq) && seq->ob_refcnt == 1) { if (args->ob_refcnt > 1) { ./Python/bltinmodule.c if (value->ob_refcnt != 1) ./Python/import.c return PyInt_FromLong((long) arg->ob_refcnt); ./Python/sysmodule.c -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
How about introducing internal macros for explicit ob_refcnt accesses in the core?
What problem does this solve?
Actually, there are a number of places where one can see "op->ob_refcnt" logic, which could be replaced with _Py_GETREF(op), _Py_SETREF(op, n) thus decoupling completely the low level refcount management defined in object.h:
#define _Py_GETREF(op) (((PyObject *)op)->ob_refcnt) #define _Py_SETREF(op, n) (((PyObject *)op)->ob_refcnt = (n))
Why the cast? It loses some type-safety, e.g. _Py_GETREF(0) will now cause a core dump instead of a compile-time error.
Comments?
I don't see how it's cleaner or saves typing: op->ob_refcnt _Py_GETREF(op) op->ob_refcnt = 1 _Py_SETREF(op, 1) --Guido van Rossum (home page: http://www.python.org/~guido/)
On Thu, 24 Jun 1999, Guido van Rossum wrote:
How about introducing internal macros for explicit ob_refcnt accesses in the core?
What problem does this solve?
I assume Vladimir was trying to leave the door open for further ob_refcnt manipulation hooks later, like having objects manage their own refcounts. Until there's an actual problem to solve that requires this, though, i'm not sure it's necessary. Are there obvious reasons to want to allow this? * * * While we're talking about refcounts and all, i've had the argument quite successfully made to me that a reasonably written garbage collector can be both (a) simple and (b) more efficient than refcounting. Having spent a good number of work days doing nothing but debugging crashes by tracing refcounting bugs, i was easily converted into a believer once a friend dispelled the notion that garbage collectors were either slow or horribly complicated. I had always been scared of them before, but less so now. Is an incremental GC being considered for a future Python? I've idly been pondering various tricks by which it could be made to work with existing extension modules -- here are some possibilities: 1. Keep the refcounts and let existing code do the usual thing; introduce a new variant of PyObject_NEW that puts an object into the "gc-able" pool rather than the "refcounted" pool. 2. Have Py_DECREF and Py_INCREF just do nothing, and let the garbage collector guess from the contents of the structure where the pointers are. (I'm told it's possible to do this safely, since you can only have false positives, never false negatives.) 3. Have Py_DECREF and Py_INCREF just do nothing, and ask the extension module to just provide (in its type object) a table of where the pointers are in its struct. And so on; mix and match. What are everyone's thoughts on this one? -- ?!ng "All models are wrong; some models are useful." -- George Box
[Ka-Ping Yee, opines about GC] Ping, I think you're not getting any responses because this has been beaten to death on c.l.py over the last month (for the 53rd time, no less <wink>). A hefty percentage of CPython users *like* the reliably timely destruction refcounting yields, and some clearly rely on it. Guido recently (10 June) posted the start of a "add GC on top of RC" scheme, in a thread with the unlikely name "fork()". The combination of cycles, destructors and resurrection is quite difficult to handle in a way both principled and useful (Java's way is principled but by most accounts unhelpful to the point of uselessness). Python experience with the Boehm collector can be found in the FAQ; note that the Boehm collector deals with finalizers in cycles by letting cycles with finalizers leak!
... While we're talking about refcounts and all, i've had the argument quite successfully made to me that a reasonably written garbage collector can be both (a) simple and (b) more efficient than refcounting.
That's a dubious claim. Sophisticated mark-and-sweep (with or without compaction) is almost universally acknowledged to beat RC, but simple M&S has terrible cache behavior (you fill up the address space before reclaiming anything, then leap all over the address space repeatedly cleaning it up). Don't discount that, in Python unlike as in most other languages, the simple loop for i in xrange(1000000): pass creates a huge amount of trash at a furious pace. Under RC it can happily reuse the same little bit of storage each time around.
Having spent a good number of work days doing nothing but debugging crashes by tracing refcounting bugs,
Yes, we can trade that for tracking down M&S bugs <0.5 wink> -- instead of INCREF/DECREF macros, you end up with M&S macros marking regions where the collector must not be run (because you're in a temporarily "inconsistent" state). That's under sophisticated M&S, though, but is an absolute nightmare when you miss a pair (the bugs only show up "sometimes", and not always the same ways -- depends on when M&S happens to run, and "how inconsistent" you happen to be at the time).
... And so on; mix and match. What are everyone's thoughts on this one?
I think Python probably needs to clean up cycles, but by some variant of Guido's scheme on top of RC; I very much dislike the property of his scheme that objects with destructors may be get destroyed without their destructors getting invoked, but it seems hard to fix. Alternatives include Java's scheme (which really has nothing going for it other than that Java does it <0.3 wink>); Scheme's "guardian" scheme (which would let the user "get at" cyclic trash with destructors, but refuses to do anything with them on its own); following Boehm by saying that cycles with destructors are immortal; following goofier historical precedent by e.g. destroying such objects in reverse order of creation; or maybe just raising an exception if a trash cycle containing a destructor is found. All of those seem a comparative pain to implement, with Java's being the most painful -- and quite possibly the least satisfying! it's-a-whale-of-a-lot-easier-in-a-self-contained-universe-or-even-an- all-c-one-ly y'rs - tim
Tim Peters wrote:
[Ka-Ping Yee, opines about GC]
Ping, I think you're not getting any responses because this has been beaten to death on c.l.py over the last month (for the 53rd time, no less <wink>).
A hefty percentage of CPython users *like* the reliably timely destruction refcounting yields, and some clearly rely on it.
[CG issue dropped, I know the thread] I know how much of a pain in the .. proper refcounting can be. Sometimes, after long debugging, I wished it would go. But finally, I think it is a *really good thing* to have to do proper refcounting. The reason is that this causes a lot of discipline, which improves the whole program. I guess with GC always there, quite a number of errors stay undetected. I can say this, since I have been through a week of debugging now, and I can now publish full blown first class continuations for Python yes I'm happy - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
[Christian Tismer]
... I can say this, since I have been through a week of debugging now, and I can now publish
full blown first class continuations for Python
yes I'm happy - chris
You should be! So how come nobody else is <wink/frown>? Let's fire some imagination here: without the stinkin' C stack snaking its way thru everything, then with the exception of external system objects (like open files), the full state of a running Python program is comprised of objects Python understands and controls. So with some amount of additional pain we could pickle them. And unpickle them. Painlessly checkpoint a long computation for possible restarting? Freeze a program while it's running on your mainframe, download it to your laptop and resume it while you're on the road? Ship a bug report with the computation frozen right before the error occurs? Take an app with gobs of expensive initialization, freeze it after it's "finally ready to go", and ship the latter instead? Capture the state of an interactive session for later resumption? Etc. Not saying those are easy, but getting the C stack out of the way means they move from impossible to plausible. Maybe it would help get past the Schemeophobia <wink> if, instead of calling them "continuations", you called 'em "platform-independent potentially picklable threads". pippt-sounds-as-good-as-it-reads<wink>-ly y'rs - tim
yes I'm happy - chris
You should be! So how come nobody else is <wink/frown>?
Im a little unhappy as this will break the Active Debugging stuff - ie, the ability for Python, Java, Perl, VBScript etc to all exist in the same process, each calling each other, and each being debuggable (makes a _great_ demo :-) Im not _really_ unhappy, Im just throwing this in as an FYI. The Active Debugging interfaces need some way of sorting a call stack. As many languages may be participating in a debugging session, there is no implicit ordering available. Inter-language calls are not made via the debugger, so it has no chance to intercept. So the solution MS came up with was, surprise surprise, the machine stack! :-) The assumption is that all languages will make _some_ use of the stack, so they ask a language to report its "stack base address" and "stack size". Using this information, the debugger sorts into the correct call sequence. Indeed, getting this information (even the half of it I did manage :-) was painful, and hard to get right. Ahh, the joys of bleeding-edge technologies :-)
Let's fire some imagination here: without the stinkin' C stack snaking its
I tried, and look what happened :-) Seriously, some if this stuff would be way cool. Bit I also understand completely the silence on this issue. When the thread started, there was much discussion about exactly what the hell these continuation/coroutine thingies even were. However, there were precious few real-world examples where they could be used. A few acedemic, theoretical places, but the only real contender I have seen brought up was Medusa. There were certainly no clear examples of "as soon as we have this, I could change abc to take advantage, and this would give us the very cool xyz" So, if anyone else if feeling at all like me about this issue, they are feeling all warm and fuzzy knowing that a few smart people are giving us the facility to do something we hope we never, ever have to do. :-) Mark.
Mark Hammond wrote:
yes I'm happy - chris
You should be! So how come nobody else is <wink/frown>?
(to Tim) I believe this comes simply since following me would force people to change their way of thinking. I am through this already, but it was hard for me. And after accepting to be stackless, there is no way to go back. Today I'm wondering about my past: "how could I think of stacks when thinking of programs?" This is so wrong. The truth is: Programs are just some data, part of it called code, part of it is local state, and! its future of computation. Out, over, roger. All the rest is artificial showstoppers.
Im a little unhappy as this will break the Active Debugging stuff - ie, the ability for Python, Java, Perl, VBScript etc to all exist in the same process, each calling each other, and each being debuggable (makes a _great_ demo :-)
Im not _really_ unhappy, Im just throwing this in as an FYI.
Well, yet I see no problem.
The Active Debugging interfaces need some way of sorting a call stack. As many languages may be participating in a debugging session, there is no implicit ordering available. Inter-language calls are not made via the debugger, so it has no chance to intercept.
So the solution MS came up with was, surprise surprise, the machine stack! :-) The assumption is that all languages will make _some_ use of the stack, so they ask a language to report its "stack base address" and "stack size". Using this information, the debugger sorts into the correct call sequence.
Now, I can give it a machine stack. There is just a frame dispatcher sitting on the stack, and it grabs frames from the current thread state.
Indeed, getting this information (even the half of it I did manage :-) was painful, and hard to get right.
I would have to see the AX interface. But for sure there will be some method hooks with which I can tell AX how to walk the frame chain. And why don't I simply publish frames as COM objects? This would give you much more than everything else, I guess. BTW, as it is now, there is no need to use AX debugging for Python, since Python can do it alone now. Of course it makes sense to have it all in the AX environment. You will be able to modify a running programs local variables, its evaluation stack, change its code, change where it returns to, all is doable. ...
Bit I also understand completely the silence on this issue. When the thread started, there was much discussion about exactly what the hell these continuation/coroutine thingies even were. However, there were precious few real-world examples where they could be used. A few acedemic, theoretical places, but the only real contender I have seen brought up was Medusa. There were certainly no clear examples of "as soon as we have this, I could change abc to take advantage, and this would give us the very cool xyz"
The problem was for me, that I had also no understanding what I was doing, actually. Implemented continuations without an idea how they work. But Tim and Sam said they were the most powerful control strucure possible, so I used all my time to find this out. Now I'm beginning to understand. And my continuation based coroutine example turns out to be twenty lines of Python code. Coming soon, after I served my whining customers.
So, if anyone else if feeling at all like me about this issue, they are feeling all warm and fuzzy knowing that a few smart people are giving us the facility to do something we hope we never, ever have to do. :-)
Think of it as just a flare gun in your hands. By reading the fine print, you will realize that you actually hold an atom bomb, with a little code taming it for you. :-) back-to-the-future - ly y'rs - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
[Mark Hammond]
Im a little unhappy as this [stackless Python] will break the Active Debugging stuff ... ... So the solution MS came up with was, surprise surprise, the machine stack! :-) The assumption is that all languages will make _some_ use of the stack, so they ask a language to report its "stack base address" and "stack size". Using this information, the debugger sorts into the correct call sequence.
Mark, you can't *really* believe Chris is incapable of hacking around this, right? It's not even clear there's something to be hacked around, since Python is only Python and there's nothing Christian can do to stop other languages that call into Python from using the machine stack, or to call other languages from Python without using the machine stack. So Python "shows up on the stack" no matter what, cross-language.
... Bit I also understand completely the silence on this issue. When the thread started, there was much discussion about exactly what the hell these continuation/coroutine thingies even were.
The Fuchs paper Sam referenced explained it in simple C terms: a continuation is exactly what C setjmp/longjmp would do if setjmp saved (& longjmp restored) the C stack *in addition* to the program counter and machine registers (which they already save/restore). That's all there is to it, at heart: objects capture data state, continuations capture control flow state. Whenever the OS services an interrupt and drops into kernel mode, it captures a continuation for user mode -- they don't *call* it that, but that's what they're doing, and it's as practical as a pencil (well, *more* practical, actually <wink>).
However, there were precious few real-world examples where they could be used.
Nobody asked for any before now <0.5 wink> -- and I see Sam provided some marvelous ones in response to this.
A few acedemic, theoretical places,
I think you undervalue those: people working on the underpinnings of languages strive very hard to come up with the simplest possible examples that don't throw away the core of the problem to be solved. That doesn't mean the theoreticians are too air-headed to understand "real world problems"; it's much more that, e.g., "if you can't compare the fringes of two trees cleanly, you can't possibly do anything harder than that cleanly either -- but if you can do this little bit cleanly, we have strong reason to believe there's a large class of difficult real problems you can also do cleanly". If you need a "practical" example of that, picture e.g. a structure-based diff engine for HTML source. Which are really trees defined by tags, and where text-based comparison can be useless (you don't care if "<LI>" moved from column 12 of line 16 to column 1 of line 17, but you care a lot if the *number* of <LI> tags changed -- so have you have to compare two trees *as* trees). But that's a problem easy enough for generators to solve cleanly. Knuth writes a large (for his books) elevator-simulation program to illustrate coroutines (which are more powerful than generators), and complains that he can't come up with a simpler example that illustrates any point worth making. And he's right! The "literature standard" text-manipulation example at the end of my coroutine module illustrates what Sam was talking about wrt writing straightforward "pull" algorithms for a "push" process, but even that one can be solved with simpler pipeline control flow. At least for *that*, nobody who ever used Unix would doubt the real-world utility of the pipeline model for a nanosecond <1e-9 wink>. If you want a coroutine example, go to a restaurant and order a meal. When you leave, glance back *real quick*. If everyone in the restaurant is dead, they were a meal-generating subroutine; but if they're still serving other customers, your meal-eating coroutine and their meal-generating coroutine worked to mutual benefit <wink>.
but the only real contender I have seen brought up was Medusa. There were certainly no clear examples of "as soon as we have this, I could change abc to take advantage, and this would give us the very cool xyz"
So, if anyone else if feeling at all like me about this issue, they are feeling all warm and fuzzy knowing that a few smart people are giving us the facility to do something we hope we never, ever have to do. :-)
Actually, you'll want to do it a lot <wink>. Christian & I have bantered about this a few times a year in pvt, usually motivated by some horrendous kludge posted to c.l.py to solve a problem that any Assistant Professor of Medieval English could solve without effort in Icon. The *uses* aren't esoteric at all. or-at-least-not-more-than-you-make-'em-ly y'rs - tim
[Tim tells me it will all be obvious if I just think a little harder <wink>] Your points about "acedemic examples" is well taken. The reality is that, even given these simple examples (which I dared deride as acedemic), the simple fact is Im not seeing "the point". I seriously dont doubt all all you say. However, as Sam and Chris have said many times, it is just a matter of changing the way to you think. Interestingly: Chris said it recently, and continues to say it. Sam said it to me _years_ ago, and said it repeatedly, but hasnt said it recently. Tim hasnt really said it yet :-) This is almost certainly because when your brain does switch, it is a revelation, and really not too hard at all. But after a while, you forget the switch ever took place. Closest analogy I can think of is OO programming. In my experience trying to _learn_ OO programming from a few misc examples and texts was pointless and very hard. You need a language to play with it in. And when you have one, your brain makes the switch, you see the light, and you can't see what was ever mysterious about it. And you tell everyone its easy; "just change the way you think about data" :-) But to all us here, OO programming is just so obvious it goes without saying. Occasionaly a newbie will have trouble with OO concepts in Python, and I personally have trouble seeing what could _possibly_ be difficult about understanding these very simple concepts. So Im just as guilty, just not in this particular case :-) So, short of all us here going and discovering the light using a different language (perish the thought :), my original point stands that until Chris' efforts give us something we can easily play with, some of use _still_ wont see what all the fuss is about. (Although I admit it has nothing to do with either the examples or the applicability of the technology to all sorts of things) Which leaves you poor guys in a catch 22 - without noise of some sort from the rest of us, its hard to keep the momentum going, but without basically a fully working Python with continuations, we wont be making much noise. But-I-will-thank-you-all-personally-and-profusely-when-I-do-see-the-light, ly Mark.
yes I'm happy - chris
You should be! So how come nobody else is <wink/frown>?
Chris and I have been through this in private, but it seems that as long as I don't fess up in public I'm afraid it will come back and I'll get pressure coming at me to endorse Chris' code. I have no problem with the general concept (see my response to Sam's post of exciting examples). But I have a problem with a megapatch like this that affects many places including very sensitive areas like the main loop in ceval.c. The problem is simply that I know this is very intricate code, and I can't accept a patch of this scale to this code before I understand every little detail of the patch. I'm just too worried otherwise that there's a reference count bug in it that will very subtly break stuff and that will take forever to track down; I feel that when I finally have the time to actually understand the whole patch I'll be able to prevent that (famous last words). Please don't expect action or endorsement of Chris' patch from me any time soon, I'm too busy. However I'd love it if others used the patch in a real system and related their experiences regarding performance, stability etc. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
yes I'm happy - chris
You should be! So how come nobody else is <wink/frown>?
Chris and I have been through this in private, but it seems that as long as I don't fess up in public I'm afraid it will come back and I'll get pressure coming at me to endorse Chris' code.
Please let me add a few comments.
I have no problem with the general concept (see my response to Sam's post of exciting examples).
This is the most worthful statement I can get. And see below.
But I have a problem with a megapatch like this that affects many places including very sensitive areas like the main loop in ceval.c.
Actually it is a rather small patch, but the implicit semantic change is rather hefty.
The problem is simply that I know this is very intricate code, and I can't accept a patch of this scale to this code before I understand every little detail of the patch. I'm just too worried otherwise that there's a reference count bug in it that will very subtly break stuff and that will take forever to track down; I feel that when I finally have the time to actually understand the whole patch I'll be able to prevent that (famous last words).
I never expected to see this patch go into Python right now. The current public version is an alpha 0.2. Meanwhile I have 0.3, with again new patches, and a completely reworked policy of frame refcounting. Even worse, there is a night mare of more work which I simply had no time for. All the instance and onbect code must be carefully changed, since they still need to call back in a recursive way. This is hard to change until I have a better mechanism to generate all the callbacks. For instance, I cannot switch tasks in an __init__ at this time. Although I can do so in regular methods. But this is all half-baked. In other words, the danger is by far not over, but still in the growing phase. I believe I should work on and maintain this until I'm convinced that there are not more refcount bugs than before, and until I have evicted every recursion which is a serious impact. This is still months of work. When I release the final version, I will pay $100 to the first person who finds a refcount bug which I introduced. But not before. I don't want to waste Guido's time, and for sure not now with this bloody fresh code. What I needed to know is wether I am on the right track or if I'm wasting my time. But since I have users already, it is no waste at all. What I really could use were some hints about API design. Guido, thank you for Python - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
participants (6)
-
Christian Tismer -
Guido van Rossum -
Ka-Ping Yee -
Mark Hammond -
Tim Peters -
Vladimir Marangozov