Stackless Preview (was: Memory leak under Idle?)
Tim Peters wrote: [see the main list on idle leaks]
if-pystone-works-ship-it-ly y'rs - tim
Well, on request of uncle Timmy, I do it. Although it's very early. A preview of stackless Python can be found under ftp://ftp.pns.cc/pub/stackless_990606.zip Current status: The main interpreter is completely stackless. Just for fun, I've set max recursion depth to 30000, so just try it. PyStone does of course run. My measures were about 3-5 percent slower than with standard Python. I think this is quite fair. As a side effect, the exec_statement now behaves better than before, since exec <anything> without globals and locals should update the current environment, which worked only for exec "string". Most of the Run_<thing> functions are stackless as well. Almost all cases could be treated tail recursively. I have just begun to work on the builtins, and there is a very bloody, new-born stackless map, which seems to behave quite well. (It is just an hour old, so don't blame me if I didn't get al refcounts right). This is a first special case, since I *had* to build a tiny interpreter from the old map code. Still quite hacky, but not so bad. It creates its own frame and bails out whenever it needs to call the interpreter. If not, it stays in the loop. Since this one is so fresh, the old map is still there, and the new one has the name "map_nr". As a little bonus, map_nr now also shows up in a traceback. I've set the line no to the iteration count. Beware, this is just a proof of concept and will most probably change. Further plans: I will make the other builtins stackless as well (reduce, filter), also the simple tail-recursive ones which I didn't do now due to lack of time. I think I will *not* think of stackless imports. After loking into this for a while, I think this is rather hairy, and also not necessary. On extensions: There will be a coroutine extension in a few days. This is now nearly a no-brainer, since I did the stackless Python with exactly that in mind. This is the real fruit where I'm after, so please let me pick it :) Documentation: Besides the few new comments, there is nothing yet. Diff files: Sorry, there are no diffs but just the modified files. I had no time to do them now. All files stem from the official Python 1.5.2 release. You might wonder about the version: In order to support extension modules which rely on some special new features of frames, I decided to name this Python "1.5.42", since I believe it will be useful at least "four two" people. :-) I consider this an Alpha 1 version. fearing the feedback :-) ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
Christian Tismer wrote: [a lot]
fearing the feedback :-) ciao - chris
I expected everything but forgot to fear "no feedback". :-) About 5 or 6 people seem to have taken the .zip file. Now I'm wondering why nobody complains. Was my code so wonderful, so disgustingly bad, or is this just boring :-? If it's none of the three above, I'd be happy to get a hint if I should continue, or if and what I should change. Maybe it would make sense to add some documentation now, and also to come up with an application which makes use of the stackless implementation, since there is now not much to wonder about than that it seems to work :-) yes-call-me-impatient - ly chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
"CT" == Christian Tismer <tismer@appliedbiometrics.com> writes:
CT> Christian Tismer wrote: [a lot]
fearing the feedback :-) ciao - chris
CT> I expected everything but forgot to fear "no feedback". :-) CT> About 5 or 6 people seem to have taken the .zip file. Now I'm CT> wondering why nobody complains. Was my code so wonderful, so CT> disgustingly bad, or is this just boring :-? CT> If it's none of the three above, I'd be happy to get a hint if I CT> should continue, or if and what I should change. I'm one of the silent 5 or 6. My reasons fall under "None of the above." They are three in number: 1. No time (the perennial excuse; next 2 weeks are quite hectic) 2. I tried to use ndiff to compare old and new ceval.c, but ran into some problems with that tool. (Tim, it looks like the line endings are identical -- all '\012'.) 3. Wasn't sure what to look at first My only suggestion would be to have an executive summary. If there was a short README file -- no more than 150 lines -- that described the essentials of the approach and told me what to look at first, I would be able to comment more quickly. Jeremy
Jeremy Hylton wrote: [...]
I'm one of the silent 5 or 6. My reasons fall under "None of the above." They are three in number: 1. No time (the perennial excuse; next 2 weeks are quite hectic) 2. I tried to use ndiff to compare old and new ceval.c, but ran into some problems with that tool. (Tim, it looks like the line endings are identical -- all '\012'.)
Yes, there are a lot of changes. As a hint: windiff from VC++ does a great job here. You can see both sources in one, in a very readable colored form.
3. Wasn't sure what to look at first
My only suggestion would be to have an executive summary. If there was a short README file -- no more than 150 lines -- that described the essentials of the approach and told me what to look at first, I would be able to comment more quickly.
Thanks a lot. Will do this tomorrow moaning as my first task. feeling much better - ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
Jeremy Hylton wrote:
CT> If it's none of the three above, I'd be happy to get a hint if I CT> should continue, or if and what I should change.
I'm one of the silent 5 or 6. My reasons fall under "None of the above." They are three in number: ... My only suggestion would be to have an executive summary. If there was a short README file -- no more than 150 lines -- that described the essentials of the approach and told me what to look at first, I would be able to comment more quickly.
Same here + a small wish: please save me the stripping of the ^M line endings typical for MSW, so that I can load the files directly in Xemacs on a Unix box. Otherwise, like Jeremy, I was a bit lost trying to read ceval.c which is already too hairy. -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
[Vladimir Marangozov]
... please save me the stripping of the ^M line endings typical for MSW, so that I can load the files directly in Xemacs on a Unix box.
Vlad, get linefix.py from Python FTP contrib's System area; converts among Unix, Windows and Mac line conventions; to Unix by default. For that matter, do a global replace of ^M in Emacs <wink>. buncha-lazy-whiners<wink>-ly y'rs - tim
"TP" == Tim Peters <tim_one@email.msn.com> writes:
TP> Vlad, get linefix.py from Python FTP contrib's System area; TP> converts among Unix, Windows and Mac line conventions; to Unix TP> by default. For that matter, do a global replace of ^M in TP> Emacs <wink>. I forgot to follow up to Vlad's original message, but in XEmacs (dunno about FSFmacs), you can visit DOS-eol files without seeing the ^M's. You will see a "DOS" in the modeline, and when you go to write the file it'll ask you if you want to write it in "plain text". I use XEmacs all the time to convert between DOS-eol and eol-The-Way-God-Intended :) To enable this, add the following to your .emacs file: (require 'crypt) -Barry
Howdy, I've begun with a first draft on Stackless Python. Didn't have enough time to finish it, but something might already be useful. (Should I better drop the fish idea?) Will write the rest tomorrow. ciao - chris http://www.pns.cc/stackless/stackless.htm -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
Howdy, I worked a little more on the docs and figured out that I could use a hint. http://www.pns.cc/stackless/stackless.htm Trying to give an example how coroutines could work, some weaknesses showed up. I wanted to write some function coroutine_transfer which swaps two frame chains. This function should return my unwind token, but unfortunately in that case a real result would be needed as well. Well, I know of several ways out, but it's a matter of design, and I'd like to find the most elegant solution for this. Could perhaps someone of those who encouraged me have a look into the problem? Do I have to add yet another field for return values and handle that in the dispatcher? thanks - chris (tired of thinking) -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
[Christian Tismer]
... If it's none of the three above, I'd be happy to get a hint if I should continue, or if and what I should change.
Sorry, Chris! Just a case of "no time" here. Of *course* you should continue, and Guido should pop in with an encouraging word too -- or a "forget it". I think this design opens the doors to a world of interesting ideas, but that's based on informed prejudice rather than careful study of your code. Cheer up: if everyone thought you were a lame ass, we all would have studied your code intensely by now <wink>. [Jeremy]
2. I tried to use ndiff to compare old and new ceval.c, but ran into some problems with that tool. (Tim, it looks like the line endings are identical -- all '\012'.)
Then let's treat this like a real bug <wink>: which version of Python did you use? And ship me the files in a tarball (I'll find a way to extract them intact). And does that specific Python+ndiff combo work OK on *other* files? Or does it fail to find any lines in common no matter what you feed it (a 1-line test case would be a real help <wink>)? I couldn't provoke a problem with the stock 1.5.2 ndiff under the stock 1.5.2 Windows Python, using the then-current CVS snapshot of ceval.c as file1 and the ceval.c from Christian's stackless_990606.zip file as file2. Both files have \r\n line endings for me, though (one thanks to CVS line translation, and the other thanks to WinZip line translation). or-were-you-running-ndiff-under-the-stackless-python<wink>?-ly y'rs - tim
Tim wrote:
Sorry, Chris! Just a case of "no time" here. Of *course* you should continue, and Guido should pop in with an encouraging word too -- or a "forget it". I think this design opens the doors to a world of interesting ideas, but that's based on informed prejudice rather than careful study of your code. Cheer up: if everyone thought you were a lame ass, we all would have studied your code intensely by now <wink>.
No time here either... I did try to have a quick peek and my first impression is that it's *very* tricky code! You know what I think of that... Here's what I think we should do first (I've mentioned this before but nobody cheered me on :-). I'd like to see this as the basis for 1.6. We should structurally split the Python Virtual Machine and related code up into different parts -- both at the source code level and at the runtime level. The core PVM becomes a replaceable component, and so do a few other parts like the parser, the bytecode compiler, the import code, and the interactive read-eval-print loop. Most object implementations are shared between all -- or at least the interfaces are interchangeable. Clearly, a few object types are specific to one or another PVM (e.g. frames). The collection of builtins is also a separate component (though some builtins may again be specific to a PVM -- details, details!). The goal of course, is to create a market for 3rd party components here, e.g. Chris' flat PVM, or Skip's bytecode optimizer, or Greg's importer, and so on. Thoughts? --Guido van Rossum (home page: http://www.python.org/~guido/)
On Tue, 8 Jun 1999, Guido van Rossum wrote:
We should structurally split the Python Virtual Machine and related code up into different parts -- both at the source code level and at the runtime level. The core PVM becomes a replaceable component, and so do a few other parts like the parser, the bytecode compiler, the import code, and the interactive read-eval-print loop.
The goal of course, is to create a market for 3rd party components here, e.g. Chris' flat PVM, or Skip's bytecode optimizer, or Greg's importer, and so on.
Thoughts?
If I understand it correctly, it means that I can fit in a third-party read-eval-print loop, which is my biggest area of frustration with the current internal structure. Sounds like a plan to me, and one which (lucky for me) I'm not qualified for! --david
Guido van Rossum wrote:
Tim wrote:
Sorry, Chris! Just a case of "no time" here. Of *course* you should continue, and Guido should pop in with an encouraging word too -- or a "forget it". I think this design opens the doors to a world of interesting ideas, but that's based on informed prejudice rather than careful study of your code. Cheer up: if everyone thought you were a lame ass, we all would have studied your code intensely by now <wink>.
No time here either...
I did try to have a quick peek and my first impression is that it's *very* tricky code! You know what I think of that...
Thanks for looking into it, thanks for saying it's tricky. Since I failed to supply proper documentation yet, this impression must come up. But it is really not true. The code is not tricky but just straightforward and consequent, after one has understood what it means to work without a stack, under the precondition to avoid too much changes. I didn't want to rewrite the world, and I just added the tiny missing bits. I will write up my documentation now, and you will understand what the difficulties were. These will not vanish, "stackless" is a brainteaser. My problem was not how to change the code, but finally it was how to change my brain. Now everything is just obvious.
Here's what I think we should do first (I've mentioned this before but nobody cheered me on :-). I'd like to see this as the basis for 1.6.
We should structurally split the Python Virtual Machine and related code up into different parts -- both at the source code level and at the runtime level. The core PVM becomes a replaceable component, and so do a few other parts like the parser, the bytecode compiler, the import code, and the interactive read-eval-print loop. Most object implementations are shared between all -- or at least the interfaces are interchangeable. Clearly, a few object types are specific to one or another PVM (e.g. frames). The collection of builtins is also a separate component (though some builtins may again be specific to a PVM -- details, details!).
Good idea, and a lot of work. Having different frames for different PVM's was too much for me. Instead, I tried to adjust frames in a way where a lot of machines can work with. I tried to show the concept of having different VM's by implementing a stackless map. Stackless map is a very tiny one which uses frames again (and yes, this was really hacked). Well, different frame flavors would make sense, perhaps. But I have a central routine which handles all calls to frames, and this is what I think is needed. I already *have* pluggable interpreters here, since a function can produce a frame which is bound to an interpreter, and push it to the frame stack.
The goal of course, is to create a market for 3rd party components here, e.g. Chris' flat PVM, or Skip's bytecode optimizer, or Greg's importer, and so on.
I'm with that component goal, of course. Much work, not for one persone, but great. While I don't think it makes sense to make a flat PVM pluggable. I would start with a flat PVM, since that opens a world of possibilities. You can hardly plug flatness in after you started with a wrong stack layout. Vice versa, plugging the old machine would be possible. later - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
[Christian Tismer, replying to Guido's enthusiasm <wink>]
Thanks for looking into it, thanks for saying it's tricky. Since I failed to supply proper documentation yet, this impression must come up.
But it is really not true. The code is not tricky but just straightforward and consequent, after one has understood what it means to work without a stack, under the precondition to avoid too much changes. I didn't want to rewrite the world, and I just added the tiny missing bits.
I will write up my documentation now, and you will understand what the difficulties were. These will not vanish, "stackless" is a brainteaser. My problem was not how to change the code, but finally it was how to change my brain. Now everything is just obvious.
FWIW, I believe you! There's something *inherently* tricky about maintaining the effect of a stack without using the stack C supplies implicitly, and from all you've said and what I've learned of your code, it really isn't the code that's tricky here. You're making formerly-hidden connections explicit, which means more stuff is visible, but also means more power and flexibility *because* "more stuff is visible". Agree too that this clearly <wink> moves in the direction of making the VM pluggable.
... I *will* continue, regardless what anybody says.
Ah, if that's how this works, then STOP! Immediately! Don't you dare waste more of our time with this crap <wink>. want-some-money?-ly y'rs - tim
Tim Peters wrote:
[Christian Tismer, replying to Guido's enthusiasm <wink>]
...
I will write up my documentation now, and you will still under some work :) understand what the difficulties were. These will not vanish, "stackless" is a brainteaser. My problem was not how to change the code, but finally it was how to change my brain. Now everything is just obvious.
FWIW, I believe you! There's something *inherently* tricky about maintaining the effect of a stack without using the stack C supplies implicitly, and from all you've said and what I've learned of your code, it really isn't the code that's tricky here. You're making formerly-hidden connections explicit, which means more stuff is visible, but also means more power and flexibility *because* "more stuff is visible".
I knew you would understand me. Feeling much, much better now :-)) After this is finalized, restartable exceptions might be interesting to explore. No, Chris, do the doco...
I *will* continue, regardless what anybody says.
Ah, if that's how this works, then STOP! Immediately! Don't you dare waste more of our time with this crap <wink>.
Thanks, you fired me a continuation. Here the way to get me into an endless loop: Give me an unsolvable problem and claim I can't do that. :) (just realized that I'm just another pluggable interpreter)
want-some-money?-ly y'rs - tim
No, but meet you at least once in my life. -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
[Guido van Rossum]
... Here's what I think we should do first (I've mentioned this before but nobody cheered me on :-). I'd like to see this as the basis for 1.6.
We should structurally split the Python Virtual Machine and related code up into different parts -- both at the source code level and at the runtime level. The core PVM becomes a replaceable component, and so do a few other parts like the parser, the bytecode compiler, the import code, and the interactive read-eval-print loop. Most object implementations are shared between all -- or at least the interfaces are interchangeable. Clearly, a few object types are specific to one or another PVM (e.g. frames). The collection of builtins is also a separate component (though some builtins may again be specific to a PVM -- details, details!).
The goal of course, is to create a market for 3rd party components here, e.g. Chris' flat PVM, or Skip's bytecode optimizer, or Greg's importer, and so on.
Thoughts?
The idea of major subsystems getting reworked to conform to well-defined and well-controlled interfaces is certainly appealing. I'm just more comfortable squeezing another 1.7% out of list.sort() <0.9 wink>. trying-to-reduce-my-ambitions-to-match-my-time-ly y'rs - tim
Tim Peters wrote:
[Christian Tismer]
... If it's none of the three above, I'd be happy to get a hint if I should continue, or if and what I should change.
Sorry, Chris! Just a case of "no time" here. Of *course* you should continue, and Guido should pop in with an encouraging word too -- or a "forget it".
Yup, I know this time problem just too good. Well, I think I got something in between. I was warned before, so I didn't try to write final code, but I managed to prove the concept. I *will* continue, regardless what anybody says.
or-were-you-running-ndiff-under-the-stackless-python<wink>?-ly y'rs - tim
I didn't use ndiff, but regular "diff", and it worked. But since theere is not much change to the code, but some significant change to the control flow, I found the diff output too confusing. Windiff was always open when I wrote that, to be sure that I didn't trample on things which I didn't want to mess up. A good tool! ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
Chris> If it's none of the three above, I'd be happy to get a hint if I Chris> should continue, or if and what I should change. Chris, My vote is for you to keep at it. I haven't looked at it because I have absolutely zero free time available. This will probably continue until at least the end of July, perhaps until Labor Day. Big doings at Musi-Cal and in the Montanaro household (look for an area code change in a month or so). Skip Montanaro | Mojam: "Uniting the World of Music" http://www.mojam.com/ skip@mojam.com | Musi-Cal: http://www.musi-cal.com/ 518-372-5583
participants (8)
-
Barry A. Warsaw -
Christian Tismer -
David Ascher -
Guido van Rossum -
Jeremy Hylton -
Skip Montanaro -
Tim Peters -
Vladimir Marangozov