Stackless Python and its tasklets would be great! Why is Python still not stackless? __________________________________ Do you Yahoo!? Friends. Fun. Try the all-new Yahoo! Messenger. http://messenger.yahoo.com/
Eyal Lotem wrote:
Stackless Python and its tasklets would be great!
Indeed. It is so handy that even I use it for my projects.
Why is Python still not stackless?
Because this question is not yet asked 10 times a day. :-) -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Eyal Lotem wrote:
Why is Python still not stackless?
But it is: http://www.stackless.com/ If you are asking: Why is python.org Python still not stackless? then I think the answer has varied over time. The most recent answer is that no volunteer has stepped forward to make Python stackless. That, in turn, likely hasn't happened because it is hard to do, and because nobody has seen any reason to do so as you could always use Stackless Python. Regards, Martin
Does this mean that the implementation of Stackless 3.0 could be acceptable to integrate into CPython 2.5? It would be absolutely wonderful to have those capabilities available in the standard distribution! -Shane Holloway Martin v. Löwis wrote:
Eyal Lotem wrote:
Why is Python still not stackless?
But it is: http://www.stackless.com/
If you are asking: Why is python.org Python still not stackless? then I think the answer has varied over time.
The most recent answer is that no volunteer has stepped forward to make Python stackless. That, in turn, likely hasn't happened because it is hard to do, and because nobody has seen any reason to do so as you could always use Stackless Python.
Regards, Martin
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/shane.holloway%40ieee.org
Shane Holloway (IEEE) wrote:
Does this mean that the implementation of Stackless 3.0 could be acceptable to integrate into CPython 2.5? It would be absolutely wonderful to have those capabilities available in the standard distribution!
Probably not. Stackless 3.0 works only on selected hardware; a number of platforms currently supported by Python would not be supported anymore. An implementation of stackless python should work on all platforms (atleast on those currently supported by Python). Regards, Martin
On May 31, 2004, at 2:50 PM, Martin v. Löwis wrote:
Shane Holloway (IEEE) wrote:
Does this mean that the implementation of Stackless 3.0 could be acceptable to integrate into CPython 2.5? It would be absolutely wonderful to have those capabilities available in the standard distribution!
Probably not. Stackless 3.0 works only on selected hardware; a number of platforms currently supported by Python would not be supported anymore. An implementation of stackless python should work on all platforms (atleast on those currently supported by Python).
Although I don't particularly agree with that, because Stackless functionality could be turned on and off with a platform dependent configure flag (like threads or unicode)... Is there a canonical list of CPU architectures and platforms that Python officially supports? -bob
Bob Ippolito wrote:
Although I don't particularly agree with that, because Stackless functionality could be turned on and off with a platform dependent configure flag (like threads or unicode)... Is there a canonical list of CPU architectures and platforms that Python officially supports?
No - although there is a list of platforms, that Python *doesn't* support (PEP 11). Python is written in C, so it should work on any system that supports ISO C, although many functions won't be available if the system doesn't support POSIX. As for turning of things: yes, unicode can be turned off, but not because the platform doesn't support it - rather because the administrator doesn't like unicode. Threads are turned off on operating systems that don't provide threads, and there is nothing that Python could do about it. Stackless' support for platforms is different: it includes assembler code, so you have to write bits of assembler code for each architecure (and perhaps for each system on a single architecture, or even each assembler on each system). [In addition, the approach taken by Stackless 3.0 is probably questionable: for example, it may be that C++ exception handling stops working in Stackless.] Regards, Martin
On May 31, 2004, at 3:33 PM, Martin v. Löwis wrote:
Bob Ippolito wrote:
Although I don't particularly agree with that, because Stackless functionality could be turned on and off with a platform dependent configure flag (like threads or unicode)... Is there a canonical list of CPU architectures and platforms that Python officially supports?
No - although there is a list of platforms, that Python *doesn't* support (PEP 11). Python is written in C, so it should work on any system that supports ISO C, although many functions won't be available if the system doesn't support POSIX.
As for turning of things: yes, unicode can be turned off, but not because the platform doesn't support it - rather because the administrator doesn't like unicode. Threads are turned off on operating systems that don't provide threads, and there is nothing that Python could do about it.
Stackless' support for platforms is different: it includes assembler code, so you have to write bits of assembler code for each architecure (and perhaps for each system on a single architecture, or even each assembler on each system).
I don't see how including a little bit of per-architecture-calling-convention assembler code is much different than depending on pthreads (or some other platform dependent thing) being present. There is also potentially a platform neutral way of doing it (see below).
[In addition, the approach taken by Stackless 3.0 is probably questionable: for example, it may be that C++ exception handling stops working in Stackless.]
I'm not quite sure you are up to date on how Stackless 3.0 works. The assembly code is only there to facilitate "hard switching", which is only required to do tasklet switching in the face of some C extension that doesn't know about Stackless. The vast majority of tasklet switches use the "soft switching" mechanism, which just means that the code they use is capable of going into non-recursive mode. All pure Python code and almost all code built-in to Python goes through "soft switching", which is implemented in pure C with no stack tricks or anything. Stackless would still be a very useful addition to Python even when "hard switching" is not available. Alternatively, Armin believes that Stackless "hard switching" can actually be implemented in mostly-platform-neutral C (either by setjmp/longjmp or alloca hacks, I forget which) and has a prototype implementation in the Stackless CVS repository (though it is not integrated with Stackless itself). It might also be possible to coerce something like libffi into doing what we need it to do. I'm not sure how well "hard switching" would play with an operating system that is very protective of its stack, like OpenBSD or future versions of Windows, though. I'm also not sure how it could "break C++ exception handling" in scenarios where it wouldn't be broken anyway. I'd like to see a test of that that passes under regular Python but fails in Stackless for this. -bob
Bob Ippolito wrote:
I'm not quite sure you are up to date on how Stackless 3.0 works. The assembly code is only there to facilitate "hard switching", which is only required to do tasklet switching in the face of some C extension that doesn't know about Stackless.
Yes, that was my understanding. "Some C extensions" includes Tcl and COM, right? Unless I'm mistaken, it also includes map, filter, and .encode/.decode of unicode objects (or any other C module that calls back into Python, except for tasklet mechanism of stackless itself). In these cases, copying out the stack will break the exception chain that the Visual C++ compiler creates through the FS segment, so that an attempt to throw an exception will cause a program crash, right?
All pure Python code and almost all code built-in to Python goes through "soft switching", which is implemented in pure C with no stack tricks or anything.
What is the control flow for, say, the builtin map() function?
Stackless would still be a very useful addition to Python even when "hard switching" is not available.
That is the point I made at the beginning of this thread: Python is currently not stackless because nobody has contributed a patch recently to make it so. A patch that takes recursion out of eval_frame might alone be valuable, but hasn't been contributed.
Alternatively, Armin believes that Stackless "hard switching" can actually be implemented in mostly-platform-neutral C (either by setjmp/longjmp or alloca hacks, I forget which)
It is setjmp/longjmp. I have implemented a coroutine library on top of setjmp/longjmp myself, so I am well aware of the implications.
I'm also not sure how it could "break C++ exception handling" in scenarios where it wouldn't be broken anyway. I'd like to see a test of that that passes under regular Python but fails in Stackless for this.
The VC++ exception mechanism puts a linked chain of exception frames on the stack, where, at the beginning of each function, the stack pointer is stored at FS:18 (or some such), and the old value of FS:18 is stored on the stack. Then, when an exception occurs, this linked list is used to traverse the stack. If multiple stacks are used in a single thread, then this linked list gets messed up. Without trying, I think the following call chain might cause a crash if Stackless is compiled with exception handling on: Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet Py-func3() -> C-func2() -> throw exception() Regards, Martin
On May 31, 2004, at 4:56 PM, Martin v. Löwis wrote:
Bob Ippolito wrote:
I'm not quite sure you are up to date on how Stackless 3.0 works. The assembly code is only there to facilitate "hard switching", which is only required to do tasklet switching in the face of some C extension that doesn't know about Stackless.
Yes, that was my understanding. "Some C extensions" includes Tcl and COM, right? Unless I'm mistaken, it also includes map, filter, and .encode/.decode of unicode objects (or any other C module that calls back into Python, except for tasklet mechanism of stackless itself).
You're definitely correct about Tcl and COM.. COM is however a total non-issue because Win32/x86 is very well supported by Christian ;) I'm believe that map and filter are non-recursive, I am not sure about unicode encode/decode. I don't think it's very common to do a tasklet switch in the middle of a unicode encode or decode, though :)
In these cases, copying out the stack will break the exception chain that the Visual C++ compiler creates through the FS segment, so that an attempt to throw an exception will cause a program crash, right?
I'm also not sure how it could "break C++ exception handling" in scenarios where it wouldn't be broken anyway. I'd like to see a test of that that passes under regular Python but fails in Stackless for this.
The VC++ exception mechanism puts a linked chain of exception frames on the stack, where, at the beginning of each function, the stack pointer is stored at FS:18 (or some such), and the old value of FS:18 is stored on the stack. Then, when an exception occurs, this linked list is used to traverse the stack. If multiple stacks are used in a single thread, then this linked list gets messed up.
Without trying, I think the following call chain might cause a crash if Stackless is compiled with exception handling on:
Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet Py-func3() -> C-func2() -> throw exception()
I don't see how it could cause problems unless this is valid: Py_Func1 -> CPlusPlus_ExceptionCatcher -> Py_Func2 -> CPlusPlus_ExceptionThrower -> throws exception() If that *is* valid, how the heck does Py_Func2 get cleaned up? I have to admit that I'm not very familiar with C++'s implementation of exceptions. -bob
Bob Ippolito wrote:
I'm believe that map and filter are non-recursive
Can you point me to the code that demonstrates this? Looking at Stackless' src/Python/bltinmodule.c:builtin_map, I see for (i = 0; ; ++i) { ... value = PyEval_CallObject(func, alist); ... PyList_SetItem(result, i, value) ... } which looks recursive to me.
I am not sure about unicode encode/decode. I don't think it's very common to do a tasklet switch in the middle of a unicode encode or decode, though :)
I thought Stackless was not only about tasklet switching, but also about running without stack... Wouldn't it perform the "hard" switching if the stack is running too deep?
Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet Py-func3() -> C-func2() -> throw exception()
I don't see how it could cause problems unless this is valid: Py_Func1 -> CPlusPlus_ExceptionCatcher -> Py_Func2 -> CPlusPlus_ExceptionThrower -> throws exception()
It crashes in the process of throwing the exception - not because it finds no exception handler. It crashes because the internal data structures have been corrupted. I should have better made this example: Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet Py-func3() -> Exception-Catcher -> C-func2() -> throw exception()
If that *is* valid, how the heck does Py_Func2 get cleaned up? I have to admit that I'm not very familiar with C++'s implementation of exceptions.
Throwing exceptions through Python functions is not supported, indeed. However, I was trying to point out that the entire exception handling mechanism gets corrupted, even if there are valid handlers on the stack. Regards, Martin
On May 31, 2004, at 5:25 PM, Martin v. Löwis wrote:
Bob Ippolito wrote:
I'm believe that map and filter are non-recursive
Can you point me to the code that demonstrates this? Looking at Stackless' src/Python/bltinmodule.c:builtin_map, I see
for (i = 0; ; ++i) { ... value = PyEval_CallObject(func, alist); ... PyList_SetItem(result, i, value) ... }
which looks recursive to me.
Ok, then I am mistaken. I thought I had seen a commit message that said it was made non-recursive. Either way, it's possible to write a non-recursive version of map or anything else, it's just a bit of work.
I am not sure about unicode encode/decode. I don't think it's very common to do a tasklet switch in the middle of a unicode encode or decode, though :)
I thought Stackless was not only about tasklet switching, but also about running without stack... Wouldn't it perform the "hard" switching if the stack is running too deep?
I don't think so, because the C stack should never run too deep, since most code that is going to recurse (from the python-view) will do it non-recursively (from the C-view).
Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet Py-func3() -> C-func2() -> throw exception() I don't see how it could cause problems unless this is valid: Py_Func1 -> CPlusPlus_ExceptionCatcher -> Py_Func2 -> CPlusPlus_ExceptionThrower -> throws exception()
It crashes in the process of throwing the exception - not because it finds no exception handler. It crashes because the internal data structures have been corrupted.
I should have better made this example:
Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet Py-func3() -> Exception-Catcher -> C-func2() -> throw exception()
If that *is* valid, how the heck does Py_Func2 get cleaned up? I have to admit that I'm not very familiar with C++'s implementation of exceptions.
Throwing exceptions through Python functions is not supported, indeed. However, I was trying to point out that the entire exception handling mechanism gets corrupted, even if there are valid handlers on the stack.
I don't think that the entire exception handling mechanism gets corrupted. Each tasklet should always have a consistent view of the C stack. Anyway, I don't think we need to discuss it any further unless someone writes a failing test... and if that is indeed the case, then the discussion should be taken to the stackless list. -bob
Bob Ippolito wrote: oops, ok, this need some words, still. ...
Ok, then I am mistaken. I thought I had seen a commit message that said it was made non-recursive. Either way, it's possible to write a non-recursive version of map or anything else, it's just a bit of work.
I had that in 1.0, and I don't plan to do it again, although it is simple, now. ...
I don't think so, because the C stack should never run too deep, since most code that is going to recurse (from the python-view) will do it non-recursively (from the C-view).
You can create such situations. Until Stackless 3.0, it was possible to do very deep explicit calls to the __call__ wrapper, for instance, and wrappers were not supported. Stackless 3.1 which is about to be announced has full support for all slots and all methods by the machinery. Supported are all tail-recursive calls (like __call__), because this is very cheap to do. But it is possible to implement as many of special method handlers and wrapper handlers as one wishes; I have the infrastructure ready for this. Anyway, for this rare artificial case, I have a check built in that measures stack size instead of recursion counting, and it will save the stack if necessary. Of course this is platform dependent, but my, if we switch this off for non-supported platforms, what do we loose? [C++ exception issues]
I don't think that the entire exception handling mechanism gets corrupted. Each tasklet should always have a consistent view of the C stack. Anyway, I don't think we need to discuss it any further unless someone writes a failing test... and if that is indeed the case, then the discussion should be taken to the stackless list.
Good idea, see you there! cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Martin v. Löwis wrote:
Bob Ippolito wrote:
I'm believe that map and filter are non-recursive
They have been. They can be, again, but why.
Can you point me to the code that demonstrates this? Looking at Stackless' src/Python/bltinmodule.c:builtin_map, I see
for (i = 0; ; ++i) { ... value = PyEval_CallObject(func, alist); ... PyList_SetItem(result, i, value) ... }
which looks recursive to me.
Yes. trying to do a minimalistic approach, I dropped all non-recursive implementations of non-trivial things (meaning non-tail-recursive things) which were not absolutely necessary. And since we have list comprehensions, map is no longer worth being supported so much.
I am not sure about unicode encode/decode. I don't think it's very common to do a tasklet switch in the middle of a unicode encode or decode, though :)
I thought Stackless was not only about tasklet switching, but also about running without stack... Wouldn't it perform the "hard" switching if the stack is running too deep?
Yes, it would, if you allow it to do so. This would happen after a very deep recursion, which you wouldn't be able to do with normal Python at all. ...
It crashes in the process of throwing the exception - not because it finds no exception handler. It crashes because the internal data structures have been corrupted.
I don't know the C++ exception mechanism. It it walks the whole stack once before taking an action, then this is a problem. If it does proper unwinding of C frames, I could probably play the game and restore things just in time. Anyway, I don't really get the point. 95% of Stackless is doing soft-switched stackless calls. Behavior is completely controllable. We can easily avoid any special C stack operation, by setting a flag that disallows it (easy to implement) or by excluding the hard switching stuff, completely (not an option now, but easy, too). The discussion, as often before, tries to find out why it is *not* possible to include Stackless, instead of finding out the useful parts which could be supported with relatively small impact. This is why Stackless exists, and will continue to exist. For me, this thread is closed -- thanks - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
At 02:09 AM 6/1/04 +0200, Christian Tismer wrote:
Anyway, I don't really get the point. 95% of Stackless is doing soft-switched stackless calls. Behavior is completely controllable. We can easily avoid any special C stack operation, by setting a flag that disallows it (easy to implement) or by excluding the hard switching stuff, completely (not an option now, but easy, too).
If soft-switching is portable (i.e. pure C, no assembly), and is exposed as a library module (so that Jython et al can avoid supporting it), then perhaps a PEP for adding that functionality to mainstream Python would be meaningful. If that gets in, then the other 5% can always sneak in later via feature creep. ;) Or, more importantly (if I understand correctly), it could be separately distributed as an add-on for platforms that can support it.
On May 31, 2004, at 8:37 PM, Phillip J. Eby wrote:
At 02:09 AM 6/1/04 +0200, Christian Tismer wrote:
Anyway, I don't really get the point. 95% of Stackless is doing soft-switched stackless calls. Behavior is completely controllable. We can easily avoid any special C stack operation, by setting a flag that disallows it (easy to implement) or by excluding the hard switching stuff, completely (not an option now, but easy, too).
If soft-switching is portable (i.e. pure C, no assembly), and is exposed as a library module (so that Jython et al can avoid supporting it), then perhaps a PEP for adding that functionality to mainstream Python would be meaningful.
Soft switching needs to be implemented in a few key places of the interpreter itself or else Stackless would surely have been maintained as an extension module. It is already pure C, no assembly or platform specific code. Supporting the interface from Jython or IronPython should be possible, though at worst case each tasklet might actually be a new thread so it might not be terribly efficient... though it would work.
If that gets in, then the other 5% can always sneak in later via feature creep. ;) Or, more importantly (if I understand correctly), it could be separately distributed as an add-on for platforms that can support it.
Hard switching could be an add-on for platforms that support it.. -bob
At 08:49 PM 5/31/04 -0400, Bob Ippolito wrote:
On May 31, 2004, at 8:37 PM, Phillip J. Eby wrote:
At 02:09 AM 6/1/04 +0200, Christian Tismer wrote:
Anyway, I don't really get the point. 95% of Stackless is doing soft-switched stackless calls. Behavior is completely controllable. We can easily avoid any special C stack operation, by setting a flag that disallows it (easy to implement) or by excluding the hard switching stuff, completely (not an option now, but easy, too).
If soft-switching is portable (i.e. pure C, no assembly), and is exposed as a library module (so that Jython et al can avoid supporting it), then perhaps a PEP for adding that functionality to mainstream Python would be meaningful.
Soft switching needs to be implemented in a few key places of the interpreter itself or else Stackless would surely have been maintained as an extension module.
I'm aware of this, which is why I said "exposed" as a library module, not "implemented" as one. :)
It is already pure C, no assembly or platform specific code. Supporting the interface from Jython or IronPython should be possible, though at worst case each tasklet might actually be a new thread so it might not be terribly efficient... though it would work.
Yes, I suppose in the simplest case one could implement switching primitives via a simple mutex: acquire it when resuming, release it when yielding. If the PEP described the semantics of tasklets in terms of threads, then a heavyweight implementation could be achieved. However, it seems to me that any Python implementation that supports generators should in principle be able to support co-operative multitasking anyway, so long as there is no C code in the call chain. Since microthreads are basically doable in pure Python 2.2 as long as you write *everything* as a generator and simulate your control stack, it seems you should be able to do this in the Python interpreter. Actually, it seems to me that if there were a way to resume execution of a frame that has thrown a special "task switching" exception (sort of a non-local 'yield' operation), you could implement lightweight threading *without* having to de-stackify Python. Of course, it's possible that the implementation of such a strategy might be just as complex as de-stacking Python. To make it work, I think you'd need a way to tell a nested execution of Python code that it's being called from Python code (as opposed to C), so that it knows where to "resume to". A chain of resumable frames could then be resumed by a special function. Then, upon catching an exception that didn't unwind via any C code, you can call a function to resume the interrupted frame (and its parents). Hm, actually, you'd have to copy the frames in order to be able to resume them, so you'd need to have a designated exception type for resumable exceptions, so as not to waste time copying frames for other kinds of errors. Okay, I think I just convinced myself that it would be better to just make the interpreter core stackless instead of trying to design all that other crud. :) (To clarify, that's not an unqualified +1 to some hypothetical stacklessizing. I'm just saying that if supporting ultra-light threading in Python is desirable, stacklessness seems at first glance like a promising way to do it, compared to trying to deal with unwinding the C stack. And, to top it off, I think you'll end up doing half of what Stackless already does in order to "resume" the interrupted frames anyhow.)
On May 31, 2004, at 10:51 PM, Phillip J. Eby wrote:
At 08:49 PM 5/31/04 -0400, Bob Ippolito wrote:
On May 31, 2004, at 8:37 PM, Phillip J. Eby wrote:
At 02:09 AM 6/1/04 +0200, Christian Tismer wrote:
Anyway, I don't really get the point. 95% of Stackless is doing soft-switched stackless calls. Behavior is completely controllable. We can easily avoid any special C stack operation, by setting a flag that disallows it (easy to implement) or by excluding the hard switching stuff, completely (not an option now, but easy, too).
If soft-switching is portable (i.e. pure C, no assembly), and is exposed as a library module (so that Jython et al can avoid supporting it), then perhaps a PEP for adding that functionality to mainstream Python would be meaningful.
Soft switching needs to be implemented in a few key places of the interpreter itself or else Stackless would surely have been maintained as an extension module.
I'm aware of this, which is why I said "exposed" as a library module, not "implemented" as one. :)
Oh, it *is* already exposed as a module (named stackless)... but it's *implemented* as a built-in module, like sys or os. -bob
If soft-switching is portable (i.e. pure C, no assembly), and is exposed as a library module (so that Jython et al can avoid supporting it), then perhaps a PEP for adding that functionality to mainstream Python would be meaningful.
Anything that can be supported by a pure extension module is fair game IMO. But it was my understanding that Stackless requires changes to the VM. And I definitely don't want people to get into a habit of writing code that relies on deep recursion that is only supported by Stackless, only to find that it doesn't work in Jython etc.
If that gets in, then the other 5% can always sneak in later via feature creep. ;) Or, more importantly (if I understand correctly), it could be separately distributed as an add-on for platforms that can support it.
Hey, that's what Stackless already does. Why is that approach suddenly not good enough any more? --Guido van Rossum (home page: http://www.python.org/~guido/)
At 08:17 PM 5/31/04 -0700, Guido van Rossum wrote:
If soft-switching is portable (i.e. pure C, no assembly), and is exposed as a library module (so that Jython et al can avoid supporting it), then perhaps a PEP for adding that functionality to mainstream Python would be meaningful.
Anything that can be supported by a pure extension module is fair game IMO. But it was my understanding that Stackless requires changes to the VM.
I did mean *exposed* as a library module, not implemented as one. I am aware of the VM change.
And I definitely don't want people to get into a habit of writing code that relies on deep recursion that is only supported by Stackless, only to find that it doesn't work in Jython etc.
From the docs of sys.setrecursionlimit(): """The highest possible limit is platform-dependent. A user may need to set the limit higher when she has a program that requires deep recursion and a platform that supports a higher limit. This should be done with care, because a too-high limit can lead to a crash.""" Seems to me that the platform-specific nature of this is already established.
If that gets in, then the other 5% can always sneak in later via feature creep. ;) Or, more importantly (if I understand correctly), it could be separately distributed as an add-on for platforms that can support it.
Hey, that's what Stackless already does. Why is that approach suddenly not good enough any more?
I didn't say that. I was just suggesting that if people had specific features of Stackless that they wanted to see in CPython, that it would be best for them to write a PEP specifically addressing the desired feature(s), rather than asking "why isn't CPython stackless?"
Anything that can be supported by a pure extension module is fair game IMO. But it was my understanding that Stackless requires changes to the VM.
I did mean *exposed* as a library module, not implemented as one. I am aware of the VM change.
Well, if it's going to be a commonly required feature, it better be implementable in Jython etc. too.
And I definitely don't want people to get into a habit of writing code that relies on deep recursion that is only supported by Stackless, only to find that it doesn't work in Jython etc.
From the docs of sys.setrecursionlimit():
"""The highest possible limit is platform-dependent. A user may need to set the limit higher when she has a program that requires deep recursion and a platform that supports a higher limit. This should be done with care, because a too-high limit can lead to a crash."""
Seems to me that the platform-specific nature of this is already established.
But in practice the current limit is pretty similar across platforms. If some platforms support virtually infinite and others stick to the common 1000 or so, that is definitely not sticking to the intent of the law, even if it is to the letter.
If that gets in, then the other 5% can always sneak in later via feature creep. ;) Or, more importantly (if I understand correctly), it could be separately distributed as an add-on for platforms that can support it.
Hey, that's what Stackless already does. Why is that approach suddenly not good enough any more?
I didn't say that. I was just suggesting that if people had specific features of Stackless that they wanted to see in CPython, that it would be best for them to write a PEP specifically addressing the desired feature(s), rather than asking "why isn't CPython stackless?"
I think the bar should be raised pretty high for Stackless, given the various objections that have been raised against it (x86-only code, no Jython support, arbitrary exceptions, murky semantics, to name a few). Together, the answer as to "why" should be pretty clean by now. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Anything that can be supported by a pure extension module is fair game IMO. But it was my understanding that Stackless requires changes to the VM.
As Armin has shown, it is possible to do the hard switching approach as a pure extension module. But that's the opposite of what I want. I want a supportive core without the need to fiddle with C stacks. That needs VM changes. Cheating and supplying a different VM as an extension module is probably no option? :-)
Hey, that's what Stackless already does. Why is that approach suddenly not good enough any more?
Philip:
I didn't say that. I was just suggesting that if people had specific features of Stackless that they wanted to see in CPython, that it would be best for them to write a PEP specifically addressing the desired feature(s), rather than asking "why isn't CPython stackless?"
Guido:
I think the bar should be raised pretty high for Stackless, given the various objections that have been raised against it (x86-only code, no Jython support, arbitrary exceptions, murky semantics, to name a few).
Huh? There is support for 8 or more platforms, and I'm not addressing the assembly parts. It is most probably possible to implement soft-switchign in Jython. No idea what you mean by arbitrary exceptions (I never had that) or what's wrong with semantics. On the latter: Well, this is a pure interface issue. I don't say that everybody must love tasklets, there are other approaches and interfaces possible. But they all rely on the non-recursive interpreter core, and that's the point, IMHO.
Together, the answer as to "why" should be pretty clean by now.
If you understand the question like "why is current Stackless not included", that's true. I understood it more like the more advanced question "why does Python still block itself from lightweight threading by keping state on the C stack". ciao - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
That needs VM changes. Cheating and supplying a different VM as an extension module is probably no option? :-)
Not unless you can write it in Java or C# as well. C-only hacks are not welcome.
I think the bar should be raised pretty high for Stackless, given the various objections that have been raised against it (x86-only code, no Jython support, arbitrary exceptions, murky semantics, to name a few).
Huh? There is support for 8 or more platforms, and I'm not addressing the assembly parts.
OK, that's good news.
It is most probably possible to implement soft-switchign in Jython.
That's a rather vague statement; I'd like to hear from Samuele about that.
No idea what you mean by arbitrary exceptions (I never had that)
I meant exceptions to the general rules. Like, you can't depend on tasklet switching when you use one of these extensions, or one of these built-ins, or ...
or what's wrong with semantics.
Last time we (PythonLabs) looked at this, there were many murky rules that were difficult to explain except by looking at the accidents of the implementation. I take it you have removed all that?
On the latter: Well, this is a pure interface issue. I don't say that everybody must love tasklets, there are other approaches and interfaces possible. But they all rely on the non-recursive interpreter core, and that's the point, IMHO.
Is it really just tasklets that the users want? Then maybe we should focus on adding tasklets to Jython too, so as to understand the requirements and limitations of the approach better outside the context of a C implementation. (Jython must remain 100% pure Java.)
Together, the answer as to "why" should be pretty clean by now.
If you understand the question like "why is current Stackless not included", that's true.
That started the thread, yes.
I understood it more like the more advanced question "why does Python still block itself from lightweight threading by keping state on the C stack".
IMO mostly because Python wants to be friendly to extensions written in C or C++, including those that frequently call back into Python. --Guido van Rossum (home page: http://www.python.org/~guido/)
At 06:34 AM 6/1/04 -0700, Guido van Rossum wrote:
It is most probably possible to implement soft-switchign in Jython.
That's a rather vague statement; I'd like to hear from Samuele about that.
Me too. The only thing I've heard of in Java that does lightweight co-operative task switching, is actually implemented using a Java VM written in Java. Sort of like PyPy, only I guess you'd call it JavaJava. :) I don't remember the name of it right off, just that it was a commercial product that claimed they had a patent pending.
No idea what you mean by arbitrary exceptions (I never had that)
I meant exceptions to the general rules. Like, you can't depend on tasklet switching when you use one of these extensions, or one of these built-ins, or ...
Perhaps it would be better if it were defined in terms of where it *does* work, rather than where it doesn't. For example, one can write co-operative multitasking code in CPython now using generators... but it only works if you have "generators all the way down". That is, any code in a task that calls a generator must itself be a generator. That's a pretty comprehensible rule, albeit draconian to follow. Adding soft-switching would change the rule to be, "you can switch as long as you haven't been called from C code." That is, you haven't been called from *any* built-in or extension module. Speaking as someone who's written some significant event-driven co-operative multitasking code using generators, I would say that going from "generators all the way down" to "Python all the way down" would be a huge improvement in flexibility.
On the latter: Well, this is a pure interface issue. I don't say that everybody must love tasklets, there are other approaches and interfaces possible. But they all rely on the non-recursive interpreter core, and that's the point, IMHO.
Is it really just tasklets that the users want?
I am not currently a Stackless user, but if I were to try it, tasklets would be my primary interest. I don't actually think that the higher recursion limit is a "feature", for two reasons: 1) Python loops run a lot faster than function calls, and 2) every time I've ever gotten an error from excessive recursion, it was because I'd accidentally created an infinite recursion. So, to me, lifting the default recursion limit in standard Python would be a really *bad* idea. The other feature besides tasklets that I find potentially interesting is thread pickling. I'm intrigued by the possibility of using them to create "long-running tasks". That is, workflow using Python scripts to define the loops, branches, etc. of the workflow. But to make a really practical workflow system based on this, one would have to have a way of dealing with changes to the code that the running workflows were based on, so I don't know if that whole idea is really worth pursuing. I would probably be better off basing such an animal on PyPy, because performance for the code in a long-running process workflow is a non-issue. Anyway, I don't know about "the users" in general, but for me at least tasklets are where it's at. :) And I only care about explicit yielding, not thread-like concurrency. (IOW, I don't want Stackless doing the scheduling, I'd want to be able to write my own scheduler, in pure Python.)
I understood it more like the more advanced question "why does Python still block itself from lightweight threading by keping state on the C stack".
IMO mostly because Python wants to be friendly to extensions written in C or C++, including those that frequently call back into Python.
That would only disable task-switching within the called Python code, not from the code that called the extension. And that, only if the hard-switching module wasn't available.
Phillip J. Eby wrote:
At 06:34 AM 6/1/04 -0700, Guido van Rossum wrote:
It is most probably possible to implement soft-switchign in Jython.
That's a rather vague statement; I'd like to hear from Samuele about that.
I believe it is doable, and I'll ask him. If java can use something like function pointers, it is possible. If you have to have compiled calls all the time, well, I did not invent Java, and I don't want to know who was it. (Actually I do :) Me:
No idea what you mean by arbitrary exceptions (I never had that)
I meant exceptions to the general rules. Like, you can't depend on tasklet switching when you use one of these extensions, or one of these built-ins, or ...
This is out of date stuff, related to 2.0 where everything was hard-switched. Soft-switching has none of these problems. Hard switching between tasklets doesn't have this, too, provided that you don't leave a context without a consistent exception state. I do this consistently with Python: never leaving a Python exception around across a switch, everything is packed up. If a user tells me about a problem in the context of C++ exceptions, I will build the necessary framework to make this consistent. it just didn't happen, yet. But in the general case, there is nothing like this. If you switch between tasklets, this is like switching between threads. If your code tries to access stack memory from a different thread while running your thread, your code is flawed, not Stackless. ... Philip:
Anyway, I don't know about "the users" in general, but for me at least tasklets are where it's at. :) And I only care about explicit yielding, not thread-like concurrency. (IOW, I don't want Stackless doing the scheduling, I'd want to be able to write my own scheduler, in pure Python.)
You can have it both, some like auto-scheduling, most dislike it (as I do). This is all configurable at runtime, and by default, nothing is scheduled unless you explicitly ask for it.
I understood it more like the more advanced question "why does Python still block itself from lightweight threading by keping state on the C stack".
IMO mostly because Python wants to be friendly to extensions written in C or C++, including those that frequently call back into Python.
That would only disable task-switching within the called Python code, not from the code that called the extension. And that, only if the hard-switching module wasn't available.
Exactly. A tasklet can always tell whether it is on top-level, or if there is some C code sitting on its stack. If you know that this C code can stand a stack switch, you can switch. If not, you shouldn't do it. This is just a little more power, to be used with care. In Stackless 1.0, and in Python still, you simply can't do it. But this is all irrelevant for a possible inclusion of parts of Stackless: These would for sure avoid all hardware issues and just do the innocent "avoid the stack" thing, raising an exception if a switch on a C stack bound context is attempted. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
At 12:31 AM 6/2/04 +0200, Christian Tismer wrote:
Phillip J. Eby wrote:
At 06:34 AM 6/1/04 -0700, Guido van Rossum wrote:
It is most probably possible to implement soft-switchign in Jython.
That's a rather vague statement; I'd like to hear from Samuele about that.
I believe it is doable, and I'll ask him. If java can use something like function pointers, it is possible.
There are various things that correspond to function pointers. Sort of. You'll need to be a bit more specific than that.
If you have to have compiled calls all the time, well, I did not invent Java, and I don't want to know who was it. (Actually I do :)
Keep in mind that Jython *compiles* Python code to Java bytecodes. It is *not* a Python interpreter written in Java. I suspect this will severely limit the applicable approaches. Googling "Java co-operative multitasking" and "Java asynchronous methods" turns up plenty of hits on people who are trying to figure out how to do them, but not much about any successes. :) Indeed, the only thing I've ever run into that does it, is that commercial system I mentioned, which does it by writing a Java VM in Java, so they can control its execution. I don't think that idea will go over very well with the Jython team. :)
On Jun 2, 2004, at 1:14 AM, Phillip J. Eby wrote:
At 12:31 AM 6/2/04 +0200, Christian Tismer wrote:
If you have to have compiled calls all the time, well, I did not invent Java, and I don't want to know who was it. (Actually I do :)
Keep in mind that Jython *compiles* Python code to Java bytecodes. It is *not* a Python interpreter written in Java. I suspect this will severely limit the applicable approaches.
Googling "Java co-operative multitasking" and "Java asynchronous methods" turns up plenty of hits on people who are trying to figure out how to do them, but not much about any successes. :) Indeed, the only thing I've ever run into that does it, is that commercial system I mentioned, which does it by writing a Java VM in Java, so they can control its execution. I don't think that idea will go over very well with the Jython team. :)
Well the tasklet/channel model of Stackless is a single threaded version of CSP, which has at least one Java implementation: http://wotug.kent.ac.uk/parallel/languages/java/jcsp/ If you add a bunch of extra mutexes, then it could act just like Stackless (no actual concurrency). That, of course, doesn't cover every single function that the Stackless API currently exposes, but tasklets/channels are the raison d'etre. You don't really need the other functions (or at least, I haven't needed them). -bob
Bob Ippolito wrote:
Well the tasklet/channel model of Stackless is a single threaded version of CSP, which has at least one Java implementation: http://wotug.kent.ac.uk/parallel/languages/java/jcsp/
However, JCSP uses java.lang.Threads to implement concurrency. So they are completely unlike Stackless' tasklets in their implementation strategy.
That, of course, doesn't cover every single function that the Stackless API currently exposes, but tasklets/channels are the raison d'etre.
That sounds strange. I would have expected that the reason for Stackless Python is to have no stack, not to have tasklets. If you only wanted tasklets, you could implement them on top of threads, without any need for extensions. Regards, Martin
On Jun 2, 2004, at 1:58 AM, Martin v. Löwis wrote:
Bob Ippolito wrote:
Well the tasklet/channel model of Stackless is a single threaded version of CSP, which has at least one Java implementation: http://wotug.kent.ac.uk/parallel/languages/java/jcsp/
However, JCSP uses java.lang.Threads to implement concurrency. So they are completely unlike Stackless' tasklets in their implementation strategy.
So what, if the API is the same?
That, of course, doesn't cover every single function that the Stackless API currently exposes, but tasklets/channels are the raison d'etre.
That sounds strange. I would have expected that the reason for Stackless Python is to have no stack, not to have tasklets.
If you only wanted tasklets, you could implement them on top of threads, without any need for extensions.
You're kidding, right? Using the stack or not is an implementation detail, what matters is having more control over the flow of your programs in a reasonably efficient manner. It just so happens that the recursive evaluation in CPython uses the stack in such a way that makes these kind of constructs impossible, so it needed to be changed for Stackless. -bob
[Bob Ippolito]
Using the stack or not is an implementation detail, what matters is having more control over the flow of your programs in a reasonably efficient manner. It just so happens that the recursive evaluation in CPython uses the stack in such a way that makes these kind of constructs impossible, so it needed to be changed for Stackless.
Sorry Bob, you've reached my limit for repeating the same point over and over without making progress. You don't seem to be getting the real point of the responses you're getting, which are that Stackless is *not* a minor change to Python, and that it severely restricts the ways in which Python can interact with other systems, be they hardware platforms, software frameworks, or 3rd party extensions. Please step off your soapbox, listen, and understand how and why Stackless gets in the way of Python's range of applicability. --Guido van Rossum (home page: http://www.python.org/~guido/)
At 07:06 AM 6/2/04 -0700, Guido van Rossum wrote:
Stackless is *not* a minor change to Python, and that it severely restricts the ways in which Python can interact with other systems, be they hardware platforms, software frameworks, or 3rd party extensions.
It's definitely not a minor change, but I think that the severe restrictions comment may perhaps be inaccurate. My understanding at this point from Bob and Christian is that for tasklets (which are apparently what most people want from Stackless), the code is portable, pure C, and has no effect on code that doesn't use tasklets, except in performance. However, rather than continue discussion on this point, it would probably be better if the people who actually understand the Stackless implementation were to write a tasklets PEP addressing what *specifically* would be the *minimum* changes to the CPython interpreter to support co-operative multitasking. By specifically, I mean listing what core functions/objects will be changed, and how those changes will impact other code. Do any interfaces/APIs change? Is there anything that people writing new core code or extensions have to be aware of? And so on. A full statement of impact, in other words. At that point, a pronouncement could be made on the basis of those specifics, and if the answer is no, then the PEP would provide a convenient place to point people to in the future, thus avoiding further reiterations of this discussion. And, in the meanwhile, it gives you and Martin a way to tell the other folks to "PEP up or shut up". :)
At 07:06 AM 6/2/04 -0700, Guido van Rossum wrote:
Stackless is *not* a minor change to Python, and that it severely restricts the ways in which Python can interact with other systems, be they hardware platforms, software frameworks, or 3rd party extensions.
[Phillip]
It's definitely not a minor change, but I think that the severe restrictions comment may perhaps be inaccurate. My understanding at this point from Bob and Christian is that for tasklets (which are apparently what most people want from Stackless), the code is portable, pure C, and has no effect on code that doesn't use tasklets, except in performance.
However, rather than continue discussion on this point, it would probably be better if the people who actually understand the Stackless implementation were to write a tasklets PEP addressing what *specifically* would be the *minimum* changes to the CPython interpreter to support co-operative multitasking.
By specifically, I mean listing what core functions/objects will be changed, and how those changes will impact other code. Do any interfaces/APIs change? Is there anything that people writing new core code or extensions have to be aware of? And so on. A full statement of impact, in other words. At that point, a pronouncement could be made on the basis of those specifics, and if the answer is no, then the PEP would provide a convenient place to point people to in the future, thus avoiding further reiterations of this discussion. And, in the meanwhile, it gives you and Martin a way to tell the other folks to "PEP up or shut up". :)
Right. That PEP better explain how one writes C code that calls into Python without involving the C stack as well! --Guido van Rossum (home page: http://www.python.org/~guido/)
At 09:23 AM 6/2/04 -0700, Guido van Rossum wrote:
Right. That PEP better explain how one writes C code that calls into Python without involving the C stack as well!
I believe the outstanding proposal is that attempting a task switch from within Python code that was called by an extension will fail with an exception. But, if you mean, "how do you write a new piece of C that calls the interpreter, such that it can allow task switching to take place from the invoked interpreter?", then that's a different question, which should definitely be answered in the PEP. The problem I see is that *any* operation on a Python object potentially involves C calling Python, which seems to mean that the changes to the core interpreter would have to be quite widespread. But maybe the PEP authors have a more clever way to deal with that than anything I've thought of.
Right. That PEP better explain how one writes C code that calls into Python without involving the C stack as well!
I believe the outstanding proposal is that attempting a task switch from within Python code that was called by an extension will fail with an exception.
But how do you define "called by an exception"? There are lots built-in data types that call back into Python, e.g. comparisons and hash for dict keys, list.sort, sets, etc... Last time we looked there were so many exceptions of this type that enumerating them all was hopeless. That's a problem for a PEP.
But, if you mean, "how do you write a new piece of C that calls the interpreter, such that it can allow task switching to take place from the invoked interpreter?", then that's a different question, which should definitely be answered in the PEP.
That's what I meant, yes.
The problem I see is that *any* operation on a Python object potentially involves C calling Python, which seems to mean that the changes to the core interpreter would have to be quite widespread. But maybe the PEP authors have a more clever way to deal with that than anything I've thought of.
My way of dealing with this is to keep Stackless out of the core. Everybody happy. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote: ... [cooperative multitasking]
Right. That PEP better explain how one writes C code that calls into Python without involving the C stack as well!
Quick reply to give you an idea: A tail recursive call is rather trivial, so here the non-trivial side of the coin: Conventional style: PyObject * some_c_code(some_args) { PyObject *ret, ret2; /* do preparation things */ /* call into Python */ ret = PyEval_CallObject(some_args); /* do post-processing */ ret2 = some_other_processing(ret); return ret2; } Stackless style (simplified and sketched, only): PyObject * some_c_code(some_args) { PyObject *ret, ret2; /* do preparation things */ create_miniframe(some_c_code_post); /* put parameters in miniframe */ /* deferred call into Python */ ret = PyEval_CallObject_defer(some_args); return <special value> } PyObject * some_c_code_post(miniframe) { /* reload variables from miniframe */ /* drop miniframe */ /* do post-processing */ ret2 = some_other_processing(ret); return ret2; } So the idea is to avoid recursions by creating frame-like tiny structures, which are then executed by a toplevel loop which does nothing than call the next frame. The <special value> (not specified here) passes through the currently involved C code as a dummy which unwinds the stack until the toplevel loop is reached. There this value is captured and unpacked, and execution continues. Well, this was rough and incomplete. A PEP would be nice and also quite much work. The protocol sketched above is a lot of work to do if you want to do it throughout all of Python, and I didn't do it in most cases, but did the simple tail-recursive things (which don't need that extra tinyframe) everywhere where possible. If adopting a similar technique for parts of Python is not feasible, then the PEP would be a waste of time. There is no other principal way to do it in plain C, IMHO. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Bob Ippolito wrote:
However, JCSP uses java.lang.Threads to implement concurrency. So they are completely unlike Stackless' tasklets in their implementation strategy.
So what, if the API is the same?
If you don't have a problem with that, I don't either.
Using the stack or not is an implementation detail, what matters is having more control over the flow of your programs in a reasonably efficient manner. It just so happens that the recursive evaluation in CPython uses the stack in such a way that makes these kind of constructs impossible, so it needed to be changed for Stackless.
That is simply not true. It is possible and straight-forward to implement Stackless' tasklets and channels on top of the standard Python, with no need to change the interpreter proper. Regards, Martin
Martin v. Löwis wrote:
Bob Ippolito wrote:
However, JCSP uses java.lang.Threads to implement concurrency. So they are completely unlike Stackless' tasklets in their implementation strategy.
So what, if the API is the same?
If you don't have a problem with that, I don't either.
Using the stack or not is an implementation detail, what matters is having more control over the flow of your programs in a reasonably efficient manner. It just so happens that the recursive evaluation in CPython uses the stack in such a way that makes these kind of constructs impossible, so it needed to be changed for Stackless.
That is simply not true. It is possible and straight-forward to implement Stackless' tasklets and channels on top of the standard Python, with no need to change the interpreter proper.
This is true, we have written demo implementations of tasklets with pure Python and threads on the last sprint. The difference is that nobody would use these for production code, when tasklets are used frequently, because real threads are at least an order of magnitude slower than tasklet switching. This high speed of context switch is what makes certain kinds of algorithms even useful. You will do things with tasklets which you never would seriously do with threads, because a context switch is as cheap as a builtin function call. This makes the big difference. The compatibility issue (same API etc.) just guarantees that such code would run on regular Python and Jython as well. It would be very slow, no doubt, but it would just work. Still, tasklets make sense in the first place if they are fast. ciao - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Martin v. Löwis wrote:
Christian Tismer wrote:
This is true, we have written demo implementations of tasklets with pure Python and threads on the last sprint.
Where can I find that implementation?
If you checked out current cvs, it is in the Stackless folder Stackless/demo/stephan/stacklessness I'm not sure how complete or correct it is, I think he was planning to do some more on it. ciao - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Christian Tismer <tismer@stackless.com> writes:
This is true, we have written demo implementations of tasklets with pure Python and threads on the last sprint.
The difference is that nobody would use these for production code,
I don't know if this is true. In a production environment where there is an existing Python installation, but no Stackless installation, and where performance is not a killer issue, tasklets *may* be useful for their organisational benefits. I say *may* because I don't know much about tasklets. A pure Python implementation may help my understanding. Is the demo downloadable from anywhere? Paul. -- This signature intentionally left blank
Paul Moore wrote:
Christian Tismer <tismer@stackless.com> writes:
This is true, we have written demo implementations of tasklets with pure Python and threads on the last sprint.
The difference is that nobody would use these for production code,
I don't know if this is true. In a production environment where there is an existing Python installation, but no Stackless installation, and where performance is not a killer issue, tasklets *may* be useful for their organisational benefits.
I say *may* because I don't know much about tasklets. A pure Python implementation may help my understanding.
Is the demo downloadable from anywhere?
Yes, in current cvs under Stackless/demo/stephan cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
At 01:23 PM 6/3/04 +0100, Paul Moore wrote:
Christian Tismer <tismer@stackless.com> writes:
This is true, we have written demo implementations of tasklets with pure Python and threads on the last sprint.
The difference is that nobody would use these for production code,
I don't know if this is true. In a production environment where there is an existing Python installation, but no Stackless installation, and where performance is not a killer issue, tasklets *may* be useful for their organisational benefits.
I say *may* because I don't know much about tasklets. A pure Python implementation may help my understanding.
Is the demo downloadable from anywhere?
Also keep in mind that tasklet-like operations can be done using state-machine callbacks (e.g. Twisted), or using nested generators (e.g. 'peak.events'). Callbacks have the problem of "twisting" what would otherwise be linear code, and if you're using generators, you have to use "generators all the way down" to code that does task switching. IOW, you can't have a non-generator call a generator and have that generator be able to task-switch. The net result of either approach is that you can't use any existing code (outside of those frameworks, of course) as part of your tasks.
On Jun 2, 2004, at 1:20 PM, Martin v. Löwis wrote:
Bob Ippolito wrote:
However, JCSP uses java.lang.Threads to implement concurrency. So they are completely unlike Stackless' tasklets in their implementation strategy. So what, if the API is the same?
If you don't have a problem with that, I don't either.
Using the stack or not is an implementation detail, what matters is having more control over the flow of your programs in a reasonably efficient manner. It just so happens that the recursive evaluation in CPython uses the stack in such a way that makes these kind of constructs impossible, so it needed to be changed for Stackless.
That is simply not true. It is possible and straight-forward to implement Stackless' tasklets and channels on top of the standard Python, with no need to change the interpreter proper.
In my quick attempt to do CSP on top of CPython threads I learned that: - It's not as easy as it sounds to get deterministic behavior out of threads, and debugging threaded python apps is no fun. - When it does work, the performance sucks compared to Stackless. Perhaps someone with more Python threading experience can prove me wrong? -bob
Bob Ippolito wrote: ...
In my quick attempt to do CSP on top of CPython threads I learned that: - It's not as easy as it sounds to get deterministic behavior out of threads, and debugging threaded python apps is no fun. - When it does work, the performance sucks compared to Stackless.
Perhaps someone with more Python threading experience can prove me wrong?
I have spent almost three weeks to implement thread-aware channels for Stackless in C, and I can tell you it was hard. They are almost exactly 10 times slower than Stackless channels, and it was not trivial, but once for all time. :-) I think you are right! -- chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Bob Ippolito wrote:
However, JCSP uses java.lang.Threads to implement concurrency. So they are completely unlike Stackless' tasklets in their implementation strategy. So what, if the API is the same?
In my quick attempt to do CSP on top of CPython threads I learned that: - It's not as easy as it sounds to get deterministic behavior out of threads, and debugging threaded python apps is no fun. - When it does work, the performance sucks compared to Stackless.
The same is probably true for JCSP. I don't want a feature that can only be implemented efficiently by using assembler or some other platform-specific hack (and most uses of longjmp() fall in that category, despite its status as an ISO standard library function!). Especially if the proponents keep using efficiency as the most important property of the feature. --Guido van Rossum (home page: http://www.python.org/~guido/)
At 10:50 AM 6/2/04 -0700, Guido van Rossum wrote:
Bob Ippolito wrote:
However, JCSP uses java.lang.Threads to implement concurrency. So they are completely unlike Stackless' tasklets in their implementation strategy. So what, if the API is the same?
In my quick attempt to do CSP on top of CPython threads I learned that: - It's not as easy as it sounds to get deterministic behavior out of threads, and debugging threaded python apps is no fun. - When it does work, the performance sucks compared to Stackless.
The same is probably true for JCSP. I don't want a feature that can only be implemented efficiently by using assembler or some other platform-specific hack (and most uses of longjmp() fall in that category, despite its status as an ISO standard library function!). Especially if the proponents keep using efficiency as the most important property of the feature.
I personally think that deterministic task-switching is more relevant than raw speed. Being able to create hundreds or even thousands of lightweight tasks is also very useful. Right now I can do both of these using generators in CPython 2.2 or better, so the only things (IMO) that Stackless is offering to add are: 1. dropping the need to write tasks solely in the form of "generators all the way down" 2. a simpler syntax: yields are expressed as calls, rather than as a 'yield' statement followed by various magical idioms. I currently have code that doesn't hesitate to spawn generator-based tasklets for simple condition monitoring tasks, and I have no idea in practice how many of these tasklets might be running in my code, but I'm pretty sure it's more than I'd want to have threads running for. However, if threads only had to be used when yielding code was called from non-yielding code, or vice versa, I'd probably be okay with it. Such boundaries aren't all that common in my experience to date using generator-based tasklets.
Guido> Especially if the proponents keep using efficiency as the most Guido> important property of the feature. True, but there's efficiency and there's efficiency. I would agree if we were just speeding up a slow algorithm. Following this thread with just one eye though it seems like the people advocating tasklets are suggesting they are efficient enough to allow algorithms which simply aren't feasible using threads (far too expensive) or are intractable for other reasons (like generators all the way down). Skip
Bob Ippolito wrote:
In my quick attempt to do CSP on top of CPython threads I learned that: - It's not as easy as it sounds to get deterministic behavior out of threads, and debugging threaded python apps is no fun.
You could have looked at JCSP for guidance.
- When it does work, the performance sucks compared to Stackless.
So the performance of JCSP sucks as well? Then tasklets aren't implementable in an acceptable way in Java? Regards, Martin
"Martin v. Löwis" <martin@v.loewis.de> writes:
Bob Ippolito wrote:
Well the tasklet/channel model of Stackless is a single threaded version of CSP, which has at least one Java implementation: http://wotug.kent.ac.uk/parallel/languages/java/jcsp/
However, JCSP uses java.lang.Threads to implement concurrency. So they are completely unlike Stackless' tasklets in their implementation strategy.
That, of course, doesn't cover every single function that the Stackless API currently exposes, but tasklets/channels are the raison d'etre.
That sounds strange. I would have expected that the reason for Stackless Python is to have no stack, not to have tasklets.
*BZZZZZZZZZZT*! :-) Cheers, mwh -- First of all, email me your AOL password as a security measure. You may find that won't be able to connect to the 'net for a while. This is normal. The next thing to do is turn your computer upside down and shake it to reboot it. -- Darren Tucker, asr
"Phillip J. Eby" <pje@telecommunity.com> wrote in message news:5.1.1.6.0.20040601001051.01f00ca0@mail.telecommunity.com...
I was just suggesting that if people had specific features of Stackless that they wanted to see in CPython, that it would be best for them to write a PEP specifically addressing the desired feature(s), rather than asking "why isn't CPython stackless?"
I agree that a PEP would be more useful than the repeated question. One criterion for adding a new module is a constituency of users. One thing that would help the Stackless in Python cause is more evidence that it is useful in practice as well as in theory and expectation. Even today, the only thing I can find on the site is " Stackless has impressive applications, like: a.. Eve, an online massive multiplayer game by CCP Games b.. ? by Iron Port " Eve was listed two years ago and ? does not say much. I went to the Eve site, downloaded several videos, and from what was presented, was impressed negatively, so it is not obviously to me an advertisement for Stackless. So other examples, less costly to try out, might be more persuasive. Terry J. Reedy
Terry Reedy wrote:
One criterion for adding a new module is a constituency of users. One thing that would help the Stackless in Python cause is more evidence that it is useful in practice as well as in theory and expectation. Even today, the only thing I can find on the site is " Stackless has impressive applications, like: a.. Eve, an online massive multiplayer game by CCP Games b.. ? by Iron Port " Eve was listed two years ago and ? does not say much. I went to the Eve site, downloaded several videos, and from what was presented, was impressed negatively, so it is not obviously to me an advertisement for Stackless. So other examples, less costly to try out, might be more persuasive.
Why did you do that? Would you download and try Zope, in order to judge about Python? Would you say C is a bad language, because you (perhaps) don't like Perl? Last time I disliked a movie, I really thought celluloid is a bad idea. These examples are shown to prove that Stackless has industrial strength, nothing else. I'm no real fan of each application, and I don't expect enthusiasm except from my kids, perhaps. But I don't need kids as Stackless users. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Terry Reedy wrote:
One criterion for adding a new module is a constituency of users. One thing that would help the Stackless in Python cause is more evidence
"Christian Tismer" <tismer@stackless.com> wrote in message news:40BD00C0.6010606@stackless.com... that
it is useful in practice as well as in theory and expectation. Even today, the only thing I can find on the site is " Stackless has impressive applications, like: a.. Eve, an online massive multiplayer game by CCP Games b.. ? by Iron Port " Eve was listed two years ago and ? does not say much. I went to the Eve site, downloaded several videos, and from what was presented, was impressed negatively, so it is not obviously to me an advertisement for Stackless. So other examples, less costly to try out, might be more persuasive.
Why did you do that?
Do what? Accept the invitation to follow the posted link to the new Stackless site to learn more about Stackless today? Click on the Wiki link and read? Click on the included link to the one and only listed example application? Among other things, I looked to see if there were currently a free trial period for Eve, as there sometines is for online games. If I were to try it, and it ran smoothly for some time without lags and crashes (which I have experience of), then that would suggest that the tools (including Stackless) and use thereof were adequate to the task. If not, that something (though not necessarily Stackless) would be shown lacking.
Would you download and try Zope, in order to judge about Python?
Given that you seem to be trying to ridicule sensible behavior, it is ironic that you should pick that particular example. In fact, several years ago, Zope (perhaps by its earlier name) was touted as Python's 'killer application'. At least a few people said something like 'To see Python in action, go visit the Python/Zope powered Zope site.' Which I (and others) did. And it appears that others went further and downloaded Zope to try on their own systems. A year or two ago, the Zope site listed Plone as an example of Zope in action. I followed the link to its site and signed up for one of the free demo accounts.
Would you say C is a bad language, because you (perhaps) don't like Perl?
The issue is performance, not liking. In any case, the proof of C was Unix, 30 years ago. I switched from Fortran to C in 80/81, long before Perl existed. However, if Perl were the only available example application for C, then yes, trying it out would one sensible way to evaluate C.
Last time I disliked a movie, I really thought celluloid is a bad idea.
If there were a celluloid movie development site that listed exacly one celluloid movie and either its price to watch were too much for me and the free stills unimpressive, or its *technical* performance crummy, then I would consider its technical virtue unproven. But that is not the case.
These examples
To me, there is only one example. I don't count '?'. But perhaps you know something you can't post.
are shown to prove that Stackless has industrial strength, nothing else.
Your biased-by-parenthood assertion is not proof to most people. The mere existence of Eve is not proof. A positive evaluation of its technical performance would be evidence. To paraphrase my original second sentence: if you want me to see that Stackless has industrial strength, show me more evidence. Terry J. Reedy
Christian Tismer wrote:
I'm believe that map and filter are non-recursive
They have been. They can be, again, but why.
I'm not saying they should be non-recursive; I was just claiming that they weren't, and that they would trigger "hard" switching.
I don't know the C++ exception mechanism. It it walks the whole stack once before taking an action, then this is a problem. If it does proper unwinding of C frames, I could probably play the game and restore things just in time.
I'm unsure also; this is part of the Visual C++ runtime (or perhaps even of the system's "structured EH"). In my coroutine library, the application would crash even if there was an exception handler that could have been found without walking the entire stack.
Anyway, I don't really get the point. 95% of Stackless is doing soft-switched stackless calls.
My point is that Stackless 3.0 is probably not acceptable for integration into C Python because of the processor-specific assembler fragments.
The discussion, as often before, tries to find out why it is *not* possible to include Stackless, instead of finding out the useful parts which could be supported with relatively small impact. This is why Stackless exists, and will continue to exist.
That's because people always ask the question "Can it be included?" They never ask "If I were to contribute this and that part, would it be accepted?". Actually, this thread *started* with the question "Why isn't Python stackless?" to which the correct answer is "Because nobody has contributed patches." Only then people get excited and think that Stackless 3.0 would be part of Python 2.5. Regards, Martin
Martin v. Löwis wrote:
My point is that Stackless 3.0 is probably not acceptable for integration into C Python because of the processor-specific assembler fragments.
Ok, that can be dropped and put into an extension.
The discussion, as often before, tries to find out why it is *not* possible to include Stackless, instead of finding out the useful parts which could be supported with relatively small impact. This is why Stackless exists, and will continue to exist.
That's because people always ask the question "Can it be included?" They never ask "If I were to contribute this and that part, would it be accepted?".
Actually, this thread *started* with the question "Why isn't Python stackless?" to which the correct answer is "Because nobody has contributed patches." Only then people get excited and think that Stackless 3.0 would be part of Python 2.5.
I didn't understand the question this way, although it was probably meant so. For me it was more like "why doesn't it aim for Stackless' features". Although I'm trying to make Stackless as clean and compatible as possible, I'm not trying to have my current version ready for inclusion. Development is more influenced by user requests. I think, the heart of the thing is to decide whether Python wants to go the way to make as much as possible calls non-recursive. This is a lot of work, and I did one possible implementation, which minimized changes to the rest of the system. If we try to modify Python to be stackless, we would probably got more rigorous ways and change more of the design. I can't do that alone. I doubt that anybody would be able to create such a re-design alone, while keeping up with Python development, that does quite much in the opposite direction. This must become a goal of the core developer group. all the best - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Hello, On Mon, May 31, 2004 at 10:56:55PM +0200, "Martin v. Löwis" wrote:
Alternatively, Armin believes that Stackless "hard switching" can actually be implemented in mostly-platform-neutral C (either by setjmp/longjmp or alloca hacks, I forget which)
It is setjmp/longjmp. I have implemented a coroutine library on top of setjmp/longjmp myself, so I am well aware of the implications.
Actually, it was alloca(), and would be a real C hack that would probably fail on some platforms, depending on the phase of the moon. The idea was that alloca() can be abused to change the stack pointer almost freely. Not-easy-to-shake-off-a-C-hacking-reputation'ly yours, Armin
Martin:
[In addition, the approach taken by Stackless 3.0 is probably questionable: for example, it may be that C++ exception handling stops working in Stackless.]
Indeed. The last time this was discussed, I seem to remember Stackless had trouble with C code that passed pointers to local variables as parameters. Since this is a perfectly legitimate thing to do in C, if Stackless can't handle it, then it's fundamentally flawed. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
Does this mean that the implementation of Stackless 3.0 could be acceptable to integrate into CPython 2.5? It would be absolutely wonderful to have those capabilities available in the standard distribution!
Let's not get too excited about this possibility. Before something is accepted into the language its semantics must be specified exactly and clearly. Last time this was proposed, the Stackless semantics were extremely murky, with lots of implementation-specific exceptions. Another requirement is that other Python implementations, like Jython and IronPython (Jim Hugunin's C# port) can also support these semantics. Until these conditions are met, Stackless is stuck in the fringe. --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (13)
-
"Martin v. Löwis" -
Armin Rigo -
Bob Ippolito -
Christian Tismer -
Eyal Lotem -
Greg Ewing -
Guido van Rossum -
Michael Hudson -
Paul Moore -
Phillip J. Eby -
Shane Holloway (IEEE) -
Skip Montanaro -
Terry Reedy