Does this mean that the implementation of Stackless 3.0 could be acceptable to integrate into CPython 2.5? It would be absolutely wonderful to have those capabilities available in the standard distribution! -Shane Holloway Martin v. Löwis wrote:

Eyal Lotem wrote:

...

Why is Python still not stackless?

But it is: http://www.stackless.com/

If you are asking: Why is python.org Python still not stackless? then I think the answer has varied over time.

The most recent answer is that no volunteer has stepped forward to make Python stackless. That, in turn, likely hasn't happened because it is hard to do, and because nobody has seen any reason to do so as you could always use Stackless Python.

Regards, Martin

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/shane.holloway%40ieee.org

On May 31, 2004, at 2:50 PM, Martin v. Löwis wrote:

Shane Holloway (IEEE) wrote:

...

Does this mean that the implementation of Stackless 3.0 could be acceptable to integrate into CPython 2.5? It would be absolutely wonderful to have those capabilities available in the standard distribution!

Probably not. Stackless 3.0 works only on selected hardware; a number of platforms currently supported by Python would not be supported anymore. An implementation of stackless python should work on all platforms (atleast on those currently supported by Python).

Although I don't particularly agree with that, because Stackless functionality could be turned on and off with a platform dependent configure flag (like threads or unicode)... Is there a canonical list of CPU architectures and platforms that Python officially supports? -bob

On May 31, 2004, at 3:33 PM, Martin v. Löwis wrote:

Bob Ippolito wrote:

...

Although I don't particularly agree with that, because Stackless functionality could be turned on and off with a platform dependent configure flag (like threads or unicode)... Is there a canonical list of CPU architectures and platforms that Python officially supports?

No - although there is a list of platforms, that Python *doesn't* support (PEP 11). Python is written in C, so it should work on any system that supports ISO C, although many functions won't be available if the system doesn't support POSIX.

As for turning of things: yes, unicode can be turned off, but not because the platform doesn't support it - rather because the administrator doesn't like unicode. Threads are turned off on operating systems that don't provide threads, and there is nothing that Python could do about it.

Stackless' support for platforms is different: it includes assembler code, so you have to write bits of assembler code for each architecure (and perhaps for each system on a single architecture, or even each assembler on each system).

I don't see how including a little bit of per-architecture-calling-convention assembler code is much different than depending on pthreads (or some other platform dependent thing) being present. There is also potentially a platform neutral way of doing it (see below).

[In addition, the approach taken by Stackless 3.0 is probably questionable: for example, it may be that C++ exception handling stops working in Stackless.]

I'm not quite sure you are up to date on how Stackless 3.0 works. The assembly code is only there to facilitate "hard switching", which is only required to do tasklet switching in the face of some C extension that doesn't know about Stackless. The vast majority of tasklet switches use the "soft switching" mechanism, which just means that the code they use is capable of going into non-recursive mode. All pure Python code and almost all code built-in to Python goes through "soft switching", which is implemented in pure C with no stack tricks or anything. Stackless would still be a very useful addition to Python even when "hard switching" is not available. Alternatively, Armin believes that Stackless "hard switching" can actually be implemented in mostly-platform-neutral C (either by setjmp/longjmp or alloca hacks, I forget which) and has a prototype implementation in the Stackless CVS repository (though it is not integrated with Stackless itself). It might also be possible to coerce something like libffi into doing what we need it to do. I'm not sure how well "hard switching" would play with an operating system that is very protective of its stack, like OpenBSD or future versions of Windows, though. I'm also not sure how it could "break C++ exception handling" in scenarios where it wouldn't be broken anyway. I'd like to see a test of that that passes under regular Python but fails in Stackless for this. -bob

On May 31, 2004, at 4:56 PM, Martin v. Löwis wrote:

Bob Ippolito wrote:

...

I'm not quite sure you are up to date on how Stackless 3.0 works. The assembly code is only there to facilitate "hard switching", which is only required to do tasklet switching in the face of some C extension that doesn't know about Stackless.

Yes, that was my understanding. "Some C extensions" includes Tcl and COM, right? Unless I'm mistaken, it also includes map, filter, and .encode/.decode of unicode objects (or any other C module that calls back into Python, except for tasklet mechanism of stackless itself).

You're definitely correct about Tcl and COM.. COM is however a total non-issue because Win32/x86 is very well supported by Christian ;) I'm believe that map and filter are non-recursive, I am not sure about unicode encode/decode. I don't think it's very common to do a tasklet switch in the middle of a unicode encode or decode, though :)

In these cases, copying out the stack will break the exception chain that the Visual C++ compiler creates through the FS segment, so that an attempt to throw an exception will cause a program crash, right?

...

I'm also not sure how it could "break C++ exception handling" in scenarios where it wouldn't be broken anyway. I'd like to see a test of that that passes under regular Python but fails in Stackless for this.

The VC++ exception mechanism puts a linked chain of exception frames on the stack, where, at the beginning of each function, the stack pointer is stored at FS:18 (or some such), and the old value of FS:18 is stored on the stack. Then, when an exception occurs, this linked list is used to traverse the stack. If multiple stacks are used in a single thread, then this linked list gets messed up.

Without trying, I think the following call chain might cause a crash if Stackless is compiled with exception handling on:

Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet Py-func3() -> C-func2() -> throw exception()

I don't see how it could cause problems unless this is valid: Py_Func1 -> CPlusPlus_ExceptionCatcher -> Py_Func2 -> CPlusPlus_ExceptionThrower -> throws exception() If that *is* valid, how the heck does Py_Func2 get cleaned up? I have to admit that I'm not very familiar with C++'s implementation of exceptions. -bob

On May 31, 2004, at 5:25 PM, Martin v. Löwis wrote:

Bob Ippolito wrote:

...

I'm believe that map and filter are non-recursive

Can you point me to the code that demonstrates this? Looking at Stackless' src/Python/bltinmodule.c:builtin_map, I see

for (i = 0; ; ++i) { ... value = PyEval_CallObject(func, alist); ... PyList_SetItem(result, i, value) ... }

which looks recursive to me.

Ok, then I am mistaken. I thought I had seen a commit message that said it was made non-recursive. Either way, it's possible to write a non-recursive version of map or anything else, it's just a bit of work.

...

I am not sure about unicode encode/decode. I don't think it's very common to do a tasklet switch in the middle of a unicode encode or decode, though :)

I thought Stackless was not only about tasklet switching, but also about running without stack... Wouldn't it perform the "hard" switching if the stack is running too deep?

I don't think so, because the C stack should never run too deep, since most code that is going to recurse (from the python-view) will do it non-recursively (from the C-view).

...

...

Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet Py-func3() -> C-func2() -> throw exception() I don't see how it could cause problems unless this is valid: Py_Func1 -> CPlusPlus_ExceptionCatcher -> Py_Func2 -> CPlusPlus_ExceptionThrower -> throws exception()

It crashes in the process of throwing the exception - not because it finds no exception handler. It crashes because the internal data structures have been corrupted.

I should have better made this example:

Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet Py-func3() -> Exception-Catcher -> C-func2() -> throw exception()

...

If that *is* valid, how the heck does Py_Func2 get cleaned up? I have to admit that I'm not very familiar with C++'s implementation of exceptions.

Throwing exceptions through Python functions is not supported, indeed. However, I was trying to point out that the entire exception handling mechanism gets corrupted, even if there are valid handlers on the stack.

I don't think that the entire exception handling mechanism gets corrupted. Each tasklet should always have a consistent view of the C stack. Anyway, I don't think we need to discuss it any further unless someone writes a failing test... and if that is indeed the case, then the discussion should be taken to the stackless list. -bob

Martin v. Löwis wrote:

Bob Ippolito wrote:

...

I'm believe that map and filter are non-recursive

They have been. They can be, again, but why.

Can you point me to the code that demonstrates this? Looking at Stackless' src/Python/bltinmodule.c:builtin_map, I see

for (i = 0; ; ++i) { ... value = PyEval_CallObject(func, alist); ... PyList_SetItem(result, i, value) ... }

which looks recursive to me.

Yes. trying to do a minimalistic approach, I dropped all non-recursive implementations of non-trivial things (meaning non-tail-recursive things) which were not absolutely necessary. And since we have list comprehensions, map is no longer worth being supported so much.

...

I am not sure about unicode encode/decode. I don't think it's very common to do a tasklet switch in the middle of a unicode encode or decode, though :)

I thought Stackless was not only about tasklet switching, but also about running without stack... Wouldn't it perform the "hard" switching if the stack is running too deep?

Yes, it would, if you allow it to do so. This would happen after a very deep recursion, which you wouldn't be able to do with normal Python at all. ...

It crashes in the process of throwing the exception - not because it finds no exception handler. It crashes because the internal data structures have been corrupted.

I don't know the C++ exception mechanism. It it walks the whole stack once before taking an action, then this is a problem. If it does proper unwinding of C frames, I could probably play the game and restore things just in time. Anyway, I don't really get the point. 95% of Stackless is doing soft-switched stackless calls. Behavior is completely controllable. We can easily avoid any special C stack operation, by setting a flag that disallows it (easy to implement) or by excluding the hard switching stuff, completely (not an option now, but easy, too). The discussion, as often before, tries to find out why it is *not* possible to include Stackless, instead of finding out the useful parts which could be supported with relatively small impact. This is why Stackless exists, and will continue to exist. For me, this thread is closed -- thanks - chris -- Christian Tismer :^) mailto:tismer@stackless.com Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

On May 31, 2004, at 8:37 PM, Phillip J. Eby wrote:

At 02:09 AM 6/1/04 +0200, Christian Tismer wrote:

...

Anyway, I don't really get the point. 95% of Stackless is doing soft-switched stackless calls. Behavior is completely controllable. We can easily avoid any special C stack operation, by setting a flag that disallows it (easy to implement) or by excluding the hard switching stuff, completely (not an option now, but easy, too).

If soft-switching is portable (i.e. pure C, no assembly), and is exposed as a library module (so that Jython et al can avoid supporting it), then perhaps a PEP for adding that functionality to mainstream Python would be meaningful.

Soft switching needs to be implemented in a few key places of the interpreter itself or else Stackless would surely have been maintained as an extension module. It is already pure C, no assembly or platform specific code. Supporting the interface from Jython or IronPython should be possible, though at worst case each tasklet might actually be a new thread so it might not be terribly efficient... though it would work.

If that gets in, then the other 5% can always sneak in later via feature creep. ;) Or, more importantly (if I understand correctly), it could be separately distributed as an add-on for platforms that can support it.

Hard switching could be an add-on for platforms that support it.. -bob

On May 31, 2004, at 10:51 PM, Phillip J. Eby wrote:

At 08:49 PM 5/31/04 -0400, Bob Ippolito wrote:

...

On May 31, 2004, at 8:37 PM, Phillip J. Eby wrote:

...

At 02:09 AM 6/1/04 +0200, Christian Tismer wrote:

...

Anyway, I don't really get the point. 95% of Stackless is doing soft-switched stackless calls. Behavior is completely controllable. We can easily avoid any special C stack operation, by setting a flag that disallows it (easy to implement) or by excluding the hard switching stuff, completely (not an option now, but easy, too).

If soft-switching is portable (i.e. pure C, no assembly), and is exposed as a library module (so that Jython et al can avoid supporting it), then perhaps a PEP for adding that functionality to mainstream Python would be meaningful.

Soft switching needs to be implemented in a few key places of the interpreter itself or else Stackless would surely have been maintained as an extension module.

I'm aware of this, which is why I said "exposed" as a library module, not "implemented" as one. :)

Oh, it *is* already exposed as a module (named stackless)... but it's *implemented* as a built-in module, like sys or os. -bob

At 08:17 PM 5/31/04 -0700, Guido van Rossum wrote:

...

If soft-switching is portable (i.e. pure C, no assembly), and is exposed as a library module (so that Jython et al can avoid supporting it), then perhaps a PEP for adding that functionality to mainstream Python would be meaningful.

Anything that can be supported by a pure extension module is fair game IMO. But it was my understanding that Stackless requires changes to the VM.

I did mean *exposed* as a library module, not implemented as one. I am aware of the VM change.

And I definitely don't want people to get into a habit of writing code that relies on deep recursion that is only supported by Stackless, only to find that it doesn't work in Jython etc.

From the docs of sys.setrecursionlimit(): """The highest possible limit is platform-dependent. A user may need to set the limit higher when she has a program that requires deep recursion and a platform that supports a higher limit. This should be done with care, because a too-high limit can lead to a crash.""" Seems to me that the platform-specific nature of this is already established.

...

If that gets in, then the other 5% can always sneak in later via feature creep. ;) Or, more importantly (if I understand correctly), it could be separately distributed as an add-on for platforms that can support it.

Hey, that's what Stackless already does. Why is that approach suddenly not good enough any more?

I didn't say that. I was just suggesting that if people had specific features of Stackless that they wanted to see in CPython, that it would be best for them to write a PEP specifically addressing the desired feature(s), rather than asking "why isn't CPython stackless?"

Guido van Rossum wrote:

...

...

Anything that can be supported by a pure extension module is fair game IMO. But it was my understanding that Stackless requires changes to the VM.

As Armin has shown, it is possible to do the hard switching approach as a pure extension module. But that's the opposite of what I want. I want a supportive core without the need to fiddle with C stacks. That needs VM changes. Cheating and supplying a different VM as an extension module is probably no option? :-)

...

...

Hey, that's what Stackless already does. Why is that approach suddenly not good enough any more?

Philip:

...

I didn't say that. I was just suggesting that if people had specific features of Stackless that they wanted to see in CPython, that it would be best for them to write a PEP specifically addressing the desired feature(s), rather than asking "why isn't CPython stackless?"

Guido:

I think the bar should be raised pretty high for Stackless, given the various objections that have been raised against it (x86-only code, no Jython support, arbitrary exceptions, murky semantics, to name a few).

Huh? There is support for 8 or more platforms, and I'm not addressing the assembly parts. It is most probably possible to implement soft-switchign in Jython. No idea what you mean by arbitrary exceptions (I never had that) or what's wrong with semantics. On the latter: Well, this is a pure interface issue. I don't say that everybody must love tasklets, there are other approaches and interfaces possible. But they all rely on the non-recursive interpreter core, and that's the point, IMHO.

Together, the answer as to "why" should be pretty clean by now.

If you understand the question like "why is current Stackless not included", that's true. I understood it more like the more advanced question "why does Python still block itself from lightweight threading by keping state on the C stack". ciao - chris -- Christian Tismer :^) mailto:tismer@stackless.com Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

At 06:34 AM 6/1/04 -0700, Guido van Rossum wrote:

...

It is most probably possible to implement soft-switchign in Jython.

That's a rather vague statement; I'd like to hear from Samuele about that.

Me too. The only thing I've heard of in Java that does lightweight co-operative task switching, is actually implemented using a Java VM written in Java. Sort of like PyPy, only I guess you'd call it JavaJava. :) I don't remember the name of it right off, just that it was a commercial product that claimed they had a patent pending.

...

No idea what you mean by arbitrary exceptions (I never had that)

I meant exceptions to the general rules. Like, you can't depend on tasklet switching when you use one of these extensions, or one of these built-ins, or ...

Perhaps it would be better if it were defined in terms of where it *does* work, rather than where it doesn't. For example, one can write co-operative multitasking code in CPython now using generators... but it only works if you have "generators all the way down". That is, any code in a task that calls a generator must itself be a generator. That's a pretty comprehensible rule, albeit draconian to follow. Adding soft-switching would change the rule to be, "you can switch as long as you haven't been called from C code." That is, you haven't been called from *any* built-in or extension module. Speaking as someone who's written some significant event-driven co-operative multitasking code using generators, I would say that going from "generators all the way down" to "Python all the way down" would be a huge improvement in flexibility.

...

On the latter: Well, this is a pure interface issue. I don't say that everybody must love tasklets, there are other approaches and interfaces possible. But they all rely on the non-recursive interpreter core, and that's the point, IMHO.

Is it really just tasklets that the users want?

I am not currently a Stackless user, but if I were to try it, tasklets would be my primary interest. I don't actually think that the higher recursion limit is a "feature", for two reasons: 1) Python loops run a lot faster than function calls, and 2) every time I've ever gotten an error from excessive recursion, it was because I'd accidentally created an infinite recursion. So, to me, lifting the default recursion limit in standard Python would be a really *bad* idea. The other feature besides tasklets that I find potentially interesting is thread pickling. I'm intrigued by the possibility of using them to create "long-running tasks". That is, workflow using Python scripts to define the loops, branches, etc. of the workflow. But to make a really practical workflow system based on this, one would have to have a way of dealing with changes to the code that the running workflows were based on, so I don't know if that whole idea is really worth pursuing. I would probably be better off basing such an animal on PyPy, because performance for the code in a long-running process workflow is a non-issue. Anyway, I don't know about "the users" in general, but for me at least tasklets are where it's at. :) And I only care about explicit yielding, not thread-like concurrency. (IOW, I don't want Stackless doing the scheduling, I'd want to be able to write my own scheduler, in pure Python.)

...

I understood it more like the more advanced question "why does Python still block itself from lightweight threading by keping state on the C stack".

IMO mostly because Python wants to be friendly to extensions written in C or C++, including those that frequently call back into Python.

That would only disable task-switching within the called Python code, not from the code that called the extension. And that, only if the hard-switching module wasn't available.

At 12:31 AM 6/2/04 +0200, Christian Tismer wrote:

Phillip J. Eby wrote:

...

At 06:34 AM 6/1/04 -0700, Guido van Rossum wrote:

...

...

It is most probably possible to implement soft-switchign in Jython.

That's a rather vague statement; I'd like to hear from Samuele about that.

I believe it is doable, and I'll ask him. If java can use something like function pointers, it is possible.

There are various things that correspond to function pointers. Sort of. You'll need to be a bit more specific than that.

If you have to have compiled calls all the time, well, I did not invent Java, and I don't want to know who was it. (Actually I do :)

Keep in mind that Jython *compiles* Python code to Java bytecodes. It is *not* a Python interpreter written in Java. I suspect this will severely limit the applicable approaches. Googling "Java co-operative multitasking" and "Java asynchronous methods" turns up plenty of hits on people who are trying to figure out how to do them, but not much about any successes. :) Indeed, the only thing I've ever run into that does it, is that commercial system I mentioned, which does it by writing a Java VM in Java, so they can control its execution. I don't think that idea will go over very well with the Jython team. :)

Bob Ippolito wrote:

Well the tasklet/channel model of Stackless is a single threaded version of CSP, which has at least one Java implementation: http://wotug.kent.ac.uk/parallel/languages/java/jcsp/

However, JCSP uses java.lang.Threads to implement concurrency. So they are completely unlike Stackless' tasklets in their implementation strategy.

That, of course, doesn't cover every single function that the Stackless API currently exposes, but tasklets/channels are the raison d'etre.

That sounds strange. I would have expected that the reason for Stackless Python is to have no stack, not to have tasklets. If you only wanted tasklets, you could implement them on top of threads, without any need for extensions. Regards, Martin

[Bob Ippolito]

Using the stack or not is an implementation detail, what matters is having more control over the flow of your programs in a reasonably efficient manner. It just so happens that the recursive evaluation in CPython uses the stack in such a way that makes these kind of constructs impossible, so it needed to be changed for Stackless.

Sorry Bob, you've reached my limit for repeating the same point over and over without making progress. You don't seem to be getting the real point of the responses you're getting, which are that Stackless is *not* a minor change to Python, and that it severely restricts the ways in which Python can interact with other systems, be they hardware platforms, software frameworks, or 3rd party extensions. Please step off your soapbox, listen, and understand how and why Stackless gets in the way of Python's range of applicability. --Guido van Rossum (home page: http://www.python.org/~guido/)

At 07:06 AM 6/2/04 -0700, Guido van Rossum wrote:

...

Stackless is *not* a minor change to Python, and that it severely restricts the ways in which Python can interact with other systems, be they hardware platforms, software frameworks, or 3rd party extensions.

[Phillip]

It's definitely not a minor change, but I think that the severe restrictions comment may perhaps be inaccurate. My understanding at this point from Bob and Christian is that for tasklets (which are apparently what most people want from Stackless), the code is portable, pure C, and has no effect on code that doesn't use tasklets, except in performance.

However, rather than continue discussion on this point, it would probably be better if the people who actually understand the Stackless implementation were to write a tasklets PEP addressing what *specifically* would be the *minimum* changes to the CPython interpreter to support co-operative multitasking.

By specifically, I mean listing what core functions/objects will be changed, and how those changes will impact other code. Do any interfaces/APIs change? Is there anything that people writing new core code or extensions have to be aware of? And so on. A full statement of impact, in other words. At that point, a pronouncement could be made on the basis of those specifics, and if the answer is no, then the PEP would provide a convenient place to point people to in the future, thus avoiding further reiterations of this discussion. And, in the meanwhile, it gives you and Martin a way to tell the other folks to "PEP up or shut up". :)

Right. That PEP better explain how one writes C code that calls into Python without involving the C stack as well! --Guido van Rossum (home page: http://www.python.org/~guido/)

...

Right. That PEP better explain how one writes C code that calls into Python without involving the C stack as well!

I believe the outstanding proposal is that attempting a task switch from within Python code that was called by an extension will fail with an exception.

But how do you define "called by an exception"? There are lots built-in data types that call back into Python, e.g. comparisons and hash for dict keys, list.sort, sets, etc... Last time we looked there were so many exceptions of this type that enumerating them all was hopeless. That's a problem for a PEP.

But, if you mean, "how do you write a new piece of C that calls the interpreter, such that it can allow task switching to take place from the invoked interpreter?", then that's a different question, which should definitely be answered in the PEP.

That's what I meant, yes.

The problem I see is that *any* operation on a Python object potentially involves C calling Python, which seems to mean that the changes to the core interpreter would have to be quite widespread. But maybe the PEP authors have a more clever way to deal with that than anything I've thought of.

My way of dealing with this is to keep Stackless out of the core. Everybody happy. --Guido van Rossum (home page: http://www.python.org/~guido/)

Bob Ippolito wrote:

...

However, JCSP uses java.lang.Threads to implement concurrency. So they are completely unlike Stackless' tasklets in their implementation strategy.

So what, if the API is the same?

If you don't have a problem with that, I don't either.

Using the stack or not is an implementation detail, what matters is having more control over the flow of your programs in a reasonably efficient manner. It just so happens that the recursive evaluation in CPython uses the stack in such a way that makes these kind of constructs impossible, so it needed to be changed for Stackless.

That is simply not true. It is possible and straight-forward to implement Stackless' tasklets and channels on top of the standard Python, with no need to change the interpreter proper. Regards, Martin

Christian Tismer wrote:

This is true, we have written demo implementations of tasklets with pure Python and threads on the last sprint.

Where can I find that implementation? Regards, Martin

Christian Tismer writes:

This is true, we have written demo implementations of tasklets with pure Python and threads on the last sprint.

The difference is that nobody would use these for production code,

I don't know if this is true. In a production environment where there is an existing Python installation, but no Stackless installation, and where performance is not a killer issue, tasklets *may* be useful for their organisational benefits. I say *may* because I don't know much about tasklets. A pure Python implementation may help my understanding. Is the demo downloadable from anywhere? Paul. -- This signature intentionally left blank

At 01:23 PM 6/3/04 +0100, Paul Moore wrote:

Christian Tismer writes:

...

This is true, we have written demo implementations of tasklets with pure Python and threads on the last sprint.

The difference is that nobody would use these for production code,

I don't know if this is true. In a production environment where there is an existing Python installation, but no Stackless installation, and where performance is not a killer issue, tasklets *may* be useful for their organisational benefits.

I say *may* because I don't know much about tasklets. A pure Python implementation may help my understanding.

Is the demo downloadable from anywhere?

Also keep in mind that tasklet-like operations can be done using state-machine callbacks (e.g. Twisted), or using nested generators (e.g. 'peak.events'). Callbacks have the problem of "twisting" what would otherwise be linear code, and if you're using generators, you have to use "generators all the way down" to code that does task switching. IOW, you can't have a non-generator call a generator and have that generator be able to task-switch. The net result of either approach is that you can't use any existing code (outside of those frameworks, of course) as part of your tasks.

Bob Ippolito wrote: ...

In my quick attempt to do CSP on top of CPython threads I learned that: - It's not as easy as it sounds to get deterministic behavior out of threads, and debugging threaded python apps is no fun. - When it does work, the performance sucks compared to Stackless.

Perhaps someone with more Python threading experience can prove me wrong?

I have spent almost three weeks to implement thread-aware channels for Stackless in C, and I can tell you it was hard. They are almost exactly 10 times slower than Stackless channels, and it was not trivial, but once for all time. :-) I think you are right! -- chris -- Christian Tismer :^) mailto:tismer@stackless.com Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

At 10:50 AM 6/2/04 -0700, Guido van Rossum wrote:

...

...

Bob Ippolito wrote:

...

...

However, JCSP uses java.lang.Threads to implement concurrency. So they are completely unlike Stackless' tasklets in their implementation strategy. So what, if the API is the same?

...

In my quick attempt to do CSP on top of CPython threads I learned that: - It's not as easy as it sounds to get deterministic behavior out of threads, and debugging threaded python apps is no fun. - When it does work, the performance sucks compared to Stackless.

The same is probably true for JCSP. I don't want a feature that can only be implemented efficiently by using assembler or some other platform-specific hack (and most uses of longjmp() fall in that category, despite its status as an ISO standard library function!). Especially if the proponents keep using efficiency as the most important property of the feature.

I personally think that deterministic task-switching is more relevant than raw speed. Being able to create hundreds or even thousands of lightweight tasks is also very useful. Right now I can do both of these using generators in CPython 2.2 or better, so the only things (IMO) that Stackless is offering to add are: 1. dropping the need to write tasks solely in the form of "generators all the way down" 2. a simpler syntax: yields are expressed as calls, rather than as a 'yield' statement followed by various magical idioms. I currently have code that doesn't hesitate to spawn generator-based tasklets for simple condition monitoring tasks, and I have no idea in practice how many of these tasklets might be running in my code, but I'm pretty sure it's more than I'd want to have threads running for. However, if threads only had to be used when yielding code was called from non-yielding code, or vice versa, I'd probably be okay with it. Such boundaries aren't all that common in my experience to date using generator-based tasklets.

Bob Ippolito wrote:

In my quick attempt to do CSP on top of CPython threads I learned that: - It's not as easy as it sounds to get deterministic behavior out of threads, and debugging threaded python apps is no fun.

You could have looked at JCSP for guidance.

- When it does work, the performance sucks compared to Stackless.

So the performance of JCSP sucks as well? Then tasklets aren't implementable in an acceptable way in Java? Regards, Martin

"Phillip J. Eby" wrote in message news:5.1.1.6.0.20040601001051.01f00ca0@mail.telecommunity.com...

I was just suggesting that if people had specific features of Stackless that they wanted to see in CPython, that it would be best for them to write a PEP specifically addressing the desired feature(s), rather than asking "why isn't CPython stackless?"

I agree that a PEP would be more useful than the repeated question. One criterion for adding a new module is a constituency of users. One thing that would help the Stackless in Python cause is more evidence that it is useful in practice as well as in theory and expectation. Even today, the only thing I can find on the site is " Stackless has impressive applications, like: a.. Eve, an online massive multiplayer game by CCP Games b.. ? by Iron Port " Eve was listed two years ago and ? does not say much. I went to the Eve site, downloaded several videos, and from what was presented, was impressed negatively, so it is not obviously to me an advertisement for Stackless. So other examples, less costly to try out, might be more persuasive. Terry J. Reedy

Terry Reedy wrote:

...

One criterion for adding a new module is a constituency of users. One thing that would help the Stackless in Python cause is more evidence

"Christian Tismer" wrote in message news:40BD00C0.6010606@stackless.com... that

...

it is useful in practice as well as in theory and expectation. Even today, the only thing I can find on the site is " Stackless has impressive applications, like: a.. Eve, an online massive multiplayer game by CCP Games b.. ? by Iron Port " Eve was listed two years ago and ? does not say much. I went to the Eve site, downloaded several videos, and from what was presented, was impressed negatively, so it is not obviously to me an advertisement for Stackless. So other examples, less costly to try out, might be more persuasive.

Why did you do that?

Do what? Accept the invitation to follow the posted link to the new Stackless site to learn more about Stackless today? Click on the Wiki link and read? Click on the included link to the one and only listed example application? Among other things, I looked to see if there were currently a free trial period for Eve, as there sometines is for online games. If I were to try it, and it ran smoothly for some time without lags and crashes (which I have experience of), then that would suggest that the tools (including Stackless) and use thereof were adequate to the task. If not, that something (though not necessarily Stackless) would be shown lacking.

Would you download and try Zope, in order to judge about Python?

Given that you seem to be trying to ridicule sensible behavior, it is ironic that you should pick that particular example. In fact, several years ago, Zope (perhaps by its earlier name) was touted as Python's 'killer application'. At least a few people said something like 'To see Python in action, go visit the Python/Zope powered Zope site.' Which I (and others) did. And it appears that others went further and downloaded Zope to try on their own systems. A year or two ago, the Zope site listed Plone as an example of Zope in action. I followed the link to its site and signed up for one of the free demo accounts.

Would you say C is a bad language, because you (perhaps) don't like Perl?

The issue is performance, not liking. In any case, the proof of C was Unix, 30 years ago. I switched from Fortran to C in 80/81, long before Perl existed. However, if Perl were the only available example application for C, then yes, trying it out would one sensible way to evaluate C.

Last time I disliked a movie, I really thought celluloid is a bad idea.

If there were a celluloid movie development site that listed exacly one celluloid movie and either its price to watch were too much for me and the free stills unimpressive, or its *technical* performance crummy, then I would consider its technical virtue unproven. But that is not the case.

These examples

To me, there is only one example. I don't count '?'. But perhaps you know something you can't post.

are shown to prove that Stackless has industrial strength, nothing else.

Your biased-by-parenthood assertion is not proof to most people. The mere existence of Eve is not proof. A positive evaluation of its technical performance would be evidence. To paraphrase my original second sentence: if you want me to see that Stackless has industrial strength, show me more evidence. Terry J. Reedy

Martin v. Löwis wrote:

My point is that Stackless 3.0 is probably not acceptable for integration into C Python because of the processor-specific assembler fragments.

Ok, that can be dropped and put into an extension.

...

The discussion, as often before, tries to find out why it is *not* possible to include Stackless, instead of finding out the useful parts which could be supported with relatively small impact. This is why Stackless exists, and will continue to exist.

That's because people always ask the question "Can it be included?" They never ask "If I were to contribute this and that part, would it be accepted?".

Actually, this thread *started* with the question "Why isn't Python stackless?" to which the correct answer is "Because nobody has contributed patches." Only then people get excited and think that Stackless 3.0 would be part of Python 2.5.

I didn't understand the question this way, although it was probably meant so. For me it was more like "why doesn't it aim for Stackless' features". Although I'm trying to make Stackless as clean and compatible as possible, I'm not trying to have my current version ready for inclusion. Development is more influenced by user requests. I think, the heart of the thing is to decide whether Python wants to go the way to make as much as possible calls non-recursive. This is a lot of work, and I did one possible implementation, which minimized changes to the rest of the system. If we try to modify Python to be stackless, we would probably got more rigorous ways and change more of the design. I can't do that alone. I doubt that anybody would be able to create such a re-design alone, while keeping up with Python development, that does quite much in the opposite direction. This must become a goal of the core developer group. all the best - chris -- Christian Tismer :^) mailto:tismer@stackless.com Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

Martin:

[In addition, the approach taken by Stackless 3.0 is probably questionable: for example, it may be that C++ exception handling stops working in Stackless.]

Indeed. The last time this was discussed, I seem to remember Stackless had trouble with C code that passed pointers to local variables as parameters. Since this is a perfectly legitimate thing to do in C, if Stackless can't handle it, then it's fundamentally flawed. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+