[Python-Dev] Stackless Python

[Phillip J. Eby]

At 06:34 AM 6/1/04 -0700, Guido van Rossum wrote:
> > It is most probably possible to implement soft-switchign in Jython.
>
>That's a rather vague statement; I'd like to hear from Samuele about
>that.

Me too.  The only thing I've heard of in Java that does lightweight 
co-operative task switching, is actually implemented using a Java VM 
written in Java.  Sort of like PyPy, only I guess you'd call it 
JavaJava.  :)  I don't remember the name of it right off, just that it was 
a commercial product that claimed they had a patent pending.


> > No idea what you mean by arbitrary exceptions (I never had that)
>
>I meant exceptions to the general rules.  Like, you can't depend on
>tasklet switching when you use one of these extensions, or one of
>these built-ins, or ...

Perhaps it would be better if it were defined in terms of where it *does* 
work, rather than where it doesn't.  For example, one can write 
co-operative multitasking code in CPython now using generators...  but it 
only works if you have "generators all the way down".  That is, any code in 
a task that calls a generator must itself be a generator.  That's a pretty 
comprehensible rule, albeit draconian to follow.  Adding soft-switching 
would change the rule to be, "you can switch as long as you haven't been 
called from C code."  That is, you haven't been called from *any* built-in 
or extension module.

Speaking as someone who's written some significant event-driven 
co-operative multitasking code using generators, I would say that going 
from "generators all the way down" to "Python all the way down" would be a 
huge improvement in flexibility.


> > On the latter: Well, this is a pure interface issue. I don't say
> > that everybody must love tasklets, there are other approaches and
> > interfaces possible.  But they all rely on the non-recursive
> > interpreter core, and that's the point, IMHO.
>
>Is it really just tasklets that the users want?

I am not currently a Stackless user, but if I were to try it, tasklets 
would be my primary interest.  I don't actually think that the higher 
recursion limit is a "feature", for two reasons: 1) Python loops run a lot 
faster than function calls, and 2) every time I've ever gotten an error 
from excessive recursion, it was because I'd accidentally created an 
infinite recursion.  So, to me, lifting the default recursion limit in 
standard Python would be a really *bad* idea.

The other feature besides tasklets that I find potentially interesting is 
thread pickling.  I'm intrigued by the possibility of using them to create 
"long-running tasks".  That is, workflow using Python scripts to define the 
loops, branches, etc. of the workflow.   But to make a really practical 
workflow system based on this, one would have to have a way of dealing with 
changes to the code that the running workflows were based on, so I don't 
know if that whole idea is really worth pursuing.  I would probably be 
better off basing such an animal on PyPy, because performance for the code 
in a long-running process workflow is a non-issue.

Anyway, I don't know about "the users" in general, but for me at least 
tasklets are where it's at.  :)  And I only care about explicit yielding, 
not thread-like concurrency.  (IOW, I don't want Stackless doing the 
scheduling, I'd want to be able to write my own scheduler, in pure Python.)


> > I understood it more like the more advanced question "why does
> > Python still block itself from lightweight threading by keping state
> > on the C stack".
>
>IMO mostly because Python wants to be friendly to extensions written
>in C or C++, including those that frequently call back into Python.

That would only disable task-switching within the called Python code, not 
from the code that called the extension.  And that, only if the 
hard-switching module wasn't available.