[Python-Dev] Re: [Stackless] comments on PEP 219

[Gordon McMillan]

[Jeremy]
>   >> If a programmer uses a library implement via coroutines, can
>   she >> call library methods from an __xxx__ method?
> 
>   GMcM> Certain situations won't work, but you knew that.
> 
> I expected that some won't work, but no one seems willing to tell
> me exactly which ones will and which ones won't.  Should the
> caveat in the documentation say "avoid using certain __xxx__
> methods" <0.9 wink>. 

Within an __xxx__ method, you cannot *use* a coroutine not 
created in that method. That is true in current Stackless and 
will be true in Stack-lite. The presence of "library" in the 
question is a distraction.

I guess if you think of a coroutine as just another kind of 
callable object, this looks like a strong limitation. But you 
don't find yourself thinking of threads as plain old callable 
objects, do you? In a threaded program, no matter how 
carefully designed, there is a lot of thread detritus lying 
around. If you don't stay concious of the transfers of control 
that may happen, you will screw up.

Despite the limitation on using coroutines in magic methods, 
coroutines have an advantage in that tranfers of control only 
happen when you want them to. So avoiding unwanted 
transfers of control is vastly easier.
 
>   >> Can coroutines or microthreads co-exist with callbacks
>   invoked by >> C extensions?
> 
>   GMcM> Again, in certain situations it won't work. Again, you
>   knew GMcM> that.
> 
> Wasn't sure.

It's exactly the same situation.
 
>   >> Can a program do any microthread IO in an __call__ method?
> 
>   GMcM> Considering you know the answer to that one too, you
>   could've GMcM> phrased it as a parsable question.
> 
> Do I know the answer?  I assume the answer is no, but I don't
> feel very certain.

What is "microthreaded IO"? Probably the attempt to yield 
control if the IO operation would block. Would doing that 
inside __call__ work with microthreads? No. 

It's not my decision over whether this particular situation 
needs to be documented. Somtime between the 2nd and 5th 
times the programmer encounters this exception, they'll say 
"Oh phooey, I can't do this in __call__, I need an explicit 
method instead."  Python has never claimed that __xxx__ 
methods are safe as milk. Quite the contrary.

 
>   >> If any of these are the sort "in theory" problems that the
>   PEP >> alludes to, then we need a full spec for what is and is
>   not >> allowed.  It doesn't make sense to tell programmers to
>   follow >> unspecified "reasonable" programming practices.
> 
>   GMcM> That's easy. In a nested invocation of the Python
>   interpreter, GMcM> you can't use a coroutine created in an
>   outer interpreter.
> 
> Can we define these situations in a way that doesn't appeal to
> the interpreter implementation? 

No, because it's implementation dependent.

> If not, can we at least come up
> with a list of what will and will not work at the python level?

Does Python attempt to catalogue all the ways you can screw 
up using magic methods? Using threads? How 'bout the 
metaclass hook? Even stronger, do we catalogue all the ways 
that an end-user-programmer can get bit by using a library 
written by someone else that makes use of these facilities?
 
>   GMcM> In the Python 2 documentation, there are 6 caveats listed
>   in GMcM> the thread module. That's a couple order of magnitudes
>   GMcM> different from the actual number of ways you can screw up
>   GMcM> using the thread module.
> 
> The caveats for the thread module seem like pretty minor stuff to
> me. If you are writing a threaded application, don't expect code
> to continue running after the main thread has exited.

Well, the thread caveats don't mention the consequences of 
starting and running a thread within an __init__ method.  

> The caveats for microthreads seems to cover a vast swath of
> territory: The use of libraries or extension modules that involve
> callbacks or instances with __xxx__ methods may lead to
> application failure. 

While your statement is true on the face of it, it is very 
misleading. Things will only fall apart when you code an 
__xxx__ method or callback that uses a pre-existing coroutine 
(or does a uthread swap). You can very easily get in trouble 
right now with threads and callbacks. But the real point is that 
it is *you* the programmer trying to do something that won't 
work (and, BTW, getting notified right away), not some library 
pulling a fast one on you. (Yes, the library could make things 
very hard for you, but that's nothing new.)

Application programmers do not need magic methods. Ever. 
They are very handy for people creating libraries for application 
programmers to use, but we already presume (naively) that 
these people know what they're doing.

> I worry about it becomes it doesn't sound
> very modular.  The use of coroutines in one library means I can't
> use that library in certain special cases in my own code.

With a little familiarity, you'll find that coroutines are a good 
deal more modular than threads.

In order for that library to violate your expectations, that library 
must be concious of multiple coroutines (otherwise, it's just a 
plain stackfull call / return). It must have kept a coroutine from 
some other call, or had you pass one in. So you (if at all 
cluefull <wink>) will be concious that something is going on 
here.

The issue is the same as if you used a framework which used 
real threads, but never documented anything about the 
threads. You code callbacks that naively and independently 
mutate a global collection. Do you blame Python?

> I'm sorry if I sound grumpy, but I feel like I can't get a
> straight answer despite several attempts.  At some level, it's
> fine to say that there are some corner cases that won't work well
> with microthreads or coroutines implemented on top of stackless
> python.  But I think the PEP should discuss the details.  I've
> never written in an application that uses stackless-based
> microthreads or coroutines so I don't feel confident in my
> judgement of the situation.

And where on the fearful to confident scale was the Jeremy 
just getting introduced to threads?
 
> Which gets back to Bernd's original question:
> 
>   GMcM> >   BR> """ Don't use classes and libraries that use
>   classes when GMcM> >   BR> IO in microthreaded programs!  """
>   GMcM> > GMcM> >   BR> which might indeed be a problem. Am I
>   overlooking something GMcM> >   BR> fundamental here?
> 
> and the synopsis of your answer:
> 
>   GMcM> Synopsis of my reply: this is more a problem with
>   uthreads GMcM> than coroutines. In any (real) thread, you're
>   limited to dealing GMcM> with one non-blocking IO technique
>   (eg, select) without going GMcM> into a busy loop. If you're
>   dedicating a (real) thread to select, it GMcM> makes more sense
>   to use coroutines than uthreads.
> 
> I don't understand how this addresses the question, but perhaps I
> haven't seen your reply yet.  Mail gets through to python-dev and
> stackless at different rates.

Coroutines only swap voluntarily. It's very obvious where these 
transfers of control take place hence simple to control when 
they take place. My suspicion is that most people use 
uthreads because they use a familiar model. Not many people 
are used to coroutines, but many situations would be more 
profitably approached with coroutines than uthreads.

- Gordon