[Python-Dev] Re: [Stackless] comments on PEP 219

[Jeremy Hylton]

>>>>> "GMcM" == Gordon McMillan <gmcm@hypernet.com> writes:

  GMcM> Can we please get the followups under control? Bernd sent me a
  GMcM> private email. I replied privately. Then he forwarded to
  GMcM> Stackless. So I forwarded my reply to Stackless. Now Jeremy
  GMcM> adds python-dev to the mix.

I had no idea what was going on with forwards and the like.  It looks
like someone "bounced" messages, i.e. sent a message to me or a list
I'm on without including me or the list in the to or cc fields.  So I
couldn't tell how I received the message!  So I restored the original
recipients list of the thread (you, stackless, python-dev).

  >> >>>>> "BR" == Bernd Rinn <Bernd.Rinn@epost.de> writes:
  >> A few other variations on the question come to mind:
  >>
  >> If a programmer uses a library implement via coroutines, can she
  >> call library methods from an __xxx__ method?

  GMcM> Certain situations won't work, but you knew that.

I expected that some won't work, but no one seems willing to tell me
exactly which ones will and which ones won't.  Should the caveat in
the documentation say "avoid using certain __xxx__ methods" <0.9
wink>. 
 
  >> Can coroutines or microthreads co-exist with callbacks invoked by
  >> C extensions?

  GMcM> Again, in certain situations it won't work. Again, you knew
  GMcM> that.

Wasn't sure.
 
  >> Can a program do any microthread IO in an __call__ method?

  GMcM> Considering you know the answer to that one too, you could've
  GMcM> phrased it as a parsable question.

Do I know the answer?  I assume the answer is no, but I don't feel
very certain.
 
  >> If any of these are the sort "in theory" problems that the PEP
  >> alludes to, then we need a full spec for what is and is not
  >> allowed.  It doesn't make sense to tell programmers to follow
  >> unspecified "reasonable" programming practices.

  GMcM> That's easy. In a nested invocation of the Python interpreter,
  GMcM> you can't use a coroutine created in an outer interpreter.

Can we define these situations in a way that doesn't appeal to the
interpreter implementation?  If not, can we at least come up with a
list of what will and will not work at the python level?

  GMcM> In the Python 2 documentation, there are 6 caveats listed in
  GMcM> the thread module. That's a couple order of magnitudes
  GMcM> different from the actual number of ways you can screw up
  GMcM> using the thread module.

The caveats for the thread module seem like pretty minor stuff to me.
If you are writing a threaded application, don't expect code to
continue running after the main thread has exited.

The caveats for microthreads seems to cover a vast swath of territory:
The use of libraries or extension modules that involve callbacks or
instances with __xxx__ methods may lead to application failure.  I
worry about it becomes it doesn't sound very modular.  The use of
coroutines in one library means I can't use that library in certain
special cases in my own code.

I'm sorry if I sound grumpy, but I feel like I can't get a straight
answer despite several attempts.  At some level, it's fine to say that
there are some corner cases that won't work well with microthreads or
coroutines implemented on top of stackless python.  But I think the
PEP should discuss the details.  I've never written in an application
that uses stackless-based microthreads or coroutines so I don't feel
confident in my judgement of the situation.

Which gets back to Bernd's original question:

  GMcM> >   BR> """ Don't use classes and libraries that use classes when
  GMcM> >   BR> IO in microthreaded programs!  """
  GMcM> > 
  GMcM> >   BR> which might indeed be a problem. Am I overlooking something
  GMcM> >   BR> fundamental here?

and the synopsis of your answer:

  GMcM> Synopsis of my reply: this is more a problem with uthreads 
  GMcM> than coroutines. In any (real) thread, you're limited to dealing 
  GMcM> with one non-blocking IO technique (eg, select) without going 
  GMcM> into a busy loop. If you're dedicating a (real) thread to select, it 
  GMcM> makes more sense to use coroutines than uthreads.

I don't understand how this addresses the question, but perhaps I
haven't seen your reply yet.  Mail gets through to python-dev and
stackless at different rates.

Jeremy