[Python-ideas] asyncore: included batteries don't fit

[Guido van Rossum]

 Tue, Oct 9, 2012 at 5:44 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Guido van Rossum wrote:
>
>> Indeed, in NDB this works great. However tracebacks don't work so
>> great: If you don't catch the exception right away, it takes work to
>> make the tracebacks look right when you catch it a few generator calls
>> down on the (conceptual) stack. I fixed this to some extent in NDB, by
>> passing the traceback explicitly along when setting an exception on a
>> Future;
>
>
> Was this before or after the recent change that was supposed
> to improve tracebacks from yield-fram chains? If there's still
> a problem after that, maybe exception handling in yield-from
> requires some more work.

Sadly it was with Python 2.5/2.7...

>> But so far when thinking about this
>> recently I have found the goal elusive --
>
>
>> Perhaps you can clear things up by
>>
>> showing some detailed (but still simple enough) example code to handle
>> e.g. a simple web client?
>
>
> You might like to take a look at this, where I develop a series of
> examples culminating in a simple multi-threaded server:
>
> http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/yf_current/Examples/Scheduler/scheduler.txt

Definitely very enlightening. Though I think you should not use
'thread' since that term is already reserved for OS threads as
supported by the threading module. In NDB I chose to use 'tasklet' --
while that also has other meanings, its meaning isn't fixed in core
Python. You could also use task, which also doesn't have a core Python
meaning. Just don't call it "process", never mind that Erlang uses
this (a number of other languages rooted in old traditions do too, I
believe).

Also I think you can now revisit it and rewrite the code to use Python 3.3.

> Code here:
>
> http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/yf_current/Examples/Scheduler/

It does bother me somehow that you're not using .send() and yield
arguments at all. I notice that you have a lot ofthree-line code
blocks like this:

      block_for_reading(sock)
      yield
      data = sock.recv(1024)

The general form seems to be:

      arrange for a callback when some operation can be done without blocking
      yield
      do the operation

This seems to be begging to be collapsed into a single line, e.g.

      data = yield sock.recv_async(1024)

(I would also prefer to see the socket wrapped in an object that makes
it hard to accidentally block.)

>> somehow it seems there *has*
>> to be a distinction between an operation you just *yield* (this would
>> be waiting for a specific low-level I/O operation) and something you
>> use with yield-from, which returns a value through StopIteration.
>
> It may be worth noting that nothing in my server example uses 'yield'
> to send or receive values -- yield is only used without argument as
> a suspension point. But the functions containing the yields *are*
> called with yield-from and may return values via StopIteration.

Yeah, but see my remark above...

> So I think there are (at least) two distinct ways of using generators,
> but the distinction isn't quite the one you're making. Rather, we
> have "coroutines" (don't yield values, do return values) and
> "iterators" (do yield values, don't return values).

But surely there's still a place for send() and other PEP 342 features?

> Moreover, it's *only* the "coroutine" variety that we need to cater
> for when designing an async event system. Does that help to
> alleviate any of your monad-induced headaches?

Not entirely, no. I now have a fair amount experience writing an async
system and helping users make sense of its error messages, and there
are some practical considerations. E.g. my users sometimes want to
treat something as a coroutine but they don't have any yields in it
(perhaps they are writing skeleton code and plan to fill in the I/O
later). Example:

def caller():
  data = yield from reader()

def reader():
    return 'dummy'
    yield

works, but if you drop the yield it doesn't work. With a decorator I
know how to make it work either way.

-- 
--Guido van Rossum (python.org/~guido)