[Python-ideas] Async API

[Yury Selivanov]

On 2012-10-29, at 9:59 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:

> Guido, Greg,
> 
> On 2012-10-27, at 7:45 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> 
>> Right.  But now I'm not sure this approach will work with yield-froms.
>> As when you yield-fromming scheduler knows nothing about the chain of 
>> generators, as it's all hidden in the yield-from implementation.
> 
> I think I've come up with a solution that should work for yield-froms too
> (if we accept my in_finally idea in 3.4).  And there should be a way
> of writing a 'protect_finally' context manager too.
> 
> I'll illustrate the approach on Guido's tulip micro-framework
> (consider it a pseudo code to illustrate the idea):
> 
>    class Interrupt(BaseException):
>        """Should penetrate all try..excepts"""
> 
>    def call_with_timeout(timeout, gen):
>        context.current_task._add_timeout(timeout, gen)
>        try:
>            return (yield from gen)
>        except Interrupt:
>            raise TimeoutError() from None
> 
>    class Task:
>        def _add_timeout(timeout, gen):
>            self.eventloop.call_later(
>                timeout,
>                partial(self._interrupt, gen))
> 
>        def _interrupt(self, gen):
>            if not gen.in_finally:
>                gen.throw(Interrupt, Interrupt(), None)
>            else:
>                # So we set a flag to watch for gen's in_finally value
>                # on each 'step' call.  And when it's 0 - Task.step
>                # will call '_interrupt' again.
>                self._watch_finally(gen)
> 
> I defined a new function 'call_with_timeout', because tulip's 'with_timeout'
> starts a new Task, whereas the former works in any generator inside the task.
> 
> So, after that you'd be able to do the following:
> 
>     yield from call_with_timeout(1.0, something())
> 
> And something's 'finally' won't ever be aborted.

Ah, the solution is wrong, I've tricked myself.

The right code would be something like that:

   class Interrupt(BaseException):
       """Should penetrate all try..excepts"""

   def call_with_timeout(timeout, gen):
       context.current_task._add_timeout(timeout, gen)
       try:
           return (yield from gen)
       except Interrupt:
           raise TimeoutError() from None

   class Task:
       def _add_timeout(timeout, gen):
           # XXX The following line is the key.  We need a reference
           # to the generator object that is yield-fromming our 'gen'
           # ('caller' for 'gen')
           current_yield_from = self.gen.yield_from
           self.eventloop.call_later(
               timeout,
               partial(self._interrupt, gen, current_yield_from))

       def _interrupt(self, gen, yf):           
           if not yf.in_finally:
               # If gen's caller is not in it's finally block - it's
               # safe for us to interrupt gen.
               gen.throw(Interrupt, Interrupt(), None)
           else:
               # So we set a flag to watch for yf's in_finally value
               # on each 'step' call.  And when it's 0 - Task.step
               # will call '_interrupt' again.
               self._watch_finally(yf, gen)

IOW, besides just 'in_finally', we also need to add 'yield_from' property
to generator object.  The latter will hold a reference to the sub-generator
that current generator is yielding from.

The logic is pretty twisted, but i'm sure that the problem is solvable.

P.S. I'm not proposing to add anything.  It's more about finding *any* way
to actually solve the problem correctly.  Once we find that way, we *maybe*
start thinking about language support of it.

-
Yury