[Python-ideas] Async API

[Guido van Rossum]

On Wed, Oct 24, 2012 at 4:03 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> Hi Guido,
>
> On 2012-10-24, at 6:43 PM, Guido van Rossum <guido at python.org> wrote:
>> What's the problem with just letting the cleanup take as long as it
>> wants to and do whatever it wants? That's how try/finally works in
>> regular Python code.

> The problem appears when you add timeouts support.
>
> Let me show you an abstract example (I won't use yield_froms, but I'm
> sure that the problem is the same with them):
>
>    @coroutine
>    def fetch_comments(app):
>        session = yield app.new_session()
>        try:
>             return (yield session.query(...))
>        finally:
>             yield session.close()
>
> and now we execute that with:
>
>    #: Get a list of comments; throw a TimeoutError if it
>    #: takes more than 1 second
>    comments = yield fetch_comments(app).with_timeout(1.0)
>
> Now, scheduler starts with 'fetch_comments', then executes
> 'new_session', then executes 'session.query' in a round-robin fashion.
>
> Imagine, that database query took a bit less than a second to execute,
> scheduler pushes the result in coroutine, and then a timeout event occurs.
> So scheduler throws a 'TimeoutError' in the coroutine, thus preventing
> the 'session.close' to be executed.  There is no way for a scheduler to
> understand, that there is no need in pushing the exception right now,
> as the coroutine is in its finally block.
>
> And this situation is a pretty common when you have such timeouts
> mechanism in place and widely used.

Ok, I can understand. But still, this is a problem with timeouts in
general, not just with timeouts in a yield-based environment. How does
e.g. Twisted deal with this?

As a work-around, I could imagine some kind of with-statement that
tells the scheduler we're already in the finally clause (it could
still send you a timeout if your cleanup takes way too long):

try:
  yield <regular code>
finally:
  with protect_finally():
    yield <cleanup code>

Of course this could be abused, but at your own risk -- the scheduler
only gives you a fixed amount of extra time and then it's quits.

-- 
--Guido van Rossum (python.org/~guido)