On Fri, Jun 3, 2016 at 4:11 PM, Yury Selivanov <yselivanov@gmail.com> wrote:
During the asyncio open space a few days ago we had a number of important questions raised, including Twisted/asyncio interop.  I’d like to start the discussion of a few related things:

For those who haven't seen it, it's already possible to mix asyncio, twisted, and tornado freely in tornado coroutines (using the `yield` syntax. `await` can be used with tornado and asyncio, but not twisted last time I looked):
https://groups.google.com/forum/#!topic/python-tulip/W9_Cj4zN1Jc
 

1. key difference between a Deferred and a Future;
2. make Deferred objects compatible with asyncio code;
3. overall strategy to make Twisted code compatible with asyncio code.

In detail:

1. Unlike Futures, Deferred objects know nothing about the event loop.  Deferred calls its callbacks directly, while Future uses ‘loop.call_soon’ to ensure fair scheduling.

asyncio Futures know about their event loop, but concurrent futures and Tornado futures do not. asyncio is the odd one out here, and I think it would be better to make asyncio futures call their callbacks directly than to introduce knowledge of the event loop to other deferred/future-like objects.
 

2. I see two issues with Deferred/Future compatibility:

2.1. There should be a way for Deferred objects to get the currently running event loop reliably. There is a pending PR to add the new ‘asyncio.get_running_loop’ function.  If we land it in 3.5.2, Deferred object will be able to get the currently running event loop and schedule callbacks in a fair fashion, similarly to Futures.

2.2. asyncio uses ‘isinstance(o, Future)’ calls in many key places, such as ‘asyncio.ensure_future’ and in the Task class.  The latter makes it impossible for anything but Future and coroutines to be awaitable in asyncio code.  We have two options to fix this: make Future an instance of ABCMeta, or introduce a new protocol for Future-like objects.  The former option will not be considered because it makes isinstance calls much slower.

+1. This was my goal in the thread linked above. 
 

As for the second option — a protocol for Future-like objects — I propose to add a new dunder attribute ‘__asyncio_future__ = True’ to asyncio.Future class, and replace all ‘isinstance(o, Future)’ calls with ‘hasattr(o, “__asyncio_future__”)’ checks.

Why make it asyncio specific? What you need is A) does it have an add_done_callback() method with the right interface, and B) are callbacks guaranteed to be run on the same thread (as with asyncio Futures) or is the thread unspecified (as in concurrent Futures). The latter isn't really a type-level check, so maybe Future instances should be associated with a thread (rather than an event loop). 
 

(2.1) and (2.2) should make it possible for Twisted to make Deferred objects fully compatible with asyncio code.  Glyph and Amber, please confirm this.

3. I’ve been on Amber’s talk about Twisted and asyncio, and wanted to share my ideas on how Twisted should approach compatibility with asyncio and async/await syntax:

- I don’t think we need ‘twisted.deferredCoroutine’ decorator — that will make async/await a lot slower. Instead, Twisted should follow asyncio and create an alternative, tailored for Twisted, implementation of ‘asyncio.Task’, along with ‘ensure_deferred’ and/or ‘create_task’ APIs.  This way users can create Deferreds for coroutines only when they need it. 

- ‘ensure_deferred’ could also accept instances of ‘asyncio.Future’, and wrap them in a Deferred object.  This should make it possible to re-use asyncio-specific code in Twisted code.

This is what Tornado's `convert_yielded` function does to allow twisted and asyncio libraries to be used from tornado applications. (I can't remember why we have to wrap asyncio futures in tornado futures; the interfaces are compatible except for tornado's extensions to better capture stack traces on python 2). 

-Ben
 

Thoughts?

Thanks,
Yury
_______________________________________________
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/