[Async-sig] asyncio <-> twisted interoperability

[Yury Selivanov]

During the asyncio open space a few days ago we had a number of important questions raised, including Twisted/asyncio interop.  I’d like to start the discussion of a few related things:

1. key difference between a Deferred and a Future;
2. make Deferred objects compatible with asyncio code;
3. overall strategy to make Twisted code compatible with asyncio code.

In detail:

1. Unlike Futures, Deferred objects know nothing about the event loop.  Deferred calls its callbacks directly, while Future uses ‘loop.call_soon’ to ensure fair scheduling.

2. I see two issues with Deferred/Future compatibility:

2.1. There should be a way for Deferred objects to get the currently running event loop reliably. There is a pending PR to add the new ‘asyncio.get_running_loop’ function.  If we land it in 3.5.2, Deferred object will be able to get the currently running event loop and schedule callbacks in a fair fashion, similarly to Futures.

2.2. asyncio uses ‘isinstance(o, Future)’ calls in many key places, such as ‘asyncio.ensure_future’ and in the Task class.  The latter makes it impossible for anything but Future and coroutines to be awaitable in asyncio code.  We have two options to fix this: make Future an instance of ABCMeta, or introduce a new protocol for Future-like objects.  The former option will not be considered because it makes isinstance calls much slower.

As for the second option — a protocol for Future-like objects — I propose to add a new dunder attribute ‘__asyncio_future__ = True’ to asyncio.Future class, and replace all ‘isinstance(o, Future)’ calls with ‘hasattr(o, “__asyncio_future__”)’ checks.

(2.1) and (2.2) should make it possible for Twisted to make Deferred objects fully compatible with asyncio code.  Glyph and Amber, please confirm this.

3. I’ve been on Amber’s talk about Twisted and asyncio, and wanted to share my ideas on how Twisted should approach compatibility with asyncio and async/await syntax:

- I don’t think we need ‘twisted.deferredCoroutine’ decorator — that will make async/await a lot slower. Instead, Twisted should follow asyncio and create an alternative, tailored for Twisted, implementation of ‘asyncio.Task’, along with ‘ensure_deferred’ and/or ‘create_task’ APIs.  This way users can create Deferreds for coroutines only when they need it.

- ‘ensure_deferred’ could also accept instances of ‘asyncio.Future’, and wrap them in a Deferred object.  This should make it possible to re-use asyncio-specific code in Twisted code.

Thoughts?

Thanks,
Yury