asyncio <-> twisted interoperability
During the asyncio open space a few days ago we had a number of important questions raised, including Twisted/asyncio interop. I’d like to start the discussion of a few related things: 1. key difference between a Deferred and a Future; 2. make Deferred objects compatible with asyncio code; 3. overall strategy to make Twisted code compatible with asyncio code. In detail: 1. Unlike Futures, Deferred objects know nothing about the event loop. Deferred calls its callbacks directly, while Future uses ‘loop.call_soon’ to ensure fair scheduling. 2. I see two issues with Deferred/Future compatibility: 2.1. There should be a way for Deferred objects to get the currently running event loop reliably. There is a pending PR to add the new ‘asyncio.get_running_loop’ function. If we land it in 3.5.2, Deferred object will be able to get the currently running event loop and schedule callbacks in a fair fashion, similarly to Futures. 2.2. asyncio uses ‘isinstance(o, Future)’ calls in many key places, such as ‘asyncio.ensure_future’ and in the Task class. The latter makes it impossible for anything but Future and coroutines to be awaitable in asyncio code. We have two options to fix this: make Future an instance of ABCMeta, or introduce a new protocol for Future-like objects. The former option will not be considered because it makes isinstance calls much slower. As for the second option — a protocol for Future-like objects — I propose to add a new dunder attribute ‘__asyncio_future__ = True’ to asyncio.Future class, and replace all ‘isinstance(o, Future)’ calls with ‘hasattr(o, “__asyncio_future__”)’ checks. (2.1) and (2.2) should make it possible for Twisted to make Deferred objects fully compatible with asyncio code. Glyph and Amber, please confirm this. 3. I’ve been on Amber’s talk about Twisted and asyncio, and wanted to share my ideas on how Twisted should approach compatibility with asyncio and async/await syntax: - I don’t think we need ‘twisted.deferredCoroutine’ decorator — that will make async/await a lot slower. Instead, Twisted should follow asyncio and create an alternative, tailored for Twisted, implementation of ‘asyncio.Task’, along with ‘ensure_deferred’ and/or ‘create_task’ APIs. This way users can create Deferreds for coroutines only when they need it. - ‘ensure_deferred’ could also accept instances of ‘asyncio.Future’, and wrap them in a Deferred object. This should make it possible to re-use asyncio-specific code in Twisted code. Thoughts? Thanks, Yury
On 3 Jun 2016, at 13:11, Yury Selivanov <yselivanov@gmail.com> wrote:
2.1. There should be a way for Deferred objects to get the currently running event loop reliably. There is a pending PR to add the new ‘asyncio.get_running_loop’ function. If we land it in 3.5.2, Deferred object will be able to get the currently running event loop and schedule callbacks in a fair fashion, similarly to Futures.
Let’s pretend I’m an idiot for a moment. ;) Why do we want Twisted to schedule callbacks in the asyncio style? Is this intended to be a compatibility mode for Twisted whereby in an asyncio runner Deferreds emulate Futures? Is it required that Deferreds behave in this way to be awaitable in Python 3.5+? Cory
On Jun 3, 2016, at 6:53 PM, Cory Benfield <cory@lukasa.co.uk> wrote:
On 3 Jun 2016, at 13:11, Yury Selivanov <yselivanov@gmail.com> wrote:
2.1. There should be a way for Deferred objects to get the currently running event loop reliably. There is a pending PR to add the new ‘asyncio.get_running_loop’ function. If we land it in 3.5.2, Deferred object will be able to get the currently running event loop and schedule callbacks in a fair fashion, similarly to Futures.
Let’s pretend I’m an idiot for a moment. ;)
Why do we want Twisted to schedule callbacks in the asyncio style? Is this intended to be a compatibility mode for Twisted whereby in an asyncio runner Deferreds emulate Futures? Is it required that Deferreds behave in this way to be awaitable in Python 3.5+?
2.1 is not strictly necessary. Although I would really prefer that any Twisted code I use within an asyncio application will behave like asyncio code (if that’s possible). Yury
On Fri, Jun 3, 2016 at 4:11 PM, Yury Selivanov <yselivanov@gmail.com> wrote:
During the asyncio open space a few days ago we had a number of important questions raised, including Twisted/asyncio interop. I’d like to start the discussion of a few related things:
For those who haven't seen it, it's already possible to mix asyncio, twisted, and tornado freely in tornado coroutines (using the `yield` syntax. `await` can be used with tornado and asyncio, but not twisted last time I looked): https://groups.google.com/forum/#!topic/python-tulip/W9_Cj4zN1Jc
1. key difference between a Deferred and a Future; 2. make Deferred objects compatible with asyncio code; 3. overall strategy to make Twisted code compatible with asyncio code.
In detail:
1. Unlike Futures, Deferred objects know nothing about the event loop. Deferred calls its callbacks directly, while Future uses ‘loop.call_soon’ to ensure fair scheduling.
asyncio Futures know about their event loop, but concurrent futures and Tornado futures do not. asyncio is the odd one out here, and I think it would be better to make asyncio futures call their callbacks directly than to introduce knowledge of the event loop to other deferred/future-like objects.
2. I see two issues with Deferred/Future compatibility:
2.1. There should be a way for Deferred objects to get the currently running event loop reliably. There is a pending PR to add the new ‘asyncio.get_running_loop’ function. If we land it in 3.5.2, Deferred object will be able to get the currently running event loop and schedule callbacks in a fair fashion, similarly to Futures.
2.2. asyncio uses ‘isinstance(o, Future)’ calls in many key places, such as ‘asyncio.ensure_future’ and in the Task class. The latter makes it impossible for anything but Future and coroutines to be awaitable in asyncio code. We have two options to fix this: make Future an instance of ABCMeta, or introduce a new protocol for Future-like objects. The former option will not be considered because it makes isinstance calls much slower.
+1. This was my goal in the thread linked above.
As for the second option — a protocol for Future-like objects — I propose to add a new dunder attribute ‘__asyncio_future__ = True’ to asyncio.Future class, and replace all ‘isinstance(o, Future)’ calls with ‘hasattr(o, “__asyncio_future__”)’ checks.
Why make it asyncio specific? What you need is A) does it have an add_done_callback() method with the right interface, and B) are callbacks guaranteed to be run on the same thread (as with asyncio Futures) or is the thread unspecified (as in concurrent Futures). The latter isn't really a type-level check, so maybe Future instances should be associated with a thread (rather than an event loop).
(2.1) and (2.2) should make it possible for Twisted to make Deferred objects fully compatible with asyncio code. Glyph and Amber, please confirm this.
3. I’ve been on Amber’s talk about Twisted and asyncio, and wanted to share my ideas on how Twisted should approach compatibility with asyncio and async/await syntax:
- I don’t think we need ‘twisted.deferredCoroutine’ decorator — that will make async/await a lot slower. Instead, Twisted should follow asyncio and create an alternative, tailored for Twisted, implementation of ‘asyncio.Task’, along with ‘ensure_deferred’ and/or ‘create_task’ APIs. This way users can create Deferreds for coroutines only when they need it.
- ‘ensure_deferred’ could also accept instances of ‘asyncio.Future’, and wrap them in a Deferred object. This should make it possible to re-use asyncio-specific code in Twisted code.
This is what Tornado's `convert_yielded` function does to allow twisted and asyncio libraries to be used from tornado applications. (I can't remember why we have to wrap asyncio futures in tornado futures; the interfaces are compatible except for tornado's extensions to better capture stack traces on python 2). -Ben
Thoughts?
Thanks, Yury _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
[..]
1. Unlike Futures, Deferred objects know nothing about the event loop. Deferred calls its callbacks directly, while Future uses ‘loop.call_soon’ to ensure fair scheduling.
asyncio Futures know about their event loop, but concurrent futures and Tornado futures do not. asyncio is the odd one out here, and I think it would be better to make asyncio futures call their callbacks directly than to introduce knowledge of the event loop to other deferred/future-like objects.
Please see my reply to Lukasz. In short there are some problems with changing how asyncio currently works. [..]
Why make it asyncio specific? What you need is A) does it have an add_done_callback() method with the right interface, and B) are callbacks guaranteed to be run on the same thread (as with asyncio Futures) or is the thread unspecified (as in concurrent Futures). The latter isn't really a type-level check, so maybe Future instances should be associated with a thread (rather than an event loop).
Yes, (A) and (B) are both important, and they define a minimal set of requirements. I’d say the Future should also be an awaitable, and support ‘set_result’ and other methods. Associating Futures with threads is a neat idea, but maybe, a special attribute like __isfuture__ is the simplest option? Yury
Being able to re-use Twisted’s vast array of existing protocols and related libraries within asyncio would be a dream come true. Likewise, being able to write Twisted code using async/await coroutines without wrappers in-between would increase Twisted adoption since that syntax is more welcoming than callback-based Deferreds. I have a few inline comments here.
On Jun 3, 2016, at 1:11 PM, Yury Selivanov <yselivanov@gmail.com> wrote:
2.1. There should be a way for Deferred objects to get the currently running event loop reliably. There is a pending PR to add the new ‘asyncio.get_running_loop’ function. If we land it in 3.5.2, Deferred object will be able to get the currently running event loop and schedule callbacks in a fair fashion, similarly to Futures.
In general, writing polyglot code sucks. I always recommend sticking to a single version for applications. As library maintainers, we’re the poor people that have to provide compatibility across versions, typically with Tox. I don’t know to what extent we can expect coroutines to be FULLY COMPATIBLE across async frameworks. They should be REUSABLE but that’s not the same thing. Specifically, having Deferreds calling their callbacks directly is perfectly fine by me. The goal here is to reuse what the Twisted ecosystem has already implemented so expecting all deferreds to now schedule callbacks on the event loop is not realistic. In the same vein, we can expect some coroutines to reuse asyncio internals (commingling with the event loop’s implementation details for instance) to the extent where compatibility with Twisted might not be possible. Actually, +1 for Ben’s suggestion to revisit if asyncio.Futures really need call_soon scheduling of callbacks. Specifically, a torrent of call_soon doesn’t really achieve “fair scheduling” as suggested by Yury anyway: https://gist.github.com/ambv/3324e3e569b9f7883dfcb9c8cb1d5445 There’s still a comment on the asyncio codebase suggesting we might be able to merge concurrent.futures.Future with asyncio.Future. This would be a worthwhile goal.
2.2. asyncio uses ‘isinstance(o, Future)’ calls in many key places, such as ‘asyncio.ensure_future’ and in the Task class. The latter makes it impossible for anything but Future and coroutines to be awaitable in asyncio code. We have two options to fix this: make Future an instance of ABCMeta, or introduce a new protocol for Future-like objects. The former option will not be considered because it makes isinstance calls much slower.
As for the second option — a protocol for Future-like objects — I propose to add a new dunder attribute ‘__asyncio_future__ = True’ to asyncio.Future class, and replace all ‘isinstance(o, Future)’ calls with ‘hasattr(o, “__asyncio_future__”)’ checks.
+1 for replacing isinstance checks with an attribute that Twisted can add to Deferreds. +1 for Ben’s suggestion that we should avoid ‘asyncio’ in the name here. __isfuture__?
- I don’t think we need ‘twisted.deferredCoroutine’ decorator — that will make async/await a lot slower. Instead, Twisted should follow asyncio and create an alternative, tailored for Twisted, implementation of ‘asyncio.Task’, along with ‘ensure_deferred’ and/or ‘create_task’ APIs. This way users can create Deferreds for coroutines only when they need it.
+1 here. Having to decorate async/await coroutines to be able to use them within Twisted is an adoption barrier. The same would be true about having to decorate Deferreds before awaiting on them in asyncio. Amber, Glyph, Ben, is asyncio.Future’s event loop callback scheduling the biggest barrier in interoperability at the moment? - Ł
[..]
Actually, +1 for Ben’s suggestion to revisit if asyncio.Futures really need call_soon scheduling of callbacks. Specifically, a torrent of call_soon doesn’t really achieve “fair scheduling” as suggested by Yury anyway:
https://gist.github.com/ambv/3324e3e569b9f7883dfcb9c8cb1d5445
One quick story about call_soon: while working on uvloop I once spent a couple of hours debugging why asyncio/sslproto.py didn’t work. Turned out that SSLProtocol, in its connection_made scheduled some callbacks with loop.call_soon; and data_received did that too. Turned out that libuv called data_received one loop iteration earlier, which resulted in a race condition. What I’m trying to say here, is that making a subtle difference in how callbacks are scheduled can introduce some hard to debug bugs. In fact, I’m thinking now that Tornado/Twisted should keep their callbacks scheduling logic as is, and asyncio should keep its current implementation. Anyways, I’ve tried to update asyncio Future to call callbacks directly: https://github.com/1st1/asyncio/commit/9da7ff868f1405893947118178d62bb7e1eb3... The result is many broken unittests, until... a segfault in test_create_ssl_connection: https://gist.github.com/1st1/797dd12932bba7b353038fb7a80148ac (I’ll investigate this later). I’d be -1 to change callbacks scheduling in 3.5, IMO this has to wait until 3.6 even if we decide this should be done. [..]
+1 for replacing isinstance checks with an attribute that Twisted can add to Deferreds. +1 for Ben’s suggestion that we should avoid ‘asyncio’ in the name here. __isfuture__?
Yes, an asyncio-agnostic name would be better. I like __isfuture__. Yury
On Jun 3, 2016, at 19:17, Yury Selivanov <yselivanov@gmail.com> wrote:
In fact, I’m thinking now that Tornado/Twisted should keep their callbacks scheduling logic as is, and asyncio should keep its current implementation.
I have to agree. There are are good reasons for both the asyncio and the Twisted ways of doing things, and they have subtly different priorities that cascade out into pretty much every application; changing it might cause things to break in more or less arbitrary ways.
+1 for replacing isinstance checks with an attribute that Twisted can add to Deferreds. +1 for Ben’s suggestion that we should avoid ‘asyncio’ in the name here. __isfuture__?
Yes, an asyncio-agnostic name would be better. I like __isfuture__.
__futuristic__? :) -glyph
On Fri, Jun 3, 2016 at 10:17 PM, Yury Selivanov <yselivanov@gmail.com> wrote:
[..]
Actually, +1 for Ben’s suggestion to revisit if asyncio.Futures really need call_soon scheduling of callbacks. Specifically, a torrent of call_soon doesn’t really achieve “fair scheduling” as suggested by Yury anyway:
https://gist.github.com/ambv/3324e3e569b9f7883dfcb9c8cb1d5445
One quick story about call_soon: while working on uvloop I once spent a couple of hours debugging why asyncio/sslproto.py didn’t work. Turned out that SSLProtocol, in its connection_made scheduled some callbacks with loop.call_soon; and data_received did that too. Turned out that libuv called data_received one loop iteration earlier, which resulted in a race condition. What I’m trying to say here, is that making a subtle difference in how callbacks are scheduled can introduce some hard to debug bugs.
If things are so sensitive to minor changes in timing, doesn't that set the bar impossibly high for interoperability? In Tornado, we have similar concerns around "fair scheduling" of coroutines, but we solved this by using our equivalent of call_soon() in the coroutine runner rather than in the Future itself. So asyncio Futures yielded in a coroutine that uses Tornado's runner will be called one iteration later as they make two trips through the scheduler, while if asyncio.Task gained the ability to understand Tornado Futures they could turn into a busy loop that let nothing else run. If we want interoperability, we're going to have to mandate that code like SSLProtocol be tolerant of changes in timing (of course, saying that is one thing and achieving it is another)
In fact, I’m thinking now that Tornado/Twisted should keep their callbacks scheduling logic as is, and asyncio should keep its current implementation.
Yeah, the risks of making any changes in this area outweigh the benefits, since we'll never get all the implementations to schedule things in exactly the same way. -Ben
Anyways, I’ve tried to update asyncio Future to call callbacks directly:
https://github.com/1st1/asyncio/commit/9da7ff868f1405893947118178d62bb7e1eb3...
The result is many broken unittests, until... a segfault in test_create_ssl_connection: https://gist.github.com/1st1/797dd12932bba7b353038fb7a80148ac (I’ll investigate this later).
I’d be -1 to change callbacks scheduling in 3.5, IMO this has to wait until 3.6 even if we decide this should be done.
[..]
+1 for replacing isinstance checks with an attribute that Twisted can add to Deferreds. +1 for Ben’s suggestion that we should avoid ‘asyncio’ in the name here. __isfuture__?
Yes, an asyncio-agnostic name would be better. I like __isfuture__.
Yury _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
On Jun 4, 2016, at 13:25, Ben Darnell <ben@bendarnell.com> wrote:
If things are so sensitive to minor changes in timing, doesn't that set the bar impossibly high for interoperability?
The sensitivity is not to changes in timing - i.e. when the wall-clock runs, or ordering of non-deterministically ordered events - but rather to reentrancy - whether certain things complete synchronously while the caller is still on the stack and can depend on them having done so upon return. The recommended way of writing tests within Twisted these days depends heavily on `.callback´ synchronously resolving a Deferred, which is what adding a call_soon breaks. -glyph
On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jun 4, 2016, at 13:25, Ben Darnell <ben@bendarnell.com> wrote:
If things are so sensitive to minor changes in timing, doesn't that set the bar impossibly high for interoperability?
The sensitivity is not to changes in timing - i.e. when the wall-clock runs, or ordering of non-deterministically ordered events - but rather to reentrancy - whether certain things complete synchronously while the caller is still on the stack and can depend on them having done so upon return.
The recommended way of writing tests within Twisted these days depends heavily on `.callback´ synchronously resolving a Deferred, which is what adding a call_soon breaks.
That's interesting, and also potentially worrisome (for interop, I'm not saying Twisted is wrong here). I think asyncio depends on the opposite: that if you add a callback to a Future that's ready it does *not* immediately run. Asyncio's promise is pretty strongly that callbacks are serialized (no callbacks running inside other callbacks). IIRC we experimented with other semantics and found that it was harder to reason about. (IMO if you *know* a Future is ready why add a callback to it rather than just calling the damn thing if that's what you want?) -- --Guido van Rossum (python.org/~guido)
On Jun 6, 2016, at 08:29, Guido van Rossum <guido@python.org> wrote:
On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jun 4, 2016, at 13:25, Ben Darnell <ben@bendarnell.com> wrote:
If things are so sensitive to minor changes in timing, doesn't that set the bar impossibly high for interoperability?
The sensitivity is not to changes in timing - i.e. when the wall-clock runs, or ordering of non-deterministically ordered events - but rather to reentrancy - whether certain things complete synchronously while the caller is still on the stack and can depend on them having done so upon return.
The recommended way of writing tests within Twisted these days depends heavily on `.callback´ synchronously resolving a Deferred, which is what adding a call_soon breaks.
That's interesting, and also potentially worrisome (for interop, I'm not saying Twisted is wrong here).
I think asyncio depends on the opposite: that if you add a callback to a Future that's ready it does *not* immediately run. Asyncio's promise is pretty strongly that callbacks are serialized (no callbacks running inside other callbacks). IIRC we experimented with other semantics and found that it was harder to reason about. (IMO if you *know* a Future is ready why add a callback to it rather than just calling the damn thing if that's what you want?)
I don't think Twisted is necessarily right here either. You're absolutely right that it's easier to reason about reentrancy in some cases if you have a might-be-fired-might-not Future vs. the same sort of Deferred. I like the property where you can do: def test(test_case): a = asynchronously_something() test_case.this_must_not_have_a_result(a) cause_a_to_fire() test_case.assertEqual(test_case.this_must_have_a_result(a), something) but this is (somewhat) opposed to the fact that call_soon means you never get the nasty surprise where the callback added in the middle of a function gets run before the rest of it does. So I think there are good properties in both cases and given some thought it is probably possible to map between them, but Deferred is lower-level here in the sense that it provides a way to do this both with the event loop and without. You can always call_soon(deferred.callback) but you can't go the other way and force a Future to resolve synchronously - right? -glyph
On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jun 6, 2016, at 08:29, Guido van Rossum <guido@python.org> wrote:
On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jun 4, 2016, at 13:25, Ben Darnell <ben@bendarnell.com> wrote:
If things are so sensitive to minor changes in timing, doesn't that set the bar impossibly high for interoperability?
The sensitivity is not to changes in timing - i.e. when the wall-clock runs, or ordering of non-deterministically ordered events - but rather to reentrancy - whether certain things complete synchronously while the caller is still on the stack and can depend on them having done so upon return.
The recommended way of writing tests within Twisted these days depends heavily on `.callback´ synchronously resolving a Deferred, which is what adding a call_soon breaks.
That's interesting, and also potentially worrisome (for interop, I'm not saying Twisted is wrong here).
I think asyncio depends on the opposite: that if you add a callback to a Future that's ready it does *not* immediately run. Asyncio's promise is pretty strongly that callbacks are serialized (no callbacks running inside other callbacks). IIRC we experimented with other semantics and found that it was harder to reason about. (IMO if you *know* a Future is ready why add a callback to it rather than just calling the damn thing if that's what you want?)
I don't think Twisted is necessarily *right* here either. You're absolutely right that it's easier to reason about reentrancy in some cases if you have a might-be-fired-might-not Future vs. the same sort of Deferred. I like the property where you can do:
def test(test_case): a = asynchronously_something() test_case.this_must_not_have_a_result(a) cause_a_to_fire() test_case.assertEqual(test_case.this_must_have_a_result(a), something)
but this is (somewhat) opposed to the fact that call_soon means you never get the nasty surprise where the callback added in the middle of a function gets run before the rest of it does. So I think there are good properties in both cases and given some thought it is probably possible to map between them, but Deferred is lower-level here in the sense that it provides a way to do this both with the event loop and without. You can always call_soon(deferred.callback) but you can't go the other way and force a Future to resolve synchronously - right?
Right. I'm still unclear on what the compelling use case for that is (other than that Twisted has always done this). Is it performance? Is it callback ordering? I suppose Deferred has a method to mark it done. Does that immediately run the callbacks? Or does it come in two flavors? Can a Deferred that's marked done ever revert back to being not done? (I believe I once read the Deferred code enough to be able to find the answers, but I'm afraid I've never really needed what I learned then, so I've forgotten...) -- --Guido van Rossum (python.org/~guido)
On Jun 6, 2016, at 14:21, Guido van Rossum <guido@python.org> wrote:
On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Jun 6, 2016, at 08:29, Guido van Rossum <guido@python.org <mailto:guido@python.org>> wrote:
On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Jun 4, 2016, at 13:25, Ben Darnell <ben@bendarnell.com <mailto:ben@bendarnell.com>> wrote:
If things are so sensitive to minor changes in timing, doesn't that set the bar impossibly high for interoperability?
The sensitivity is not to changes in timing - i.e. when the wall-clock runs, or ordering of non-deterministically ordered events - but rather to reentrancy - whether certain things complete synchronously while the caller is still on the stack and can depend on them having done so upon return.
The recommended way of writing tests within Twisted these days depends heavily on `.callback´ synchronously resolving a Deferred, which is what adding a call_soon breaks.
That's interesting, and also potentially worrisome (for interop, I'm not saying Twisted is wrong here).
I think asyncio depends on the opposite: that if you add a callback to a Future that's ready it does *not* immediately run. Asyncio's promise is pretty strongly that callbacks are serialized (no callbacks running inside other callbacks). IIRC we experimented with other semantics and found that it was harder to reason about. (IMO if you *know* a Future is ready why add a callback to it rather than just calling the damn thing if that's what you want?)
I don't think Twisted is necessarily right here either. You're absolutely right that it's easier to reason about reentrancy in some cases if you have a might-be-fired-might-not Future vs. the same sort of Deferred. I like the property where you can do:
def test(test_case): a = asynchronously_something() test_case.this_must_not_have_a_result(a) cause_a_to_fire() test_case.assertEqual(test_case.this_must_have_a_result(a), something)
but this is (somewhat) opposed to the fact that call_soon means you never get the nasty surprise where the callback added in the middle of a function gets run before the rest of it does. So I think there are good properties in both cases and given some thought it is probably possible to map between them, but Deferred is lower-level here in the sense that it provides a way to do this both with the event loop and without. You can always call_soon(deferred.callback) but you can't go the other way and force a Future to resolve synchronously - right?
Right. I'm still unclear on what the compelling use case for that is (other than that Twisted has always done this). Is it performance? Is it callback ordering?
Very, very early in the development of Deferreds, they worked the way Futures do; we changed it mainly to reduce coupling to the event loop so that we could test general-purpose algorithms (like gatherResults) without needing to spin an event loop to do it. So the main use-case is testing.
I suppose Deferred has a method to mark it done.
Yep; ".callback".
Does that immediately run the callbacks?
It runs callbacks up to the point that the first one returns a Deferred, and then it waits for that one to be fired to continue running the chain. There is an exception here, where Deferred effectively opts in to Future-like behavior in a very specific case: if you are recursively giving results to a Deferred X that would un-block a Deferred Y inside a callback on Y, Y will not execute its own callbacks reentrantly; it waits until the current callback is done. So while the semantics of .callback() on a Deferred are clear-cut with respect to that Deferred itself, "continue any Deferreds waiting upon it" is a callback-like structure that is slightly squirrely in a very call_soon-like way to avoid surprise reentrancy and RecursionError explosions when having a structure like an asynchronous 'for' loop.
Or does it come in two flavors? Can a Deferred that's marked done ever revert back to being not done?
No. A Deferred that has been called back stays called back; callbacking it again is always an error. However, it may pause running its chain if you return another Deferred in the middle someplace; the way to resume it is to give the inner Deferred a result; the outer one cannot be otherwise compelled to continue.
(I believe I once read the Deferred code enough to be able to find the answers, but I'm afraid I've never really needed what I learned then, so I've forgotten...)
Happy to fill in these blanks; they're (mostly, modulo the weird exception for callbacks-in-callbacks) straightforward :). -glyph
Glyph, Ben, Amber, So what’s the resolution on Future.__isfuture__ and fixing the isinstance(obj, Future) checks from asyncio? 3.5.2 RC is only few days away, I can still make the change if it’s a blocker for Twisted and Tornado. Yury
On Jun 6, 2016, at 5:35 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jun 6, 2016, at 14:21, Guido van Rossum <guido@python.org> wrote:
On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jun 6, 2016, at 08:29, Guido van Rossum <guido@python.org> wrote:
On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jun 4, 2016, at 13:25, Ben Darnell <ben@bendarnell.com> wrote:
If things are so sensitive to minor changes in timing, doesn't that set the bar impossibly high for interoperability?
The sensitivity is not to changes in timing - i.e. when the wall-clock runs, or ordering of non-deterministically ordered events - but rather to reentrancy - whether certain things complete synchronously while the caller is still on the stack and can depend on them having done so upon return.
The recommended way of writing tests within Twisted these days depends heavily on `.callback´ synchronously resolving a Deferred, which is what adding a call_soon breaks.
That's interesting, and also potentially worrisome (for interop, I'm not saying Twisted is wrong here).
I think asyncio depends on the opposite: that if you add a callback to a Future that's ready it does *not* immediately run. Asyncio's promise is pretty strongly that callbacks are serialized (no callbacks running inside other callbacks). IIRC we experimented with other semantics and found that it was harder to reason about. (IMO if you *know* a Future is ready why add a callback to it rather than just calling the damn thing if that's what you want?)
I don't think Twisted is necessarily right here either. You're absolutely right that it's easier to reason about reentrancy in some cases if you have a might-be-fired-might-not Future vs. the same sort of Deferred. I like the property where you can do:
def test(test_case): a = asynchronously_something() test_case.this_must_not_have_a_result(a) cause_a_to_fire() test_case.assertEqual(test_case.this_must_have_a_result(a), something)
but this is (somewhat) opposed to the fact that call_soon means you never get the nasty surprise where the callback added in the middle of a function gets run before the rest of it does. So I think there are good properties in both cases and given some thought it is probably possible to map between them, but Deferred is lower-level here in the sense that it provides a way to do this both with the event loop and without. You can always call_soon(deferred.callback) but you can't go the other way and force a Future to resolve synchronously - right?
Right. I'm still unclear on what the compelling use case for that is (other than that Twisted has always done this). Is it performance? Is it callback ordering?
Very, very early in the development of Deferreds, they worked the way Futures do; we changed it mainly to reduce coupling to the event loop so that we could test general-purpose algorithms (like gatherResults) without needing to spin an event loop to do it. So the main use-case is testing.
I suppose Deferred has a method to mark it done.
Yep; ".callback".
Does that immediately run the callbacks?
It runs callbacks up to the point that the first one returns a Deferred, and then it waits for that one to be fired to continue running the chain.
There is an exception here, where Deferred effectively opts in to Future-like behavior in a very specific case: if you are recursively giving results to a Deferred X that would un-block a Deferred Y inside a callback on Y, Y will not execute its own callbacks reentrantly; it waits until the current callback is done.
So while the semantics of .callback() on a Deferred are clear-cut with respect to that Deferred itself, "continue any Deferreds waiting upon it" is a callback-like structure that is slightly squirrely in a very call_soon-like way to avoid surprise reentrancy and RecursionError explosions when having a structure like an asynchronous 'for' loop.
Or does it come in two flavors? Can a Deferred that's marked done ever revert back to being not done?
No. A Deferred that has been called back stays called back; callbacking it again is always an error. However, it may pause running its chain if you return another Deferred in the middle someplace; the way to resume it is to give the inner Deferred a result; the outer one cannot be otherwise compelled to continue.
(I believe I once read the Deferred code enough to be able to find the answers, but I'm afraid I've never really needed what I learned then, so I've forgotten...)
Happy to fill in these blanks; they're (mostly, modulo the weird exception for callbacks-in-callbacks) straightforward :).
-glyph
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
On Wed, Jun 8, 2016 at 1:52 PM, Yury Selivanov <yselivanov@gmail.com> wrote:
Glyph, Ben, Amber,
So what’s the resolution on Future.__isfuture__ and fixing the isinstance(obj, Future) checks from asyncio?
3.5.2 RC is only few days away, I can still make the change if it’s a blocker for Twisted and Tornado.
None of this is blocking Tornado - we shipped asyncio integration six months ago. There's some room for improvement, but I don't think there's a clear enough mandate to squeeze something in for this release. I'd rather take the time to sort out a more complete plan before the next release. -Ben
Yury
On Jun 6, 2016, at 5:35 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jun 6, 2016, at 14:21, Guido van Rossum <guido@python.org> wrote:
On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jun 6, 2016, at 08:29, Guido van Rossum <guido@python.org> wrote:
On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph@twistedmatrix.com>
On Jun 4, 2016, at 13:25, Ben Darnell <ben@bendarnell.com> wrote:
If things are so sensitive to minor changes in timing, doesn't that
set the
bar impossibly high for interoperability?
The sensitivity is not to changes in timing - i.e. when the wall-clock runs, or ordering of non-deterministically ordered events - but rather to reentrancy - whether certain things complete synchronously while the caller is still on the stack and can depend on them having done so upon return.
The recommended way of writing tests within Twisted these days depends heavily on `.callback´ synchronously resolving a Deferred, which is what adding a call_soon breaks.
That's interesting, and also potentially worrisome (for interop, I'm not saying Twisted is wrong here).
I think asyncio depends on the opposite: that if you add a callback to a Future that's ready it does *not* immediately run. Asyncio's promise is pretty strongly that callbacks are serialized (no callbacks running inside other callbacks). IIRC we experimented with other semantics and found that it was harder to reason about. (IMO if you *know* a Future is ready why add a callback to it rather than just calling the damn thing if that's what you want?)
I don't think Twisted is necessarily right here either. You're absolutely right that it's easier to reason about reentrancy in some cases if you have a might-be-fired-might-not Future vs. the same sort of Deferred. I like the property where you can do:
def test(test_case): a = asynchronously_something() test_case.this_must_not_have_a_result(a) cause_a_to_fire() test_case.assertEqual(test_case.this_must_have_a_result(a), something)
but this is (somewhat) opposed to the fact that call_soon means you never get the nasty surprise where the callback added in the middle of a function gets run before the rest of it does. So I think there are good
wrote: properties in both cases and given some thought it is probably possible to map between them, but Deferred is lower-level here in the sense that it provides a way to do this both with the event loop and without. You can always call_soon(deferred.callback) but you can't go the other way and force a Future to resolve synchronously - right?
Right. I'm still unclear on what the compelling use case for that is
(other than that Twisted has always done this). Is it performance? Is it callback ordering?
Very, very early in the development of Deferreds, they worked the way Futures do; we changed it mainly to reduce coupling to the event loop so that we could test general-purpose algorithms (like gatherResults) without needing to spin an event loop to do it. So the main use-case is testing.
I suppose Deferred has a method to mark it done.
Yep; ".callback".
Does that immediately run the callbacks?
It runs callbacks up to the point that the first one returns a Deferred, and then it waits for that one to be fired to continue running the chain.
There is an exception here, where Deferred effectively opts in to Future-like behavior in a very specific case: if you are recursively giving results to a Deferred X that would un-block a Deferred Y inside a callback on Y, Y will not execute its own callbacks reentrantly; it waits until the current callback is done.
So while the semantics of .callback() on a Deferred are clear-cut with respect to that Deferred itself, "continue any Deferreds waiting upon it" is a callback-like structure that is slightly squirrely in a very call_soon-like way to avoid surprise reentrancy and RecursionError explosions when having a structure like an asynchronous 'for' loop.
Or does it come in two flavors? Can a Deferred that's marked done ever revert back to being not done?
No. A Deferred that has been called back stays called back; callbacking it again is always an error. However, it may pause running its chain if you return another Deferred in the middle someplace; the way to resume it is to give the inner Deferred a result; the outer one cannot be otherwise compelled to continue.
(I believe I once read the Deferred code enough to be able to find the answers, but I'm afraid I've never really needed what I learned then, so I've forgotten...)
Happy to fill in these blanks; they're (mostly, modulo the weird exception for callbacks-in-callbacks) straightforward :).
-glyph
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
Similarly, txtulip has existed for some time. It would definitely be more performant and simpler if we could skip the subclass check, but we don't need this to happen in this cycle. -glyph
On Jun 8, 2016, at 11:17 AM, Ben Darnell <ben@bendarnell.com> wrote:
On Wed, Jun 8, 2016 at 1:52 PM, Yury Selivanov <yselivanov@gmail.com <mailto:yselivanov@gmail.com>> wrote: Glyph, Ben, Amber,
So what’s the resolution on Future.__isfuture__ and fixing the isinstance(obj, Future) checks from asyncio?
3.5.2 RC is only few days away, I can still make the change if it’s a blocker for Twisted and Tornado.
None of this is blocking Tornado - we shipped asyncio integration six months ago. There's some room for improvement, but I don't think there's a clear enough mandate to squeeze something in for this release. I'd rather take the time to sort out a more complete plan before the next release.
-Ben
Yury
On Jun 6, 2016, at 5:35 PM, Glyph <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Jun 6, 2016, at 14:21, Guido van Rossum <guido@python.org <mailto:guido@python.org>> wrote:
On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Jun 6, 2016, at 08:29, Guido van Rossum <guido@python.org <mailto:guido@python.org>> wrote:
On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Jun 4, 2016, at 13:25, Ben Darnell <ben@bendarnell.com <mailto:ben@bendarnell.com>> wrote:
If things are so sensitive to minor changes in timing, doesn't that set the bar impossibly high for interoperability?
The sensitivity is not to changes in timing - i.e. when the wall-clock runs, or ordering of non-deterministically ordered events - but rather to reentrancy - whether certain things complete synchronously while the caller is still on the stack and can depend on them having done so upon return.
The recommended way of writing tests within Twisted these days depends heavily on `.callback´ synchronously resolving a Deferred, which is what adding a call_soon breaks.
That's interesting, and also potentially worrisome (for interop, I'm not saying Twisted is wrong here).
I think asyncio depends on the opposite: that if you add a callback to a Future that's ready it does *not* immediately run. Asyncio's promise is pretty strongly that callbacks are serialized (no callbacks running inside other callbacks). IIRC we experimented with other semantics and found that it was harder to reason about. (IMO if you *know* a Future is ready why add a callback to it rather than just calling the damn thing if that's what you want?)
I don't think Twisted is necessarily right here either. You're absolutely right that it's easier to reason about reentrancy in some cases if you have a might-be-fired-might-not Future vs. the same sort of Deferred. I like the property where you can do:
def test(test_case): a = asynchronously_something() test_case.this_must_not_have_a_result(a) cause_a_to_fire() test_case.assertEqual(test_case.this_must_have_a_result(a), something)
but this is (somewhat) opposed to the fact that call_soon means you never get the nasty surprise where the callback added in the middle of a function gets run before the rest of it does. So I think there are good properties in both cases and given some thought it is probably possible to map between them, but Deferred is lower-level here in the sense that it provides a way to do this both with the event loop and without. You can always call_soon(deferred.callback) but you can't go the other way and force a Future to resolve synchronously - right?
Right. I'm still unclear on what the compelling use case for that is (other than that Twisted has always done this). Is it performance? Is it callback ordering?
Very, very early in the development of Deferreds, they worked the way Futures do; we changed it mainly to reduce coupling to the event loop so that we could test general-purpose algorithms (like gatherResults) without needing to spin an event loop to do it. So the main use-case is testing.
I suppose Deferred has a method to mark it done.
Yep; ".callback".
Does that immediately run the callbacks?
It runs callbacks up to the point that the first one returns a Deferred, and then it waits for that one to be fired to continue running the chain.
There is an exception here, where Deferred effectively opts in to Future-like behavior in a very specific case: if you are recursively giving results to a Deferred X that would un-block a Deferred Y inside a callback on Y, Y will not execute its own callbacks reentrantly; it waits until the current callback is done.
So while the semantics of .callback() on a Deferred are clear-cut with respect to that Deferred itself, "continue any Deferreds waiting upon it" is a callback-like structure that is slightly squirrely in a very call_soon-like way to avoid surprise reentrancy and RecursionError explosions when having a structure like an asynchronous 'for' loop.
Or does it come in two flavors? Can a Deferred that's marked done ever revert back to being not done?
No. A Deferred that has been called back stays called back; callbacking it again is always an error. However, it may pause running its chain if you return another Deferred in the middle someplace; the way to resume it is to give the inner Deferred a result; the outer one cannot be otherwise compelled to continue.
(I believe I once read the Deferred code enough to be able to find the answers, but I'm afraid I've never really needed what I learned then, so I've forgotten...)
Happy to fill in these blanks; they're (mostly, modulo the weird exception for callbacks-in-callbacks) straightforward :).
-glyph
_______________________________________________ Async-sig mailing list Async-sig@python.org <mailto:Async-sig@python.org> https://mail.python.org/mailman/listinfo/async-sig <https://mail.python.org/mailman/listinfo/async-sig> Code of Conduct: https://www.python.org/psf/codeofconduct/ <https://www.python.org/psf/codeofconduct/>
_______________________________________________ Async-sig mailing list Async-sig@python.org <mailto:Async-sig@python.org> https://mail.python.org/mailman/listinfo/async-sig <https://mail.python.org/mailman/listinfo/async-sig> Code of Conduct: https://www.python.org/psf/codeofconduct/ <https://www.python.org/psf/codeofconduct/>
On Fri, Jun 3, 2016 at 8:46 PM, Łukasz Langa <lukasz@langa.pl> wrote:
Amber, Glyph, Ben, is asyncio.Future’s event loop callback scheduling the biggest barrier in interoperability at the moment?
For Tornado, this was not an issue. Things are perhaps not as efficient as they could be (and this is why tornado.concurrent.Future cannot be an alias for asyncio.Future), but it basically just works. The biggest problem that arises when people use Tornado and asyncio together is that some asyncio functions (such as asyncio.gather()) are written to accept bare coroutines and run them in asyncio.Task. This works in a pure asyncio world, but fails in an application that tries to mix the two frameworks. Since Tornado's coroutine runner is essentially a superset of asyncio's, the workaround is simple: use the Tornado version of any functions that exhibit this problem (e.g. tornado.gen.multi() instead of asyncio.gather()). And while I'm happy to promote Tornado's more flexible coroutine runner over asyncio's, it's clearly problematic for standardization and interop. This approach only works so long as there is some coroutine runner that is a superset of all the others. The root of the problem is that without a standardized coroutine runner, async functions are difficult to use: each asynchronous function embeds assumptions about what runner will be used, and these assumptions will need to be documented along with instructions for what to do when you need to cross runner boundaries. We kind of have a standardized coroutine runner in async.Task, but it's inflexible, dealing only in asyncio.Futures. I think the right thing to do is to give asyncio a functools.singledispatch-based hook like the one in Tornado (tornado.gen.convert_yielded) for converting yielded objects into Futures. If we could register Tornado's Futures with asyncio then I don't think there would be a reason to prefer Tornado's coroutine runner when asyncio is available. -Ben
On Jun 4, 2016, at 5:06 PM, Ben Darnell <ben@bendarnell.com> wrote:
The root of the problem is that without a standardized coroutine runner, async functions are difficult to use: each asynchronous function embeds assumptions about what runner will be used, and these assumptions will need to be documented along with instructions for what to do when you need to cross runner boundaries. We kind of have a standardized coroutine runner in async.Task, but it's inflexible, dealing only in asyncio.Futures. I think the right thing to do is to give asyncio a functools.singledispatch-based hook like the one in Tornado (tornado.gen.convert_yielded) for converting yielded objects into Futures. If we could register Tornado's Futures with asyncio then I don't think there would be a reason to prefer Tornado's coroutine runner when asyncio is available.
Looks like you, Glyph and I should work on a PEP to standardize asyncio.Task and make it more flexible. I can draft a first version (in a couple of weeks if that works), that will basically describe the current Task implementation, but I’ll need your help to make it useful for Tornado and Twisted. Thanks, Yury
participants (6)
-
Ben Darnell -
Cory Benfield -
Glyph -
Guido van Rossum -
Yury Selivanov -
Łukasz Langa