[Async-sig] asyncio <-> twisted interoperability

[Yury Selivanov]

Glyph, Ben, Amber,

So what’s the resolution on Future.__isfuture__ and fixing the
isinstance(obj, Future) checks from asyncio?

3.5.2 RC is only few days away, I can still make the change if
it’s a blocker for Twisted and Tornado.

Yury

> On Jun 6, 2016, at 5:35 PM, Glyph <glyph at twistedmatrix.com> wrote:
> 
>> 
>> On Jun 6, 2016, at 14:21, Guido van Rossum <guido at python.org> wrote:
>> 
>> On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph at twistedmatrix.com> wrote:
>> 
>> 
>>> On Jun 6, 2016, at 08:29, Guido van Rossum <guido at python.org> wrote:
>>> 
>>> On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph at twistedmatrix.com> wrote:
>>>> 
>>>> On Jun 4, 2016, at 13:25, Ben Darnell <ben at bendarnell.com> wrote:
>>>> 
>>>> If things are so sensitive to minor changes in timing, doesn't that set the
>>>> bar impossibly high for interoperability?
>>>> 
>>>> 
>>>> The sensitivity is not to changes in timing - i.e. when the wall-clock runs,
>>>> or ordering of non-deterministically ordered events - but rather to
>>>> reentrancy - whether certain things complete synchronously while the caller
>>>> is still on the stack and can depend on them having done so upon return.
>>>> 
>>>> The recommended way of writing tests within Twisted these days depends
>>>> heavily on `.callback´ synchronously resolving a Deferred, which is what
>>>> adding a call_soon breaks.
>>> 
>>> That's interesting, and also potentially worrisome (for interop, I'm
>>> not saying Twisted is wrong here).
>>> 
>>> I think asyncio depends on the opposite: that if you add a callback to
>>> a Future that's ready it does *not* immediately run. Asyncio's promise
>>> is pretty strongly that callbacks are serialized (no callbacks running
>>> inside other callbacks). IIRC we experimented with other semantics and
>>> found that it was harder to reason about. (IMO if you *know* a Future
>>> is ready why add a callback to it rather than just calling the damn
>>> thing if that's what you want?)
>> 
>> I don't think Twisted is necessarily right here either.  You're absolutely right that it's easier to reason about reentrancy in some cases if you have a might-be-fired-might-not Future vs. the same sort of Deferred.  I like the property where you can do:
>> 
>> def test(test_case):
>>     a = asynchronously_something()
>>     test_case.this_must_not_have_a_result(a)
>>     cause_a_to_fire()
>>     test_case.assertEqual(test_case.this_must_have_a_result(a), something)
>> 
>> but this is (somewhat) opposed to the fact that call_soon means you never get the nasty surprise where the callback added in the middle of a function gets run before the rest of it does.  So I think there are good properties in both cases and given some thought it is probably possible to map between them, but Deferred is lower-level here in the sense that it provides a way to do this both with the event loop and without.  You can always call_soon(deferred.callback) but you can't go the other way and force a Future to resolve synchronously - right?
>> 
>> Right. I'm still unclear on what the compelling use case for that is (other than that Twisted has always done this). Is it performance? Is it callback ordering?
> 
> Very, very early in the development of Deferreds, they worked the way Futures do; we changed it mainly to reduce coupling to the event loop so that we could test general-purpose algorithms (like gatherResults) without needing to spin an event loop to do it.  So the main use-case is testing.
> 
>> I suppose Deferred has a method to mark it done.
> 
> Yep; ".callback".
> 
>> Does that immediately run the callbacks?
> 
> It runs callbacks up to the point that the first one returns a Deferred, and then it waits for that one to be fired to continue running the chain.
> 
> There is an exception here, where Deferred effectively opts in to Future-like behavior in a very specific case: if you are recursively giving results to a Deferred X that would un-block a Deferred Y inside a callback on Y, Y will not execute its own callbacks reentrantly; it waits until the current callback is done.
> 
> So while the semantics of .callback() on a Deferred are clear-cut with respect to that Deferred itself, "continue any Deferreds waiting upon it" is a callback-like structure that is slightly squirrely in a very call_soon-like way to avoid surprise reentrancy and RecursionError explosions when having a structure like an asynchronous 'for' loop.
> 
>> Or does it come in two flavors? Can a Deferred that's marked done ever revert back to being not done?
> 
> No.  A Deferred that has been called back stays called back; callbacking it again is always an error.  However, it may pause running its chain if you return another Deferred in the middle someplace; the way to resume it is to give the inner Deferred a result; the outer one cannot be otherwise compelled to continue.
> 
>> (I believe I once read the Deferred code enough to be able to find the answers, but I'm afraid I've never really needed what I learned then, so I've forgotten...)
> 
> Happy to fill in these blanks; they're (mostly, modulo the weird exception for callbacks-in-callbacks) straightforward :).
> 
> -glyph
> 
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/