[Async-sig] asyncio <-> twisted interoperability

Wed Jun 8 14:17:54 EDT 2016

On Wed, Jun 8, 2016 at 1:52 PM, Yury Selivanov <yselivanov at gmail.com> wrote:

> Glyph, Ben, Amber,
>
> So what’s the resolution on Future.__isfuture__ and fixing the
> isinstance(obj, Future) checks from asyncio?
>
> 3.5.2 RC is only few days away, I can still make the change if
> it’s a blocker for Twisted and Tornado.
>

None of this is blocking Tornado - we shipped asyncio integration six
months ago. There's some room for improvement, but I don't think there's a
clear enough mandate to squeeze something in for this release. I'd rather
take the time to sort out a more complete plan before the next release.

-Ben

>
> Yury
>
> > On Jun 6, 2016, at 5:35 PM, Glyph <glyph at twistedmatrix.com> wrote:
> >
> >>
> >> On Jun 6, 2016, at 14:21, Guido van Rossum <guido at python.org> wrote:
> >>
> >> On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph at twistedmatrix.com> wrote:
> >>
> >>
> >>> On Jun 6, 2016, at 08:29, Guido van Rossum <guido at python.org> wrote:
> >>>
> >>> On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph at twistedmatrix.com>
> wrote:
> >>>>
> >>>> On Jun 4, 2016, at 13:25, Ben Darnell <ben at bendarnell.com> wrote:
> >>>>
> >>>> If things are so sensitive to minor changes in timing, doesn't that
> set the
> >>>> bar impossibly high for interoperability?
> >>>>
> >>>>
> >>>> The sensitivity is not to changes in timing - i.e. when the
> wall-clock runs,
> >>>> or ordering of non-deterministically ordered events - but rather to
> >>>> reentrancy - whether certain things complete synchronously while the
> caller
> >>>> is still on the stack and can depend on them having done so upon
> return.
> >>>>
> >>>> The recommended way of writing tests within Twisted these days depends
> >>>> heavily on `.callback´ synchronously resolving a Deferred, which is
> what
> >>>> adding a call_soon breaks.
> >>>
> >>> That's interesting, and also potentially worrisome (for interop, I'm
> >>> not saying Twisted is wrong here).
> >>>
> >>> I think asyncio depends on the opposite: that if you add a callback to
> >>> a Future that's ready it does *not* immediately run. Asyncio's promise
> >>> is pretty strongly that callbacks are serialized (no callbacks running
> >>> inside other callbacks). IIRC we experimented with other semantics and
> >>> found that it was harder to reason about. (IMO if you *know* a Future
> >>> is ready why add a callback to it rather than just calling the damn
> >>> thing if that's what you want?)
> >>
> >> I don't think Twisted is necessarily right here either.  You're
> absolutely right that it's easier to reason about reentrancy in some cases
> if you have a might-be-fired-might-not Future vs. the same sort of
> Deferred.  I like the property where you can do:
> >>
> >> def test(test_case):
> >>     a = asynchronously_something()
> >>     test_case.this_must_not_have_a_result(a)
> >>     cause_a_to_fire()
> >>     test_case.assertEqual(test_case.this_must_have_a_result(a),
> something)
> >>
> >> but this is (somewhat) opposed to the fact that call_soon means you
> never get the nasty surprise where the callback added in the middle of a
> function gets run before the rest of it does.  So I think there are good
> properties in both cases and given some thought it is probably possible to
> map between them, but Deferred is lower-level here in the sense that it
> provides a way to do this both with the event loop and without.  You can
> always call_soon(deferred.callback) but you can't go the other way and
> force a Future to resolve synchronously - right?
> >>
> >> Right. I'm still unclear on what the compelling use case for that is
> (other than that Twisted has always done this). Is it performance? Is it
> callback ordering?
> >
> > Very, very early in the development of Deferreds, they worked the way
> Futures do; we changed it mainly to reduce coupling to the event loop so
> that we could test general-purpose algorithms (like gatherResults) without
> needing to spin an event loop to do it.  So the main use-case is testing.
> >
> >> I suppose Deferred has a method to mark it done.
> >
> > Yep; ".callback".
> >
> >> Does that immediately run the callbacks?
> >
> > It runs callbacks up to the point that the first one returns a Deferred,
> and then it waits for that one to be fired to continue running the chain.
> >
> > There is an exception here, where Deferred effectively opts in to
> Future-like behavior in a very specific case: if you are recursively giving
> results to a Deferred X that would un-block a Deferred Y inside a callback
> on Y, Y will not execute its own callbacks reentrantly; it waits until the
> current callback is done.
> >
> > So while the semantics of .callback() on a Deferred are clear-cut with
> respect to that Deferred itself, "continue any Deferreds waiting upon it"
> is a callback-like structure that is slightly squirrely in a very
> call_soon-like way to avoid surprise reentrancy and RecursionError
> explosions when having a structure like an asynchronous 'for' loop.
> >
> >> Or does it come in two flavors? Can a Deferred that's marked done ever
> revert back to being not done?
> >
> > No.  A Deferred that has been called back stays called back; callbacking
> it again is always an error.  However, it may pause running its chain if
> you return another Deferred in the middle someplace; the way to resume it
> is to give the inner Deferred a result; the outer one cannot be otherwise
> compelled to continue.
> >
> >> (I believe I once read the Deferred code enough to be able to find the
> answers, but I'm afraid I've never really needed what I learned then, so
> I've forgotten...)
> >
> > Happy to fill in these blanks; they're (mostly, modulo the weird
> exception for callbacks-in-callbacks) straightforward :).
> >
> > -glyph
> >
> > _______________________________________________
> > Async-sig mailing list
> > Async-sig at python.org
> > https://mail.python.org/mailman/listinfo/async-sig
> > Code of Conduct: https://www.python.org/psf/codeofconduct/
>
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160608/33480cb6/attachment-0001.html>