On Jun 6, 2016, at 14:21, Guido van Rossum <guido@python.org> wrote:

On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph@twistedmatrix.com> wrote:

On Jun 6, 2016, at 08:29, Guido van Rossum <guido@python.org> wrote:

On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph@twistedmatrix.com> wrote:

On Jun 4, 2016, at 13:25, Ben Darnell <ben@bendarnell.com> wrote:

If things are so sensitive to minor changes in timing, doesn't that set the
bar impossibly high for interoperability?

The sensitivity is not to changes in timing - i.e. when the wall-clock runs,
or ordering of non-deterministically ordered events - but rather to
reentrancy - whether certain things complete synchronously while the caller
is still on the stack and can depend on them having done so upon return.

The recommended way of writing tests within Twisted these days depends
heavily on `.callback´ synchronously resolving a Deferred, which is what
adding a call_soon breaks.

That's interesting, and also potentially worrisome (for interop, I'm
not saying Twisted is wrong here).

I think asyncio depends on the opposite: that if you add a callback to
a Future that's ready it does *not* immediately run. Asyncio's promise
is pretty strongly that callbacks are serialized (no callbacks running
inside other callbacks). IIRC we experimented with other semantics and
found that it was harder to reason about. (IMO if you *know* a Future
is ready why add a callback to it rather than just calling the damn
thing if that's what you want?)

I don't think Twisted is necessarily right here either. You're absolutely right that it's easier to reason about reentrancy in some cases if you have a might-be-fired-might-not Future vs. the same sort of Deferred. I like the property where you can do:

def test(test_case):
  a = asynchronously_something()
  test_case.this_must_not_have_a_result(a)
  cause_a_to_fire()
  test_case.assertEqual(test_case.this_must_have_a_result(a), something)

but this is (somewhat) opposed to the fact that call_soon means you never get the nasty surprise where the callback added in the middle of a function gets run before the rest of it does. So I think there are good properties in both cases and given some thought it is probably possible to map between them, but Deferred is lower-level here in the sense that it provides a way to do this both with the event loop and without. You can always call_soon(deferred.callback) but you can't go the other way and force a Future to resolve synchronously - right?

Right. I'm still unclear on what the compelling use case for that is (other than that Twisted has always done this). Is it performance? Is it callback ordering?

Very, very early in the development of Deferreds, they worked the way Futures do; we changed it mainly to reduce coupling to the event loop so that we could test general-purpose algorithms (like gatherResults) without needing to spin an event loop to do it. So the main use-case is testing.

I suppose Deferred has a method to mark it done.

Yep; ".callback".

Does that immediately run the callbacks?

It runs callbacks up to the point that the first one returns a Deferred, and then it waits for that one to be fired to continue running the chain.

There is an exception here, where Deferred effectively opts in to Future-like behavior in a very specific case: if you are recursively giving results to a Deferred X that would un-block a Deferred Y inside a callback on Y, Y will not execute its own callbacks reentrantly; it waits until the current callback is done.

So while the semantics of .callback() on a Deferred are clear-cut with respect to that Deferred itself, "continue any Deferreds waiting upon it" is a callback-like structure that is slightly squirrely in a very call_soon-like way to avoid surprise reentrancy and RecursionError explosions when having a structure like an asynchronous 'for' loop.

Or does it come in two flavors? Can a Deferred that's marked done ever revert back to being not done?

No. A Deferred that has been called back stays called back; callbacking it again is always an error. However, it may pause running its chain if you return another Deferred in the middle someplace; the way to resume it is to give the inner Deferred a result; the outer one cannot be otherwise compelled to continue.

(I believe I once read the Deferred code enough to be able to find the answers, but I'm afraid I've never really needed what I learned then, so I've forgotten...)