On Oct 18, 2016, at 4:05 PM, Kevin Conway <kevinjacobconway@gmail.com> wrote:
This is why twisted.python.context came to exist in the first place; I always wanted to attach it to Deferred somehow
Well, it's not something we've announced yet through any official channel, but we had to solve the context propagation problem at Atlassian to instrument our services with traceable logging. We open sourced our solution at https://bitbucket.org/hipchat/txlocal <https://bitbucket.org/hipchat/txlocal>. The answer for us was an extension for the reactor, thread pool, and inline callbacks that maintain the needed state. There's a readme with some insight into how we tool our services.
This is (A) very cool, and (B) making such aggressive use of private APIs that it could win a contest about how to ensure that you break on every new release of Twisted :). I'm super impressed that you tracked the introduction of twisted._threads and support both old- and new-style thread pools!
We've had it on our backlog to address the mailing list and, possibly even, discuss what it would take to put this into Twisted. I guess now is as good if a time as any. Feel free to spin off a another thread or reach out to me off list with any questions or feedback
This seems like as good a time to talk about it as any! Integrating this into the core in some fashion would be good, but I imagine that it's a non-trivial impact to performance, so it would be worthwhile to track that. Speaking of performance - I found the long digression on "Don't Switch In The Core" interesting, since we actually _do_ tracking this kind of context already, for logging. I was a little surprised you didn't integrate with this at all. To be fair, this is something that our friends over at PyPy have been bugging us about since forever; when you're benchmarking raw wire speed it does tend to show up in profiling. I also take it from the performance notes that you're not using PyPy? __slots__ shouldn't make much of a difference there. (In fact I'm given to believe it's a slight _decrease_ in performance on pypy...) -glyph
On Tue, Oct 18, 2016, 14:47 Glyph Lefkowitz <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Oct 18, 2016, at 5:50 AM, Itamar Turner-Trauring <itamar@itamarst.org <mailto:itamar@itamarst.org>> wrote:
Not been doing much Twisted lately, but have been doing async stuff elsewhere, and I've learned some useful things.
Thanks for writing these up, Itamar! This sort of reflection is rare and it's always helpful :).
1. Callbacks should be sync or async, but never sometimes-one-sometimes-the-other. For details go read http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/ <http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/>. For example, Deferred.addCallback(f) really should never run f() immediately.
This has come up a lot in a compare-and-contrast of Twisted vs. asyncio.
I agree that the problems with synchronous callbacks are not insignificant (reentrancy is a degenerate form of preemption, and as we all know preemption is the corrupt wellspring of all bugs). However, the benefit, i.e. consistency of behavior with respect to reentrancy, comes with a cost: tight coupling to an event loop. In asyncio, Future's tight coupling to call_soon is a source of problems; it makes it hard to write a test without setting up an elaborate scheduling trampoline, whereas successResultOf/failureResultOf are quite simple to work with.
I think Deferred as it is today is a pretty good compromise between the two positions. On the one hand it is decoupled from the event loop. On the other - and this is important - no Deferred-returning API will ever call your callbacks synchronously. Deferred.addCallback will, of course, but savvy Twisted programmers can (and should) do this, if they have dependent state changes:
self.manipulateSomeStateForSetup() d = doSomethingPotentiallySynchronous() self.manipulateSomeStateForProcessing() d.addCallback(completeOperation)
As a caller, you can always decide whether you can safely be re-entered or not. In most cases, simply moving the 'addCallback' to the end of the function (a-la Go's "defer", oddly enough) is fine. In more complex cases where you really need to unwind reentrancy completely, you can do your own callLater(0) or callFromThread() from an object with a reference to a reactor.
3.
What happened to '2'? :)
By instrumenting all callbacks it manages, which may or may not require item #1, Twisted can have a context that automatically follows callbacks. Node has this and it is extremely useful. http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls... <http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls...> is best summary I've found with a bit of searching.
This was _always_ supposed to be the way that Twisted worked, but frankly I just wasn't smart enough to figure it out. This is why twisted.python.context came to exist in the first place; I always wanted to attach it to Deferred somehow. I will watch this talk intently; if #1 really is required to address this, my opinion might change. A PR would be intensely appreciated.
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python <http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python> _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python