[Twisted-Python] Some things I've learned: safer callbacks, better t.p.context
Not been doing much Twisted lately, but have been doing async stuff elsewhere, and I've learned some useful things. 1. Callbacks should be sync or async, but never sometimes-one-sometimes-the-other. For details go read http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/. For example, Deferred.addCallback(f) really should never run f() immediately. 3. By instrumenting all callbacks it manages, which may or may not require item #1, Twisted can have a context that automatically follows callbacks. Node has this and it is extremely useful. http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls... is best summary I've found with a bit of searching. -- Itamar Turner-Trauring
On Oct 18, 2016, at 5:50 AM, Itamar Turner-Trauring <itamar@itamarst.org> wrote:
Not been doing much Twisted lately, but have been doing async stuff elsewhere, and I've learned some useful things.
Thanks for writing these up, Itamar! This sort of reflection is rare and it's always helpful :).
1. Callbacks should be sync or async, but never sometimes-one-sometimes-the-other. For details go read http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/. For example, Deferred.addCallback(f) really should never run f() immediately.
This has come up a lot in a compare-and-contrast of Twisted vs. asyncio. I agree that the problems with synchronous callbacks are not insignificant (reentrancy is a degenerate form of preemption, and as we all know preemption is the corrupt wellspring of all bugs). However, the benefit, i.e. consistency of behavior with respect to reentrancy, comes with a cost: tight coupling to an event loop. In asyncio, Future's tight coupling to call_soon is a source of problems; it makes it hard to write a test without setting up an elaborate scheduling trampoline, whereas successResultOf/failureResultOf are quite simple to work with. I think Deferred as it is today is a pretty good compromise between the two positions. On the one hand it is decoupled from the event loop. On the other - and this is important - no Deferred-returning API will ever call your callbacks synchronously. Deferred.addCallback will, of course, but savvy Twisted programmers can (and should) do this, if they have dependent state changes: self.manipulateSomeStateForSetup() d = doSomethingPotentiallySynchronous() self.manipulateSomeStateForProcessing() d.addCallback(completeOperation) As a caller, you can always decide whether you can safely be re-entered or not. In most cases, simply moving the 'addCallback' to the end of the function (a-la Go's "defer", oddly enough) is fine. In more complex cases where you really need to unwind reentrancy completely, you can do your own callLater(0) or callFromThread() from an object with a reference to a reactor.
3.
What happened to '2'? :)
By instrumenting all callbacks it manages, which may or may not require item #1, Twisted can have a context that automatically follows callbacks. Node has this and it is extremely useful. http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls... is best summary I've found with a bit of searching.
This was _always_ supposed to be the way that Twisted worked, but frankly I just wasn't smart enough to figure it out. This is why twisted.python.context came to exist in the first place; I always wanted to attach it to Deferred somehow. I will watch this talk intently; if #1 really is required to address this, my opinion might change. A PR would be intensely appreciated. -glyph
This is why twisted.python.context came to exist in the first place; I always wanted to attach it to Deferred somehow
Well, it's not something we've announced yet through any official channel, but we had to solve the context propagation problem at Atlassian to instrument our services with traceable logging. We open sourced our solution at https://bitbucket.org/hipchat/txlocal. The answer for us was an extension for the reactor, thread pool, and inline callbacks that maintain the needed state. There's a readme with some insight into how we tool our services. We've had it on our backlog to address the mailing list and, possibly even, discuss what it would take to put this into Twisted. I guess now is as good if a time as any. Feel free to spin off a another thread or reach out to me off list with any questions or feedback. On Tue, Oct 18, 2016, 14:47 Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Oct 18, 2016, at 5:50 AM, Itamar Turner-Trauring <itamar@itamarst.org> wrote:
Not been doing much Twisted lately, but have been doing async stuff elsewhere, and I've learned some useful things.
Thanks for writing these up, Itamar! This sort of reflection is rare and it's always helpful :).
1. Callbacks should be sync or async, but never sometimes-one-sometimes-the-other. For details go read http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/. For example, Deferred.addCallback(f) really should never run f() immediately.
This has come up a lot in a compare-and-contrast of Twisted vs. asyncio.
I agree that the problems with synchronous callbacks are not insignificant (reentrancy is a degenerate form of preemption, and as we all know preemption is the corrupt wellspring of all bugs). However, the benefit, i.e. consistency of behavior with respect to reentrancy, comes with a cost: tight coupling to an event loop. In asyncio, Future's tight coupling to call_soon is a source of problems; it makes it hard to write a test without setting up an elaborate scheduling trampoline, whereas successResultOf/failureResultOf are quite simple to work with.
I think Deferred as it is today is a pretty good compromise between the two positions. On the one hand it is decoupled from the event loop. On the other - and this is important - *no Deferred-returning API will ever call your callbacks synchronously*. Deferred.addCallback will, of course, but savvy Twisted programmers can (and should) do this, if they have dependent state changes:
self.manipulateSomeStateForSetup()
d = doSomethingPotentiallySynchronous() *self.manipulateSomeStateForProcessing()* d.addCallback(completeOperation)
As a caller, you can always decide whether you can safely be re-entered or not. In most cases, simply moving the 'addCallback' to the end of the function (a-la Go's "defer", oddly enough) is fine. In more complex cases where you really need to unwind reentrancy completely, you can do your own callLater(0) or callFromThread() from an object with a reference to a reactor.
3.
What happened to '2'? :)
By instrumenting all callbacks it manages, which may or may not require item #1, Twisted can have a context that automatically follows callbacks. Node has this and it is extremely useful.
http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls... is best summary I've found with a bit of searching.
This was _always_ supposed to be the way that Twisted worked, but frankly I just wasn't smart enough to figure it out. This is why twisted.python.context came to exist in the first place; I always wanted to attach it to Deferred somehow. I will watch this talk intently; if #1 really is required to address this, my opinion might change. A PR would be intensely appreciated.
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Oct 18, 2016, at 4:05 PM, Kevin Conway <kevinjacobconway@gmail.com> wrote:
This is why twisted.python.context came to exist in the first place; I always wanted to attach it to Deferred somehow
Well, it's not something we've announced yet through any official channel, but we had to solve the context propagation problem at Atlassian to instrument our services with traceable logging. We open sourced our solution at https://bitbucket.org/hipchat/txlocal <https://bitbucket.org/hipchat/txlocal>. The answer for us was an extension for the reactor, thread pool, and inline callbacks that maintain the needed state. There's a readme with some insight into how we tool our services.
This is (A) very cool, and (B) making such aggressive use of private APIs that it could win a contest about how to ensure that you break on every new release of Twisted :). I'm super impressed that you tracked the introduction of twisted._threads and support both old- and new-style thread pools!
We've had it on our backlog to address the mailing list and, possibly even, discuss what it would take to put this into Twisted. I guess now is as good if a time as any. Feel free to spin off a another thread or reach out to me off list with any questions or feedback
This seems like as good a time to talk about it as any! Integrating this into the core in some fashion would be good, but I imagine that it's a non-trivial impact to performance, so it would be worthwhile to track that. Speaking of performance - I found the long digression on "Don't Switch In The Core" interesting, since we actually _do_ tracking this kind of context already, for logging. I was a little surprised you didn't integrate with this at all. To be fair, this is something that our friends over at PyPy have been bugging us about since forever; when you're benchmarking raw wire speed it does tend to show up in profiling. I also take it from the performance notes that you're not using PyPy? __slots__ shouldn't make much of a difference there. (In fact I'm given to believe it's a slight _decrease_ in performance on pypy...) -glyph
On Tue, Oct 18, 2016, 14:47 Glyph Lefkowitz <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Oct 18, 2016, at 5:50 AM, Itamar Turner-Trauring <itamar@itamarst.org <mailto:itamar@itamarst.org>> wrote:
Not been doing much Twisted lately, but have been doing async stuff elsewhere, and I've learned some useful things.
Thanks for writing these up, Itamar! This sort of reflection is rare and it's always helpful :).
1. Callbacks should be sync or async, but never sometimes-one-sometimes-the-other. For details go read http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/ <http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/>. For example, Deferred.addCallback(f) really should never run f() immediately.
This has come up a lot in a compare-and-contrast of Twisted vs. asyncio.
I agree that the problems with synchronous callbacks are not insignificant (reentrancy is a degenerate form of preemption, and as we all know preemption is the corrupt wellspring of all bugs). However, the benefit, i.e. consistency of behavior with respect to reentrancy, comes with a cost: tight coupling to an event loop. In asyncio, Future's tight coupling to call_soon is a source of problems; it makes it hard to write a test without setting up an elaborate scheduling trampoline, whereas successResultOf/failureResultOf are quite simple to work with.
I think Deferred as it is today is a pretty good compromise between the two positions. On the one hand it is decoupled from the event loop. On the other - and this is important - no Deferred-returning API will ever call your callbacks synchronously. Deferred.addCallback will, of course, but savvy Twisted programmers can (and should) do this, if they have dependent state changes:
self.manipulateSomeStateForSetup() d = doSomethingPotentiallySynchronous() self.manipulateSomeStateForProcessing() d.addCallback(completeOperation)
As a caller, you can always decide whether you can safely be re-entered or not. In most cases, simply moving the 'addCallback' to the end of the function (a-la Go's "defer", oddly enough) is fine. In more complex cases where you really need to unwind reentrancy completely, you can do your own callLater(0) or callFromThread() from an object with a reference to a reactor.
3.
What happened to '2'? :)
By instrumenting all callbacks it manages, which may or may not require item #1, Twisted can have a context that automatically follows callbacks. Node has this and it is extremely useful. http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls... <http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls...> is best summary I've found with a bit of searching.
This was _always_ supposed to be the way that Twisted worked, but frankly I just wasn't smart enough to figure it out. This is why twisted.python.context came to exist in the first place; I always wanted to attach it to Deferred somehow. I will watch this talk intently; if #1 really is required to address this, my opinion might change. A PR would be intensely appreciated.
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python <http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python> _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
making such aggressive use of private APIs that it could win a contest about how to ensure that you break on every new release of Twisted :)
We're very aware of that! It's one of the reasons we have the test matrix set up to run multiple versions of Python and Twisted. I have not started on 16.X compatibility yet.
I imagine that it's a non-trivial impact to performance, so it would be worthwhile to track that.
We put this this through some extensive benchmarks and testing to measure the performance impact. For example, the details are logged in a commit message but, we initially implemented the @inlineCallbacks extension as a coroutine wrapper. However, we found that the way t.p.Failure tries to serialize itself, and its local+global scopes, to a dictionary caused enormous memory and CPU consumption when triggered because of the added objects in those spaces. The negative impact grew exponentially with levels of nested coroutines. Very bad day. Once we pivoted to a small fork of @inlineCallbacks, we measured the overall performance hit to be negligible in our services. I'll dig around to see if I can find where we documented the actual numbers we saw. At a macro level, our service wide stats showed no meaningful growth of runtime or memory consumption.
digression on "Don't Switch In The Core"
I was surprised at how much switching this context implementation was when we put it in the lower level read/write callbacks. Each of our services process a large amount of continually streaming data and our profiles show, IIRC, that one of the top 5 consumers of CPU time was calling the read/write callbacks. When we added this to those paths it increased overall CPU usage by double digit percentage points. If this feature were available as an opt-in reactor extension then providers could capacity plan around the performance hit. We found it more valuable to move the switching closer to application protocol code where switches happen less frequently.
I also take it from the performance notes that you're not using PyPy?
We're still on cPython. PyPy is something we've talked about before but haven't invested much time into yet. I don't know to what extent PyPy might change the performance characteristics of the project. On Tue, Oct 18, 2016 at 7:50 PM Glyph Lefkowitz <glyph@twistedmatrix.com> wrote: On Oct 18, 2016, at 4:05 PM, Kevin Conway <kevinjacobconway@gmail.com> wrote:
This is why twisted.python.context came to exist in the first place; I always wanted to attach it to Deferred somehow
Well, it's not something we've announced yet through any official channel, but we had to solve the context propagation problem at Atlassian to instrument our services with traceable logging. We open sourced our solution at https://bitbucket.org/hipchat/txlocal. The answer for us was an extension for the reactor, thread pool, and inline callbacks that maintain the needed state. There's a readme with some insight into how we tool our services. This is (A) very cool, and (B) making such aggressive use of private APIs that it could win a contest about how to ensure that you break on every new release of Twisted :). I'm super impressed that you tracked the introduction of twisted._threads and support both old- and new-style thread pools! We've had it on our backlog to address the mailing list and, possibly even, discuss what it would take to put this into Twisted. I guess now is as good if a time as any. Feel free to spin off a another thread or reach out to me off list with any questions or feedback This seems like as good a time to talk about it as any! Integrating this into the core in some fashion would be good, but I imagine that it's a non-trivial impact to performance, so it would be worthwhile to track that. Speaking of performance - I found the long digression on "Don't Switch In The Core" interesting, since we actually _do_ tracking this kind of context already, for logging. I was a little surprised you didn't integrate with this at all. To be fair, this is something that our friends over at PyPy have been bugging us about since forever; when you're benchmarking raw wire speed it does tend to show up in profiling. I also take it from the performance notes that you're not using PyPy? __slots__ shouldn't make much of a difference there. (In fact I'm given to believe it's a slight _decrease_ in performance on pypy...) -glyph On Tue, Oct 18, 2016, 14:47 Glyph Lefkowitz <glyph@twistedmatrix.com> wrote: On Oct 18, 2016, at 5:50 AM, Itamar Turner-Trauring <itamar@itamarst.org> wrote: Not been doing much Twisted lately, but have been doing async stuff elsewhere, and I've learned some useful things. Thanks for writing these up, Itamar! This sort of reflection is rare and it's always helpful :). 1. Callbacks should be sync or async, but never sometimes-one-sometimes-the-other. For details go read http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/. For example, Deferred.addCallback(f) really should never run f() immediately. This has come up a lot in a compare-and-contrast of Twisted vs. asyncio. I agree that the problems with synchronous callbacks are not insignificant (reentrancy is a degenerate form of preemption, and as we all know preemption is the corrupt wellspring of all bugs). However, the benefit, i.e. consistency of behavior with respect to reentrancy, comes with a cost: tight coupling to an event loop. In asyncio, Future's tight coupling to call_soon is a source of problems; it makes it hard to write a test without setting up an elaborate scheduling trampoline, whereas successResultOf/failureResultOf are quite simple to work with. I think Deferred as it is today is a pretty good compromise between the two positions. On the one hand it is decoupled from the event loop. On the other - and this is important - *no Deferred-returning API will ever call your callbacks synchronously*. Deferred.addCallback will, of course, but savvy Twisted programmers can (and should) do this, if they have dependent state changes: self.manipulateSomeStateForSetup() d = doSomethingPotentiallySynchronous() *self.manipulateSomeStateForProcessing()* d.addCallback(completeOperation) As a caller, you can always decide whether you can safely be re-entered or not. In most cases, simply moving the 'addCallback' to the end of the function (a-la Go's "defer", oddly enough) is fine. In more complex cases where you really need to unwind reentrancy completely, you can do your own callLater(0) or callFromThread() from an object with a reference to a reactor. 3. What happened to '2'? :) By instrumenting all callbacks it manages, which may or may not require item #1, Twisted can have a context that automatically follows callbacks. Node has this and it is extremely useful. http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls... is best summary I've found with a bit of searching. This was _always_ supposed to be the way that Twisted worked, but frankly I just wasn't smart enough to figure it out. This is why twisted.python.context came to exist in the first place; I always wanted to attach it to Deferred somehow. I will watch this talk intently; if #1 really is required to address this, my opinion might change. A PR would be intensely appreciated. -glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Oct 18, 2016, at 7:09 PM, Kevin Conway <kevinjacobconway@gmail.com> wrote:
making such aggressive use of private APIs that it could win a contest about how to ensure that you break on every new release of Twisted :)
We're very aware of that! It's one of the reasons we have the test matrix set up to run multiple versions of Python and Twisted. I have not started on 16.X compatibility yet.
I imagine that it's a non-trivial impact to performance, so it would be worthwhile to track that.
We put this this through some extensive benchmarks and testing to measure the performance impact. For example, the details are logged in a commit message but, we initially implemented the @inlineCallbacks extension as a coroutine wrapper. However, we found that the way t.p.Failure tries to serialize itself, and its local+global scopes, to a dictionary caused enormous memory and CPU consumption when triggered because of the added objects in those spaces. The negative impact grew exponentially with levels of nested coroutines. Very bad day.
What are you referring to as a "coroutine" here? A generator? And exponential growth, you say? That sounds very surprising.
Once we pivoted to a small fork of @inlineCallbacks, we measured the overall performance hit to be negligible in our services. I'll dig around to see if I can find where we documented the actual numbers we saw. At a macro level, our service wide stats showed no meaningful growth of runtime or memory consumption.
Does this mean you only get context tracking against inlineCallbacks, and not other usages of Deferred?
digression on "Don't Switch In The Core"
I was surprised at how much switching this context implementation was when we put it in the lower level read/write callbacks. Each of our services process a large amount of continually streaming data and our profiles show, IIRC, that one of the top 5 consumers of CPU time was calling the read/write callbacks. When we added this to those paths it increased overall CPU usage by double digit percentage points. If this feature were available as an opt-in reactor extension then providers could capacity plan around the performance hit. We found it more valuable to move the switching closer to application protocol code where switches happen less frequently.
Maybe "switching" is more expensive than I realized. Where is this implemented?
I also take it from the performance notes that you're not using PyPy?
We're still on cPython. PyPy is something we've talked about before but haven't invested much time into yet. I don't know to what extent PyPy might change the performance characteristics of the project.
As I always tell people - if you care about performance, PyPy should be step zero. Optimizing for CPython looks like adding weird implementation-specific hacks that might start working or be backwards in the next version; optimizing for PyPy means making the code simpler and more readable so the JIT can figure out what to do ;). So the pressure that optimizing for PyPy exerts on your code is generally a lot healthier. -glyph
On Tue, Oct 18, 2016, at 03:44 PM, Glyph Lefkowitz wrote:
I think Deferred as it is today is a pretty good compromise between the two positions. On the one hand it is decoupled from the event loop. On the other - and this is important - *no Deferred-returning API will ever call your callbacks synchronously*. Deferred.addCallback will, of course, but savvy Twisted programmers can (and should) do this, if they have dependent state changes:
self.manipulateSomeStateForSetup()
d = doSomethingPotentiallySynchronous() *self.manipulateSomeStateForProcessing()* d.addCallback(completeOperation)
As a caller, you can always decide whether you can safely be re- entered or not. In most cases, simply moving the 'addCallback' to the end of the function (a-la Go's "defer", oddly enough) is fine. In more complex cases where you really need to unwind reentrancy completely, you can do your own callLater(0) or callFromThread() from an object with a reference to a reactor.
Well... I had a test that went through synchronous Deferred path. And yeah, it was easier to write than async test. But it failed to catch a bug that was only in async case. So the problem I see is that supporting both in Deferred means you need twice the number of tests each time you use Deferreds.
3.
What happened to '2'? :)
There were only two points :)
On Wed, Oct 19, 2016, at 05:45 PM, Itamar Turner-Trauring wrote:
Well... I had a test that went through synchronous Deferred path. And yeah, it was easier to write than async test. But it failed to catch a bug that was only in async case. So the problem I see is that supporting both in Deferred means you need twice the number of tests each time you use Deferreds.
Er, that was unclear. I had a bug that wasn't caught by tests because it passed with sync Deferred and failed with async Deferred callback, and I didn't have tests for latter. -- Itamar Turner-Trauring
On Oct 19, 2016, at 2:47 PM, Itamar Turner-Trauring <itamar@itamarst.org> wrote:
On Wed, Oct 19, 2016, at 05:45 PM, Itamar Turner-Trauring wrote:
Well... I had a test that went through synchronous Deferred path. And yeah, it was easier to write than async test. But it failed to catch a bug that was only in async case. So the problem I see is that supporting both in Deferred means you need twice the number of tests each time you use Deferreds.
Er, that was unclear. I had a bug that wasn't caught by tests because it passed with sync Deferred and failed with async Deferred callback, and I didn't have tests for latter.
To be clear: I do see this as a downside to Deferred's architecture; it's a tradeoff. I see the loose coupling with the event loop as a worthwhile upside. However, it's totally possible to write the async Deferred callback case as well, just by having the test fire the Deferred after returning to the test body instead of firing it before :-). -glyph
It's also possible to write a little bit of testing library code and get both versions of the test nearly for free. Unfortunately, I very rarely see test suites written that way. I think many people don't realize there are two cases to handle or believe testing both cases is too expensive for them (perhaps because they don't realize you can reuse the code for doing so to save on the cost). Jean-Paul On Wed, Oct 19, 2016 at 5:54 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Oct 19, 2016, at 2:47 PM, Itamar Turner-Trauring <itamar@itamarst.org> wrote:
On Wed, Oct 19, 2016, at 05:45 PM, Itamar Turner-Trauring wrote:
Well... I had a test that went through synchronous Deferred path. And yeah, it was easier to write than async test. But it failed to catch a bug that was only in async case. So the problem I see is that supporting both in Deferred means you need twice the number of tests each time you use Deferreds.
Er, that was unclear. I had a bug that wasn't caught by tests because it passed with sync Deferred and failed with async Deferred callback, and I didn't have tests for latter.
To be clear: I do see this as a *downside* to Deferred's architecture; it's a tradeoff. I see the loose coupling with the event loop as a worthwhile upside.
However, it's totally possible to write the async Deferred callback case as well, just by having the test fire the Deferred after returning to the test body instead of firing it before :-).
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Oct 19, 2016, at 4:25 PM, Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:
It's also possible to write a little bit of testing library code and get both versions of the test nearly for free. Unfortunately, I very rarely see test suites written that way. I think many people don't realize there are two cases to handle or believe testing both cases is too expensive for them (perhaps because they don't realize you can reuse the code for doing so to save on the cost).
I don't think even I've ever written a test suite that way. Could we provide any utilities in Twisted to make this easier, or even more importantly, more obvious? -glyph
On 20 Oct. 2016, at 08:54, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Oct 19, 2016, at 2:47 PM, Itamar Turner-Trauring <itamar@itamarst.org <mailto:itamar@itamarst.org>> wrote:
On Wed, Oct 19, 2016, at 05:45 PM, Itamar Turner-Trauring wrote:
Well... I had a test that went through synchronous Deferred path. And yeah, it was easier to write than async test. But it failed to catch a bug that was only in async case. So the problem I see is that supporting both in Deferred means you need twice the number of tests each time you use Deferreds.
Er, that was unclear. I had a bug that wasn't caught by tests because it passed with sync Deferred and failed with async Deferred callback, and I didn't have tests for latter.
To be clear: I do see this as a downside to Deferred's architecture; it's a tradeoff. I see the loose coupling with the event loop as a worthwhile upside.
One thing that I have still not figured out is how Futures (which are tightly tied to an event loop) will possibly ever work over multiple event loops. I think since you can't really chain them, this is less of a problem, but Deferreds like to absorb other Deferreds and make one big one, which would make the tight coupling problematic if you wanted to use two event loops (like, say, a GTK one and an IOCP one on Windows). - Amber
However, it's totally possible to write the async Deferred callback case as well, just by having the test fire the Deferred after returning to the test body instead of firing it before :-).
-glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Oct 19, 2016, at 4:43 PM, Amber Hawkie Brown <hawkowl@atleastfornow.net> wrote:
One thing that I have still not figured out is how Futures (which are tightly tied to an event loop) will possibly ever work over multiple event loops. I think since you can't really chain them, this is less of a problem, but Deferreds like to absorb other Deferreds and make one big one, which would make the tight coupling problematic if you wanted to use two event loops (like, say, a GTK one and an IOCP one on Windows).
I've never seen this as a problem, because "multiple event loops" implicitly means "multiple threads"; and you really kinda need to call back a Deferred on the same event loop where it was originated. When would you want to do this? -glyph
What are you referring to as a "coroutine" here? A generator?
Generator, yes, but used as a coroutine that both emits values and receives them. The introduction of dedicated coroutines in Python 3.5 is something I haven't updated my personal language for very well. Generator-coroutine is the term used in there Python docs, IIRC.
And exponential growth, you say?
Well... That may not be perfectly accurate and I shouldn't have used that term. It grew large enough, fast enough that the same level of nesting was measured in microseconds for the control and and over an hour for the wrapper so I cancelled the test. Let me put together a reproducible experiment and let others decide the growth rate.
Where is this implemented?
Unfortunately, our production use of the lib is not something we've open sourced. Similar to the above, I'll make available a reproducible experiment so you can draw your own conclusion.
Optimizing for CPython looks like adding weird implementation-specific hacks that might start working or be backwards in the next version
This went into the rationale behind making most of the components pluggable. The choice of optimizing for cPython is internal to us and we chose to document it for others. It's not required and not the default. As far as the possibility of integrating this feature into Twisted, let me chat with my cohorts over here. Well come back with some benchmark code and a proposal of changes for further discussion. I'll probably start a new thread to focus on the context bits. On Wed, Oct 19, 2016, 16:56 Glyph Lefkowitz <glyph@twistedmatrix.com> wrote: On Oct 19, 2016, at 2:47 PM, Itamar Turner-Trauring <itamar@itamarst.org> wrote: On Wed, Oct 19, 2016, at 05:45 PM, Itamar Turner-Trauring wrote: Well... I had a test that went through synchronous Deferred path. And yeah, it was easier to write than async test. But it failed to catch a bug that was only in async case. So the problem I see is that supporting both in Deferred means you need twice the number of tests each time you use Deferreds. Er, that was unclear. I had a bug that wasn't caught by tests because it passed with sync Deferred and failed with async Deferred callback, and I didn't have tests for latter. To be clear: I do see this as a *downside* to Deferred's architecture; it's a tradeoff. I see the loose coupling with the event loop as a worthwhile upside. However, it's totally possible to write the async Deferred callback case as well, just by having the test fire the Deferred after returning to the test body instead of firing it before :-). -glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Oct 19, 2016, at 5:12 PM, Kevin Conway <kevinjacobconway@gmail.com> wrote:
As far as the possibility of integrating this feature into Twisted, let me chat with my cohorts over here. Well come back with some benchmark code and a proposal of changes for further discussion. I'll probably start a new thread to focus on the context bits.
The Twisted benchmarks are not super discoverable, so for your reference, they're available here <https://github.com/twisted-infra/twisted-benchmarks> and PRs should be submitted there. They're separate from the repo because newer benchmarks often have to be run against older versions to get a trend line. (Also, if you didn't know, the results from these benchmarks are presented at <http://speed.twistedmatrix.com> :)). -glyph
participants (5)
-
Amber "Hawkie" Brown -
Glyph Lefkowitz -
Itamar Turner-Trauring -
Jean-Paul Calderone -
Kevin Conway