Re: [Twisted-Python] A pseudo-deferred class that can be canceled
"Glyph" == Glyph Lefkowitz <glyph@twistedmatrix.com> writes:
Glyph> And let's not forget, in long-running servers, it's possible to Glyph> "leak" operations, and the associated resources like file Glyph> descriptors. When the caller says that they're not interested in Glyph> the result of an operation any more, the library should clean up as Glyph> much as possible, otherwise you'll start running out of those Glyph> resources eventually. Yes, but it shouldn't be the job of the caller to clean up something it has no knowledge of. I'm not sure if it's yet clear that I'm not trying *at all* to address somehow stopping operations that are in progress. On the contrary, my code always lets them run to their natural conclusion. It's just that if the caller decides they're no longer interested in the result or that they want the thing to fire right now, they can arrange for it. The originally called code, including any err/callbacks that may be on the deferred it gave you, is unaffected. Using the word "cancel" in my original posting & class name was I guess quite misleading. Apologies for that. As usual I thought everyone would know what I was talking about :-) Is the simplified class that really is a Deferred (subclass) clearer? Terry
On Jan 5, 2010, at 4:53 PM, Terry Jones wrote:
I understand what you're saying: you're interested in a subset of what I'm interested in, here. The point I'm trying to make is that once you've gone to the trouble of providing an API for *clients* of an operation to declare that they are no longer interested in its results, then it's wasteful for the underlying engine to continue executing the operation only to discard its result. I think that coming up with a good API and semantics for "I no longer care about the result here" has a huge amount of overlap with this anyway. I grant that it may well be easier to implement without worrying about the underlying operation though, and the semantics you've defined by explicitly ignoring the actually-stopping case are much simpler. But that also means that you still have to go to the trouble of figuring out when you're no longer interested in the result any more, but after going to the trobule ... what's the benefit? I know what the use-cases are for stopping the underlying operation (notifying the peer that you're not going to accept it, reclaiming the resources); but if you're just going to let the operation complete eventually anyway, why wouldn't you want to just finish processing the result when it arrives regardless?
On Jan 6, 2010, at 5:23 AM, Glyph Lefkowitz wrote:
I know what the use-cases are for stopping the underlying operation (notifying the peer that you're not going to accept it, reclaiming the resources); but if you're just going to let the operation complete eventually anyway, why wouldn't you want to just finish processing the result when it arrives regardless?
Please forgive an old real-time jock here, but one way of handling this, and yes, I know how nasty, dirty, and ugly it is, is to patch the response chain so that all further notifications route to an ABORT handler that explicitly knows the details of how to clean old things up and stop new things from happening. I have had to use a mechanism like this in multi-processor, low bandwidth, high priority systems where a corrupted message can not be allowed to propagate and use up bandwidth or processor time that could be used to regenerate a correct message (think naval guidance system where one or more transducers just stopped working properly, like, got hit by a torpedo or started taking on sea water under pressure) Obviously, this is very radical and the ABORT handler needs to be completely aware of exactly how to interrupt all things in the chain properly in all possible states, but if it's that important that the process be interrupted, then it is. As far as I know, in my limited spelunking into Twisted's guts, Deferred's pretty much only know about themselves and their callbacks so I'm not sure whether there's a "ringmaster" that knows any more than that that could make sure that all things are cleaned up properly. S
On Jan 6, 2010, at 9:46 AM, ssteinerX@gmail.com wrote:
As far as I know, in my limited spelunking into Twisted's guts, Deferred's pretty much only know about themselves and their callbacks so I'm not sure whether there's a "ringmaster" that knows any more than that that could make sure that all things are cleaned up properly.
Indeed not. And I don't think there could be; in your ABORT handler description, you are describing a system where pretty much all the constraints are known, and all the code is being integrated into a single unit, together with hardware. Twisted is supposed to be general-purpose enough that a "ringmaster" packaged with Twisted could not know about every possible different thing that might need to be cancelled and could not account for them... (Unless of course you're talking about having a plugin API where things could register interest in cancellation, and that's pretty much what Terry and I seem to be converging on in the other fork of this thread.)
On Jan 5, 2010, at 4:53 PM, Terry Jones wrote:
I understand what you're saying: you're interested in a subset of what I'm interested in, here. The point I'm trying to make is that once you've gone to the trouble of providing an API for *clients* of an operation to declare that they are no longer interested in its results, then it's wasteful for the underlying engine to continue executing the operation only to discard its result. I think that coming up with a good API and semantics for "I no longer care about the result here" has a huge amount of overlap with this anyway. I grant that it may well be easier to implement without worrying about the underlying operation though, and the semantics you've defined by explicitly ignoring the actually-stopping case are much simpler. But that also means that you still have to go to the trouble of figuring out when you're no longer interested in the result any more, but after going to the trobule ... what's the benefit? I know what the use-cases are for stopping the underlying operation (notifying the peer that you're not going to accept it, reclaiming the resources); but if you're just going to let the operation complete eventually anyway, why wouldn't you want to just finish processing the result when it arrives regardless?
On Jan 6, 2010, at 5:23 AM, Glyph Lefkowitz wrote:
I know what the use-cases are for stopping the underlying operation (notifying the peer that you're not going to accept it, reclaiming the resources); but if you're just going to let the operation complete eventually anyway, why wouldn't you want to just finish processing the result when it arrives regardless?
Please forgive an old real-time jock here, but one way of handling this, and yes, I know how nasty, dirty, and ugly it is, is to patch the response chain so that all further notifications route to an ABORT handler that explicitly knows the details of how to clean old things up and stop new things from happening. I have had to use a mechanism like this in multi-processor, low bandwidth, high priority systems where a corrupted message can not be allowed to propagate and use up bandwidth or processor time that could be used to regenerate a correct message (think naval guidance system where one or more transducers just stopped working properly, like, got hit by a torpedo or started taking on sea water under pressure) Obviously, this is very radical and the ABORT handler needs to be completely aware of exactly how to interrupt all things in the chain properly in all possible states, but if it's that important that the process be interrupted, then it is. As far as I know, in my limited spelunking into Twisted's guts, Deferred's pretty much only know about themselves and their callbacks so I'm not sure whether there's a "ringmaster" that knows any more than that that could make sure that all things are cleaned up properly. S
On Jan 6, 2010, at 9:46 AM, ssteinerX@gmail.com wrote:
As far as I know, in my limited spelunking into Twisted's guts, Deferred's pretty much only know about themselves and their callbacks so I'm not sure whether there's a "ringmaster" that knows any more than that that could make sure that all things are cleaned up properly.
Indeed not. And I don't think there could be; in your ABORT handler description, you are describing a system where pretty much all the constraints are known, and all the code is being integrated into a single unit, together with hardware. Twisted is supposed to be general-purpose enough that a "ringmaster" packaged with Twisted could not know about every possible different thing that might need to be cancelled and could not account for them... (Unless of course you're talking about having a plugin API where things could register interest in cancellation, and that's pretty much what Terry and I seem to be converging on in the other fork of this thread.)
participants (3)
-
Glyph Lefkowitz -
ssteinerX＠gmail.com -
Terry Jones