Re: [Twisted-web] Is there a way to implement a deferred sequence?
"Alexey" == twisted-web <twisted-web@udmvt.ru> writes:
Hi Alexey Alexey> I want a sequence which is Deferred with two callbacks: one as Alexey> usual - for the whole result called at the end of the sequence, Alexey> another - called for every element of the sequence (as they Alexey> arrive). I'm pretty sure I don't understand exactly what you want. Maybe you could write it in pseudo-code, showing both how you'd call the thing you want and what the code to implement it would look like, at least conceptually. I can't promise to help, if you can be a bit more specific it might be easier to be more helpful. You might find something of worth in one of my Twisted mailing list soliloquies, which starts here: http://www.twistedmatrix.com/pipermail/twisted-python/2008-June/017904.html I have feeling that the code Esteve Fernandez and I came up with to tackle that situation might be something like what you're after. You may also like to think about whether you can use twisted.internet.task.LoopingCall It's more flexible than it may appear at first, and it also fits your pattern at least at a high-level description. Terry
On Thu, Apr 09, 2009 at 09:18:39PM +0200, Terry Jones wrote:
"Alexey" == twisted-web <twisted-web@udmvt.ru> writes:
Hi Alexey
Alexey> I want a sequence which is Deferred with two callbacks: one as Alexey> usual - for the whole result called at the end of the sequence, Alexey> another - called for every element of the sequence (as they Alexey> arrive).
I'm pretty sure I don't understand exactly what you want. Maybe you could write it in pseudo-code, showing both how you'd call the thing you want and what the code to implement it would look like, at least conceptually. I can't promise to help, if you can be a bit more specific it might be easier to be more helpful.
You might find something of worth in one of my Twisted mailing list soliloquies, which starts here:
http://www.twistedmatrix.com/pipermail/twisted-python/2008-June/017904.html
I have feeling that the code Esteve Fernandez and I came up with to tackle that situation might be something like what you're after. Well, that is close to my problem, but I have a feature request with somewhat more generic requirements.
Current Deferred is used to defer a simple process, that only have its 'start' and 'end', returning some single result object at the end. And there is nothing between start and end, it is not allowed to look inside that process. Once started it can only fail or finish returning a single result. But what about deferring some more complex process, that may have zero to more itermediate results before finishing? What if we are inside that process and have some event, but we are not finished the process? Why can't the consumers of the Deferred be interested in receiving that event too? Why not augment the Deferred for that? As an example consider running deferred the final state machine. Having only a current Deferred it is not possible to trace the state change if the state is not changed to the final one, since there is no callback that can be called several times for every state change. We can only know, when the machine is entered the final state and stopped or failed. It is not possible to defer a process that run in stages if each stage does return some result or if we want to trace the completion of stages, not only the whole process. (Not possible without further complications, described in the link above.) It would be nice if Deferred be extended to have another callback to be called for updating the result or notifying about state change of some process. In addition to d.callback(result) we will have something like the d.update_state(args) How this may be used depends on what the process is and what are the intermediate results. For example d.update_state may be invoked to tell the percentage of completion of the whole deferred process. Or d.update_state may be invoked to append new item to the resulting sequence as it arrives. Or d.update_state may be invoked to say that intermediate result should be replaced by a new one or somehow updated in other fashion. That could also depend on the type of the final result: If we are returning a list, then we may want to append items to it. If we are returning a tree we may want to build that tree node by node. (may we want that?) If we are aggregating some data series into some single result we may want to present current intermediate result to the observer. Any other process, that could be useful to observe in its dynamics, not only in its final state. It is just the way to send/receive the events produced by some process that is deferred in the case that events are not the 'process_finished' events. I think that Deferred class can be extended in that way preserving full backward compatibility, since if noone is interested in recieving intermediate events, noone will register a callback for that and no callback will be called - noone will notice that.
You may also like to think about whether you can use twisted.internet.task.LoopingCall It's more flexible than it may appear at first, and it also fits your pattern at least at a high-level description.
Well, it can solve only a specific problem, as far as I understand.
Terry
Thank for your attention and interest. PS: Should I file a feature request? Or that feature is absolutely unnecessary to have? PPS: Does anybody need some more detailed explanation? -- Alexey S.
On 08:12 am, twisted-web@udmvt.ru wrote:
But what about deferring some more complex process, that may have zero to more itermediate results before finishing? What if we are inside that process and have some event, but we are not finished the process? Why can't the consumers of the Deferred be interested in receiving that event too? Why not augment the Deferred for that?
Because Deferreds are intended to be an asynchronous mechanism for control flow. They are the way to return a value which is not yet present. In other words they are an asynchronous version of exit semantics; "return" and "raise". They are very explicitly *not* designed to simultaneously implement asynchronous versions of every other potential control flow - looping, for example, which seems to be what you want. There are other mechanisms for doing those things. For example, if you want to get a stream of data and handle it incrementally, you want to implement an IProtocol or ITransport. Trying to cram all of IProtocol into Deferred is a mistake. If you want some abstraction to support an incremental stream of results with a size known in advance, for example to support drawing a progress bar on an upload or download, then please feel free to propose that as a *separate* mechanism from Deferred, and we can debate its merits and usefulness on its own. But I doubt very much that Deferred should do such a thing itself.
On Fri, Apr 10, 2009 at 08:43:39AM -0000, glyph@divmod.com wrote:
Because Deferreds are intended to be an asynchronous mechanism for control flow. They are the way to return a value which is not yet present. In other words they are an asynchronous version of exit semantics; "return" and "raise".
They are very explicitly *not* designed to simultaneously implement asynchronous versions of every other potential control flow - looping, for example, which seems to be what you want.
Yes, but there is no synchronous version of what I want, I can't name any. Asynchronousness adds some more capabilities to be introduced into flow control.
From my perception Deferred is a point where two entities exchange events (one send, other receive) And if we think about Deferred as an event router, then my question was "why there is only two types of events possible and both of them are 'final' events?" I just asked about subclassing that event router to include capability to route some other events in addition to two existing. It is worth noting, that such improvement does not forbid Deferred from being "the way to return a value which is not yet present". It just make the term "value" more complex, allows it to be dispersed in time, not only concentrated at a single moment. I think there is no natural analogy for synchronous flow control and can't be - results are not synchronous. Perhaps the thing I am talking about should not be called "Deferred", since it is only asynchronous version of syncronous flow control. But for me extending Deferred looks natural, that is why I am asking.
There are other mechanisms for doing those things. For example, if you want to get a stream of data and handle it incrementally, you want to implement an IProtocol or ITransport. Trying to cram all of IProtocol into Deferred is a mistake.
IProtocol and ITransport looks like being a byte-oriented interfaces, not an arbitrary-object-oriented.
If you want some abstraction to support an incremental stream of results with a size known in advance, for example to support drawing a progress
^^^^^ - did you mean unknown?
bar on an upload or download, then please feel free to propose that as a *separate* mechanism from Deferred, and we can debate its merits and usefulness on its own. But I doubt very much that Deferred should do such a thing itself. I am asking for an half-duplex Event-pipe which can be closed from the sending end. Just as variant it can be used to deliver cumulative results, but not fixed to that. I said "Deferred" because the implementation of such thing will be identical to Deferred's. It will be a Deferred with additional callback(s) which is(are) not restricted to be invoked only once.
Thank for your time. -- Alexey S.
Alexey S. wrote:
On Fri, Apr 10, 2009 at 08:43:39AM -0000, glyph@divmod.com wrote:
Because Deferreds are intended to be an asynchronous mechanism for control flow. They are the way to return a value which is not yet present. In other words they are an asynchronous version of exit semantics; "return" and "raise".
They are very explicitly *not* designed to simultaneously implement asynchronous versions of every other potential control flow - looping, for example, which seems to be what you want.
Yes, but there is no synchronous version of what I want, I can't name any. Asynchronousness adds some more capabilities to be introduced into flow control.
From my perception Deferred is a point where two entities exchange events (one send, other receive)
A Deferred's purpose is to provide a mechanism for one entity to communicate a single result to another. You're asking for it to do something else, therefore (by definition, basically) you're asking for something other than a Deferred. What Deferred provides is pretty similar to just passing in a callback to an async function, except it's much more convenient. And like passing callbacks directly it's a building block for richer control flow, not an all-in-one answer to every problem.
And if we think about Deferred as an event router, then my question was "why there is only two types of events possible and both of them are 'final' events?"
If you need more events, you can make multiple Deferreds. in your case you wanted an event to fire when a set of work has completed — ok, that's one Deferred right there, with a simple object to track the outstanding work and fire that Deferred when the amount of remaining work is zero. Each of those individual items of work may also have a corresponding Deferred to report their individual results. Or as Glyph says, you may simply want to have an object that is called directly every time some event happens. Or some combination of the above.
I just asked about subclassing that event router to include capability to route some other events in addition to two existing. It is worth noting, that such improvement does not forbid Deferred from being "the way to return a value which is not yet present". It just make the term "value" more complex, allows it to be dispersed in time, not only concentrated at a single moment. I think there is no natural analogy for synchronous flow control and can't be - results are not synchronous. Perhaps the thing I am talking about should not be called "Deferred", since it is only asynchronous version of syncronous flow control. But for me extending Deferred looks natural, that is why I am asking.
No, Deferred is very definitely not about multiple events (which is what multiple values over time are, unless you only want to deal with a final, single, aggregated value). Deferred may be a useful building block for a the thing you are talking about, but it is definitely not a natural extension of it.
There are other mechanisms for doing those things. For example, if you want to get a stream of data and handle it incrementally, you want to implement an IProtocol or ITransport. Trying to cram all of IProtocol into Deferred is a mistake. IProtocol and ITransport looks like being a byte-oriented interfaces, not an arbitrary-object-oriented.
They were just an example. It's not hard to imagine examples using richer objects than bytes, e.g. with methods like “recordReceived”, “accountCreated” or “buttonPressed” or “cpuOnFire” rather than “bytesReceived”. The interesting part of e.g. IProtocol is that “bytesReceived” can be (and usually is) called many times rather than once. By the way, I can imagine a similar transformation to passing in an IProtocol class, analagous to the transformation of replacing passing in a callback function with making that function return a Deferred which you then add the callback to. So instead of e.g.: clientCreator = ClientCreator(reactor, MyProtocolClass) d = clientCreator.connectTCP(host, port) d.addCallback(doSomethingWithProtocolInstance) I suppose you could avoid passing the protocol up front: clientCreator = HypotheticalDeferringClientCreator(reactor) d = clientCreator.connectTCP(host, port) def connectionMade(transport): protocolInstance = MyProtocolClass() protocolInstance.makeConnection(transport) return protocolInstance d.addCallback(connectionMade) d.addCallback(doSomethingWithProtocolInstance) But really that's just reinventing twisted.internet.protocol.Factory in a less convenient form... My point here is basically just that there isn't a single API or design pattern that is best in every situation. [...]
I am asking for an half-duplex Event-pipe which can be closed from the sending end. Just as variant it can be used to deliver cumulative results, but not fixed to that. I said "Deferred" because the implementation of such thing will be identical to Deferred's. It will be a Deferred with additional callback(s) which is(are) not restricted to be invoked only once.
So, “identical” except for some fundamental bits that are radically different? :) Seriously, the extra complexity of the semantics you want is way beyond the scope of what's reasonable for Deferred. Handling multiple events is vastly more complicated in the general case than handling a single result (consider for instance the various options on DeferredList, and still that API can't even handle common situations like a dynamically sized list of results, as you've noticed). That's not to say that what you want is unreasonable, just that what you want isn't called “Deferred”. Deferred does one thing, and does it very well. -Andrew.
participants (4)
-
Alexey S. -
Andrew Bennetts -
glyph＠divmod.com -
Terry Jones