[Twisted-Python] Re-Occuring Operations

Hello. I was wondering if there is anything which handles re-occuring functions; that is, an auto-rescheduler or something of that nature. Here is what I came up with, I'm sure there is a shorter 'lambda' form, but I was going for clarity here. class MultiRun: """ apply a callable while it returns true """ def __init__(self, callable, waitInterval = 0): self.callable = callable self.waitInterval = waitInterval self.iterate(starting = 1) def iterate(self, starting = 0): if starting or self.callable(): from twisted.internet import reactor reactor.callLater(self.waitInterval, self.iterate) Something like this would be useful, it's already emerged in two different contexts in my code: (a) a background operation (which never returns true) that runs every 5 minutes, (b) an operation which keeps pooling a resource, if the resource is busy it returns true, otherwise it processes the resource and returns false to signal that it is done Perhaps calling this an "Operation" would be good. Best, clark

On Mon, Mar 17, 2003 at 06:42:17AM +0000, Clark C. Evans wrote:
There is. This is copied straight from doc/howto/time.html: """ If we want a task to run every X seconds repeatedly, we can just re-add it every time it's run: from twisted.internet import reactor def runEverySecond(): print "a second has passed" reactor.callLater(1, runEverySecond) reactor.callLater(1, runEverySecond) """
Or: from __future__ import nested_scopes # For you retro Python 2.1 hippies def multiRun(interval, func): from twisted.internet import reactor def iterate(): if func(): reactor.callLater(interval, iterate) iterate() Except to fit in with the Twisted naming scheme, you'd have to call it 'mindlessReruns' ;) On the other hand, being a fairly straightforward six-line function, it probably doesn't need to be in Twisted.
That's amply catered for by the first example, possibly with a try/finally thrown in.
I don't quite understand your terminology. I think you mean there's a queue of operations that can only be processed if there's some sort of resource available to process them, and you wish to occasionally poll the resource pool to see if there's one free. If so, it's probably better to arrange your code to avoid polling, i.e. make the resource pool check the queue automatically when a resource is freed. This reduces latency and is usually simpler. Regardless, I'm not sure this deserves or requires special support in Twisted. Merely documenting the necessary idioms is probably better. After all there are lots of possible small variations (e.g. dynamically varying the interval based on the function's return value) that would be tedious to build into a generic class or function, but are dead simple to build into your own six-line implementation. Or perhaps I'm wrong, and there's a nice class with lots of use-cases just waiting to be implemented :) -Andrew.

On Mon, Mar 17, 2003 at 11:46:45PM +1100, Andrew Bennetts wrote:
Or perhaps I'm wrong, and there's a nice class with lots of use-cases just waiting to be implemented :)
Oh, and I forgot to mention: we used to have such a beast, called Delayed. It was awkward and confusing. So any solution you propose will have to, as a minimum, suck less than Delayeds did :) -Andrew.

On Mon, Mar 17, 2003 at 11:46:45PM +1100, Andrew Bennetts wrote: | On the other hand, being a fairly straightforward six-line function, it | probably doesn't need to be in Twisted. Yes, it is a straight-foward 6-liner, but there are quite a few 6-liners in Twisted. Anyway, I think it'd be useful. | > (b) an operation which keeps pooling a resource, if | > the resource is busy it returns true, otherwise | > it processes the resource and returns false to | > signal that it is done | | I don't quite understand your terminology. I think you mean there's a queue | of operations that can only be processed if there's some sort of resource | available to process them, and you wish to occasionally poll the resource | pool to see if there's one free. If so, it's probably better to arrange | your code to avoid polling, i.e. make the resource pool check the queue | automatically when a resource is freed. This reduces latency and is usually | simpler. Cool. Is there an example? I have a database select statement, and I need to reschedule several operations till it produces information. Further, it may consume faster than the database can produce rows; so it'be be nice to have it add back to the wait queue every once and a while. For a simple example, assume I have an array; and a thread which is populating the array, using callFromThread(array.append, value) in the main IO thread, I'd like to have my function wait till the array has items in it, and pause when the array is empty. I'm using the array as the 'input queue' so to speak. I'd love to know the 'preferred' way to do this, right now I'm pooling the array to see if it's size is > 0. Not clean. | Regardless, I'm not sure this deserves or requires special support in | Twisted. Merely documenting the necessary idioms is probably better. After | all there are lots of possible small variations (e.g. dynamically varying | the interval based on the function's return value) that would be tedious to | build into a generic class or function, but are dead simple to build into | your own six-line implementation. I don't think that there are many permutations on this case; and yes, allowing the function to specify how long to wait would be a perfect example. In fact, it fits in really nicely with the iterator pattern; make the loop break with a StopIteration exception. And, take the return value of the function to figure out how long from now the next invocation should be; None could be equivalent to 0. | Or perhaps I'm wrong, and there's a nice class with lots of use-cases just | waiting to be implemented :) The advantage of having a standard 6-liner is that it helps in documentation this seems to be common enough that it reoccurs in various projects. Clark

On Mon, Mar 17, 2003 at 06:42:17AM +0000, Clark C. Evans wrote:
I think we need to add a new rule to the coding standard. "Don't add new utility code that nothing else in Twisted uses" -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/

On Mon, Mar 17, 2003 at 06:42:17AM +0000, Clark C. Evans wrote:
There is. This is copied straight from doc/howto/time.html: """ If we want a task to run every X seconds repeatedly, we can just re-add it every time it's run: from twisted.internet import reactor def runEverySecond(): print "a second has passed" reactor.callLater(1, runEverySecond) reactor.callLater(1, runEverySecond) """
Or: from __future__ import nested_scopes # For you retro Python 2.1 hippies def multiRun(interval, func): from twisted.internet import reactor def iterate(): if func(): reactor.callLater(interval, iterate) iterate() Except to fit in with the Twisted naming scheme, you'd have to call it 'mindlessReruns' ;) On the other hand, being a fairly straightforward six-line function, it probably doesn't need to be in Twisted.
That's amply catered for by the first example, possibly with a try/finally thrown in.
I don't quite understand your terminology. I think you mean there's a queue of operations that can only be processed if there's some sort of resource available to process them, and you wish to occasionally poll the resource pool to see if there's one free. If so, it's probably better to arrange your code to avoid polling, i.e. make the resource pool check the queue automatically when a resource is freed. This reduces latency and is usually simpler. Regardless, I'm not sure this deserves or requires special support in Twisted. Merely documenting the necessary idioms is probably better. After all there are lots of possible small variations (e.g. dynamically varying the interval based on the function's return value) that would be tedious to build into a generic class or function, but are dead simple to build into your own six-line implementation. Or perhaps I'm wrong, and there's a nice class with lots of use-cases just waiting to be implemented :) -Andrew.

On Mon, Mar 17, 2003 at 11:46:45PM +1100, Andrew Bennetts wrote:
Or perhaps I'm wrong, and there's a nice class with lots of use-cases just waiting to be implemented :)
Oh, and I forgot to mention: we used to have such a beast, called Delayed. It was awkward and confusing. So any solution you propose will have to, as a minimum, suck less than Delayeds did :) -Andrew.

On Mon, Mar 17, 2003 at 11:46:45PM +1100, Andrew Bennetts wrote: | On the other hand, being a fairly straightforward six-line function, it | probably doesn't need to be in Twisted. Yes, it is a straight-foward 6-liner, but there are quite a few 6-liners in Twisted. Anyway, I think it'd be useful. | > (b) an operation which keeps pooling a resource, if | > the resource is busy it returns true, otherwise | > it processes the resource and returns false to | > signal that it is done | | I don't quite understand your terminology. I think you mean there's a queue | of operations that can only be processed if there's some sort of resource | available to process them, and you wish to occasionally poll the resource | pool to see if there's one free. If so, it's probably better to arrange | your code to avoid polling, i.e. make the resource pool check the queue | automatically when a resource is freed. This reduces latency and is usually | simpler. Cool. Is there an example? I have a database select statement, and I need to reschedule several operations till it produces information. Further, it may consume faster than the database can produce rows; so it'be be nice to have it add back to the wait queue every once and a while. For a simple example, assume I have an array; and a thread which is populating the array, using callFromThread(array.append, value) in the main IO thread, I'd like to have my function wait till the array has items in it, and pause when the array is empty. I'm using the array as the 'input queue' so to speak. I'd love to know the 'preferred' way to do this, right now I'm pooling the array to see if it's size is > 0. Not clean. | Regardless, I'm not sure this deserves or requires special support in | Twisted. Merely documenting the necessary idioms is probably better. After | all there are lots of possible small variations (e.g. dynamically varying | the interval based on the function's return value) that would be tedious to | build into a generic class or function, but are dead simple to build into | your own six-line implementation. I don't think that there are many permutations on this case; and yes, allowing the function to specify how long to wait would be a perfect example. In fact, it fits in really nicely with the iterator pattern; make the loop break with a StopIteration exception. And, take the return value of the function to figure out how long from now the next invocation should be; None could be equivalent to 0. | Or perhaps I'm wrong, and there's a nice class with lots of use-cases just | waiting to be implemented :) The advantage of having a standard 6-liner is that it helps in documentation this seems to be common enough that it reoccurs in various projects. Clark

On Mon, Mar 17, 2003 at 06:42:17AM +0000, Clark C. Evans wrote:
I think we need to add a new rule to the coding standard. "Don't add new utility code that nothing else in Twisted uses" -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/
participants (3)
-
Andrew Bennetts
-
Christopher Armstrong
-
Clark C. Evans