[Twisted-Python] Interleaving long running loops

Hello, I have been developing a medium/large scale application using twisted and very happy about it. It is composed of two parts, an application server and a PyQt based client. Perspective broker is used for all communication between client and server. Server mainly talks with a database, formats results and sends to clients (roughly). Most formatting is done in simple for loops (there are many different kinds of operations which a client can choose): @defer.inlineCallbacks def operation(): dbQueryResult = yield runSomeSQL() for item in dbQueryResult: process(item) Again, most of the time these are light operations which complete in sub-seconds. However, rarely, results may be so large that it takes several seconds to process, and during this period all other server processing stops (i.e. accepting new client connections). I've written a utility function, deferredSleep(), which I call occasionally to avoid blocking: def deferredSleep(seconds=0.01): d = defer.Deferred() reactor.callLater(seconds, d.callback, None) return d And I use it like below to avoid blocking for very large result sets: @defer.inlineCallbacks def operation(): dbQueryResult = yield runSomeSQL() for i, item in enumerate(dbQueryResult): if i % 1000 == 0: # Breathe after a thousand iterations yield deferredSleep(0.01) process(item) So my question is, is this an acceptable solution? And is there a more generic way, i.e. for item in magicalIterator(dbQueryResult): process(item) I have lots of similar loops all over the place and a more generic solution would be great. I am also aware of deferToThread but I am trying to avoid using threads. Regards, -- Umit

On 02/11/11 10:40, Umit Oztosun wrote:
Well, deferToThread gives you all kinds of problems re: talking to the reactor from threads, so avoid. You could try it as a cooperator from twisted.internet import task @defer.inlineCallbacks def operation(): result = yield somesql() # n.b. generator expression! work = (process(item) for item in result) yield task.coiterate(work)

On 02/11/11 10:40, Umit Oztosun wrote:
Well, deferToThread gives you all kinds of problems re: talking to the reactor from threads, so avoid. You could try it as a cooperator from twisted.internet import task @defer.inlineCallbacks def operation(): result = yield somesql() # n.b. generator expression! work = (process(item) for item in result) yield task.coiterate(work)
participants (2)
-
Phil Mayers
-
Umit Oztosun