[Twisted-Python] looping in a non-blocking way
Hi everybody. I have an application where I have several polling protocol instances servicing an array of COM ports driven by timers using reactor.callLater with typically 50 millisecond timing constants. I'm also using FTP client protocol instances to concurrently refresh locally cached data from ftp servers. These files may be quite large and during the text parsing routines I end up with blocking loops which are undesirable. Text files my be parsed in a loop with something like: def importFromText(self,data): datalines=data.split("\r\n") for line in datalines: arrin=[] arrin.insert(0,line[0:10]) etc.... For large files the above blocks - starving the other COM port protocols drivers of processing time. I have tried to "yield" to the reactor by calling reactor.iterate() in the above loop trying to do what you would normally accomplish with sleep(0) in a multithreaded application. This solution seemed to partially solve the problem but I'm experiencing intermittent stability problems with my polling protocol drivers that are normally quite robust simply hanging up. I'm now considering rewriting the above loop as a recursive function using reactor.callLater(0,recursiveTexParsingFunction(params)). I'm aware of twisted.flow but the examples given are a bit sparse and obscure and having tried something like http://twistedmatrix.com/documents/current/api/twisted.flow.flow.html - it appears I don't understand how to apply twisted.flow (iterators/coroutines) solve my particular problem. I hope I'm missing something obvious and more elegant since the above possible solutions seems like a lot of rework to solve a very simple problem. Any thoughts/advise would be much appreciated. regards, Eugene Coetzee Web -> www.reedflute.com ===============================================
Eugene Coetzee wrote:
def importFromText(self,data): datalines=data.split("\r\n") for line in datalines: arrin=[] arrin.insert(0,line[0:10]) etc....
from twisted.internet import reactor CHUNKSIZE = 100 def _parseSome(lines): for i in xrange(CHUNKSIZE): try: line = lines.pop(0) except IndexError: break # parse line here, maybe mutate a data # structured passed as an argument if lines: reactor.callLater(0, _parseSome, lines) else: # all done, probably should notify someone def importFromText(self, data): datalines=data.split("\r\n") _parseSome(datalines)
participants (2)
-
Eugene Coetzee -
Tommi Virtanen