Re: [Twisted-Python] Re-working a synchronous iterator to use Twisted
For the record, here's a followup to my own posting, with working code. The earlier untested code was a bit of a mess. The below runs fine. In case it wasn't clear before, you're pulling "results" (e.g., from a search engine) in off the web. Each results pages comes with an indicator to tell you whether there are more results. I wanted to write a function (see processResults below) that, when called, would call the process function below on each result, all done asynchronously. This solution feels cumbersome, but it does work (aka prints the expected output). Comments welcome (just don't tell me to use version control :-)) Terry from twisted.internet import defer def process(result): # Stub: process a single result. print 'processed result', result def getPage(uri, offset=0): # Stub: return some results and an indicator of whether more are available. data = [[0, 1, 2], [4, 7, 8, 10], [12, 14, 18, 30]] results = data[offset] more = offset < len(data) - 1 return defer.succeed((results, more)) def getResultsFromPage(page): # Stub: get the results from page, return them in a list return page[0] def needToCallAgain(page): # Stub: determine if there are more results, given current page. return bool. return page[1] # ASYNCHRONOUS result producer def getResults(uri, offset=0): def parsePage(page, offset): results = getResultsFromPage(page) if needToCallAgain(page): d = getResults(uri, offset + 1) else: d = None return results, d def returnTheseResults(page, offset): resultIterator, done = parsePage(page, offset) return resultIterator, done return getPage(uri, offset).addCallback(returnTheseResults, offset) # ASYNCHRONOUS calling def processResults(uri): def cb((resultIterator, deferred)): for result in resultIterator: process(result) if deferred is not None: deferred.addCallback(cb) return getResults(uri).addCallback(cb) if __name__ == '__main__': def finished(x): print 'finished' processResults('uri').addCallback(finished) The above prints $ python xxx.py processed result 0 processed result 1 processed result 2 processed result 4 processed result 7 processed result 8 processed result 10 processed result 12 processed result 14 processed result 18 processed result 30 finished
participants (1)
-
Terry Jones