Itamar Shtull-Trauring wrote:
On Tue, 2008-06-03 at 17:03 +1000, Justin Warren wrote:
Greetings twisted gurus!
I'm hoping someone with a better algorithmic brain than I could offer some advice on a good way to do something in a twisty/pythonic way.
The pseudocode for what I want to do is:
for item in generator_that_fetches_rows_from_database: ... do_stuff() ...
where generator_that_fetches_rows_from_database is an object that uses enterprise.adbapi (or anything else) to fetch rows from a database and return them to the for..in.. loop.
runInteraction is your friend:
Sure, but that's exactly what the OP didn't want; doing a bulk fetch from the SQL (which yes will be non-blocking for the most part) I think he wanted: def query(txn): txn.execute("...") while True: rs = txn.fetchone() if blah: continue if foo: transform data yield data @defer.inlineCallbacks def sqlQuery(): deferred_iter = runInteraction(query) for def in deferred_iter: row = yield def # do a thing with the row e.g. push it to an Athena page i.e. the function that runs inside the thread pool to be able to yield values, and the function that runs in the reactor to be able to iterate through them in a deferred-compatible fashion. AFAIK there is no such thing, which is a shame because for *really* huge queries it has the potential to significantly reduce memory usage. On a not-very related issue, it would be highly useful IMHO for