On Fri, Oct 19, 2012 at 7:41 PM, Steve Dower <Steve.Dower@microsoft.com> wrote:
I'm not entirely sure whether I'm hijacking the thread here... I have to admit I've somewhat lost track with all the renames. The discussion has been very interesting (I really like the 'codef' idea, and decorators can provide this without requiring syntax changes) regardless of which thread is active.
I have spent a bit of time writing up the approach that we (Dino, who posted it here originally, myself and with some advice from colleagues who are working on a similar API for C++) came up with and implemented.
I must apologise for the length - I got a little carried away with background information, but I do believe that it is important for us to understand exactly what problem we're trying to solve so that we aren't distracted by "new toys".
The write-up is here: http://stevedower.id.au/blog/async-api-for-python/
I included code, since there have been a few people asking for prototype implementations, so if you want to skip ahead to the code (which is quite heavily annotated) it is at http://stevedower.id.au/blog/async-api-for-python/#thecode or http://stevedower.id.au/downloads/PythonAsync.zip (I based my example on Greg's socket spam, so thanks for that!)
And no, I'm not collecting any ad revenue from the page, so feel free to visit as often as you like and use up my bandwidth.
Let the tearing apart of my approach commence! :)
Couple of questions and comments. - You mention a query interface a few times but there are no details in your example code; can you elaborate? (Or was that a typo for queue?) - This is almost completely isomorphic with NDB's tasklets, except that you borrow the Future class implementation from concurrent.futures -- I think that's the wrong building block to start with, because it is linked too closely to threads. - There is a big speed difference between yield from <generator> and yield <future>. With yield <future>, the scheduler has to do significant work for each yield at an intermediate level, whereas with yield from, the schedule is only involved when actual blocking needs to be performed. In my experience, real code has lots of intermediate levels. Therefore I would like to use yield from. You can already do most things with yield from that you can do with Futures; there are a few operations that need a helper (in particular spawning truly concurrent tasks), but the helper code can be much simpler than the Future object, and isn't needed as often, so it's still a bare win. - Nit: I don't like calling the event loop context; there are too many things called context (e.g. context managers in Python), so I prefer to call it what it is -- event loop or I/O loop. - Nittier nit: you have a few stray colons, e.g. "it = iter(fn(*args, **kwargs)):" . -- --Guido van Rossum (python.org/~guido)