[Python-ideas] Async API

[Yury Selivanov]

Sam,

BTW, kudos for shrapnel!

On 2012-10-23, at 5:25 PM, Sam Rushing <sam-pydeas at rushing.nightmare.com> wrote:
[snip]

> On 10/23/12 12:33 PM, Yury Selivanov wrote:
>> 
>> What we really need now (and I think Guido has already mentioned that) is a 
>> callback-based (Deferreds, Futures, plain callbacks) design that is easy to 
>> plug-and-play in any coroutine-framework.  It has to be low-level and simple.  
>> Sort of WSGI for async frameworks ;)
> 
> I've been trying to play catch-up since being told about this thread a
> couple of days ago.  If I understand it correctly, 'yield-from' looks
> like it can help make generator-based-concurrency a little more sane by
> cutting back on endless chains of 'for x in ...: yield ...', right? 
> That certainly sounds like an improvement, but does the generator nature
> of the API bubble all the way up to the top?  Can you send an email with
> a function call?

Well, I guess so.  Let's say, urllib is rewritten internally in async-style,
exposing publicly its old API, like::

   def urlopen(*args, **kwargs):
       return run_coro(urlopen_async, args, kwargs)

where 'run_coro' takes care of setting up a Scheduler/event-loop and running
yield-style or callback-style private code.  So that 'urllib' is blocking, but
there is an option of using 'urlopen_async' for those who need it.

For basic library functions that will work.  And that's already a huge win.
But developing a complicated library will become twice as hard, as you'll need
to maintain two versions of API - sync & async all the way through the code.

There is only one way to 'magically' make existing code both sync- & async-
friendly--greenlets, but I think there is no chance for them (or stackless) to 
land in cpython in the foreseeable future (although it would be awesome.)

BTW, why didn't you use greenlets in shrapnel and ended up with your own
implementation?

>> We also need to work on the stdlib, so that it is easy to inject a custom socket 
>> in any object.  Ideally, by passing it in the constructor (as everybody hates 
>> monkey-patching.)
>> 
> 
> I second this one.  Having a way to [optionally] pass in a factory for
> sockets would help with portability, and would cut down on the
> temptation to monkey-patch.

Great.

Let's see - if nobody is opposed to this we can start with submitting patches :)

Or is there a need for a separate small PEP?

Thanks,
Yury