[Python-ideas] The async API of the future: PEP 3153 (async-pep)

[Guido van Rossum]

On Sat, Oct 13, 2012 at 10:54 AM, Laurens Van Houtven <_ at lvh.cc> wrote:
> On Sat, Oct 13, 2012 at 1:22 AM, Guido van Rossum <guido at python.org> wrote:
>>
>> [Hopefully this is the last spin-off thread from "asyncore: included
>> batteries don't fit"]
>>
>> So it's totally unfinished?
>
>
> At the time, the people I talked to placed significantly more weight in
> "explain why this is necessary" than "get me something I can play with".

Odd. Were those people experienced in writing / reviewing PEPs?

>> > Do you feel that there should be less talk about rationale?
>>
>> No, but I feel that there should be some actual specification. I am
>> also looking forward to an actual meaty bit of example code -- ISTR
>> you mentioned you had something, but that it was incomplete, and I
>> can't find the link.
>
> Just examples of how it would work, nothing hooked up to real code. My
> memory of it is more of a drowning-in-politics-and-bikeshedding kind of
> thing, unfortunately :) Either way, I'm okay with letting bygones be bygones
> and focus on how we can get this show on the road.

Shall I just reject PEP 3153 so it doesn't distract people? Of course
we can still refer to it when people ask for a rationale for the
separation between transports and protocols, but it doesn't seem the
PEP itself is going to be finished (correct me if I'm wrong), and as
it stands it is not useful as a software specification.

>> > It's not that there's *no* reference to IO: it's just that that
>> > reference is
>> > abstracted away in data_received and the protocol's transport object,
>> > just
>> > like Twisted's IProtocol.
>>
>> The words "data_received" don't even occur in the PEP.
>
>
> See above.
>
> What thread should I reply in about the pull APIs?

Probably the yield-from thread; or the Twisted/Deferred thread.

>> I just want to make sure that we don't *completely* paint ourselves into
>> the wrong corner when it comes to that.
>
>
> I don't think we have to worry about it too much. Any reasonable API I can
> think of makes this completely doable.

Agreed that we needn't constantly worry about it. It should be enough
to have some kind of reality check closer to PEP accept time.

>> But I'm really hoping you'll make good on your promise of redoing
>> async-pep, giving some actual specifications and example code, so I
>> can play with it.
>
>
> Takeaways:
>
> - The async API of the future is very important, and too important to be
> left to chance.

That's why we're discussing it here.

> - It requires a lot of very experienced manpower.

It also requires (a certain level of) *agreement* between people with
different preferences, since it's no good if the community fragments
or the standard solution gets ignored by Twisted and Tornado, for
example. Ideally those packages (that is, their Python 3.4 versions)
would build on and extend the standard API, and for "boring" stuff
(like possibly the event loop) they would just use the standard
solution.

> - It requires a lot of effort to handle the hashing out of it (as we're
> doing here) as well as it deserves to be.

Right.

> I'll take as proactive a role as I can afford to take in this process, but I
> don't think I can do it by myself.

I hope I didn't come across as asking you that! I am just hoping that
you can give some concrete, working example code showing how to do
protocols and transports.

>  Furthermore, it's a risk nobody wants to
> take: a repeat performance wouldn't be good for anyone, in particular not
> for Python nor myself.

A repeat of what? Of the failure of PEP 3153? Don't worry about that.
This time around I'm here, and since then I have got a lot of
experience implementing and using a solid async library (albeit of a
quite different nature than the typical socket-based stuff that most
people do).

> I've asked JP Calderone and Itamar Turner-Trauring if they would be
> interested in carrying this forward professionally, and they have
> tentatively said yes. JP's already familiar with a large part of the problem
> space with the implementation of the ssl module. JP and Itamar have worked
> together for years and have recently set up a consulting firm.

Insight in the right way to support SSL would be huge; it is an
excellent example of a popular transport that does *not* behave like
sockets, even though its abstract conceptual model is similar (a setup
phase, followed by two bidirectional byte streams).

> Given that this is emphatically important to Python, I intend to apply for a
> PSF grant on their behalf to further this goal. Given their experience in
> the field, I expect this to be a fairly low risk endeavor.

Famous last words. :-)

-- 
--Guido van Rossum (python.org/~guido)