[Python-ideas] Async API: some code to review

[Guido van Rossum]

On Mon, Oct 29, 2012 at 7:00 AM, Steve Dower <Steve.Dower at microsoft.com> wrote:
> Richard Oudkerk wrote:
>> On 28/10/2012 11:52pm, Guido van Rossum wrote:
>>> I'm most interested in feedback on the design of polling.py and
>>> scheduling.py, and to a lesser extent on the design of sockets.py;
>>> main.py is just an example of how this style works out in practice.
>>
>> What happens if two tasks try to do a read op (or two tasks try to do a
>> write op) on the same file descriptor?  It looks like the second one to
>> do scheduling.block_r(fd) will cause the first task to be forgotten,
>> causing the first task to block forever.
>
> I know I haven't posted my own code yet (coming very soon), but I'd like to put out there that I don't think this is an important sort of question at this time.

Kind of. I think if it was an important use case it might affect the
shape of the API. However I can't think of a use case where it might
make sense for two tasks to read or write the same file descriptor
without some higher-level mediation. (Even at a higher level I find it
hard to imagine, except for writing to a common log file -- but even
there you want to be sure that individual lines aren't spliced into
each other, and the semantics of send() don't prevent that.)

> We both have sample schedulers that work well enough to demonstrate the API, but aren't meant to be production ready.
>
> IMO, the important questions are:
>
>  - how easy/difficult/flexible/restrictive is it to write a new scheduler as a core Python developer?
>  - how easy/difficult/flexible/restrictive is it to write a new scheduler as an end user?
>  - how easy/difficult/flexible/restrictive is it to write async operations as a core Python developer?
>  - how easy/difficult/flexible/restrictive is it to write async operations as an end user?
>  - how straightforward is it to consume async operations?
>  - how easy is it to write async code that is correct?

Yes, these are all important questions. I'm not sure that end users
would be writing new schedulers -- but 3rd party library developers
will be, and I suppose that's what you are referring to.

My own approach to answering these is to first try to figure out what
a typical application would be trying to accomplish. That's why I made
a point of implementing a 100% async HTTP client -- it's just quirky
enough that it exercises various issues (e.g. switching between
line-mode and blob mode, and the need to invoke getaddrinfo()).

> Admittedly, I am writing this preemptively knowing that there are a lot of distractions like this in my code (some people are going to be horrified at what I did with file IO :-) Don't worry, it's only for trying the API). Once we know what interface we'll be coding against we can worry about getting the implementation perfect. Also, I imagine we'll find some more volunteers for coding (hopefully people who have done non-blocking stuff in C or similar before) who are currently avoiding the higher-level ideas discussion.

I'm looking forward to it! I suspect we'll be merging our designs shortly...

-- 
--Guido van Rossum (python.org/~guido)