[Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted

[Guido van Rossum]

[Markus sent this to me off-list, but agreed to me responding on-list,
quoting his entire message.]

On Wed, Dec 26, 2012 at 2:38 PM, Markus <nepenthesdev at gmail.com> wrote:
> Hi,

Hi Markus,

I don't believe we've met before, have we? It would probably help if
you introduced yourself and your experience, since our past
experiences color our judgment.

> as I've been waiting for this to happen, I decided to speak up.
> While I really look forward to this, I disagree with the PEP.

Heh, we can't all agree on everything. :-)

> First shoot should be getting a well established event loop into python.

Perhaps. What is your definition of an event loop?

> libev is great, it takes care of operating system specialities, and
> only does a single job, providing an event loop.

It is also written for C, and I presume much of its API design was
influenced by the conventions and affordabilities of that language.

> This event loop can take care of timers, sockets and signals,

But sockets are not native on Windows, and I am making some effort
with PEP 3156 to efficiently support higher-level abstractions without
tying them to sockets. (The plan is to support IOCP on Windows. The
previous version of Tulip already had a branch that did support that,
as a demonstration of the power of this abstraction.)

> pyev, a
> great python wrapper for libev already provides this simple eventing
> facility in python.

But, being a libev wrapper, it is likely also strongly influenced by C.

> In case you embed python in a c program, the libev default loop of the
> python code and c code can even be shared, providing a great amount of
> flexibility.

Only if the C code also uses libev, of course. But C programs may use
other event mechanisms -- e.g. AFAIK there are alternatives to libev
(during the early stages of Tulip development I chatted a bit with one
of the original authors of libevent, Niels Provos, and I believe
there's also something called libuv), and GUI frameworks (e.g. X, Qt,
Gtk, Wx) tend to have their own event loop.

PEP 3156 is designed to let alternative *implementations* of the same
*interface* be selected at run time. Hopefully it is possible to
provide a conforming implementation using libev -- then your goal
(smooth interoperability with C code using libev) is obtained.

It's possible that in order to do that the PEP 3156 interface may have
to be refactored into separate pieces. The Tulip implementation
already has separate "pollster" implementations (which concern
themselves *only* with polling for I/O using select, poll, or other
alternatives). It probably makes sense to factor the part that
implements transports out as well. However, the whole point of
including transports and protocols (and futures) in the PEP is that
some platforms may want to implement the same high-level API (e.g.
create a transport that connects to a certain host/port) using a
different approach altogether, e.g. on Windows the transport might not
even use sockets. OTOH on UNIX it may be possible to add file
descriptors representing pipes and pseudo-ttys.

> libev is great as it is small - it provides exactly what's required,
> and nothing beyond.

Depending on your requirements. :-)

> getaddrinfo/getnameinfo/create_transport are out of scope from a event
> loop point of view.
> This functionality already exists in python, it just does not use a
> event loop and is blocking, as every other io related api.

It wasn't random to add these. The "event loop" in PEP 3156 provides
abstractions that leave the platform free to implement connections
using the appropriate native constructs without letting those
constructs "leak" into the application -- after all, whether you're on
UNIX or on Windows, a TCP connection represents the same abstraction,
but the network stack may have a very different interface.

> I'd propose not to replicate the functionality in the event loop
> namespace, but to extend the existing implementations - by allowing to
> provide an event loop/callback/ctx as optional args which get used.

That's an interface choice that I would regret (I really don't like
writing code using callbacks).

(It would also be harder to implement initially as a 3rd party
framework. At the lowest level, no changes to Python itself are needed
-- it already supports non-blocking sockets, for example. But adding
optional callbacks to existing low-level APIs would require changes
throughout the stdlib.)

> If you specify something like pyev as PEP, you can still come up with
> another PEP which defines the semantics for upper layer protocols like
> udp/tcp on IPv4/6, which can be used to take care of dns and
> 'echo-server-connections'.

I could split up the PEP, but that wouldn't really change anything,
since to me it is still a package deal. I am willing to put an effort
into specifying a low-level event loop because I know that I can still
write high-level code which is (mostly) free of callbacks, using
futures, tasks and the yield-from construct. And in order to do that I
need a minimum set of high-level abstractions such as getaddrinfo()
and transport creation (the exact names of the transport creation
methods are still under debate, as are the details of their
signatures, but the need for them is established without a doubt in my
mind).

I note that the stdlib socket module has roughly the same set of
abstractions bundled together:

- socket objects
- getaddrinfo(), getnameinfo()
- create_connection()
- the makefile() methods on socket objects, which create buffered streams

PEP 3156 offers alternatives for all of these, using higher-level
abstractions that have been developed and proven in practice by
Twisted, *and* offers a path to interop to frameworks that previously
couldn't very well interoperate -- Twisted, Tornado, and others have
traditionally been pretty segregated, but with PEP 3156 they can
interoperate both through the event loop and through Futures (which
are friendly both to a callback style and to yield-from).

> Anyway, I really hope you'll have a look on libev and pyev, both is
> great and well tested software and may give you an idea what people
> who dedicate themselves to event loops came up with already in terms
> of names, subclassing, requirements, guarantees and workarounds for
> platform specific failures (kqueue, epoll ...).

I will certainly have a look! I am not so concerned about naming (it
seems inevitable that everyone uses somewhat different terminology
anyway, and it is probably better not to reuse terms when the meaning
is different), but I do like to look at guarantees (or the absence
thereof!) and best practices for dealing with the differences between
platforms.

> http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod
> http://code.google.com/p/pyev/
>
> All together, I'd limit the scope of the PEP to the API of the event
> loop, just focussing on io/timers/signals and propose to extend
> existing API to  be usable with an event loop, instead of replicating
> it.

You haven't convinced me about this. However, you can help me by
comparing the event loop part of PEP 3156 (ignoring anything that
returns or takes a Future) to libev and pointing out things (either
specific APIs or certain guarantees or requirements) that would be
hard to implement using libev, as well as useful features in libev
that you think every event loop should have.

> For naming I'd prefer 'watcher' over 'Handler'.

Hm, 'watcher' to me sounds more active than the behavior I have in
mind for this class. It is just a reification of a specific function
and some arguments to pass to it, with the ability to cancel the call
altogether.

Thanks for writing!

-- 
--Guido van Rossum (python.org/~guido)