[Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface

[Ben Darnell]

On Mon, Feb 4, 2013 at 2:02 PM, Guido van Rossum <guido at python.org> wrote:

> I'm going to try and snip as much as I can to get the the heart of this...
>
>
Me too...


>
> Thinking about what you could mean by "more esoteric protocols",
> there's really not much at the level of TCP and UDP that comes to
> mind. UNIX-domain sockets, and perhaps the (root-only) protocol for
> sniffing packets (raw sockets?).
>

Looking at twisted as an example, the things that are supported across both
PosixReactorBase and IOCPReactor are TCP, UDP, SSL, multicast, and
subprocesses.  Unix domain sockets aren't that interesting here since any
system that supports them will (presumably?) support add_reader and things
will just work.


>
> A new feature just landed in Tulip (I still have to update PEP 3156)
> where you can pass a pre-constructed socket object to
> create_connection() and start_serving(), which will make it a little
> easier to support esoteric ways of setting up the socket; however,
> create_connection() is still limited to sockets that implement a byte
> stream, because of the way the transport/protocol API works.
>

Cool.  That's definitely useful (especially on the server side), but we'll
still need a separate interface for datagrams.


>
> > Right.  Third-party extensions to the event loop interface are inherently
> > problematic, so we'll have to provide them in some other way.  I'm
> proposing
> > a pattern for that "some other way" and then realizing that I like it
> even
> > for first-party interfaces.
>
> Glad that is out of the way. But I'm still skeptical -- first, as I
> explained before, I am actually in favor of using different styles for
> 1st and 3rd party interfaces, so the status of the interface used is
> obvious to the reader (and the coder, in case they are copy-pasting
> recipes :-); second, I don't expect there will be too many
> opportunities to put the pattern at work.
>

You can't argue about style. :-)



> > The
> > idea is that someone can propose a transport interface in a third-party
> > module (mymodule.listen_udp in this example), implement it themselves for
> > some event loop implementations, and other event loops can declare
> > themselves compatible with it.
>
> Aha! This is the executive summary of your proposal, or at least your
> goal for it.
>
> This is hypothesizing rather a lot of goodwill and coordination
> between different 3rd party developers.


Yeah, history is unfortunately not very supportive of the idea that
developers of asynchronous frameworks will coordinate on this kind of thing
:)


> And the registry offered by
> the event loop comes down to not much more than a dictionary with keys
> that follow a certain convention (e.g. fully-qualified package+module
> name plus some identifier for the feature) and nothing can be said
> about what the items stored in the registry are (since a packet
> transport and a stream transport are not interchangeable, and even two
> stream transports may not be).
>
> Given that for each 3rd party transport the details of how to
> implement a compatible version of it will vary hugely, both depending
> on what the transport is trying to do and how the event loop works, I
> expect that the market for this registry will be rather small. And
> when a particular 3rd party transport wants to enable other 3rd party
> events to support them, they can implement their own registry, which
> the other 3rd party could then plug into. (But see below.)
>

True.


>
> > (And in an admittedly far-fetched scenario, if there were two third-party
> > UDP interfaces and LibUVEventLoop implemented one of them, yet another
> party
> > could build a bridge between the two, and then they'd plug it in with
> > register_implementation)
>
> I do see one argument in favor of having a standard registry on the
> event loop, even if it's just a dict with register/lookup APIs and a
> naming convention, and no semantics assigned to the items registered.
> That argument is to make 3rd party transport implementers aware of the
> possibility that some other 3rd party might want to offer a compatible
> implementation aimed at an event loop that's not supported natively by
> the (former) 3rd party transport. And I could even be convinced that
> the standard protocols should use this registry so that the source
> code serves as an example of best practices.
>
> Still, it's a pretty weak argument IMO -- I don't expect there to be a
> significant cottage industry cranking out 3rd party protocol
> implementations, assuming we add UDP to PEP 3156, which is my plan.
>

Yeah, as long as we get TCP, UDP, SSL, and pipes (at least for
subprocesses), I'm hard pressed to imagine anything else that would be in
so much demand it would need to be supported by all event loops.


> And I don't think that create_connection() should be used to create
> UDP connections -- the signature of the protocol factory passed in
> would be quite different, for starters, and the set of options needed
> to configure the transport is also different (assuming we want to
> support both connected and connection-less UDP).
>

Of course.  (I may have been unclear somewhere along the way, but it was
never my intention for create_connection to work for both TCP and UDP)


>
> > I don't think the status quo prevents the development of third-party
> > transports, but it does implicitly encourage two bad habits:  A) adding
> > methods to the event loop, inviting name collisions, or B) just building
> on
> > add_reader and friends without thinking about non-posix platforms.  Of
> > course, no one expects a groundswell of third-party development at the
> event
> > loop and transport level, so this could just be so much overengineering,
> but
> > I like it from a stylistic perspective even without the third-party
> > benefits.
>
> Yeah, so that's the rub: I'm not so keen on adding extra machinery to
> the PEP that I don't expect to be used much. I have more important
> fish to fry (such as adding UDP :-). And adding a registry one whole
> Python release cycle later (e.g. in 3.5, assuming PEP 3156 is
> standardized and included in 3.4) doesn't strike me as such a bad
> thing -- I don't think we're painting ourselves into much of a corner
> by not having a registry right from the start.
>

Fair enough.  I may use this pattern when/if I retrofit Tornado to be
IOCP-friendly since we currently create transports with static functions,
but for Tulip let's wait and see if the problem this proposal is trying to
solve ever materializes.

-Ben


>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130204/c2ee1ddd/attachment.html>