On Mon, Feb 4, 2013 at 2:02 PM, Guido van Rossum
I'm going to try and snip as much as I can to get the the heart of this...
Me too...
Thinking about what you could mean by "more esoteric protocols", there's really not much at the level of TCP and UDP that comes to mind. UNIX-domain sockets, and perhaps the (root-only) protocol for sniffing packets (raw sockets?).
Looking at twisted as an example, the things that are supported across both PosixReactorBase and IOCPReactor are TCP, UDP, SSL, multicast, and subprocesses. Unix domain sockets aren't that interesting here since any system that supports them will (presumably?) support add_reader and things will just work.
A new feature just landed in Tulip (I still have to update PEP 3156) where you can pass a pre-constructed socket object to create_connection() and start_serving(), which will make it a little easier to support esoteric ways of setting up the socket; however, create_connection() is still limited to sockets that implement a byte stream, because of the way the transport/protocol API works.
Cool. That's definitely useful (especially on the server side), but we'll still need a separate interface for datagrams.
Right. Third-party extensions to the event loop interface are inherently problematic, so we'll have to provide them in some other way. I'm proposing a pattern for that "some other way" and then realizing that I like it even for first-party interfaces.
Glad that is out of the way. But I'm still skeptical -- first, as I explained before, I am actually in favor of using different styles for 1st and 3rd party interfaces, so the status of the interface used is obvious to the reader (and the coder, in case they are copy-pasting recipes :-); second, I don't expect there will be too many opportunities to put the pattern at work.
You can't argue about style. :-)
The idea is that someone can propose a transport interface in a third-party module (mymodule.listen_udp in this example), implement it themselves for some event loop implementations, and other event loops can declare themselves compatible with it.
Aha! This is the executive summary of your proposal, or at least your goal for it.
This is hypothesizing rather a lot of goodwill and coordination between different 3rd party developers.
Yeah, history is unfortunately not very supportive of the idea that developers of asynchronous frameworks will coordinate on this kind of thing :)
And the registry offered by the event loop comes down to not much more than a dictionary with keys that follow a certain convention (e.g. fully-qualified package+module name plus some identifier for the feature) and nothing can be said about what the items stored in the registry are (since a packet transport and a stream transport are not interchangeable, and even two stream transports may not be).
Given that for each 3rd party transport the details of how to implement a compatible version of it will vary hugely, both depending on what the transport is trying to do and how the event loop works, I expect that the market for this registry will be rather small. And when a particular 3rd party transport wants to enable other 3rd party events to support them, they can implement their own registry, which the other 3rd party could then plug into. (But see below.)
True.
(And in an admittedly far-fetched scenario, if there were two third-party UDP interfaces and LibUVEventLoop implemented one of them, yet another party could build a bridge between the two, and then they'd plug it in with register_implementation)
I do see one argument in favor of having a standard registry on the event loop, even if it's just a dict with register/lookup APIs and a naming convention, and no semantics assigned to the items registered. That argument is to make 3rd party transport implementers aware of the possibility that some other 3rd party might want to offer a compatible implementation aimed at an event loop that's not supported natively by the (former) 3rd party transport. And I could even be convinced that the standard protocols should use this registry so that the source code serves as an example of best practices.
Still, it's a pretty weak argument IMO -- I don't expect there to be a significant cottage industry cranking out 3rd party protocol implementations, assuming we add UDP to PEP 3156, which is my plan.
Yeah, as long as we get TCP, UDP, SSL, and pipes (at least for subprocesses), I'm hard pressed to imagine anything else that would be in so much demand it would need to be supported by all event loops.
And I don't think that create_connection() should be used to create UDP connections -- the signature of the protocol factory passed in would be quite different, for starters, and the set of options needed to configure the transport is also different (assuming we want to support both connected and connection-less UDP).
Of course. (I may have been unclear somewhere along the way, but it was never my intention for create_connection to work for both TCP and UDP)
I don't think the status quo prevents the development of third-party transports, but it does implicitly encourage two bad habits: A) adding methods to the event loop, inviting name collisions, or B) just building on add_reader and friends without thinking about non-posix platforms. Of course, no one expects a groundswell of third-party development at the event loop and transport level, so this could just be so much overengineering, but I like it from a stylistic perspective even without the third-party benefits.
Yeah, so that's the rub: I'm not so keen on adding extra machinery to the PEP that I don't expect to be used much. I have more important fish to fry (such as adding UDP :-). And adding a registry one whole Python release cycle later (e.g. in 3.5, assuming PEP 3156 is standardized and included in 3.4) doesn't strike me as such a bad thing -- I don't think we're painting ourselves into much of a corner by not having a registry right from the start.
Fair enough. I may use this pattern when/if I retrofit Tornado to be IOCP-friendly since we currently create transports with static functions, but for Tulip let's wait and see if the problem this proposal is trying to solve ever materializes. -Ben
-- --Guido van Rossum (python.org/~guido)