<div dir="ltr">On Sun, Feb 3, 2013 at 12:30 PM, Guido van Rossum <span dir="ltr"><<a href="mailto:guido@python.org" target="_blank">guido@python.org</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Ben: I wrote a long reply, it is inline below. However it's possible<br>
that I am not seeing the use case right. Your proposal is written in<br>
very general terms; perhaps you can come up with a more specific<br>
example to defend it further? The UDP example somehow doesn't seem<br>
very compelling to me.<br></blockquote><div><br></div><div style>UDP is a real-life example from tornado - we don't have any built-in support for UDP, but people who need it have been able to build it without touching tornado itself. The same argument would apply to pipes or any number of other (admittedly much more esoteric) network protocols. I'll elaborate on the UDP example below.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
On Sat, Feb 2, 2013 at 12:10 PM, Ben Darnell <<a href="mailto:ben@bendarnell.com">ben@bendarnell.com</a>> wrote:<br>
> The event loop interface in PEP 3156 has an extensibility problem. It seems<br>
> likely that it will have a method like listen_udp() by the time it's done,<br>
> but suppose that doesn't make it into the first official release.<br>
> Third-party event loop implementations may want to provide UDP support as an<br>
> extension, but the most consistent way to provide that extension is by<br>
> adding new methods on the event loop object, where various extensions risk<br>
> conflicting with each other or with new methods that become standardized<br>
> later.<br>
<br>
</div>This may be based on a misunderstanding. If a specific event loop<br>
implementation wants to offer a new transport, there is no reason to<br>
add it as a method to the event loop. The app that wants to use that<br>
transport has to import that event loop too, so the app might as well<br>
call a module-level function that's specific to that event loop, in<br>
order to instantiate the new transport.<br>
<br>
We may have to point this out in the PEP, since it is likely that<br>
implementers wanting to offer new features will think of adding new<br>
methods to their event loop first. But that's a precious namespace<br>
(since it's shared by all event loop implementations), whereas their<br>
own implementation's module namespace is less precious.<br></blockquote><div><br></div><div style>Right. Third-party extensions to the event loop interface are inherently problematic, so we'll have to provide them in some other way. I'm proposing a pattern for that "some other way" and then realizing that I like it even for first-party interfaces.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
> The PEP specifies the add_reader family of methods in part so that core<br>
> protocol implementations can be shared across all EventLoop implementations<br>
> that support these methods. However, if those transports are implemented in<br>
> a common base class (like Twisted's PosixReactorBase), there is no way for<br>
> third-party transports to take advantage of a similar structure.<br>
<br>
</div>I'm not sure I understand this (the "there is no way" part). Is the<br>
problem that that base class is private, or that the add_reader<br>
methods may not be there, or what?<br></blockquote><div><br></div><div style>Suppose twisted did not have UDP support built in. Most reactor implementations subclass PosixReactorBase (with IOCPReactor as the notable exception). Twisted can add UDP support and implement listenUDP in PosixReactorBase and IOCPReactor, and suddenly most reactors (even third-party ones like TornadoReactor) support UDP for free. Those that don't (a hypothetical LibUVReactor?) can implement it themselves and interoperate with everything else. </div>
<div style><br></div><div style>If a third party wanted to add UDP support separately from twisted's release schedule, they can't do with an interface that is generically usable across all reactors. They could make a static function listenUDP() that works with any IReactorFDSet, and maybe special-case IOCPReactor, but then there'd be no way for a third-party LibUVReactor to participate.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
> I'd like<br>
> to make it possible for transports to be developed independent of a<br>
> particular EventLoop implementation in a way that is consistent with the way<br>
> the core transports work.<br>
<br>
</div>Hm, now we're talking about something else. Transports implemented<br>
*independently* from an event loop implementation should not assume<br>
more than the standardized API. This feels rather limiting unless<br>
we're talking about transports built on top of other protocols (e.g.<br>
the mythical TCP-over-HTTP transport :-).</blockquote><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
My expectation is that most transport implementations (as opposed to<br>
protocols) are probably tied closely to an event loop implementation<br>
(and note that the various Tulip event loop implementations using<br>
select, poll, epoll, kqueue, are a single implementation for this<br>
purpose -- OTOH the IocpEventLoop (in the iocp branch) is a different<br>
implementation and, indeed, has different transport implementations!<br></blockquote><div><br></div><div><div>add_reader is not very limiting except for its platform-specificity. It's possible to have a generic protocol across all posixy event loops and then special-case the small number of interesting non-posixy ones (or maybe there is some other class of methods that could be standardized for other platforms? Is there some set of methods analogous to add_reader that multiple IOCP-based loops could share?)</div>
</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
> (This is a bigger concern for tulip than it is<br>
> for twisted because twisted can update PosixReactorBase more frequently than<br>
> the stdlib can change)<br>
<br>
</div>I think what you're really getting at here is that there is no 3rd<br>
significant party cottage industry creating new transports. Transports<br>
in practice are all part of the Twisted distribution, so development<br>
is only gated by backwards compatibility requirements with user apps<br>
(APIs, once offered, must remain supported and stable), not by<br>
compatibilities with older versions of the rest of the framework (a<br>
new transport introduced in Twisted 12.1 doesn't have to work with<br>
Twisted 12.0).<br></blockquote><div><br></div><div style>I was thinking more about release schedules. Twisted has several releases a year, but the standard library moves much more slowly. </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
> I propose turning the interface around so that transport creation uses<br>
> static functions that take the event loop as an argument, with a<br>
> double-dispatch mechanism to allow the event loop to provide the actual<br>
> implementation when it can:<br>
><br>
> def create_connection(protocol_factory, host, port, event_loop=None):<br>
> if event_loop is None:<br>
> event_loop = get_event_loop()<br>
> # Note the use of a fully-qualified name in the registry<br>
> impl = event_loop.get_implementation('tulip.create_connection')<br>
> return impl(protocol_factory, host, port)<br>
<br>
</div>Hm. I don't see what this adds. It still always gets the protocol from<br>
the event loop so moving this standardized method out of the event<br>
loop class doesn't seem to buy anything. The implementation-dependent<br>
work is just moved into get_implementation(). I also don't see why we<br>
need a registry.<br></blockquote><div><br></div><div style>This version doesn't change much, it's mainly to set the stage for the following variations. However, it does have a few nice properties - it keeps the (public) event loop interface small and manageable, and callers don't need to touch actual event loop objects unless they want to have more than one. From a stylistic perspective I like this style of interface more than using dozens of methods on the event loop object itself (even if those dozens of methods are still there but hidden as an implementation detail).</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
> New third-party transports could provide fallbacks for event loops that<br>
> don't have their own implementations:<br>
><br>
> if impl is None:<br>
> # These supports_*() functions are placeholders for a to-be-determined<br>
> # introspection interface.<br>
> if supports_fd_interface(event_loop):<br>
> return posix_udp_implementation(*args)<br>
> elif supports_iocp_interface(event_loop):<br>
> return iocp_udp_implementation(*args)<br>
> else:<br>
> raise Exception("This transport is not supported on this event loop")<br>
<br>
</div>It seems you want each transport implementation to provide its own<br>
create_connection() function, right? There's nothing wrong with that,<br>
and I don't see that just because 3rd party transports will be<br>
instantiated through a module-level function (in a specific module)<br>
that means that the standard transports specified by the PEP (standard<br>
in semantics, not in implementation!) can't be instantiated through<br>
methods on the event loop.<br>
<br>
Perhaps you are placing a higher value on consistency between standard<br>
and non-standard transports? To me, it is actually positive to be<br>
inconsistent here, so readers are made fully aware that a non-standard<br>
transport is being used.<br></blockquote><div><br></div><div style>When third-party modules get absorbed into the standard library, it's often possible to support both just by trying different imports until one works (unittest.mock vs mock, json vs simplejson, etc). Sometimes a module's interface gets cleaned up and rearranged in the process, but that seems to be less common. It would be nice if a third-party transport could get standardized and the only thing callers would need to change is their imports. However, this is a minor concern; as I wrote up this design I realized I liked it for first-party work even if there were no third-party modules to be consistent with.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
(The idea of introspection functions is fine, however.)<br>
<div class="im"><br>
> Or they could plug into the event loop's implementation registry:<br>
><br>
> LibUVEventLoop.register_implementation('mymodule.listen_udp',<br>
> libuv_udp_implementation)<br>
<br>
</div>Yeah, but wouldn't this likely be a private affair between UV's event<br>
loop and UV's UDP transport, which are being distributed together?<br></blockquote><div><br></div><div style>But libuv doesn't necessarily contain the transport creation function. The idea is that someone can propose a transport interface in a third-party module (mymodule.listen_udp in this example), implement it themselves for some event loop implementations, and other event loops can declare themselves compatible with it.</div>
<div style><br></div><div style>(And in an admittedly far-fetched scenario, if there were two third-party UDP interfaces and LibUVEventLoop implemented one of them, yet another party could build a bridge between the two, and then they'd plug it in with register_implementation)</div>
<div> </div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Users who wish to use UDP (assuming PEP 3156 ends up not specifying<br>
UDP support) are required to depend on a non-standard feature of their<br>
event loop, you can't hide that with a registry. (Note that a UDP<br>
transport has a different API than a TCP transport, and requires the<br>
protocol to implement different methods as well.)<br></blockquote><div><br></div><div style>Yes, third-party transports will be non-standard, but in practice the add_reader family makes it easy to get broad coverage in a quasi-standard way, and the registry makes it possible to fill in the gaps.</div>
<div style><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
> This does introduce a little magic (is there any precedent for this kind of<br>
> multiple-dispatch in the standard library?), but I like the way it keeps the<br>
> event loop interface from getting too big and monolithic. Third-party<br>
> transports can avoid naming conflicts without looking fundamentally<br>
> different from standard ones, and there's a clean path from doing something<br>
> that's platform-specific (e.g. with add_reader and friends) to supporting<br>
> multiple event loops to full standardization.<br>
<br>
</div>I actually consider it a good thing that when a concept is<br>
standardized, the "name" of the API changes. When we adopt a 3rd party<br>
module in the stdlib we typically give it a new name too, to avoid<br>
confusion about which version is meant (since inevitably the 3rd party<br>
has more variability than the version adopted into the stdlib).<br></blockquote><div><br></div><div style>The *module* gets a new name (and that is indeed a good thing), but the functions and classes within (usually) stay the same.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
With all that said, if a particular event loop implementation prefers<br>
to add non-standard methods to their event loop object, and then<br>
starts lobbying for its adoption in the standard, I can't stop them,<br>
and they may even have a good shot at getting adopted in the next<br>
Python version. But they should be aware of the risk they run, that<br>
the next version of the stdlib might expose a different API under<br>
their chose name. It will be easier for their users to transition if<br>
they choose a way to spell their extension that is *not* likely to be<br>
standardized, e.g. a function in their own module, or an event loop<br>
method name with a custom prefix like uv_listen_udp().<br></blockquote><div><br></div><div style>Agreed.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
I'm looking forward to explanations of why I am preventing the<br>
developing of 3rd party transports with this response....<br></blockquote><div><br></div><div style>I don't think the status quo prevents the development of third-party transports, but it does implicitly encourage two bad habits: A) adding methods to the event loop, inviting name collisions, or B) just building on add_reader and friends without thinking about non-posix platforms. Of course, no one expects a groundswell of third-party development at the event loop and transport level, so this could just be so much overengineering, but I like it from a stylistic perspective even without the third-party benefits. </div>
<div style><br></div><div style>-Ben</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span class=""><font color="#888888"><br>
--<br>
--Guido van Rossum (<a href="http://python.org/~guido" target="_blank">python.org/~guido</a>)<br>
</font></span></blockquote></div><br></div></div>