[Python-ideas] async: feedback on EventLoop API

[Guido van Rossum]

On Mon, Dec 17, 2012 at 11:21 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Tue, Dec 18, 2012 at 2:01 PM, Guido van Rossum <guido at python.org> wrote:
>>
>> On Mon, Dec 17, 2012 at 7:20 PM, Nick Coghlan <ncoghlan at gmail.com>
>> wrote:Also, event loop implementations are allowed to offer additional APIs
>> on their implementation. If the need for multiple handlers per FD only
>> exists on those platforms where the platform's event loop supports it,
>> no harm is done if the functionality is only available through a
>> platform-specific API.
>
>
> Sure, but since we know this capability is offered by multiple event loops,
> it would be good if there was a defined way to go about exposing it.

Only if there's a use case.

>> But still, I don't understand the use case. Possibly it is using file
>> descriptors as a more general signaling mechanism? That sounds pretty
>> platform specific anyway (on Windows, FDs must represent sockets).
>>
>> If someone shows me a real-world use case I may change my mind.

> The most likely use case that comes to mind is monitoring and debugging
> (i.e. the event loop equivalent of a sys.settrace). Being able to tap into a
> datastream (e.g. to dump it to a console or pipe it to a monitoring process)
> can be really powerful, and being able to do it at the Python level means
> you have this kind of capability even without root access to the machine to
> run Wireshark.

I can't see how that would work. Once one callback reads the data the
other callback won't see it. There's also the issue of ordering.

Solving this seems easier by implementing a facade for the event loop
that wraps certain callbacks, and installing it using a custom event
loop policy.

So, I still don't see the use case.

> There are other more obscure signal analysis use cases that occur to me, but
> those could readily be handled with a custom transport implementation that
> duplicated that data stream, so I don't think there's any reason to worry
> about those.

Right, that seems a better way to go about it.

>> Twisted supports this for writing through its writeSequence(), which
>> appears in Tulip and PEP 3156 as writelines(). (Though IIRC Glyph told
>> me that Twisted rarely uses the platform's scatter/gather primitives,
>> because they are so damn hard to use, and the kernel implementation
>> often just joins the buffers together before passing it to the regular
>> send()...)
>>
>> But regardless, I don't think scatter/gather would use multiple
>> callbacks per FD.
>>
>> I think it would be really hard to benefit from reading into multiple
>> buffers in Python.

> Sorry, I wasn't quite clear on what I meant by gather/scatter and it's more
> a protocol thing than an event loop thing.
>
> Specifically, gather/scatter interfaces are most useful for multiplexed
> transports. The ones I'm particularly familiar with are traditional
> telephony transports like E1 links, with 15 time-division-multiplexed
> channels on the wire (and a signalling timeslot), as well a few different HF
> comms protocols. When reading from one of those, you have a demultiplexing
> component which is reading the serial data coming in on the wire and making
> it look like 15 distinct data channels from the application's point of view.
> Similarly, the output multiplexer takes 15 streams of data from the
> application and interleaves them into the single stream on the wire.
>
> The rise of packet switching means that sharing connections like that is
> increasingly less common, though, so gather/scatter devices are
> correspondingly less useful in a networking context. The only modern use
> cases I can think of that someone might want to handle with Python are
> things like sharing a single USB or classic serial connection amongst
> multiple data streams. However, I suspect the standard transport and
> protocol API definitions already proposed should also suffice for the
> gather/scatter use case, as such a component would largely work like any
> other protocol-as-transport adapter, with the difference being that there
> would be a many-to-one relationship between the number of interfaces on the
> application side and those on the communications side.
>
> (Technically, gather/scatter components can also be used the other way
> around to distribute a single data stream across multi transports, but that
> use case is even less likely to come up when programming in Python.
> Multi-channel HF data comms is the only possibility that really comes to
> mind)

I'm glad you talked yourself out of that objection. :-)

-- 
--Guido van Rossum (python.org/~guido)