[Python-ideas] async: feedback on EventLoop API

[Guido van Rossum]

On Mon, Dec 17, 2012 at 7:20 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Tue, Dec 18, 2012 at 10:40 AM, Guido van Rossum <guido at python.org> wrote:

[A better name for DelayedCall]
>> Anyway, Handler sounds like a pretty good name. Let me think it over.

> Is DelayedCall a subclass of Future, like Task? If so, FutureCall might
> work.

No, they're completely related. (I'm even thinking of renaming its
cancel() to avoid the confusion?

I still like Handler best. In fact, if I'd thought of Handler before,
I wouldn't have asked for a better name. :-)

Going once, going twice...

[Wall-clock timers]
> If someone really does want a wall-clock timer with a given granularity, it
> can be handled by adding a repeating timer with that granularity (with the
> obvious consequences for low power modes).

+1.

[Multiple calls per FD]
>> That makes sense. If we wanted to be fancy we could have several
>> different APIs: add (must not be set), set (may be set), replace (must
>> be set). But I think just offering the add and remove APIs is nicely
>> minimalistic and lets you do everything else with ease. (I'll make the
>> remove API return True if it did remove something, False otherwise.)

> Perhaps the best bet would be to have the standard API allow multiple
> callbacks, and emulate that on systems which don't natively support multiple
> callbacks for a single event?

Hm. AFAIK Twisted doesn't support this either. Antoine, do you know? I
didn't see it in the Tornado event loop either.

> Otherwise, I don't see how an event loop could efficiently expose access to
> the multiple callback APIs without requiring awkward fallbacks in the code
> interacting with the event loop. Given that the natural fallback
> implementation is reasonably clear (i.e. a single callback that calls all of
> the other callbacks), why force reimplementing that on users rather than
> event loop authors?

But what's the use case?

I don't think our goal should be to offer APIs for any feature that
any event loop might offer. It's not quite a least-common denominator
either though -- it's about offering commonly needed functionality,
and interoperability.

Also, event loop implementations are allowed to offer additional APIs
on their implementation. If the need for multiple handlers per FD only
exists on those platforms where the platform's event loop supports it,
no harm is done if the functionality is only available through a
platform-specific API.

But still, I don't understand the use case. Possibly it is using file
descriptors as a more general signaling mechanism? That sounds pretty
platform specific anyway (on Windows, FDs must represent sockets).

If someone shows me a real-world use case I may change my mind.

> Related, the protocol/transport API design may end up needing to consider
> the gather/scatter problem (i.e. fanning out data from a single transport to
> multiple consumers, as well as feeding data from multiple producers into a
> single underlying transport). Actual *implementations* of such tools
> shouldn't be needed in the standard suite, but at least understanding how
> you would go about writing multiplexers and demultiplexers can be a good
> test of a stacked I/O design.

Twisted supports this for writing through its writeSequence(), which
appears in Tulip and PEP 3156 as writelines(). (Though IIRC Glyph told
me that Twisted rarely uses the platform's scatter/gather primitives,
because they are so damn hard to use, and the kernel implementation
often just joins the buffers together before passing it to the regular
send()...)

But regardless, I don't think scatter/gather would use multiple
callbacks per FD.

I think it would be really hard to benefit from reading into multiple
buffers in Python.

>> > Just enabling/disabling these events is a bit more friendly to the
>> > programmer IMHO than having to cancel and recreate them when needed.
>>
>> The methods on the Transport class take care of this at a higher
>> level: pause() and resume() to suspend reading, and the write() method
>> takes care of buffering and so on.

> And the main advantage of handling that at a higher level is that suitable
> buffering designs are going to be transport specific.

+1

-- 
--Guido van Rossum (python.org/~guido)