[Python-ideas] Add a generic async IO poller/reactor to select module

[Giampaolo Rodolà]

2012/5/24 Antoine Pitrou <solipsis at pitrou.net>:
> On Thu, 24 May 2012 13:50:27 +0200
> Giampaolo Rodolà <g.rodola at gmail.com>
> wrote:
>> 2012/5/24 Ronald Oussoren <ronaldoussoren at mac.com>:
>> >
>> > On 24 May, 2012, at 3:32, Giampaolo Rodolà wrote:
>> >>>>>
>> >>
>> >> The handler is supposed to provide 3 methods:
>> >> - handle_read_event
>> >> - handle_write_event
>> >> - handle_error_event
>> >>
>> >> Users willing to support multiple event loops such as wx, gtk etc can do:
>> >>
>> >>>>> while 1:
>> >> ...       poller.poll(timeout=0.1, blocking=False)
>> >> ...       otherpoller.poll()
>> >>
>> >>
>> >> Basically, this would be the whole API.
>> >
>> > Isn't this a limited version of asyncore? (poller.poll == asyncore.loop, the handler is a subset of asyncore.dispatcher).
>> >
>> > Ronald
>>
>> poller.poll serves the same purpose of asyncore.loop, yes, but this is
>> supposed to be independent from asyncore.
>
> I agree with Ronald that it looks like a less-braindead version of
> asyncore. I don't think the select module is the right place.

Yeah, probably. Usually when I post here I'm the first one not being
sure whether what I propose is a good idea or not. =)
Anyway, it must be clear that what I have in mind is not related to
asyncore per-se.
The proposal is to add a *generic* poller/reactor to select module as
an abstraction layer on top of select(), poll(), epoll() and kqueue(),
that's all.

> Also, I don't know why you would specify poller.READ or poller.WRITE
> explicitly. Usually you are interested in all events, no?

Nope, that's what asyncore does and that's why it is significantly
slower compared to more modern and clever async loops (independenly
from the lack of epoll() / kqueue() support in asyncore).
You should only be interested in reading for accepting sockets
(servers) or when you want to receive data.
You should only be interested in writing for connecting sockets
(clients) or when you want to send data.
Being interested in both when, say, you only intend to receive data is
a considerable waste of time, especially when there are many
concurrent connections.
The performance degradation if you wildly look for both read and write
events is *huge*, see benchmarks referring to old vs. new select()
implementation here (~8.5x slowdown with 200 concurrent clients):
http://code.google.com/p/pyftpdlib/issues/detail?id=203#c6


--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/
http://code.google.com/p/pysendfile/