[Python-Dev] epoll implementation

[Ross Cohen]

On Sat, May 27, 2006 at 02:27:20AM +0100, Steve Holden wrote:
> Greg Ewing wrote:
> > Rather than adding yet another platform-dependent module,
> > I'd like to see a unified Python interface in the stdlib
> > that uses whichever is the best one available.
> > 
> Of course that would mean establishing which *was* the best available 
> which, as we've seen this week, may not be easy.

Lots of systems (Windows? what's that?) implement both select and poll. I
have yet to see a system which implements more than 1 of epoll, kqueue,
/dev/poll, I/O completion ports, etc.

You'd have to search long and hard for a case where the non-select/poll
interfaces didn't perform better. Correctness/completeness is, of course,
a stronger criterion than performance, so if kqueue on OS X doesn't support
all relevant types of file descriptors, then it shouldn't used.

Fortunately, the python select.poll interface is a good match for all of
epoll, kqueue and /dev/poll. kqueue supports watching a lot more than file
descriptors, but that doesn't matter much for this discussion. kqueue also
mashes register/unregister/poll into a single call, which contributes to
how confusing the interface is, though from what I can tell it really is
basically the same as epoll.

As for Windows, the WSASelect interface is truly awful. The python library
currently sets it so you can pass in up to 512 descriptors, but don't count
on getting even that many. There is a hard limit on the number of sockets
a *user* can have open (varies based on OS version) with WSASelect and it
will just start returning WSAENOBUFS once you go over. You then have the
task of guessing how many you need to close before it will start working
again. Firewalls especially can eat into the pool.

IOCP has neither the WSAENOBUFS problem nor the 512 descriptor limit. It
would be nice (if you or your customers run Windows) if someone did a
select.poll interface with IOCP as a backend.

Ross