[Python-Dev] epoll implementation

[Jean-Paul Calderone]

On Fri, 26 May 2006 11:47:43 -0500, skip at pobox.com wrote:
>
>    Ross> I wrote an epoll implementation which can be used as a drop-in
>    Ross> replacement for parts of the select module
>    ...
>    Ross> Is there any interest in incorporating this into the standard
>    Ross> python distribution?
>
>Without going to the trouble of downloading epoll (always an adventure with
>SourceForget), can you explain what epoll does, how it's better than (parts
>of) select, how widely it's used and how stable it is?

epoll is a high-performance io notification mechanism provided by linux 2.6.  It supports more sockets than select: it has no FD_SETSIZE or equivalent. It is more efficient than select: it scales roughly with the number of events instead of with the number of sockets.  There is little or no controversy over its superiority to select and poll.

The idea was first introduced to linux about five years ago, but the API has changed a lot in the interval.  The epoll(4) API has been stable for all of linux 2.6.

Including a wrapper for this functionality would be quite useful for many python network apps.  However, I think with ctypes going into 2.5, it might be better to consider providing epoll support using a ctypes-based module rather than an extension module.

Of course, if there is a volunteer to maintain and support an extension module, that's better than nothing.  PyEpoll is missing a couple features I would like to see - the size of the epoll set is hard-coded to FD_SETSIZE, for example: while this makes little difference to Python 2.4.3 (since you cannot use sockets with fileno >= FD_SETSIZE at all in that version of Python), it is a serious limitation for other versions of Python.  Similarly, the number of events to retrieve when invoking epoll_wait() is also hardcoded to FD_SETSIZE.  Real applications often want to tune this value to avoid being overwhelmed by io events.

These features could easily be added, I suspect, since they are primarily just extra integer parameters to various methods.

Of course the other standard things should be added as well - documentation and test coverage, neither of which seem to be present at all in PyEpoll.

Jean-Paul