NT select.select?

[Bruce Dodson]

It does cause a huge load, because in Winsock an fd_set is represented as an
array of SOCKETs, not as a bitmap.  So in a normal program, which uses
select for simple things like polling a single FD, a large FD_SETSIZE will
degrade performance.  For example, if FD_SETSIZE were 4096, then
select.select() would allocate about 48KB (~16KB per fd_set) on the stack
whenever it is called, even if there are just a few calls.

The good news is that, since those fake fd_sets are always passed by
reference, it is safe for the core Python to define a small FD_SETSIZE like
64 while some module overrides the builtin select with another that defines
a larger FD_SETSIZE.  Then no changes to "core" python would be required,
and only those who want the bigger fd_sets will get them.

Another thing that you can do is just dynamically allocate an array of
32-bit integers: the first holds the count, and the others hold the SOCKETs.
You can fill this without using the FD_SET macros if you are careful to
avoid duplicate FDs, or you can define FD_SETSIZE to be huge, cast the
allocated memory to fd_set *, and use the macros.  (It won't matter that
your buffer is smaller than FD_SETSIZE, because you know you're going to run
out of elements before you run out of buffer.)  If your malloc does a
reasonable job of handling small and large objects, and of re-using memory
quickly, this could be pretty much ideal.

Bruce