[Python-Dev] Pervasive socket failures on Windows

["Martin v. Löwis"]

Thomas Wouters wrote:
> I doubt it will have problems on other platforms. As Tim said, FD_SETSIZE is
> mandated by POSIX. Perhaps some platforms do allow larger sizes, by
> replacing the FD_* macros with functions that dynamically grow whatever
> magic is the 'fdset' datatype. I sincerely doubt it's a common approach,
> though, and for them to be POSIX they would need to have FD_SETSIZE set to
> some semi-sane value. So at worst, on those platforms (if any), we're
> reducing the number of sockets you can actually select() on, from some
> undefined platform maximum to whatever the platform *claims* is the maximum.

I think the Windows interpretation is actually well-designed: FD_SETSIZE
shouldn't be the number of the largest descriptor, but instead be the
maximum size of the set. So FD_SETSIZE is 64 on Windows, but you still
can have much larger file descriptor numbers.

The implementation strategy of Windows is to use an array of integers,
rather than the bit mask, and an index telling you how many slots have
already been filled. With FD_SETSIZE being 64, the fd_set requires
256 bytes.

This strategy has a number of interesting implications:
- a naive implementation of FD_SET is not idempotent; old winsock
  implementations where so naive. So you might fill the set by
  setting the same descriptor 64 times. Current implementations
  use a linear search to make the operation idempotent.
- FD_CLR needs to perform a linear scan for the descriptor,
  and then shift all subsequent entries by one (it could actually
  just move the very last entry to the deleted slot, but doesn't)

In any case, POSIX makes it undefined what FD_SET does when the
socket is larger than FD_SETSIZE, and apparently clearly expects
an fd_set to be a bit mask.

Regards,
Martin