
Hi Markus, It turns out that NT has a default 64 fd limit on arguments to select(). The good news is that you can actually bump the limit up to whatever number you want by specifying a define when compiling python15.dll. If you have the ability to rebuild your python15.dll, you can add the define: FD_SETSIZE=1024 to the preprocessor options for the python15 project to raise the limit to 1024 fds. The default 64 fd limit is too low for anyone trying to run an async server that handles even a modest load, so I've submitted a bug report to python.org asking that the define above find its way into the next python release... Brian Lloyd brian@digicool.com Software Engineer 540.371.6909 Digital Creations http://www.digicool.com

Brian, (Also in response to your bug report.) I'm a little worried that upping the limit to 1024 would cause some performance problems if you're making a lot of select() calls. The select allocates three arrays of length FD_SETSIZE+3; each array item is 12 bytes. This is a total allocation of more than 36K for a meager select() call! And all that memory also has to be cleared by the FD_ZERO() call. If you actually have that many sockets, that's worth paying for (the socket objects themselves use up just as much memory, and your Python data structures for the sockets, no matter how small, are probably several times bigger), but for a more typical program, I see this as a lot of overhead. Is there a way that this can be done more dynamically, e.g. by making the set size as big as needed on windows but no bigger? (Before you suggest allocating that memory statically, remember it's possible to call select from multiple threads. Allocating 36K of thread-local space for each thread also doesn't sound too pleasant.) --Guido van Rossum (home page: http://www.python.org/~guido/)

Brian, (Also in response to your bug report.) I'm a little worried that upping the limit to 1024 would cause some performance problems if you're making a lot of select() calls. The select allocates three arrays of length FD_SETSIZE+3; each array item is 12 bytes. This is a total allocation of more than 36K for a meager select() call! And all that memory also has to be cleared by the FD_ZERO() call. If you actually have that many sockets, that's worth paying for (the socket objects themselves use up just as much memory, and your Python data structures for the sockets, no matter how small, are probably several times bigger), but for a more typical program, I see this as a lot of overhead. Is there a way that this can be done more dynamically, e.g. by making the set size as big as needed on windows but no bigger? (Before you suggest allocating that memory statically, remember it's possible to call select from multiple threads. Allocating 36K of thread-local space for each thread also doesn't sound too pleasant.) --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (2)
-
Brian Lloyd
-
Guido van Rossum