[ python-Bugs-1494314 ] Cannot use high-numbered sockets in 2.4.3

[SourceForge.net]

Bugs item #1494314, was opened at 2006-05-24 09:51
Message generated for change (Comment added) made by akuchling
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1494314&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Extension Modules
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Michael Smith (mlrsmith)
Assigned to: Nobody/Anonymous (nobody)
Summary: Cannot use high-numbered sockets in 2.4.3

Initial Comment:
Python 2.4.3 introduced (in Modules/socketmodule.c) the
IS_SELECTABLE() macro, to check whether calling
select() on a given file descriptor is permissible.

However, it executes this check even when it won't
actually call select(). Specifically, select() is
called ONLY when s->sock_timeout > 0 (in
internal_select mostly, but also internal_connect).

I have a twisted application which uses many FDs
(several thousand), and upgrading python to 2.4.3 makes
it hit this limit (at 1024 file descriptors),
regardless of ulimit settings. Twisted itself always
uses non-blocking I/O on sockets, so with older
versions of python this worked fine.

A simple solution relies on the fact that select is
only ever actually called, and changes the
IS_SELECTABLE macro as in the attached fairly trivial
patch. This is sufficient to restore my application to
its previous state (where it works fine).

This doesn't solve the more general problem in
socketmodule - that we simply can't do all operations
properly with the current reliance on select(), though,
and it seems like a bit of a hack...  If I wrote up a
patch to use poll() on systems where it's available,
throughout socketmodule.c, in preference to select(),
would this be accepted?

Mike



----------------------------------------------------------------------

>Comment By: A.M. Kuchling (akuchling)
Date: 2006-05-31 14:39

Message:
Logged In: YES 
user_id=11375

I expect such a patch would be acceptable.  The largest
issue is probably whether poll() is available everywhere, or
if we'd be stuck maintaining both select() and poll() based
versions of internal_select().  Does Windows support poll()
at all?

----------------------------------------------------------------------

Comment By: Michael Smith (mlrsmith)
Date: 2006-05-29 10:13

Message:
Logged In: YES 
user_id=1488997

Yes, I had my ulimit set appropriately.

There's no problem with _opening_ many sockets (at least, I
don't think there is) - the problem is with trying to do
something (like call socket.recv()) with them.

The code in socketmodule.c is pretty clear - and having
upgraded to 2.4.3 with ubuntu dapper, I _am_ running into this.

For now, we're having to keep all our production machines on
2.4.2.


----------------------------------------------------------------------

Comment By: Gabriel Wicke (gabrielwicke)
Date: 2006-05-29 10:11

Message:
Logged In: YES 
user_id=956757

Please disregard my comments completely- just opening more
than 1024 files does not trigger this bug, but doing a
socket.send() on the 1025th socket does.

----------------------------------------------------------------------

Comment By: Gabriel Wicke (gabrielwicke)
Date: 2006-05-29 10:00

Message:
Logged In: YES 
user_id=956757

Never mind- you already tried ulimit. It still works for me
though, 2.4.3 from Ubuntu Dapper.

----------------------------------------------------------------------

Comment By: Gabriel Wicke (gabrielwicke)
Date: 2006-05-29 09:57

Message:
Logged In: YES 
user_id=956757

Try "ulimit -n 8096" (only permitted in a root shell) to set
your max socket usage to something larger. Opening more than
1024 sockets works for me in python 2.4.3 after changing the
limit for the terminal in question and restarting the
interpreter. Just ulimit -n will show you your current
limit. Ulimits are enforced by the Linux kernel and can
likely be configured system-wide in /etc/sysctl.cfg.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1494314&group_id=5470