[Twisted-Python] twisted run out of file descriptors
Hi I am running twisted on 2.4.21 Kernel. My chat server has been puking and dying the last couple days and it seemed pretty clear that it was dying when it was reaching 1024 open file descriptors. I noticed the default ulimit -n was 1024, so I added a 'ulimit -n 4096' before starting the chat server. That has apparently allowed it to surpass 1024 descriptors (it has 1263 at the moment) but now it is failing to call select (see sample errors below). There is this value FD_SETSIZE in the C headers that determines the maximum number of sockets a proc can select() on, and it's set to 1024. It it possible to increase this limit, short of recompiling the Kernel? Sample Errors: ----- File "/usr/local/encap/Twisted-1.2.0/lib/python2.3/site-packages/twisted/ internet/default.py", line 487, in doSelect [], timeout) exceptions.ValueError: filedescriptor out of range in select() 2004/10/25 16:35 PDT [-] Malformed file descriptor found. Preening lists. 2004/10/25 16:35 PDT [-] bad descriptor Thanks Alec matusis
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 check your /proc/sys/fs/file-max file. It's the max number of file descriptors the kernel will handle. if it's 1024 just set it to something higher (2048 or 4096) I don't know if that's really the reason, but it doesn't hurt either to give it a shot On Monday 25 October 2004 09:40 pm, Alec Matusis wrote:
- -- UC - -- Open Source Solutions 4U, LLC 2570 Fleetwood Drive Phone: +1 650 872 2425 San Bruno, CA 94066 Cell: +1 650 302 2405 United States Fax: +1 650 872 2417 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) iD8DBQFBfeKBjqGXBvRToM4RAulDAKC3SoSVvIWld/fk4o0cr4i2iVJRFgCglSWc ejWGhr5mlcVj76u2q70TlF0= =LY3Q -----END PGP SIGNATURE-----
Hi- thanks - that number i have changed- but that's total number for the ENTIRE system alecm@web0:~> cat /proc/sys/fs/file-max 200000 I also changed the number of fds available PER child PROCESS in a shell ulimit -n 4096 select() call has it's own hardcoded number apparently... -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com]On Behalf Of Uwe C. Schroeder Sent: Monday, October 25, 2004 10:37 PM To: Twisted general discussion Subject: Re: [Twisted-Python] twisted run out of file descriptors -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 check your /proc/sys/fs/file-max file. It's the max number of file descriptors the kernel will handle. if it's 1024 just set it to something higher (2048 or 4096) I don't know if that's really the reason, but it doesn't hurt either to give it a shot On Monday 25 October 2004 09:40 pm, Alec Matusis wrote:
- -- UC - -- Open Source Solutions 4U, LLC 2570 Fleetwood Drive Phone: +1 650 872 2425 San Bruno, CA 94066 Cell: +1 650 302 2405 United States Fax: +1 650 872 2417 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) iD8DBQFBfeKBjqGXBvRToM4RAulDAKC3SoSVvIWld/fk4o0cr4i2iVJRFgCglSWc ejWGhr5mlcVj76u2q70TlF0= =LY3Q -----END PGP SIGNATURE----- _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Alec Matusis wrote:
Yes (in some libcs) but you don't want to play with that thing. Switch to pollreactor. Poll has no builtin limits, but will start to get slow after a few thousand fds. But, do you know why your app has over a thousand fds open? If there's no really good reason, you have a leak and should fix that. A thousand fds is quite a lot.
My application is a chat, and it has > 1024 simultaneously connected chat clients at the peak of usage. I checked the numbers of fds in /proc/PID/fd before, during and after a client connect- there is no leak, I just have > 1024 people connected... I was trying to avoid using poll() but I will have to resort to it if there is no other choice. Do you know how where to find instructions to change this in glibc ? I have been looking in Google, but to no avail... Thanks. -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com]On Behalf Of Tommi Virtanen Sent: Tuesday, October 26, 2004 12:25 AM To: Twisted general discussion Subject: Re: [Twisted-Python] twisted run out of file descriptors Alec Matusis wrote:
Yes (in some libcs) but you don't want to play with that thing. Switch to pollreactor. Poll has no builtin limits, but will start to get slow after a few thousand fds. But, do you know why your app has over a thousand fds open? If there's no really good reason, you have a leak and should fix that. A thousand fds is quite a lot. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
I thought it would be faster with select() rather than poll() - lower load on the machine? -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com]On Behalf Of Tommi Virtanen Sent: Tuesday, October 26, 2004 1:28 AM To: Twisted general discussion Subject: Re: [Twisted-Python] twisted run out of file descriptors Alec Matusis wrote:
I was trying to avoid using poll() but I will have to resort to it if there is no other choice.
Why would you want to do that? _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Alec Matusis wrote:
I thought it would be faster with select() rather than poll() - lower load on the machine?
Not for any realistic application. On some platforms, select is just a wrapper around poll, anyway. select is fundamentally flawed for large numbers of FDs. Use poll. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
On Tue, 2004-10-26 at 01:35 -0700, Alec Matusis wrote:
I thought it would be faster with select() rather than poll() - lower load on the machine?
Sounds like you might be confused - if someone had told you that select() is faster than *polling*, that would be accurate - the terminology is confusing because "polling" means "repeatedly checking for input in a busy-loop". In this context, "polling" is actually the opposite of "calling poll()". poll() is simply a better version of select(). The only reason it is not the default reactor is that there are some obscure places where select() exists and it does not. I am not sure if this is even relevant any more, since I don't know if any such place can build recent versions of Python.
On Oct 26, 2004, at 11:31, Glyph Lefkowitz wrote:
Mac OS X prior to 10.3 does not ship with an implementation of poll. Mac OS X 10.3 has an implementation of poll, but it is a wrapper for select. #ifndef POLL_NO_WARN #warning "poll() functionality for Mac OS X is implemented via an emulation layer on top of select(), not in the kernel directly. It is recommended that programs running under OS X 10.3 prefer select() over poll(). Configure scripts should look for the _POLL_EMUL_H_ define (instead of _POLL_H_ or _SYS_POLL_H_) and avoid implementations where poll is not implemented in the kernel." #endif -bob
I changed reactor to poll(), and it has sucessfully surpassed 1024 file descriptors limit, while the load on the machine did not increase significantly from select() based reactor. Thanks everybody for the help Alec Matusis -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com]On Behalf Of Glyph Lefkowitz Sent: Tuesday, October 26, 2004 8:31 AM To: Twisted general discussion Subject: RE: [Twisted-Python] twisted run out of file descriptors On Tue, 2004-10-26 at 01:35 -0700, Alec Matusis wrote:
I thought it would be faster with select() rather than poll() - lower load on the machine?
Sounds like you might be confused - if someone had told you that select() is faster than *polling*, that would be accurate - the terminology is confusing because "polling" means "repeatedly checking for input in a busy-loop". In this context, "polling" is actually the opposite of "calling poll()". poll() is simply a better version of select(). The only reason it is not the default reactor is that there are some obscure places where select() exists and it does not. I am not sure if this is even relevant any more, since I don't know if any such place can build recent versions of Python. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 check your /proc/sys/fs/file-max file. It's the max number of file descriptors the kernel will handle. if it's 1024 just set it to something higher (2048 or 4096) I don't know if that's really the reason, but it doesn't hurt either to give it a shot On Monday 25 October 2004 09:40 pm, Alec Matusis wrote:
- -- UC - -- Open Source Solutions 4U, LLC 2570 Fleetwood Drive Phone: +1 650 872 2425 San Bruno, CA 94066 Cell: +1 650 302 2405 United States Fax: +1 650 872 2417 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) iD8DBQFBfeKBjqGXBvRToM4RAulDAKC3SoSVvIWld/fk4o0cr4i2iVJRFgCglSWc ejWGhr5mlcVj76u2q70TlF0= =LY3Q -----END PGP SIGNATURE-----
Hi- thanks - that number i have changed- but that's total number for the ENTIRE system alecm@web0:~> cat /proc/sys/fs/file-max 200000 I also changed the number of fds available PER child PROCESS in a shell ulimit -n 4096 select() call has it's own hardcoded number apparently... -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com]On Behalf Of Uwe C. Schroeder Sent: Monday, October 25, 2004 10:37 PM To: Twisted general discussion Subject: Re: [Twisted-Python] twisted run out of file descriptors -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 check your /proc/sys/fs/file-max file. It's the max number of file descriptors the kernel will handle. if it's 1024 just set it to something higher (2048 or 4096) I don't know if that's really the reason, but it doesn't hurt either to give it a shot On Monday 25 October 2004 09:40 pm, Alec Matusis wrote:
- -- UC - -- Open Source Solutions 4U, LLC 2570 Fleetwood Drive Phone: +1 650 872 2425 San Bruno, CA 94066 Cell: +1 650 302 2405 United States Fax: +1 650 872 2417 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) iD8DBQFBfeKBjqGXBvRToM4RAulDAKC3SoSVvIWld/fk4o0cr4i2iVJRFgCglSWc ejWGhr5mlcVj76u2q70TlF0= =LY3Q -----END PGP SIGNATURE----- _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Alec Matusis wrote:
Yes (in some libcs) but you don't want to play with that thing. Switch to pollreactor. Poll has no builtin limits, but will start to get slow after a few thousand fds. But, do you know why your app has over a thousand fds open? If there's no really good reason, you have a leak and should fix that. A thousand fds is quite a lot.
My application is a chat, and it has > 1024 simultaneously connected chat clients at the peak of usage. I checked the numbers of fds in /proc/PID/fd before, during and after a client connect- there is no leak, I just have > 1024 people connected... I was trying to avoid using poll() but I will have to resort to it if there is no other choice. Do you know how where to find instructions to change this in glibc ? I have been looking in Google, but to no avail... Thanks. -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com]On Behalf Of Tommi Virtanen Sent: Tuesday, October 26, 2004 12:25 AM To: Twisted general discussion Subject: Re: [Twisted-Python] twisted run out of file descriptors Alec Matusis wrote:
Yes (in some libcs) but you don't want to play with that thing. Switch to pollreactor. Poll has no builtin limits, but will start to get slow after a few thousand fds. But, do you know why your app has over a thousand fds open? If there's no really good reason, you have a leak and should fix that. A thousand fds is quite a lot. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
I thought it would be faster with select() rather than poll() - lower load on the machine? -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com]On Behalf Of Tommi Virtanen Sent: Tuesday, October 26, 2004 1:28 AM To: Twisted general discussion Subject: Re: [Twisted-Python] twisted run out of file descriptors Alec Matusis wrote:
I was trying to avoid using poll() but I will have to resort to it if there is no other choice.
Why would you want to do that? _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Alec Matusis wrote:
I thought it would be faster with select() rather than poll() - lower load on the machine?
Not for any realistic application. On some platforms, select is just a wrapper around poll, anyway. select is fundamentally flawed for large numbers of FDs. Use poll. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
On Tue, 2004-10-26 at 01:35 -0700, Alec Matusis wrote:
I thought it would be faster with select() rather than poll() - lower load on the machine?
Sounds like you might be confused - if someone had told you that select() is faster than *polling*, that would be accurate - the terminology is confusing because "polling" means "repeatedly checking for input in a busy-loop". In this context, "polling" is actually the opposite of "calling poll()". poll() is simply a better version of select(). The only reason it is not the default reactor is that there are some obscure places where select() exists and it does not. I am not sure if this is even relevant any more, since I don't know if any such place can build recent versions of Python.
On Oct 26, 2004, at 11:31, Glyph Lefkowitz wrote:
Mac OS X prior to 10.3 does not ship with an implementation of poll. Mac OS X 10.3 has an implementation of poll, but it is a wrapper for select. #ifndef POLL_NO_WARN #warning "poll() functionality for Mac OS X is implemented via an emulation layer on top of select(), not in the kernel directly. It is recommended that programs running under OS X 10.3 prefer select() over poll(). Configure scripts should look for the _POLL_EMUL_H_ define (instead of _POLL_H_ or _SYS_POLL_H_) and avoid implementations where poll is not implemented in the kernel." #endif -bob
I changed reactor to poll(), and it has sucessfully surpassed 1024 file descriptors limit, while the load on the machine did not increase significantly from select() based reactor. Thanks everybody for the help Alec Matusis -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com]On Behalf Of Glyph Lefkowitz Sent: Tuesday, October 26, 2004 8:31 AM To: Twisted general discussion Subject: RE: [Twisted-Python] twisted run out of file descriptors On Tue, 2004-10-26 at 01:35 -0700, Alec Matusis wrote:
I thought it would be faster with select() rather than poll() - lower load on the machine?
Sounds like you might be confused - if someone had told you that select() is faster than *polling*, that would be accurate - the terminology is confusing because "polling" means "repeatedly checking for input in a busy-loop". In this context, "polling" is actually the opposite of "calling poll()". poll() is simply a better version of select(). The only reason it is not the default reactor is that there are some obscure places where select() exists and it does not. I am not sure if this is even relevant any more, since I don't know if any such place can build recent versions of Python. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (8)
-
Alec Matusis -
Anthony Baxter -
Bob Ippolito -
George Schlossnagle -
Glyph Lefkowitz -
Itamar Shtull-Trauring -
Tommi Virtanen -
Uwe C. Schroeder