[Twisted-Python] Maximum number of SSL connections to the server
Hi all, I am doing load testing on my twisted SSL server. I spawn threads(~2000) from the application and each thread will act as a SSL client and connect to the server. Client code is straight forward. It creates a socket, does SSL wrap and connects to the server. Once all threads connect to the server, every thread closes its socket. During my initial tests, I found that after 245 successful connections, all remaining threads started to fail with this error on the client side. *SSLError: [Errno 8] _ssl.c:503: EOF occurred in violation of protocol* Then I figured out that its an issue with number of open files which was by default - 256. I changed the limit for server and client to 10000. It successfully completed the 500 connections test. Now it fails after 1010 successful connections. I checked the number of open files of the server process it remains at 1010 and there are no stale open files. Client shows the same error *SSLError: [Errno 8] _ssl.c:503: EOF occurred in violation of protocol* Twisted logs on the server shows that it connects first and then disconnects with this error. *<127.0.0.1:5549 <http://127.0.0.1:5549>> connected* *<127.0.0.1:55249 <http://127.0.0.1:55249>> lost connection: [('SSL routines', 'SSL23_READ', 'ssl handshake failure')]* How can I fix this ? How should I approach to debug ? Thanks -Sumanth
Am Montag, 13. Januar 2014, 18:58:01 schrieb Sumanth Puram:
Now it fails after 1010 successful connections
which limit did you raise? Per process or systemwide? 1010 sound like you have a limit of 1024 somewhere -- Wolfgang
Hi Sumanth, Still sounds like an open file problem. What platform? How did you set the number of files higher that 256? Take a look at /etc/security/limits.conf :-) cheers lvh
Yes, 1010 looks more like a 1024 limit. I am using a Mac OSX 10.8. I have set the limit with 'ulimit -n 10000' for the current shell where I was running the server and the same for client. I have tried to increase the limit for number of files, but could not find any other way than this. sumanth$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 10000 pipe size (512 bytes, -p) 1 stack size (kbytes, -s) 65532 cpu time (seconds, -t) unlimited max user processes (-u) 709 virtual memory (kbytes, -v) unlimited On Mon, Jan 13, 2014 at 7:18 PM, Laurens Van Houtven <_@lvh.io> wrote:
Hi Sumanth,
Still sounds like an open file problem. What platform? How did you set the number of files higher that 256? Take a look at /etc/security/limits.conf :-)
cheers lvh
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi Sumanth, You probably want to look at the limits launchd is imposing. Consider: launchctl limit maxfiles 5000 5000 You can make this permanent by pumping that line (minus launchctl) into /etc/launchd.conf. hth lvh
On 01:28 pm, spuram@kiwiup.com wrote:
Hi all,
I am doing load testing on my twisted SSL server. I spawn threads(~2000) from the application and each thread will act as a SSL client and connect to the server.
[snip]
Now it fails after 1010 successful connections. I checked the number of open files of the server process it remains at 1010 and there are no stale open files.
select() has a hard limit that is often 1024. Make sure you're not using the select()-based reactor (which I think is probably the default on OS X). Jean-Paul
Client shows the same error *SSLError: [Errno 8] _ssl.c:503: EOF occurred in violation of protocol*
Twisted logs on the server shows that it connects first and then disconnects with this error.
*<127.0.0.1:5549 <http://127.0.0.1:5549>> connected* *<127.0.0.1:55249 <http://127.0.0.1:55249>> lost connection: [('SSL routines', 'SSL23_READ', 'ssl handshake failure')]*
How can I fix this ? How should I approach to debug ?
Thanks -Sumanth
On Mon, Jan 13, 2014 at 3:19 PM, <exarkun@twistedmatrix.com> wrote:
select() has a hard limit that is often 1024. Make sure you're not using the select()-based reactor (which I think is probably the default on OS X).
Jean-Paul
What is the A-grade reactor on OS X? I thought it was cfreactor, but the docs appear to brand it as a GUI reactor (that doesn't mean it can't be better than select, of course); perhaps it's kqueuereactor but I hear kqueue is kind of gimpy on OS X compared to FreeBSD (but I have no idea what that actually entails). cheers lvh
On Jan 13, 2014, at 6:22 AM, Laurens Van Houtven <_@lvh.io> wrote:
On Mon, Jan 13, 2014 at 3:19 PM, <exarkun@twistedmatrix.com> wrote: select() has a hard limit that is often 1024. Make sure you're not using the select()-based reactor (which I think is probably the default on OS X).
Jean-Paul
What is the A-grade reactor on OS X? I thought it was cfreactor, but the docs appear to brand it as a GUI reactor (that doesn't mean it can't be better than select, of course); perhaps it's kqueuereactor but I hear kqueue is kind of gimpy on OS X compared to FreeBSD (but I have no idea what that actually entails).
Actually, by default, cfreactor doesn't even support a GUI; you need to specifically install a different main loop (as the docs explain, "install(runner=PyObjCTools.AppHelper.runEventLoop)") in order to integrate with a GUI. The reason the docs brand it as such is that you cannot have an OS X GUI cooperate with your Twisted main loop using a different reactor. KQueue's gimpiness is entirely to do with deficiencies in its behavior with regard to weird file descriptor types, mostly PTYs. (I've seen some people talking about difficulties with pipes, but I've never had an issue with them personally; I don't know what that's about.) If all you need to do is talk to a lot of sockets, KQueue is just fine. -glyph
On Jan 13, 2014, at 3:37 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
KQueue's gimpiness is entirely to do with deficiencies in its behavior with regard to weird file descriptor types, mostly PTYs.
Hmm. And actually, I believe, this was fixed on Mavericks but we haven't updated our test configuration yet. I just enabled the kqueue reactor on my Mavericks machine, and: PTYProcessTestsBuilder_KQueueReactor test_changeGID ... [SKIPPED] test_changeUID ... [SKIPPED] test_openFileDescriptors ... [OK] test_processExitedRaises ... [OK] test_processExitedWithSignal ... [OK] test_processTransportInterface ... [OK] test_spawnProcessEarlyIsReaped ... [OK] test_systemCallUninterruptedByChildExit ... [OK] test_timelyProcessExited ... [OK] test_write ... [OK] test_writeSequence ... [OK] test_writeToChild ... [OK] test_writeToChildBadFileDescriptor ... [OK] -glyph
Thanks Jean. Changing reactor kQueue on OSX fixed this issue. On Mon, Jan 13, 2014 at 7:49 PM, <exarkun@twistedmatrix.com> wrote:
On 01:28 pm, spuram@kiwiup.com wrote:
Hi all,
I am doing load testing on my twisted SSL server. I spawn threads(~2000) from the application and each thread will act as a SSL client and connect to the server.
[snip]
Now it fails after 1010 successful connections. I checked the number of open files of the server process it remains at 1010 and there are no stale open files.
select() has a hard limit that is often 1024. Make sure you're not using the select()-based reactor (which I think is probably the default on OS X).
Jean-Paul
Client shows the same error *SSLError: [Errno 8] _ssl.c:503: EOF occurred in violation of protocol*
Twisted logs on the server shows that it connects first and then disconnects with this error.
*<127.0.0.1:5549 <http://127.0.0.1:5549>> connected* *<127.0.0.1:55249 <http://127.0.0.1:55249>> lost connection: [('SSL routines', 'SSL23_READ', 'ssl handshake failure')]*
How can I fix this ? How should I approach to debug ?
Thanks -Sumanth
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (5)
-
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
Laurens Van Houtven -
Sumanth Puram -
Wolfgang Rohdewald