[Twisted-Python] Performance and select.select

Hello, I profiled my application with cProfiler. Below is an excerpt from the resulting report: Fri Jan 04 01:07:39 2008 profiling_data_cProfile 15123197 function calls (14579262 primitive calls) in 228.517 CPU seconds Ordered by: internal time List reduced from 2933 to 50 due to restriction <50> ncalls tottime percall cumtime percall filename:lineno(function) 138305 135.419 0.001 135.817 0.001 {select.select} 26168 13.159 0.001 14.044 0.001 Sources\Server\Plugins\RKSimulation.py:218(UpdatePhysics) As you can see select.select takes up ~136 of ~229 seconds. This is the most expensive function. The row below shows the next expensive function which is one of my own. Now I am wondering whether a) the select calls are really taking up that much time (more than 50%!) b) if they are taking up that much time I'd like to find out how to reduce their execution time (either by minimizing the number it gets called or the time it takes to execute etc) My application doesn't invoke select.select itself so it is most likely called within twisted. I am on Windows and using the default reactor ("from twisted.internet import reactor"). -Matthias

a) the select calls are really taking up that much time (more than 50%!)
They're certainly going to take up lots and lots of *clock* time, not necessarily CPU time: select() is the way Twisted (in default reactors) waits for events to happen. So if your program does nothing at all (e.g. just reactor.run(), select() will actually be close to 100% of clock time, even though CPU usage will be minuscule. If you have many TCP connections (or really, file descriptors), hundreds or thousands, select() will also start taking up CPU time. Using poll reactor, or in next version of Twisted (2.5 had some bugs) epoll reactor on Linux will use less CPU for this scenario.

Am 04.01.2008, 04:18 Uhr, schrieb Itamar Shtull-Trauring <itamar@itamarst.org>:
a) the select calls are really taking up that much time (more than 50%!)
They're certainly going to take up lots and lots of *clock* time, not necessarily CPU time: select() is the way Twisted (in default reactors) waits for events to happen. So if your program does nothing at all (e.g. just reactor.run(), select() will actually be close to 100% of clock time, even though CPU usage will be minuscule.
I guess this is what cProfile is actually outputting. Clock time and not cpu time. If the app is under more load, the percentage of the time spent in select() calls decreases to a minimum. So select() isn't bad then.
If you have many TCP connections (or really, file descriptors), hundreds or thousands, select() will also start taking up CPU time. Using poll reactor, or in next version of Twisted (2.5 had some bugs) epoll reactor on Linux will use less CPU for this scenario.
I have very few connections, so this was not the problem. Only cProfile being a bit strange. -Matthias

On Fri, 04 Jan 2008 03:42:10 +0100, Nitro <nitro@dr-code.org> wrote:
Hello,
I profiled my application with cProfiler. Below is an excerpt from the resulting report:
Fri Jan 04 01:07:39 2008 profiling_data_cProfile
15123197 function calls (14579262 primitive calls) in 228.517 CPU seconds
Ordered by: internal time List reduced from 2933 to 50 due to restriction <50>
ncalls tottime percall cumtime percall filename:lineno(function) 138305 135.419 0.001 135.817 0.001 {select.select} 26168 13.159 0.001 14.044 0.001 Sources\Server\Plugins\RKSimulation.py:218(UpdatePhysics)
As you can see select.select takes up ~136 of ~229 seconds. This is the most expensive function. The row below shows the next expensive function which is one of my own. Now I am wondering whether
a) the select calls are really taking up that much time (more than 50%!) b) if they are taking up that much time I'd like to find out how to reduce their execution time (either by minimizing the number it gets called or the time it takes to execute etc)
My application doesn't invoke select.select itself so it is most likely called within twisted. I am on Windows and using the default reactor ("from twisted.internet import reactor").
Twisted does use select by default on Windows. One thing you might try is one of the other reactors available on Windows; there are two, win32eventreactor and iocpreactor. You can read about how to do this here: http://twistedmatrix.com/projects/core/documentation/howto/choosing-reactor.... Depending on what your application does, iocpreactor may not be complete enough to actually run your code. It lacks some features provided by the default reactor. Whether either of these reactors will actually improve the performance of your application, I don't know. You pointed out that a lot of time is spent in select, but this doesn't mean that spending less time inside the event notification function will make your overall application faster. Also, you are right to be suspicious of these profile results. Python has a very spotty history when it comes to profilers. They are often inaccurate or completely wrong. cProfile is supposed to be better than the previous attempts, but I haven't personally used it extensively enough (nor at all on Windows) to say whether this is true. Some things to consider are: * How many simultaneous connections does your application have? select usually scales linearly with this number. I'm not sure exactly what it does on Windows, though. * How often do connections have events? Is data passing through them constantly or are they mostly idle? More events means more trips into select, which incurs setup costs more. Fewer events means select should be spending more of its time idling, waiting for events. * How often do timed events go off? These also cause more trips into select, since each one causes select to return when it is time to run. Do you use reactor.callLater() with a 0 timeout a lot? Hope this helps, Jean-Paul

Am 04.01.2008, 04:19 Uhr, schrieb Jean-Paul Calderone <exarkun@divmod.com>:
Twisted does use select by default on Windows. One thing you might try is one of the other reactors available on Windows; there are two, win32eventreactor and iocpreactor. You can read about how to do this here:
http://twistedmatrix.com/projects/core/documentation/howto/choosing-reactor....
Depending on what your application does, iocpreactor may not be complete enough to actually run your code. It lacks some features provided by the default reactor.
Whether either of these reactors will actually improve the performance of your application, I don't know. You pointed out that a lot of time is spent in select, but this doesn't mean that spending less time inside the event notification function will make your overall application faster.
Also, you are right to be suspicious of these profile results. Python has a very spotty history when it comes to profilers. They are often inaccurate or completely wrong. cProfile is supposed to be better than the previous attempts, but I haven't personally used it extensively enough (nor at all on Windows) to say whether this is true.
As I explained in my other post, cProfile's output wasn't what I expected it to be. I tried IOCP reactor nevertheless and it worked for my application. Can't see a real performance difference, I guess I am not performance limited by twisted anymore. At least for the moment being :-)
* How often do timed events go off? These also cause more trips into select, since each one causes select to return when it is time to run. Do you use reactor.callLater() with a 0 timeout a lot?
What is a lot? I am probably using it 10-50 times per iteration. Just out of curiosity - since my real problem is solved - which difference do those callLater(0, ...) calls make? Thanks for helping, also thanks to Itamar. -Matthias

On Fri, 2008-01-04 at 03:42 +0100, Nitro wrote:
My application doesn't invoke select.select itself so it is most likely called within twisted. I am on Windows and using the default reactor ("from twisted.internet import reactor").
Ah. In general: 1. Linux is better ;) 2. On Windows IOCP is much better than select() in terms of scalability (i.e. low CPU usage with many connections), but IOCP reactor isn't finished and thus has some limitations in functionality and potentially some bugs.
participants (3)
-
Itamar Shtull-Trauring
-
Jean-Paul Calderone
-
Nitro