[Twisted-Python] Can I benefit from multi-processor hardware?
Our current Windows server, written thoroughly in C++, is spawning a few threads in order to benefit from multi-processor hardware. Coz our app logic is not so flexible, now we consider moving to Twisted. Can we still benefit from multi-processor hardware with Twisted's basic event-driven architecture?
At 01:03 PM 5/24/04 +0900, June Jae-jun Kang wrote:
Can we still benefit from multi-processor hardware with Twisted s basic event-driven architecture?
Only if you use multiple processes. The Python interpreter has a global lock that's used to protect potentially shared data structures, so this inhibits multi-processor execution of pure Python code. Portions of a Python program that are written in C or C++ can be free-threaded, but if you are doing code that's pure Python, you'll need multiple processes. So, to take advantage of multi-processor hardware, you can do your calculation-intensive work in separate processes, with the network communications being handled by an event-driven front end. Note that this is basically true for *any* Python program, not just Twisted.
On Mon, May 24, 2004 at 12:56:38AM -0400, Phillip J. Eby wrote: [...]
So, to take advantage of multi-processor hardware, you can do your calculation-intensive work in separate processes, with the network communications being handled by an event-driven front end. Note that this is basically true for *any* Python program, not just Twisted.
Or, potentially, inside free-threaded C or C++ extensions... it depends on the situation. I wrote a Twisted server on Windows that replaced a C++ one quite comfortably, partly due to much of the work being talking to an SQL database, which you can do in threads easily (see the twisted.enterprise package), while handling the networking with Twisted. -Andrew.
On Mon, 24 May 2004 13:03:02 +0900 "June Jae-jun Kang" <june@digicell.co.kr> wrote:
Our current Windows server, written thoroughly in C++,
is spawning a few threads in order to benefit from multi-processor hardware.
Coz our app logic is not so flexible, now we consider moving to Twisted.
Can we still benefit from multi-processor hardware with Twisted's basic event-driven architecture?
Well, iocpreactor, which uses IO completion ports, could be extended to use a threadpool (running C code exclusively) to handle IO events. I am not sure whether that will give a performance gain, though, because the main thread might still be a bottleneck. In any case, give iocpreactor a try. It is currently the only way for Twisted to scale up on Windows. It's a bit immature, so if you see anything weird, yell at me.
participants (4)
-
Andrew Bennetts -
June Jae-jun Kang -
Pavel Pergamenshchik -
Phillip J. Eby