[Twisted-Python] scaling with lots of clients
Hi, I have a twisted Perspective Broker based server. Recently with an increased number of clients we are seeing a decrease in throughput. Profiling the server it seems that it is spending the majority of time sending messages to the clients (unsuprisingly). Messages are sent to the client from the server using a callRemote, invoked from the event queue using reactor.callFromThread - nice and simple. What designs have people used to improve throughput in a heavily loaded server? My experience is telling me that I need to implement some sort of thread/process pool to divide the messaging to clients. Various documents on the twisted website strongly suggets that this is not a good idea. Thoughts? Cheers, G.
PyPy might speed this up quite a bit; http://speed.pypy.org suggests PyPy can send PB messages 3 times as fast as CPython. You could also switch to a protocol that has a faster serialization, which shouldn't be too hard if your usage of PB is sufficiently simple.
Threading won't help much if you're CPU bound, since Python can only run one Python thread at once. Process pooling probably would help, though will involve a more complex infrastructure. Which part of the docs made you think that process pooling is a bad idea?
Hi Itamar, on re-reading I see nothing particular about process pooling being a bad idea. I see the explanation of spawning processes from the reactor here: http://twistedmatrix.com/documents/current/core/howto/process.html This seems to be designed for spawning processes for compute purposes which no doubt has a lot of use cases. I'd like ot spawn processes to handle user communication however. THe only thing that I can think of is multiple processes each with their own reactor handling communication, and one or more other processes to handle compute. Thoughts? THanks for your help. G
Hi everyone, Problems might arise if you use multiple processes, each with each own reactor. Speaking from (limited, admittedly experience), using say two reactors in two processes respectively, if there is a need to communicate between them might lead to all kinds of bugs, *if* you do not do things properly. Another workaround would be the following: Python's GIL lock applies only to executing Python bytecode. Therefore, if you offload some of it to a C-based module, you are GIL-lock free. On 08/04/2011 01:42 AM, Grant Mckenzie wrote:
-- What is the air-speed velocity of an unladen swallow?
PyPy might speed this up quite a bit; http://speed.pypy.org suggests PyPy can send PB messages 3 times as fast as CPython. You could also switch to a protocol that has a faster serialization, which shouldn't be too hard if your usage of PB is sufficiently simple.
Threading won't help much if you're CPU bound, since Python can only run one Python thread at once. Process pooling probably would help, though will involve a more complex infrastructure. Which part of the docs made you think that process pooling is a bad idea?
Hi Itamar, on re-reading I see nothing particular about process pooling being a bad idea. I see the explanation of spawning processes from the reactor here: http://twistedmatrix.com/documents/current/core/howto/process.html This seems to be designed for spawning processes for compute purposes which no doubt has a lot of use cases. I'd like ot spawn processes to handle user communication however. THe only thing that I can think of is multiple processes each with their own reactor handling communication, and one or more other processes to handle compute. Thoughts? THanks for your help. G
Hi everyone, Problems might arise if you use multiple processes, each with each own reactor. Speaking from (limited, admittedly experience), using say two reactors in two processes respectively, if there is a need to communicate between them might lead to all kinds of bugs, *if* you do not do things properly. Another workaround would be the following: Python's GIL lock applies only to executing Python bytecode. Therefore, if you offload some of it to a C-based module, you are GIL-lock free. On 08/04/2011 01:42 AM, Grant Mckenzie wrote:
-- What is the air-speed velocity of an unladen swallow?
participants (3)
-
AK -
Grant Mckenzie -
Itamar Turner-Trauring