You should break the loop into small parts.
Can you explain this? Do you mean, use reactor.callLater to process say 100 messages at a time? Or maybe use deferToThread for the whole loop?
You see the memory growing because the Python interpreter will not release the memory to the operating system (however the memory should not grow indefinitely)
The memory is actually growing indefinitely. We run 4 identical servers on one machine. The load is not balanced very symmetrically. So one server can have 4500 connections, while another has 6000. There is a certain threshold on the number of connections, when a server starts to leak memory uncontrollably, and it's not set by the highest usage. For example, a server with 4500 connections remains at 100mb RSS for days, while a server that reaches 6000 connections leaks memory to 2gb in 10 hrs (I did not try beyond that, since the machine starts swapping!)
-----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python- bounces@twistedmatrix.com] On Behalf Of Manlio Perillo Sent: Saturday, December 01, 2007 2:06 AM To: Twisted general discussion Subject: Re: [Twisted-Python] Advise for heavy concurrency
Alec Matusis ha scritto:
What causes the CPU overload in such a case
It looks like the for loop that loops over connections to send the message is the main source of the load. The biggest problem is not the CPU however. When the process takes more than 70% of the CPU (as displayed by top), python seems to start skipping garbage collection, and the memory size of the process just keeps growing. We have no control over this.
You should break the loop into small parts.
If you send a message to a lot of clients, then Twisted needs to keep all the buffers in memory (since your loop is blocking the reactor and Twisted is unable to flush the buffers).
You see the memory growing because the Python interpreter will not release the memory to the operating system (however the memory should not grow indefinitely)
[...]
Manlio Perillo
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python