Alec Matusis wrote:
In desperation of not finding the real memory leak on the production server,
I wrote a test server that I can push to arbitrary high RSS memory. I am far from sure if this the same leak that I observe in production, but I would like to understand what this one is. This is the server code:
Server.py:
import twisted.protocols.basic from twisted.internet.protocol import Factory from twisted.internet import reactor
class HiRate(twisted.protocols.basic.LineOnlyReceiver): MAX_LENGTH = 20000
def lineReceived(self, line): if line == 'get': out = 'a'*4000+'\r\r' self.transport.write(out)
factory = Factory() factory.protocol = HiRate
reactor.listenTCP(8007, factory, backlog=50, interface='10.18.0.2') reactor.run() [...] To reproduce the memory leak, I either need two machines with fast LAN between them (since the client program takes 100% CPU), or possibly one machine with a dual core CPU (I have not tried that). It is important that client.py is given a separate CPU to run. When the length of the response from the server is sufficient, (out = 'a'*4000+'\r\r' , 4000 is enough in my case), the RSS of the server process starts to leak without a bound. If you introduce a small delay in the client (#time.sleep(.001)), the leak does not occur.
Hi Alec, I wouldn't call that a memory leak. In your example ( SC/s = (n* CS) / s ; n > 1, SC: bytes sent to the client, CS: bytes sent to the server ) there is some CS that satifies the condition SC/s > [available bandwidth]. For all CS that exceed that limit, the data will be first queued in the Send-Q of the socket and then inside twisted, because the Send-Q is full. That's the reason why you see increasing memory usage, but it's not a leak, it is a server that can not cope with this kind of DOS attack. Twisted can not handle this situation for you because that would mean it had to decide which packets, connections, etc. to drop. Such decisions have to be made by your application. regards, Johann