[Twisted-Python] How to get protocol function do not block server?
Hello, I am a newbie in twisted, sorry if this was asked thousands times before. I have wrote a simple protocol (over TCP sockets) with crc32. Suppose one client made a handshake.. and then sends to server 10Mb of data... at the end of that 10Mb there is an crc32.. crc32 is a pretty time consuming task on a slow machine.. while server calculates crc32 of 10Mb sent by client one, other clients can not communicate with server (i tryed by myself to log in while transfer.. and i was wating about a 10 seconds) And also i have a UnCryptAFrame time consuming function. How to make those function do not block server, to run in background, and when result is avalable call a chain of other functions (for example it would be nice to add a callback to Factory.handleFrame(data) with the return results of Protocol.UnCryptAFrame())? Is it have to be deferToThread ? And also does dataReceived blocks server by itself? Is twisted.TCPServer scalable, can it serve 1000 - 10000 request in a minute? (in a secound would be great :) ) -------------------------------------------------------------------------------- server.py class MyProtocolServer(protocol.Protocol): # ... connectionMade .. def dataReceived(self, data): self.inp_buf += data if self.initialized != 0: self.handleData() else: self.handShake() # handshake is fast.. but i would gladly # make async if i know how. def handleData(self): if self.w_len is None and len(self.inp_buf) >= 5: self.w_len = struct.unpack("L",self.UncryptLen(self.inp_buf[:5]))[0] if len(self.inp_buf) == self.w_len: uncrypt_frame = self.UnCryptAFrame(self.inp_buf) self.NullInpBufer() def UnCryptAFrame(self): # a very slow function! pass class MyFactory(protocol.ServerFactory): protocol = MyProtocolServer def buildProtocol(self, addr): p = MyProtocolServer() p.factory = self p.addr = addr # ... return p factory = MyFactory() application = service.Application('myservice', uid=1, gid=1) internet.TCPServer(9090, factory).setServiceParent(service.IServiceCollection(application)) -------------------------------------------------------------------------------- end server.py $ twistd -ny ./f_server.py 2006/09/26 14:18 SST [-] Log opened. 2006/09/26 14:18 SST [-] twistd 2.4.0 (/usr/bin/python 2.4.2) starting up 2006/09/26 14:18 SST [-] reactor class: <class 'twisted.internet.selectreactor.SelectReactor'> 2006/09/26 14:18 SST [-] Loading ./f_server.py... 2006/09/26 14:18 SST [-] Loaded. 2006/09/26 14:18 SST [-] __builtin__.UpSTPFactory starting on 9090 2006/09/26 14:18 SST [-] Starting factory <__builtin__.UpSTPFactory instance at 0x407ec54c> 2006/09/26 14:18 SST [-] set uid/gid 1/1 ------------------------------------------------------- In other words if i do something slow in dataReceived (of my Protocol) it seems that it blocks server. how to avoid this? sorry for my english :) -- Best regards, Norman mailto:NormanTindall@zdisk.net
Is it have to be deferToThread ? I think so.since if you don't want to block server ,you should let the time-consuming process run in a separate thread from the main one And also does dataReceived blocks server by itself? yes, if you have ever hacked implementation of reactor,you will see that reactor just dispatches the data recevied event to correspondent protocol's dataReceived handle. there is a threadable reactor available,but seemingly it is not stable
Norman Tindall ha scritto:
Hello, I am a newbie in twisted, sorry if this was asked thousands times before.
I have wrote a simple protocol (over TCP sockets) with crc32.
Suppose one client made a handshake.. and then sends to server 10Mb of data... at the end of that 10Mb there is an crc32..
You can require a max size for the packets. A more general solution is to do the job in chunks:
From Python documentation: print binascii.crc32("hello world") # Or, in two pieces: crc = binascii.crc32("hello") crc = binascii.crc32(" world", crc) print crc
Here is an example of computing the edonkey2k hash: def waitFor(seconds): """Wait for the specified amount of seconds. To be used in a deferred generator. """ from twisted.internet import reactor d = defer.Deferred() reactor.callLater(seconds, lambda: d.callback(None)) return defer.waitForDeferred(d) @defer.deferredGenerator def getFileHash(path): """XXX TODO use a smaller file buffer size (8096 bytes?) use a thread? XXX TODO add support for md5, sha1 and crc32. """ from Crypto.Hash import MD4 # this is used as file buffer size, too CHUNK_SIZE = 9728000 # XXX delay to give the reactor a chance DELAY = 0 hashList = [] fp = file(path, 'rb') while True: hash = MD4.new() data = fp.read(CHUNK_SIZE) yield waitFor(DELAY) if len(data) == 0: hash.update('') hashList.append(hash.digest()) hash.update(data) yield waitFor(DELAY) hashList.append(hash.digest()) if len(data) < CHUNK_SIZE: break # compute the hash of the hash list hash = MD4.new() hash.update(''.join(hashList)) yield waitFor(DELAY) yield hash.hexdigest() You can do the same, using cStringIO.
crc32 is a pretty time consuming task on a slow machine.. while server calculates crc32 of 10Mb sent by client one, other clients can not communicate with server (i tryed by myself to log in while transfer.. and i was wating about a 10 seconds) And also i have a UnCryptAFrame time consuming function.
Check if the encryption can be done in chunks. Regards Manlio Perillo
participants (3)
-
aerodew -
Manlio Perillo -
Norman Tindall