[Twisted-Python] non-blocking .read

It is often convenient in protocol implementations to just .read() as much as you want to. However, it is impossible in Twisted, becase .read() is a blocking operation. Happily, we can make blocking operations seem non-blocking with the use of deferred. The following protocol does just that: class ReadProtocol(protocol.Protocol) toRead = 0 until = '' def rawDataReceived(self, data): pass def dataReceived(self, data): while data: if self.toRead: read, data = data[:self.toRead], data[self.toRead:] self.buffer.write(read) self.toRead -= len(read) if self.toRead == 0: self.deferred.callback(self.buffer.getvalue()) self.buffer.close() elif self.until: idx = data.find(self.until) if idx == -1: self.buffer.write(data) data = '' else: read, data = data[:idx], data[idx:] self.until = '' self.buffer.write(read): self.deferred.callback(self.buffer.getvalue()) self.buffer.close() else: self.rawDataReceived(data) data = '' def readUntil(self, c): self.toRead = 0 self.until = c self.buffer = StringIO() self.deferred = defer.Deferred() return self.deferred def readLength(self, len): self.toRead = len self.until = '' self.buffer = StringIO() self.deferred = defer.Deferred() return self.deferred # How to use, e.g., to implement netstrings: class NetstringProtocol(ReadProtocol): def connectionMade(self): self.readString() def readString(self): self.readUntil(':').callback( lambda s: self.readLength(int(s)).callback( self.stringReceived).callback( lambda _: self.readLength(1).callback( lambda x: x==',' or self.transport.loseConnection()).callback( lambda _: self.readString)) def stringReceived(self, s): pass

On Wed, 02 Apr 2003 01:13:41 -0600 Moshe Zadka <moshez@twistedmatrix.org> wrote:
I suspect it will be waaaaaaaay slower than the equivalent netstring implementation. -- Itamar Shtull-Trauring http://itamarst.org/ http://www.zoteca.com -- Python & Twisted consulting

On Wed, Apr 02, 2003 at 03:34:15PM -0500, Itamar Shtull-Trauring wrote:
And way more DEMONIC. You're a sick man, moshez. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/

Just to bring generators back up.. Note that of course, the wrappers for read and the actual flow.BufferedProtocol class don't exist.. but their implementation is possible with the given syntax. I'm imagining that a "flow" version of Int32Receiver could be coded like this: class NetstringProtocol(flow.BufferedProtocol): def connectionMade(self): pass def handleConnection(self): self.connectionMade() reason = None # the string receive loop while 1: # read int32, size of int32 is 4 result = self.read(4) yield result if result.isFailure(): reason = result.get() break strlen, = struct.unpack('!I', result.get()) if strlen <= 0: endFailure = Failure(ValueError('Zero length strings are not valid')) break # read string result = self.read(strlen) yield result if result.isFailure(): reason = result.get() break self.stringReceived(result.get()) # exception caused connection to drop, or connection dropped on its own self.connectionLost(reason) def connectionLost(self, reason): pass

On Wed, 02 Apr 2003 01:13:41 -0600 Moshe Zadka <moshez@twistedmatrix.org> wrote:
I suspect it will be waaaaaaaay slower than the equivalent netstring implementation. -- Itamar Shtull-Trauring http://itamarst.org/ http://www.zoteca.com -- Python & Twisted consulting

On Wed, Apr 02, 2003 at 03:34:15PM -0500, Itamar Shtull-Trauring wrote:
And way more DEMONIC. You're a sick man, moshez. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/

Just to bring generators back up.. Note that of course, the wrappers for read and the actual flow.BufferedProtocol class don't exist.. but their implementation is possible with the given syntax. I'm imagining that a "flow" version of Int32Receiver could be coded like this: class NetstringProtocol(flow.BufferedProtocol): def connectionMade(self): pass def handleConnection(self): self.connectionMade() reason = None # the string receive loop while 1: # read int32, size of int32 is 4 result = self.read(4) yield result if result.isFailure(): reason = result.get() break strlen, = struct.unpack('!I', result.get()) if strlen <= 0: endFailure = Failure(ValueError('Zero length strings are not valid')) break # read string result = self.read(strlen) yield result if result.isFailure(): reason = result.get() break self.stringReceived(result.get()) # exception caused connection to drop, or connection dropped on its own self.connectionLost(reason) def connectionLost(self, reason): pass
participants (4)
-
Bob Ippolito
-
Christopher Armstrong
-
Itamar Shtull-Trauring
-
Moshe Zadka