anurag uniyal <anuraguniyal@yahoo.com> writes:

Is it possible to send and recieve compressed content using twisted.web?

I have set Accept-encoding to gzip but it doesn't make any difference. Will I have to cater for this myself?

Yes, in twisted.web. In twisted.web2, there's a filter that supports it. Here's a small wrapper class I wrote recently that I've used to handle this in twisted.web, with the goal of minimal changes to existing resource code. I use it by noticing, in the original request handler, the ability to handle gzip encodings, and then wrap the original request in this class (since there's no built-in filtering of the output stream in twisted.web). It's worked fine in my tests to date, and in limited production use, but to be honest it hasn't seen large scale, long term use because my original use case only yielded ~15% compression (multimedia files) and using this negates the ability to know the full length in advance, thus clients can't give projected download times. (My files are large enough that compressing twice is high overhead, and it wouldn't play well with the wrapper approach anyway). In any event, hopefully it'll be a useful starting point for you. Here's the wrapper: - - - - - - - - - - - - - - - - - - - - - - - - - import struct import zlib class GzipRequest(object): """Wrapper for a request that applies a gzip content encoding""" def __init__(self, request, compressLevel=6): self.request = request self.request.setHeader('Content-Encoding', 'gzip') # Borrowed from twisted.web2 gzip filter self.compress = zlib.compressobj(compressLevel, zlib.DEFLATED, -zlib.MAX_WBITS, zlib.DEF_MEM_LEVEL,0) def __getattr__(self, attr): if 'request' in self.__dict__: return getattr(self.request, attr) else: raise AttributeError, attr def __setattr__(self, attr, value): if 'request' in self.__dict__: return setattr(self.request, attr, value) else: self.__dict__[attr] = value def write(self, data): if not self.request.startedWriting: self.crc = zlib.crc32('') self.size = self.csize = 0 # XXX: Zap any length for now since we don't know final size if 'content-length' in self.request.headers: del self.request.headers['content-length'] # Borrow header information from twisted.web2 gzip filter self.request.write('\037\213\010\000' '\0\0\0\0' '\002\377') self.crc = zlib.crc32(data, self.crc) self.size += len(data) cdata = self.compress.compress(data) self.csize += len(cdata) if cdata: self.request.write(cdata) elif self.request.producer: # Simulate another pull even though it hasn't really made it # out to the consumer yet. self.request.producer.resumeProducing() def finish(self): remain = self.compress.flush() self.csize += len(remain) if remain: self.request.write(remain) self.request.write(struct.pack('<LL', self.crc & 0xFFFFFFFFL, self.size & 0xFFFFFFFFL)) self.request.finish() - - - - - - - - - - - - - - - - - - - - - - - - - and here's a sample of using it. This code is from one of my Resource objects - if I were to use it more generally in my site I'd extract that into a mix-in class of some sort - currently there was only one specific resource (the multimedia files) I wanted to support it for: def render_GET(self, request): # (... argument validation ...) accept_encoding = request.getHeader('accept-encoding') if accept_encoding: encodings = accept_encoding.split(',') for encoding in encodings: name = encoding.split(';')[0].strip() if name == 'gzip': request = GzipRequest(request) break # At this point, use 'request' as normal -- David