On Fri, 2010-07-30 at 11:28 +0100, Michele - wrote:
Hi,
I have written a small utility function to replace "twisted.web.client.getPage", to be able to read the response header.
I have to say that the ever improving documentation made it quite easy for me to do it using the new twisted.web.client.Agent, so well done to all!
Since my wrapper works quite well, I decided to add gzip response support, as it's another feature lacking from the original getPage. Again, it was quite simple and it looks it works quite well, in proof of concept scenario.
Then it came my dilemma. What I'm doing now is a synchronous decompression as shown below:
compressedstream = StringIO.StringIO(inzip) gzipper = gzip.GzipFile(fileobj=compressedstream) _data = gzipper.read() return _data
In the standard Agent API, streaming data is downloaded to a protocol. So a gunzipping version would do the same: you have a wrapper protocol that uncompresses data, then delivers to underlying protocol. The basic logic would require reimplementing a small part of the gzip module: first few bytes of data are gzip header, which you skip. Then, use the zlib module to decompress data as it arrives (specifically you'd want a decompression object) and deliver it to the wrapped protocol's dataReceived.