On 9 Aug, 11:34 am, sergei.vokdin@yandex.ru wrote:
Hello,
I've putted together examples from the web (see below) to replace getPage with web.client.Agent and it works fine. Now, I'd like to get gzipped response, but I can't get gunzipping to work before returning result.
Thanks for help!
What are you having trouble with? Your code looks okay, more or less. If I were writing it, I'd try to make the gzip support more transparent, but the way you've done it seems like it should probably work. A few simple things I notice that could cause problems, but won't necessarily... * Using repeated string concatenation to buffer the response is going to be extremely slow for responses of any significant size. * In general, there's no guarantee you'll be able to decode the un- gzipped bytes using utf-8. You could easily have downloaded a gzipped TIFF image. * Similarly, there's no guarantee that the un-gzipping will succeed if you got a truncated response (represented by the PotentialDataLoss failure). * The server might have sent back un-gzipped contents. You have to check one of the response headers to see if it's appropriate to do the decompression. Jean-Paul
class StringGzipReceiver(Protocol): def __init__(self): self.string = None self.deferred = defer.Deferred()
def dataReceived(self, bytes): print "dataReceived" print type(bytes) if self.string: self.string += bytes else: self.string = bytes
def connectionLost(self, reason): if reason.check(ResponseDone) or reason.check(PotentialDataLoss): gzipper = gzip.GzipFile(fileobj=self.string) gz = gzipper.read() result = unicode(gz, 'UTF-8') self.deferred.callback(result) else: self.deferred.errback(reason)
class StringReceiver(Protocol): def __init__(self): self.string_io = codecs.getwriter('utf_8')(StringIO()) self.deferred = defer.Deferred()
def dataReceived(self, bytes): self.string_io.write(bytes)
def connectionLost(self, reason): if reason.check(ResponseDone) or reason.check(PotentialDataLoss): self.deferred.callback(self.string_io.getvalue()) else: self.deferred.errback(reason)
class StringProducer(object): implements(IBodyProducer)
def __init__(self, body): self.body = body self.length = len(body)
def startProducing(self, consumer): consumer.write(self.body) return succeed(None)
def pauseProducing(self): pass
def stopProducing(self): pass
def SearchHotelsByID(): host = 'demo.com' postdata = 'some data' headers = { 'Host' : [host], 'Accept-Encoding' : ['gzip'] }
def cbRequest(response): stringReceiver = StringGzipReceiver() response.deliverBody(stringReceiver) return stringReceiver.deferred
def _noPage(failure): print "Error: %s" % failure.getErrorMessage() print failure.getTraceback() return failure
agent = Agent(reactor) d = agent.request( 'POST', url, headers=Headers(headers), bodyProducer=StringProducer(postdata) ) d.addCallback(cbRequest) d.addErrback(_noPage) d.addBoth(finish)
return d
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web