Bug in twisted.web2.client when receiving non-chunked responses of indeterminate length
twisted.web2.client cuts responses short if they are not in chunked encoding and do not have a Content-Length header. I have attached a patch that fixes this bug. I am posting it to the list because I couldn't register on the site to post it to the bug tracker. diff -r -C 3 TwistedWeb2-svn20071201.orig/twisted/web2/channel/http.py TwistedWeb2-svn20071201/twisted/web2/channel/http.py *** TwistedWeb2-svn20071201.orig/twisted/web2/channel/http.py Sat Dec 1 04:10:51 2007 --- TwistedWeb2-svn20071201/twisted/web2/channel/http.py Wed Mar 26 03:33:09 2008 *************** *** 163,176 **** self.handleContentChunk(data[:self.length]) extraneous = data[self.length:] channel = self.channel # could go away from allContentReceived. ! if not self.chunkedIn: ! self.allContentReceived() ! else: ! # NOTE: in chunked mode, self.length is the size of the current chunk, ! # so we still have more to read. ! self.chunkedIn = 2 # Read next chunksize ! channel.setLineMode(extraneous) def headerReceived(self, line): """Store this header away. Check for too much header data --- 163,177 ---- self.handleContentChunk(data[:self.length]) extraneous = data[self.length:] channel = self.channel # could go away from allContentReceived. ! if not (self.length is None and not self.chunkedIn and self.parseCloseAsEnd): ! if not self.chunkedIn: ! self.allContentReceived() ! else: ! # NOTE: in chunked mode, self.length is the size of the current chunk, ! # so we still have more to read. ! self.chunkedIn = 2 # Read next chunksize ! channel.setLineMode(extraneous) def headerReceived(self, line): """Store this header away. Check for too much header data diff -r -C 3 TwistedWeb2-svn20071201.orig/twisted/web2/client/http.py TwistedWeb2-svn20071201/twisted/web2/client/http.py *** TwistedWeb2-svn20071201.orig/twisted/web2/client/http.py Sat Dec 1 04:11:46 2007 --- TwistedWeb2-svn20071201/twisted/web2/client/http.py Wed Mar 26 03:15:48 2008 *************** *** 8,13 **** --- 8,14 ---- from zope.interface import implements + from twisted.internet.error import ConnectionDone from twisted.internet.defer import Deferred from twisted.protocols.basic import LineReceiver from twisted.protocols.policies import TimeoutMixin *************** *** 165,171 **** self._error(ProtocolError(text)) def connectionLost(self, reason): ! self._error(reason) def gotInitialLine(self, initialLine): parts = initialLine.split(' ', 2) --- 166,177 ---- self._error(ProtocolError(text)) def connectionLost(self, reason): ! # If the response was non-chunked and of indeterminate length, treat the ! # connection being closed cleanly as the end of the response ! if self.length is None and not self.chunkedIn and reason.check(ConnectionDone): ! self.allContentReceived() ! else: ! self._error(reason) def gotInitialLine(self, initialLine): parts = initialLine.split(' ', 2)
Andrew Warkentin wrote:
twisted.web2.client cuts responses short if they are not in chunked encoding and do not have a Content-Length header. I have attached a patch that fixes this bug. I am posting it to the list because I couldn't register on the site to post it to the bug tracker.
The previous patch introduced anotther bug which prevented the Deferred returned from HTTPClient.submit from being fired if the connection was closed cleanly before the data was sent. I have attached an updated version of the patch which does not have that bug. Only in TwistedWeb2-svn20071201: build diff -r -C 3 TwistedWeb2-svn20071201.orig/twisted/web2/channel/http.py TwistedWeb2-svn20071201/twisted/web2/channel/http.py *** TwistedWeb2-svn20071201.orig/twisted/web2/channel/http.py Sat Dec 1 04:10:51 2007 --- TwistedWeb2-svn20071201/twisted/web2/channel/http.py Fri Mar 28 04:40:01 2008 *************** *** 163,176 **** self.handleContentChunk(data[:self.length]) extraneous = data[self.length:] channel = self.channel # could go away from allContentReceived. ! if not self.chunkedIn: ! self.allContentReceived() ! else: ! # NOTE: in chunked mode, self.length is the size of the current chunk, ! # so we still have more to read. ! self.chunkedIn = 2 # Read next chunksize ! channel.setLineMode(extraneous) def headerReceived(self, line): """Store this header away. Check for too much header data --- 163,177 ---- self.handleContentChunk(data[:self.length]) extraneous = data[self.length:] channel = self.channel # could go away from allContentReceived. ! if not (self.length is None and not self.chunkedIn and self.parseCloseAsEnd): ! if not self.chunkedIn: ! self.allContentReceived() ! else: ! # NOTE: in chunked mode, self.length is the size of the current chunk, ! # so we still have more to read. ! self.chunkedIn = 2 # Read next chunksize ! channel.setLineMode(extraneous) def headerReceived(self, line): """Store this header away. Check for too much header data diff -r -C 3 TwistedWeb2-svn20071201.orig/twisted/web2/client/http.py TwistedWeb2-svn20071201/twisted/web2/client/http.py *** TwistedWeb2-svn20071201.orig/twisted/web2/client/http.py Sat Dec 1 04:11:46 2007 --- TwistedWeb2-svn20071201/twisted/web2/client/http.py Fri Mar 28 04:39:49 2008 *************** *** 8,13 **** --- 8,14 ---- from zope.interface import implements + from twisted.internet.error import ConnectionDone from twisted.internet.defer import Deferred from twisted.protocols.basic import LineReceiver from twisted.protocols.policies import TimeoutMixin *************** *** 165,171 **** self._error(ProtocolError(text)) def connectionLost(self, reason): ! self._error(reason) def gotInitialLine(self, initialLine): parts = initialLine.split(' ', 2) --- 166,177 ---- self._error(ProtocolError(text)) def connectionLost(self, reason): ! # If the response was non-chunked and of indeterminate length, treat the ! # connection being closed cleanly as the end of the response ! if self.length is None and not self.chunkedIn and reason.check(ConnectionDone) and hasattr(self, 'stream'): ! self.allContentReceived() ! else: ! self._error(reason) def gotInitialLine(self, initialLine): parts = initialLine.split(' ', 2)
On Wed, 26 Mar 2008 03:38:35 -0600, Andrew Warkentin <andreww@datanet.ab.ca> wrote:
twisted.web2.client cuts responses short if they are not in chunked encoding and do not have a Content-Length header. I have attached a patch that fixes this bug. I am posting it to the list because I couldn't register on the site to post it to the bug tracker.
Why couldn't you register? The bug tracker is the right place for this sort of thing. Jean-Paul
Jean-Paul Calderone wrote:
Why couldn't you register? The bug tracker is the right place for this sort of thing.
I guess it did work to register, and I can log in. There is still something wrong with the login script, because it is redirecting back to itself in an endless loop after I log in (that's why I thought login wasn't working).
On Saturday 29 March 2008, Andrew Warkentin wrote:
Jean-Paul Calderone wrote:
Why couldn't you register? The bug tracker is the right place for this sort of thing.
I guess it did work to register, and I can log in. There is still something wrong with the login script, because it is redirecting back to itself in an endless loop after I log in (that's why I thought login wasn't working).
This often happens if you have configured your browser to decline cookies and the server does not check whether or not the browser has acccepted the cookie. Because it does not see a cookie, the server assumes you have not logged in yet and sends another HTTP auth challenge. The browser has cached the password and will automatically respond, logging you in again and again. If this is the cause, please file a bug against Trac as well ;) Bye, Maarten
Maarten ter Huurne wrote:
This often happens if you have configured your browser to decline cookies and the server does not check whether or not the browser has acccepted the cookie. Because it does not see a cookie, the server assumes you have not logged in yet and sends another HTTP auth challenge. The browser has cached the password and will automatically respond, logging you in again and again.
If this is the cause, please file a bug against Trac as well ;)
My browser is set to reject third-party cookies, but I don't think that should be interfering. I also use a filtering proxy, but again, I doubt that is the problem. I should still make sure that those aren't cuasing any problems, though.
participants (3)
-
Andrew Warkentin
-
Jean-Paul Calderone
-
Maarten ter Huurne