Re: URL length limits in twisted-web
--- Tommi Virtanen wrote:
There's this DoS-avoidance bit:
class LineReceiver(protocol.Protocol,
_PauseableMixin):
... MAX_LENGTH = 16384
Thanks, Tommi. That appears to be it (16898 - len('http://') and the non "path" part of my URL = the more sane number of 16384). The 'silent failure' feature doesn't seem quite right to me. LineReceiver.dataReceived() returns self.lineLengthExceeded(line), which calls self.transport.loseConnection(), which means that neither http.HTTPClient.lineReceived() nor http.HTTPChannel.lineReceived() ever get called. If I'm following the code correctly, loseConnection() does its job silently, and the caller never knows what happened, but is led to believe that the request was sent successfully. I changed LineReceiver.lineLengthReceived() as follows: def lineLengthExceeded(self, line): """Called when the maximum line length has been reached. Override if it needs to be dealt with in some special way. """ #return self.transport.loseConnection() self.transport.loseConnection() raise error.ConnectionLost('Line length exceeded') Which now does what I would expect (raises an exception if I try to send URLs that exceed the maximum), but I'm sure there are reasons not to raise an exception here (other classes that rely on LineReceiver, etc). What is the proper solution to this issue? Lenny ____________________________________________________ Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs
On Thu, 25 Aug 2005 11:13:23 -0700 (PDT), Lenny G Arbage <alengarbage@yahoo.com> wrote:
--- Tommi Virtanen wrote:
There's this DoS-avoidance bit:
class LineReceiver(protocol.Protocol,
_PauseableMixin):
... MAX_LENGTH = 16384
Thanks, Tommi. That appears to be it (16898 - len('http://') and the non "path" part of my URL = the more sane number of 16384).
The 'silent failure' feature doesn't seem quite right to me.
It isn't a silent failure. The connection is dropped, as you notice. The client is entirely capable of noticing that the server never responded to its request. If no response is received, the correct thing to do is /not/ to assume everything worked perfectly. Especially if the response was supposed to be a page. twisted.web.client may well be buggy in its handling of this case (I've noticed several other bugs of this variety, though I'm not sure I've seen this one in particular).
[snip]
I changed LineReceiver.lineLengthReceived() as follows:
def lineLengthExceeded(self, line): """Called when the maximum line length has been reached. Override if it needs to be dealt with in some special way. """ #return self.transport.loseConnection() self.transport.loseConnection() raise error.ConnectionLost('Line length exceeded')
Which now does what I would expect (raises an exception if I try to send URLs that exceed the maximum), but I'm sure there are reasons not to raise an exception here (other classes that rely on LineReceiver, etc).
This just throws a random exception up into the reactor. This is not allowed, but Twisted kindly catches it, logs it, and makes sure the protocol is not given any further data. It's not an error condition you can reasonably deal with, it's just loud and ugly and obnoxious to make sure you notice your program is horribly broken and should be fixed.
What is the proper solution to this issue?
A better HTTP client. Some people have been working in this area. Perhaps they will grace us with a progress report. Jp
participants (2)
-
Jp Calderone -
Lenny G Arbage