[Python-Dev] httplib and bad response chunking

[Greg Ward]

[me, on 25 July]
> I have
> discovered other hypothetical cases of bad chunking that cause httplib
> to go into an infinite loop or block forever on socket.readline().
> Should we worry about those cases as well, despite not having seen them
> happen in the wild?  More annoying, I can reproduce the "block forever"
> case using a real socket, but not using the StringIO-based FakeSocket
> class in test_httplib.

[John J Lee]
> They have been seen in the wild :-)
> 
> http://python.org/sf/1411097

Thanks -- that was really all the encouragement I needed to keep banging
away at this bug.

Did you look at the crude attempt at testing for this bug that I hacked
into test_httplib.py?  I posted it to bug #1486335 here:

  http://sourceforge.net/tracker/download.php?group_id=5470&atid=105470&file_id=186245&aid=1486335

The idea is simple: put various chunked responses into strings and then
feed those strings to HTTPConnection.  The trouble is that StringIO does
not behave the same as a real socket: where HTTPResponse fails one way
reading from a real socket (eg. infinite loop), it fails differently (or
not at all) reading from a StringIO.  Makes testing with the FakeSocket
class in test_httplib.py problematic.

Maybe the right way to test httplib is to spawn a server process
(thread?) to listen on some random port, feed various HTTP responses at
HTTPConnection/HTTPResponse, and see what happens.  I'm not sure how to
do that portably, though.  Well, I'll see if I can whip up a Unix-y
solution and see if anyone knows how to make it portable.

        Greg
-- 
Greg Ward <gward at python.net>                         http://www.gerg.ca/
Be careful: sometimes, you're only standing on the shoulders of idiots.