[Python-Dev] httplib and bad response chunking

[John J Lee]

On Sun, 30 Jul 2006, Greg Ward wrote:
[...]
> Did you look at the crude attempt at testing for this bug that I hacked
> into test_httplib.py?  I posted it to bug #1486335 here:
>
>  http://sourceforge.net/tracker/download.php?group_id=5470&atid=105470&file_id=186245&aid=1486335
>
> The idea is simple: put various chunked responses into strings and then
> feed those strings to HTTPConnection.  The trouble is that StringIO does
> not behave the same as a real socket: where HTTPResponse fails one way
> reading from a real socket (eg. infinite loop), it fails differently (or
> not at all) reading from a StringIO.  Makes testing with the FakeSocket
> class in test_httplib.py problematic.

There are always ways around unit testing problems, but...


> Maybe the right way to test httplib is to spawn a server process
> (thread?) to listen on some random port, feed various HTTP responses at
> HTTPConnection/HTTPResponse, and see what happens.  I'm not sure how to
> do that portably, though.  Well, I'll see if I can whip up a Unix-y
> solution and see if anyone knows how to make it portable.

I think adding this kind of functional test is extremely valuable, given 
that we don't seem to have any for httplib at present 
(Lib/test/test_httplib.py does not send any network packets).  Maybe you 
could add a basic smoke test that does a simple successful GET, while 
you're going about adding your chunked encoding regression test(s)?

Oh, wait: there is a functional test that uses the network, but it's at 
the end of httplib.py rather than being part of the test suite, and it 
follows the "Guru Watches Output" pattern :-)

I tried to add a test for urllib2 recently that relied on a python.org URL 
being set up, but the python.org guys are pretty overworked already and 
haven't had time to enable that yet -- so I think that simply from that 
point of view your run-your-own-server approach is better.  Why does it 
need to be unix-y, though?  We have SimpleHTTPServer.  We're not trying to 
test the OS, so I don't see a problem with using loopback and a single 
process -- and that would keep test run times down.  Um, except that MS 
OSes seem a bit odd re the loopback interface -- ISTR that, at least back 
with NT4, you just didn't get a loopback i/face unless you had an ethernet 
driver installed (back then, I was on dialup).

More unit tests would also be valuable, of course.


John