httplib request is slow on read()

[Garth Grimm]

John Hunter wrote:

> I am using httplib to make a direct GET request, and then dumping the
> data to file with 
> 
> of = open('temp.dat', 'w')
> f = h.getfile()
> data = f.read()
> of.write( h.getfile().read() )
> 
> where h is the httplib object.
> 
> The read() part is really slow.  When I request the same data from my
> browser, it loads very quickly, which suggests to me that there may be
> a faster way to get this data into the file.


Check to make sure that that comparison is valid.  Browsers often cache DNS lookups, images, and 
entire web pages.  Your python runs won't do this unless you've coded that support in.  Also, some 
browsers will start rendering before all of the file is received.  You'll have to do some special 
coding if you want your file to start being written before the full response has been received.


> 
> I don't know if read() is waiting for some end of file that is
> delayed... or what...
> 
> Any suggestions?
> 
> Thanks,
> John Hunter
>