[New-bugs-announce] [issue2576] httplib read() very slow due to lack of socket buffer

[Aren Olson]

New submission from Aren Olson <reacocard at gmail.com>:

This is a reposting of issue 508157, as requested by gvanrossum.

The socket file object in httplib is opened without any buffering
resulting in very slow performance of read().  The specific problem is
in the httplib.HTTPResponse constructor.  It calls sock.makefile() with
a 0 for the buffer size.  changing the buffer size to 4096 improved the
time needed to download 10MB from 15.5s to 1.78s, almost 9x faster.
Repeat downloads of the same file (meaning the server now has the file
cached in memory), yield times of 15.5s and 0.03s, a 500x improvement.
When fetching from a server on the local network, rather than from
localhost, these times become 15.5s and 0.9s in both cases, a 17x
speedup. Real-world situations will likely be a mix of these, however it
is safe to say the speed improvement will be substantial. Adding an
option to adjust the buffer size would be very welcome, though the
default value should still be zero, to avoid the issues already
mentioned in issue 508157.

These speed results were obtained with python2.5 and apache2 under
Ubuntu linux, using the code found here: http://pastebin.ca/973578

----------
components: Library (Lib)
messages: 65121
nosy: reacocard
severity: normal
status: open
title: httplib read() very slow due to lack of socket buffer
versions: Python 2.6

__________________________________
Tracker <report at bugs.python.org>
<http://bugs.python.org/issue2576>
__________________________________