[Python-Dev] http.client Nagle/delayed-ack optimization

[Ben Leslie]

The http.client HTTPConnection._send_output method has an optimization for
avoiding bad interactions between delayed-ack and the Nagle algorithm:

http://hg.python.org/cpython/file/f32f67d26035/Lib/http/client.py#l884

Unfortunately this interacts rather poorly if the case where the
message_body is a bytes instance and is rather large.

If the message_body is bytes it is appended to the headers, which causes a
copy of the data. When message_body is large this duplication of data can
cause a significant spike in memory usage.

(In my particular case I was uploading a 200MB file to 30 hosts at the same
leading to memory spikes over 6GB.

I've solved this by subclassing and removing the optimization, however I'd
appreciate thoughts on how this could best be solved in the library itself.

Options I have thought of are:

1: Have some size threshold on the copy. A little bit too much magic.
Unclear what the size threshold should be.

2: Provide an explicit argument to turn the optimization on/off. This is
ugly as it would need to be attached up the call chain to the request
method.

3: Provide a property on the HTTPConnection object which enables the
optimization or not. Optionally configured as part of __init__.

4: Add a class level attribute (similar to auto_open, default_port, etc)
which controls the optimization.

Be very interested to get some feedback so I can craft the appropriate
patch.

Thanks,

Benno
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20121215/6b7885ba/attachment.html>