[New-bugs-announce] [issue23377] HTTPResponse may drop buffer holding next response

[Martin Panter]

New submission from Martin Panter:

This is the same issue raised at <https://bugs.python.org/issue4879#msg91597>. Currently, every time a new response is to be received, HTTPConnection passes its raw socket object to HTTPResponse, which calls sock.makefile("rb") and creates a BufferedReader. The BufferedReader is used to parse the header section and read the response body. The problem is that the BufferedReader is closed at the end of reading the response, potentially losing buffered data read from a subsequent response.

Normally no data is lost, because most users would read the full response before triggering a new request, and the server would wait for a request before sending a response. But if a user pipelined a second request without reading all of the first response, and the server happened to send the end of the first response and the start of the second response in the same packet, it could trigger the problem. I have added a test called test_httplib.ServerTest.testDoubleResponse() which emulates this scenario. The problem also makes it hard to detect misbehaving servers, or use HTTPConnection to test that a server is behaving correctly.

I am adding a patch which creates the BufferedReader once for each connection. This involves changing the API of the HTTPResponse constructor. I think this should be okay because even though it is documented, it says “Not instantiated directly by user”. It did require changing the tests that call the HTTPResponse constructor though. If absolutely necessary, it may be possible to maintain backwards compatibility if we added a new constructor parameter, and carefully juggled how the close() calls work.

----------
components: Library (Lib)
files: http-buffer.patch
keywords: patch
messages: 235251
nosy: vadmium
priority: normal
severity: normal
status: open
title: HTTPResponse may drop buffer holding next response
type: behavior
versions: Python 3.5
Added file: http://bugs.python.org/file37977/http-buffer.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue23377>
_______________________________________