[New-bugs-announce] [issue41002] HTTPResponse.read with amt is slow

[Bruce Merry]

New submission from Bruce Merry <bmerry at gmail.com>:

I've run into this on 3.8, but the code on Git master doesn't look significantly different so I assume it still applies. I'm happy to work on a PR for this.

When http.client.HTTPResponse.read is called with a specific amount to read, it goes down this code path:
```
if amt is not None:
    # Amount is given, implement using readinto
    b = bytearray(amt)
    n = self.readinto(b)
    return memoryview(b)[:n].tobytes()
```
That's pretty inefficient, because
- `bytearray(amt)` will first zero-fill some memory
- `tobytes()` will make an extra copy of this memory
- if amt is big enough, it'll cause the temporary memory to be allocated from the kernel, which will *also* zero-fill the pages for security.

A better approach would be to use the read method of the underlying fp.

I have a micro-benchmark (that I'll attach) showing that for a 1GB body and reading the whole body with or without the amount being explicit, performance is reduced from 3GB/s to 1GB/s.

For some unknown reason the requests library likes to read the body in 10KB chunks even if the user has requested the entire body, so this will help here (although the gains probably won't be as big because 10KB is really too small to amortise all the accounting overhead).

Output from my benchmark, run against a 1GB file on localhost:

httpclient-read: 3019.0 ± 63.8 MB/s
httpclient-read-length: 1050.3 ± 4.8 MB/s
httpclient-read-raw: 3150.3 ± 5.3 MB/s
socket-read: 3134.4 ± 7.9 MB/s

----------
components: Library (Lib)
files: httpbench-simple.py
messages: 371732
nosy: bmerry
priority: normal
severity: normal
status: open
title: HTTPResponse.read with amt is slow
versions: Python 3.8
Added file: https://bugs.python.org/file49239/httpbench-simple.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue41002>
_______________________________________