[New-bugs-announce] [issue31945] Configurable blocksize in HTTP(S)Connection

[Nir Soffer]

New submission from Nir Soffer <nirsof at gmail.com>:

blocksize is hardcoded to 8192 in send() and _read_readable(), preventing
efficient upload when using file-like body.

Users of the module that are not interested in chunked encoding can rewrite
the copy loop using HTTPConnection.send():

   conn = HTTPSConnection(...)
   conn.putrequest(...)
   conn.putheader(...)
   conn.endheaders()

   while True:
       chunk = file.read(512*1024)
       if not chunk:
          break
       conn.send(chunk)

But fixing send() to use a configurable blocksize seems more useful.

Also, users of requests do not have access the underlying connection, so
they cannot use preferred buffer size.

When reading from /dev/zero and uploading to server that drop the received
data, larger buffer size gives 3X more throughput *and* 1/3 of cpu time.
With real storage and network, the effect will probably be much smaller.

----------
components: Library (Lib)
messages: 305571
nosy: brett.cannon, haypo, nirs, serhiy.storchaka, yselivanov
priority: normal
severity: normal
status: open
title: Configurable blocksize in HTTP(S)Connection
versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue31945>
_______________________________________