[New-bugs-announce] [issue43332] def _tunnel(self): - uses multiple network writes, possibly causing unnecessary implementation complexity on the receiving end

[Zveinn]

New submission from Zveinn <sveinn at zkynet.io>:

Hey, some time ago I ran into some code in the cpython code I thought might be possible to improve it a little bit.

https://github.com/python/cpython/blob/master/Lib/http/client.py#L903

This code specifically. 

Notice how the self.send() method is used multiple time to construct the CONNECT request. When the network load is high, these different parts actually get split into separate network frames.

This causes additional meta data to be sent, effectively costing more bandwidth and causing the request to be split into multiple network frames.

This has some interesting behavior on the receiving end as well. 

If you send everything as a single network frame, then the receiver can read the entire thing in a single read call. If you send multiple frames, the main reader pipe now needs a temporary buffer to encapsulate the multiple calls. 

Because of this, sending requests as many network frames actually causes a rise in processing complexity on the receiving end. 

Here is a github issue I made about this problem some time ago: https://github.com/psf/requests/issues/5384
In this issue you will find detailed information and screenshots.

My recommendation would be to construct the query as a whole before using a single self.send() to send the whole payload in one network frame. Even if we ignore the added complexity on the receivers end, the gain in network performance is worth it.

----------
components: C API
messages: 387742
nosy: zveinn
priority: normal
severity: normal
status: open
title: def _tunnel(self): - uses multiple network writes, possibly causing unnecessary implementation complexity on the receiving end
type: performance
versions: Python 3.10

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue43332>
_______________________________________