[Python-Dev] how to debug httplib slowness

[Antoine Pitrou]

Simon Cross <hodgestar+pythondev <at> gmail.com> writes:
> 
> Well, since the source for _read_chunked includes the comment
> 
>         # XXX This accumulates chunks by repeated string concatenation,
>         # which is not efficient as the number or size of chunks gets big.
> 
> you might gain some speed improvement with minimal effort by gathering
> the read data chunks into a list and then returning "".join(chunks) at
> the end.

+1 for trying this. Given differences between platforms in realloc()
performance, it might be the reason why it goes unnoticed under Linux but
degenerates under Windows.

As a sidenote, it is interesting that even an stdlib module makes this mistake
and acknowledges it without trying to fix it.