Simon Cross <hodgestar+pythondev <at> gmail.com> writes:
Well, since the source for _read_chunked includes the comment
# XXX This accumulates chunks by repeated string concatenation, # which is not efficient as the number or size of chunks gets big.
you might gain some speed improvement with minimal effort by gathering the read data chunks into a list and then returning "".join(chunks) at the end.
+1 for trying this. Given differences between platforms in realloc() performance, it might be the reason why it goes unnoticed under Linux but degenerates under Windows.
As a sidenote, it is interesting that even an stdlib module makes this mistake and acknowledges it without trying to fix it.