"".join(string_generator()) fails to be magic

[Marc 'BlackJack' Rintsch]

On Thu, 11 Oct 2007 07:02:10 +0000, thebjorn wrote:

> On Oct 11, 8:53 am, Marc 'BlackJack' Rintsch <bj_... at gmx.net> wrote:
>> Even if `str.join()` would not convert the generator into a list first,
>> you would have overallocation.  You don't know the final string size
>> beforehand so intermediate strings must get moved around in memory while
>> concatenating.  Worst case: all but the last string are already
>> concatenated and the last one does not fit into the allocated memory
>> anymore, so there is new memory allocates that can hold both strings ->
>> double amount of memory needed.
>
> Perhaps realloc() could be used to avoid this?  I'm guessing that's
> what cStringIO does, although I'm too lazy to check (I don't have
> source on this box). Perhaps a cStringIO.getvalue() implementation
> that doesn't copy memory would solve the problem?

How could `realloc()` solve that problem?  Doesn't `realloc()` copy the
memory too if the current memory block can't hold the new size!?

And `StringIO` has the very same problem, if the `getvalue()`
method doesn't copy you have to make copies while writing to the `StringIO`
object and the buffer is not large enough.

Ciao,
	Marc 'BlackJack' Rintsch