"".join(string_generator()) fails to be magic

[Sion Arrowsmith]

Matt Mackal  <mpm at selenic.com> wrote:
>I have an application that occassionally is called upon to process
>strings that are a substantial portion of the size of memory. For
>various reasons, the resultant strings must fit completely in RAM.

Do you mean physical RAM, or addressable memory? If the former,
there's an obvious solution....

>Occassionally, I need to join some large strings to build some even
>larger strings.
>
>Unfortunately, there's no good way of doing this without using 2x the
>amount of memory as the result.

I think you can get better than 2x if you've got a reasonable number
of (ideally similarly sized) large strings with something along the
lines of:

for i in range(0, len(list_of_strings), 3): #tune step
    result_string += (list_of_strings[i] + 
                      list_of_strings[i+1] +
                      list_of_strings[i+2])
    list_of_strings[i] = ""
    list_of_strings[i+1] = ""
    list_of_strings[i+2] = ""

remembering the recent string concatenation optimisations. Beyond
that, your most reliable solution may be the (c)StringIO approach
but with a real file (see the tempfile module, if you didn't know
about it).

-- 
\S -- siona at chiark.greenend.org.uk -- http://www.chaos.org.uk/~sion/
   "Frankly I have no feelings towards penguins one way or the other"
        -- Arthur C. Clarke
   her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump