[Python-ideas] Create a StringBuilder class and use it everywhere

[Masklinn]

On 2011-08-29, at 11:27 , M.-A. Lemburg wrote:
> Dirkjan Ochtman wrote:
>> On Thu, Aug 25, 2011 at 11:45, M.-A. Lemburg <mal at egenix.com> wrote:
>>> I think you should use cStringIO in your class implementation.
>>> The list + join idiom is nice, but it has the disadvantage of
>>> creating and keeping alive many small string objects (with all
>>> the memory overhead and fragmentation that goes along with it).
>> 
>> AFAIK using cStringIO just for string building is much slower than
>> using list.append() + join(). IIRC we tested some micro-benchmarks on
>> this for Mercurial output (where it was a significant part of the
>> profile for some commands). That was on Python 2, of course, it may be
>> better in io.StringIO and/or Python 3.
> 
> Turns our you're right (list.append must have gotten a lot faster
> since I last tested this years ago, or I simply misremembered
> the results).
> 
>> python2.6 teststringbuilding.py array cstringio listappend
> Running test array ...
>   669.68 ms
> Running test cstringio ...
>   563.95 ms
> Running test listappend ...
>   389.22 ms
> 
>> python2.7 teststringbuilding.py array cstringio listappend
> Running test array ...
>   775.32 ms
> Running test cstringio ...
>   679.88 ms
> Running test listappend ...
>   375.19 ms

Converting your code straight to bytes (so array still works) yields this on Python 3.2.1:

    > python3.2 timetest.py io array listappend
    Running test io ...
       334.03 ms
    Running test array ...
       776.66 ms
    Running test listappend ...
       314.90 ms

For string (excluding array):

     > python3.2 timetest.py io listappend
     Running test io ...
        451.45 ms
     Running test listappend ...
        356.39 ms