[Python-ideas] Create a StringBuilder class and use it everywhere

[M.-A. Lemburg]

Masklinn wrote:
> On 2011-08-29, at 11:27 , M.-A. Lemburg wrote:
>> Dirkjan Ochtman wrote:
>>> On Thu, Aug 25, 2011 at 11:45, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> I think you should use cStringIO in your class implementation.
>>>> The list + join idiom is nice, but it has the disadvantage of
>>>> creating and keeping alive many small string objects (with all
>>>> the memory overhead and fragmentation that goes along with it).
>>>
>>> AFAIK using cStringIO just for string building is much slower than
>>> using list.append() + join(). IIRC we tested some micro-benchmarks on
>>> this for Mercurial output (where it was a significant part of the
>>> profile for some commands). That was on Python 2, of course, it may be
>>> better in io.StringIO and/or Python 3.
>>
>> Turns our you're right (list.append must have gotten a lot faster
>> since I last tested this years ago, or I simply misremembered
>> the results).
>>
>>> python2.6 teststringbuilding.py array cstringio listappend
>> Running test array ...
>>   669.68 ms
>> Running test cstringio ...
>>   563.95 ms
>> Running test listappend ...
>>   389.22 ms
>>
>>> python2.7 teststringbuilding.py array cstringio listappend
>> Running test array ...
>>   775.32 ms
>> Running test cstringio ...
>>   679.88 ms
>> Running test listappend ...
>>   375.19 ms
> 
> Converting your code straight to bytes (so array still works) yields this on Python 3.2.1:
> 
>     > python3.2 timetest.py io array listappend
>     Running test io ...
>        334.03 ms
>     Running test array ...
>        776.66 ms
>     Running test listappend ...
>        314.90 ms
> 
> For string (excluding array):
> 
>      > python3.2 timetest.py io listappend
>      Running test io ...
>         451.45 ms
>      Running test listappend ...
>         356.39 ms

Unicode works with the array module as well. Just use 'u' as
array code and replace fromstring/tostring with
fromunicode/tounicode.

In any case, the array module approach appears to the
be slowest of all three tests.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 29 2011)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2011-10-04: PyCon DE 2011, Leipzig, Germany                36 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/