Fastest technique for string concatenation

[Peter Otten]

python at bdurham.com wrote:

> My understanding is that appending to a list and then joining
> this list when done is the fastest technique for string
> concatenation. Is this true?
> 
> The 3 string concatenation techniques I can think of are:
> 
> - append to list, join
> - string 'addition' (s = s + char)
> - cStringIO
> 
> The code that follows my signature confirms that the list
> append/join technique is indeed the fastest of these 3
> approaches, but perhaps there are other techniques I should
> consider?
> 
> Malcolm
> 
> # test various techniques for string concatenation
> import cStringIO
> import timeit
> source = 'x' * 5000000
> def testListAppend():
>     output = list()
>     for char in source:
>         output.append( char )
>     output = ''.join( output )

You are measuring list creation here, not string concatenation:

$ python -m timeit -s's = "x"*10**5' 'd = []' 'for c in s: d.append(c)'
10 loops, best of 3: 20.9 msec per loop
$ python -m timeit -s's = ["x"]*10**5' '"".join(s)'
100 loops, best of 3: 2.47 msec per loop

To fix that I'd start with a list, not a string:

source = ["x"] * 5000000

> def testStringIO():
>     output = cStringIO.StringIO()
>     for char in source:
>         output.write( char )
>     output = output.getvalue()

You can simplify that to

def test_stringio():
    output = cStringIO.StringIO()
    output.writelines(source) # doesn't add newlines
    output = output.getvalue()

which is almost as fast as ''.join:

$ python -m timeit -s's = ["x"]*10**5' '"".join(s)'
100 loops, best of 3: 2.47 msec per loop

$ python -m timeit -s's = ["x"]*10**5' -s 'from cStringIO import StringIO' 
'o = StringIO(); o.writelines(s); o.getvalue()'
100 loops, best of 3: 2.88 msec per loop

Peter