[pypy-dev] [Python-Dev] efficient string concatenation (yep, from 2004)

[Steven D'Aprano]

On 13/02/13 10:53, Christian Tismer wrote:
> Hi friends,
>
> _efficient string concatenation_ has been a topic in 2004.
> Armin Rigo proposed a patch with the name of the subject,
> more precisely:
>
> /[Patches] [ python-Patches-980695 ] efficient string concatenation//
> //on sourceforge.net, on 2004-06-28.//
> /
> This patch was finally added to Python 2.4 on 2004-11-30.
>
> Some people might remember the larger discussion if such a patch should be
> accepted at all, because it changes the programming style for many of us
> from "don't do that, stupid" to "well, you may do it in CPython", which has quite
> some impact on other implementations (is it fast on Jython, now?).

I disagree. If you look at the archives on the python-list@ and tutor at python.org
mailing lists, you will see that whenever string concatenation comes up, the common
advice given is to use join.

The documentation for strings is also clear that you should not rely on this
optimization:

http://docs.python.org/2/library/stdtypes.html#typesseq

And quadratic performance for repeated concatenation is not unique to Python:
it applies to pretty much any language with immutable strings, including Java,
C++, Lua and Javascript.


> It changed for instance my programming and teaching style a lot, of course!

Why do you say, "Of course"? It should not have changed anything.

Best practice remains the same:

- we should still use join for repeated concatenations;

- we should still avoid + except for small cases which are not performance critical;

- we should still teach beginners to use join;

- while this optimization is nice to have, we cannot rely on it being there
   when it matters.

It's not just Jython and IronPython that can't make use of this optimization. It
can, and does, fail on CPython as well, as it is sensitive to memory
allocation details. See for example:

http://utcc.utoronto.ca/~cks/space/blog/python/ExaminingStringConcatOpt

and here for a cautionary tale about what can happen when the optimization fails
under CPython:

http://mail.python.org/pipermail/python-dev/2009-August/091125.html



-- 
Steven