[issue14744] Use _PyUnicodeWriter API in str.format() internals

[STINNER Victor]

STINNER Victor <victor.stinner at gmail.com> added the comment:

To prepare a deeper change, here is a first simple patch. Change how the size of the _PyUnicodeWriter buffer is handled:

 * overallocate by 100% (instead of 25%) and allocate at least 100 characters
 * don't overallocate when possible

It is possible to not overallocate when the format string has no prefix nor suffix and when there is only one argument. For example, "%s" % "abc" does directly allocate the exact buffer size and so don't need to call realloc() at the end.

The patch does also micro-optimize (cleanup?) _PyUnicodeWriter_Finish.

When it's possible to not overallocate, the speed up is around 10% for short strings (I suppose that it's much better to longer strings).

--

Output of the attached benchmark.py script:

Linux-3.3.2-1.fc16.x86_64-x86_64-with-fedora-16-Verne
Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz

=== 3.3 ===

16 ms: N=200; fmt="{0}-"*N; arg="abc"; fmt.format(arg)
5.77 ms: N=200; L=3; fmt="%s"*N; args=("a"*L,)*N; fmt % args
648 ns: s="The {k1} is {k2} the {k3}."; args={"k1": "x", "k2": "y", "k3": "z"}; s.format(**args)
427 ns: s="The %(k1)s is %(k2)s the %(k3)s."; args={"k1":"x","k2":"y","k3":"z",}; s%args
531 ns: s="x={}, y={}, z={}"; args=(123, 456, 789); s.format(*args)
453 ns: s="x=%s, y=%u, z=%x"; args=(123, 456, 789); s%args
180 ns: fmt="{}"; arg="abc"; fmt.format(arg)
228 ns: fmt="{}"; arg=123; fmt.format(arg)
196 ns: fmt="x={}"; arg="abc"; fmt.format(arg)
244 ns: fmt="x={}"; arg=123; fmt.format(arg)
175 ns: str(12345)
233 ns: '{}'.format(12345)
243 ns: 'A{}'.format(12345)
276 ns: '\x80{}'.format(12345)
292 ns: '\u0100{}'.format(12345)
285 ns: '\U00010000{}'.format(12345)
417 ns: '{:-10}'.format(12345)
425 ns: 'A{:-10}'.format(12345)
461 ns: '\x80{:-10}'.format(12345)
488 ns: '\u0100{:-10}'.format(12345)
461 ns: '\U00010000{:-10}'.format(12345)
403 ns: '{:,}'.format(12345)
416 ns: 'A{:,}'.format(12345)
455 ns: '\x80{:,}'.format(12345)
469 ns: '\u0100{:,}'.format(12345)
448 ns: '\U00010000{:,}'.format(12345)
108 ns: fmt="%s"; arg="abc"; fmt % arg
163 ns: fmt="%s"; arg=123; fmt % arg
121 ns: fmt="%s:"; arg="abc"; fmt % arg

=== 3.3 + dont_overallocate.patch ===

15.5 ms: N=200; fmt="{0}-"*N; arg="abc"; fmt.format(arg)
 6 ms: N=200; L=3; fmt="%s"*N; args=("a"*L,)*N; fmt % args
599 ns: s="The {k1} is {k2} the {k3}."; args={"k1": "x", "k2": "y", "k3": "z"}; s.format(**args)
424 ns: s="The %(k1)s is %(k2)s the %(k3)s."; args={"k1":"x","k2":"y","k3":"z",}; s%args
531 ns: s="x={}, y={}, z={}"; args=(123, 456, 789); s.format(*args)
441 ns: s="x=%s, y=%u, z=%x"; args=(123, 456, 789); s%args
155 ns: fmt="{}"; arg="abc"; fmt.format(arg)
206 ns: fmt="{}"; arg=123; fmt.format(arg)
204 ns: fmt="x={}"; arg="abc"; fmt.format(arg)
247 ns: fmt="x={}"; arg=123; fmt.format(arg)
172 ns: str(12345)
209 ns: '{}'.format(12345)
248 ns: 'A{}'.format(12345)
257 ns: '\x80{}'.format(12345)
265 ns: '\u0100{}'.format(12345)
263 ns: '\U00010000{}'.format(12345)
354 ns: '{:-10}'.format(12345)
404 ns: 'A{:-10}'.format(12345)
415 ns: '\x80{:-10}'.format(12345)
457 ns: '\u0100{:-10}'.format(12345)
430 ns: '\U00010000{:-10}'.format(12345)
369 ns: '{:,}'.format(12345)
418 ns: 'A{:,}'.format(12345)
437 ns: '\x80{:,}'.format(12345)
459 ns: '\u0100{:,}'.format(12345)
448 ns: '\U00010000{:,}'.format(12345)
76 ns: fmt="%s"; arg="abc"; fmt % arg
133 ns: fmt="%s"; arg=123; fmt % arg
118 ns: fmt="%s:"; arg="abc"; fmt % arg

----------
Added file: http://bugs.python.org/file25557/dont_overallocate.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue14744>
_______________________________________