[New-bugs-announce] [issue16147] Rewrite PyUnicode_FromFormatV() to use the _PyUnicodeWriter API

[STINNER Victor]

New submission from STINNER Victor:

Attached patch rewrites PyUnicode_FromFormatV():
 * simplify the code: replace 4 steps with one unique step. PyUnicode_Format() has the same design. It avoids to store intermediate results which require to allocate an array of pointers in the heap.
 * use the _PyUnicodeWriter API for speed (and its convinient API): overallocate the buffer to reduce the number of "realloc()"
 * Implement "width" and "precision" in Python, don't rely on sprintf(). It avoids to need of a temporary buffer allocated on the heap: only use a small buffer allocated in the stack.
 * Detect integer overflow when parsing width and precision, as done in PyUnicode_Format()
 * Add _PyUnicodeWriter_WriteCstr() function
 * Split PyUnicode_FromFormatV() into smaller functions: add unicode_fromformat_arg(). It requires to copy vargs using Py_VA_COPY: without Py_VA_COPY, the function does crash. I don't understand why.
 * Inline parse_format_flags(): the format of an argument is now only parsed once, it's no more needed to have a subfunction.
 * Optimize PyUnicode_FromFormatV() for characters between two arguments: search the next "%" and copy the substring in one chunk, instead of copying character per character.
 * Replace "prec too big" with "precision too big" in error messages

_tescapi.test_string_from_format() is 20% faster with the patch according to timeit. I don't know how to write better benchmarks because PyUnicode_FromV() is not exposed in Python. I wrote a benchmark using ctypes to call the function, but it looks like the ctypes overhead is too high.

I wrote the patch to simplify the code, but it may be faster thanks to the _PyUnicodeWriter API and some optimizations implemented in the patch.

----------
files: unicode_fromformat.patch
keywords: patch
messages: 172138
nosy: haypo
priority: normal
severity: normal
status: open
title: Rewrite PyUnicode_FromFormatV() to use the _PyUnicodeWriter API
type: performance
versions: Python 3.4
Added file: http://bugs.python.org/file27440/unicode_fromformat.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue16147>
_______________________________________