New GitHub issue #119182 from vstinner:<br>
<hr>
<pre>
# Feature or enhancement
Creating a Python string object in an efficient way is complicated. Python has **private** `_PyUnicodeWriter` API. It's being used by these projects:
Affected projects (5):
* Cython (3.0.9)
* asyncpg (0.29.0)
* catboost (1.2.3)
* frozendict (2.4.0)
* immutables (0.20)
I propose making the API public to promote it and help C extensions maintainers to write more efficient code to create Python string objects.
API:
```c
typedef struct PyUnicodeWriter PyUnicodeWriter;
PyAPI_FUNC(PyUnicodeWriter*) PyUnicodeWriter_Create(void);
PyAPI_FUNC(void) PyUnicodeWriter_Free(PyUnicodeWriter *writer);
PyAPI_FUNC(PyObject*) PyUnicodeWriter_Finish(PyUnicodeWriter *writer);
PyAPI_FUNC(void) PyUnicodeWriter_SetOverallocate(
PyUnicodeWriter *writer,
int overallocate);
PyAPI_FUNC(int) PyUnicodeWriter_WriteChar(
PyUnicodeWriter *writer,
Py_UCS4 ch);
PyAPI_FUNC(int) PyUnicodeWriter_WriteStr(
PyUnicodeWriter *writer,
PyObject *str);
PyAPI_FUNC(int) PyUnicodeWriter_WriteSubstring(
PyUnicodeWriter *writer,
PyObject *str,
Py_ssize_t start,
Py_ssize_t stop);
PyAPI_FUNC(int) PyUnicodeWriter_WriteASCIIString(
PyUnicodeWriter *writer,
const char *ascii,
Py_ssize_t len);
```
The internal writer buffer is **overallocated by default**. `PyUnicodeWriter_Finish()` truncates the buffer to the exact size if the buffer was overallocated.
Overallocation reduces the cost of exponential complexity when adding short strings in a loop. Use `PyUnicodeWriter_SetOverallocate(writer, 0)` to disable overallocation just before the last write.
The writer takes care of the internal buffer kind: Py_UCS1 (latin1), Py_UCS2 (BMP) or Py_UCS4 (full Unicode Character Set). It also implements an optimization if a single write is made using `PyUnicodeWriter_WriteStr()`: it returns the string unchanged without any copy.
---
Example of usage (simplified code from Python/unionobject.c):
```c
static int
union_repr_item(PyUnicodeWriter *writer, PyObject *p)
{
PyObject *r = PyObject_Repr(p);
if (r == NULL) {
return -1;
}
rc = PyUnicodeWriter_WriteStr(writer, r);
Py_DECREF(r);
return rc;
}
static PyObject *
union_repr(PyObject *self)
{
unionobject *alias = (unionobject *)self;
Py_ssize_t len = PyTuple_GET_SIZE(alias->args);
PyUnicodeWriter *writer = PyUnicodeWriter_Create();
if (writer == NULL) {
return NULL;
}
for (Py_ssize_t i = 0; i < len; i++) {
if (i > 0 && PyUnicodeWriter_WriteASCIIString(writer, " | ", 3) < 0) {
goto error;
}
PyObject *p = PyTuple_GET_ITEM(alias->args, i);
if (union_repr_item(writer, p) < 0) {
goto error;
}
}
return PyUnicodeWriter_Finish(writer);
error:
PyUnicodeWriter_Free(writer);
return NULL;
}
```
</pre>
<hr>
<a href="https://github.com/python/cpython/issues/119182">View on GitHub</a>
<p>Labels: type-feature, topic-C-API</p>
<p>Assignee: </p>