[docs] [issue35295] Please clarify whether PyUnicode_AsUTF8AndSize() or PyUnicode_AsUTF8String() is preferred

[Marcin Kowalczyk]

New submission from Marcin Kowalczyk <qrczakmk at gmail.com>:

The documentation is silent whether PyUnicode_AsUTF8AndSize() or PyUnicode_AsUTF8String() is preferred.

We are under the assumption that both are acceptable for the given caller, i.e. the caller wants to access just the sequence of UTF-8 code units (e.g. for calling a C++ function which takes std::string_view or std::string as a parameter), and the caller either will copy the UTF-8 code units immediately or is willing to own a temporary object to ensure a lifetime of the UTF-8 code units.

File comments in unicodeobject.h about PyUnicode_AsUTF8AndSize() have a warning:

   *** This API is for interpreter INTERNAL USE ONLY and will likely
   *** be removed or changed in the future.
   *** If you need to access the Unicode object as UTF-8 bytes string,
   *** please use PyUnicode_AsUTF8String() instead.

The discrepancy between these comments and the documentation should be fixed. Either the documentation is correct and the comment is outdated, or the comment is correct and the documentation is lacking guidance.

It is not even clear which function is better technically:

- PyUnicode_AsUTF8String() always allocates the string. PyUnicode_AsUTF8AndSize() does not allocate the string if the unicode object is ASCII-only (this is common) or if PyUnicode_AsUTF8AndSize() was already called before.

- If conversion must be performed, then PyUnicode_AsUTF8String() makes a single allocation, while PyUnicode_AsUTF8AndSize() first calls PyUnicode_AsUTF8String() and then copies the string.

- If the string is converted multiple times, then PyUnicode_AsUTF8AndSize() caches the result - faster. If the string is converted once, then the result persists as long as the string persists - wastes memory.

I see the following possible resolutions:

1a. Declare both functions equally acceptable. Remove comments claiming that PyUnicode_AsUTF8AndSize() should be avoided.

1b. 1a, and change the implementation of PyUnicode_AsUTF8AndSize() to avoid allocating the string twice if it needs to be materialized, so that PyUnicode_AsUTF8AndSize() is never significantly slower than PyUnicode_AsUTF8String().

2a. Declare PyUnicode_AsUTF8String() preferred. Indicate this in the documentation.

2b. 2a, and provide a public interface to check and access UTF-8 code units without allocating a new string in case this is possible (I think PyUnicode_READY() + PyUnicode_IS_ASCII() + PyUnicode_DATA() + PyUnicode_GET_LENGTH() would work, but they are not documented; or possibly also check if the string has a cached UTF-8 representation without populating that cached representation), so that a combination of the check with PyUnicode_AsUTF8String() is rarely or never significantly slower than PyUnicode_AsUTF8AndSize().

----------
assignee: docs at python
components: Documentation, Interpreter Core, Unicode
messages: 330249
nosy: Marcin Kowalczyk, docs at python, ezio.melotti, vstinner
priority: normal
severity: normal
status: open
title: Please clarify whether PyUnicode_AsUTF8AndSize() or PyUnicode_AsUTF8String() is preferred
type: performance
versions: Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35295>
_______________________________________