PEP: 9999
Title: Remove wstr from Unicode
Author: Inada Naoki
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 18-Jun-2020
Python-Version: TBD
Abstract
========
PEP 393 deprecated some unicode APIs, and introduced ``wchar_t *wstr``,
and ``Py_ssize_t wstr_length`` in unicode implementation for backward
compatibility of these deprecated APIs. [1]_
This PEP is planning removal of ``wstr``, and ``wstr_length`` with
deprecated APIs using these members.
Motivation
==========
Memory usage
------------
``str`` is one of the most used types in Python. Even most simple ASCII
strings have a ``wstr`` member. It consumes 8 bytes on 64bit systems.
Runtime overhead
----------------
To support legacy Unicode object created by
``PyUnicode_FromUnicode(NULL, length)``, many Unicode APIs has
``PyUnicode_READY()`` check.
When we drop support of legacy unicode object, We can reduce this overhead
too.
Simplicity
----------
Support of legacy Unicode object makes Unicode implementation complex.
Until we drop legacy Unicode object, it is very hard to try other Unicode
implementation like UTF-8 based implementation in PyPy.
Specification
=============
Affected APIs
--------------
From the Unicode implementation, ``wstr`` and ``wstr_length`` members are
removed.
Macros and functions to be removed:
* PyUnicode_GET_SIZE
* PyUnicode_GET_DATA_SIZE
* Py_UNICODE_WSTR_LENGTH
* PyUnicode_AS_UNICODE
* PyUnicode_AS_DATA
* PyUnicode_AsUnicode
* PyUnicode_AsUnicodeAndSize
Behaviors to be removed:
* PyUnicode_FromUnicode -- ``PyUnicode_FromUnicode(NULL, size)`` where
``size > 0`` cause RuntimeError instead of creating legacy Unicode
object. While this API is deprecated by PEP 393, this API will be kept
when ``wstr`` is removed. This API will be removed later.
* PyUnicode_FromStringAndSize -- Like PyUnicode_FromUnicode,
``PyUnicode_FromStringAndSize(NULL, size)`` cause RuntimeError
instead of creating legacy unicode object.
* PyArg_ParseTuple, PyArg_ParseTupleAndKeywords -- 'u', 'u#', 'Z', and
'Z#' format will be removed.
Deprecation
-----------
All APIs to be removed should have compiler deprecation warning
(e.g. `Py_DEPRECATED(3.3)`) from Python 3.9. [2]_
All APIs to be changed should raise DeprecationWarning for behavior to be
removed. Note that ``PyUnicode_FromUnicode`` has both of compiler deprecation
warning and runtime DeprecationWarning. [3]_, [4]_.
Plan
-----
All deprecations will be implemented in Python 3.10.
Some deprecations will be backported in Python 3.9.
Actual removal will happen in Python 3.12.
Alternative Ideas
=================
Advanced Schedule
-----------------
Backport warnings in 3.9, and do the removal in early development phase
in Python 3.11. If many third packages are broken by this change, we will
revert the change and back to the regular schedule.
Pros: There is a chance to remove ``wstr`` in Python 3.11. Even if we need
to revert it, third party maintainers can have more time to prepare the
removal and we can get feedback from the community early.
Cons: Adding warnings in beta period will make some confusion. Note that
we need to avoid the warning from CPython core and stdlib.
Use hashtable to store wstr
---------------------------
Store the ``wstr`` in a hashtable, instead of Unicode structure.
Pros: We can save memory usage even from Python 3.10. We can have
more longer timeline to remove the ``wstr``.
Cons: This implementation will increase the complexity of Unicode
implementation.
References
==========
A collection of URLs used as references through the PEP.
.. [1] PEP 393 -- Flexible String Representation
(https://www.python.org/dev/peps/pep-0393/)
.. [2] GH-20878 -- Add Py_DEPRECATED to deprecated unicode APIs
(https://github.com/python/cpython/pull/20878)
.. [3] GH-20933 -- Raise DeprecationWarning when creating legacy Unicode
(https://github.com/python/cpython/pull/20933)
.. [4] GH-20927 -- Raise DeprecationWarning for getargs with 'u', 'Z' #20927
(https://github.com/python/cpython/pull/20927)
Copyright
=========
This document has been placed in the public domain.
--
Inada Naoki
