[Python-Dev] PyUnicode_EncodeDecimal

Victor Stinner victor.stinner at haypocalc.com
Mon Nov 21 21:39:53 CET 2011


I'm trying to rewrite PyUnicode_EncodeDecimal() to upgrade it to the new 
Unicode API. The problem is that the function is not accessible in Python nor 
tested. Should we document and test it, leave it unchanged and deprecate it, 
or simply remove it?


Python has a PyUnicode_EncodeDecimal() function. It was used in Python 2 by 
int, long and complex constructors. In Python 3, the function is no more used: 
it has been replaced by PyUnicode_TransformDecimalToASCII() in Python <= 3.2 
and _PyUnicode_TransformDecimalAndSpaceToASCII() in Python 3.3.

PyUnicode_EncodeDecimal() goes into an unlimited loop if there is more than 
one unencodable character. It's a known bug and there is a patch:

PyUnicode_EncodeDecimal() is undocumented and not tested:

Stefan Krah uses PyUnicode_EncodeDecimal() in its cdecimal project.

See also "Malformed error message from float()" issue:

Python 3.3 has now 3 encoders to decimal:
 - PyUnicode_EncodeDecimal()
 - PyUnicode_TransformDecimalToASCII()
 - _PyUnicode_TransformDecimalAndSpaceToASCII() (new in 3.3)

_PyUnicode_TransformDecimalAndSpaceToASCII() replaces also Unicode spaces with 
ASCII spaces. PyUnicode_EncodeDecimal() and 
PyUnicode_TransformDecimalToASCII() take Py_UNICODE* strings.

PyUnicode_EncodeDecimal() requires an output buffer and it has no argument for 
the size of the output buffer. It is unsafe: it leads to buffer overflow if the 
buffer is too small.


More information about the Python-Dev mailing list