[Python-Dev] PyUnicode_EncodeDecimal

[Victor Stinner]

Hi,

I'm trying to rewrite PyUnicode_EncodeDecimal() to upgrade it to the new 
Unicode API. The problem is that the function is not accessible in Python nor 
tested. Should we document and test it, leave it unchanged and deprecate it, 
or simply remove it?

--

Python has a PyUnicode_EncodeDecimal() function. It was used in Python 2 by 
int, long and complex constructors. In Python 3, the function is no more used: 
it has been replaced by PyUnicode_TransformDecimalToASCII() in Python <= 3.2 
and _PyUnicode_TransformDecimalAndSpaceToASCII() in Python 3.3.

PyUnicode_EncodeDecimal() goes into an unlimited loop if there is more than 
one unencodable character. It's a known bug and there is a patch:
http://bugs.python.org/issue13093

PyUnicode_EncodeDecimal() is undocumented and not tested:
http://bugs.python.org/issue8646

Stefan Krah uses PyUnicode_EncodeDecimal() in its cdecimal project.

See also "Malformed error message from float()" issue:
http://bugs.python.org/issue10557

Python 3.3 has now 3 encoders to decimal:
 - PyUnicode_EncodeDecimal()
 - PyUnicode_TransformDecimalToASCII()
 - _PyUnicode_TransformDecimalAndSpaceToASCII() (new in 3.3)

_PyUnicode_TransformDecimalAndSpaceToASCII() replaces also Unicode spaces with 
ASCII spaces. PyUnicode_EncodeDecimal() and 
PyUnicode_TransformDecimalToASCII() take Py_UNICODE* strings.

PyUnicode_EncodeDecimal() requires an output buffer and it has no argument for 
the size of the output buffer. It is unsafe: it leads to buffer overflow if the 
buffer is too small.

Victor