PyUnicode_GetMax() and PyUnicode_FromOrdinal() Was: Breaking undocumented API
On Tue, Nov 16, 2010 at 1:57 PM, M.-A. Lemburg <mal@egenix.com> wrote:
Alexander Belopolsky wrote:
On Tue, Nov 16, 2010 at 1:06 PM, M.-A. Lemburg <mal@egenix.com> wrote: ..
Now, we can't use a macro for [PyUnicode_GetMax()], since the information has to be available as callable in order to applications or extensions to use it (without recompile).
.. but it *is* a macro resolving to either PyUnicodeUCS2_GetMax or PyUnicodeUCS4_GetMax.
That doesn't count :-) It's only a trick to prevent external code from using the wrong Unicode APIs.
There still is a real function behind the renaming.
What is the scenario when may want to change what PyUnicodeUCS?_GetMax return and have extensions pick up the change without a recompile?
If an extensions uses the stable ABI, it will want to know whether the interpreter was built for UCS2 or UCS4 (even if it doesn't use the Unicode APIs directly).
UCS2 case will certainly never change since it is already 0xFFFF. Is it possible that USC4 will be expanded beyond 0x10FFFF?
Well, the Unicode Consortium decided to not go beyond 0x10FFFF, but then you never know... when they started out on the quest, 16 bits appeared more than enough, but they found out relatively quickly that the Asian scripts had enough code points to easily fill that space.
Once space is available, it tends to get used sooner or later :-)
Note that we can have both a macro and a function version. This is fairly standard practice in Python C-API.
Sure, but what for ?
Note that PyUnicode_FromOrdinal() is documented (in unicodeobject.h) as follows without a reference to PyUnicode_GetMax(): """ Create a Unicode Object from the given Unicode code point ordinal. The ordinal must be in range(0x10000) on narrow Python builds (UCS2), and range(0x110000) on wide builds (UCS4). A ValueError is raised in case it is not. """ The actual implementation actually checks UCS4 range only. if (ordinal < 0 || ordinal > 0x10ffff) { PyErr_SetString(PyExc_ValueError, "chr() arg not in range(0x110000)"); return NULL; } This actually looks like a bug:
len(chr(0x10FFFF)) 2
(on a USC2 build.) Also, I think PyUnicode_FromOrdinal() should take Py_UNICODE argument rather than int.
Alexander Belopolsky wrote:
On Tue, Nov 16, 2010 at 1:57 PM, M.-A. Lemburg <mal@egenix.com> wrote:
Alexander Belopolsky wrote:
On Tue, Nov 16, 2010 at 1:06 PM, M.-A. Lemburg <mal@egenix.com> wrote: ..
Now, we can't use a macro for [PyUnicode_GetMax()], since the information has to be available as callable in order to applications or extensions to use it (without recompile).
.. but it *is* a macro resolving to either PyUnicodeUCS2_GetMax or PyUnicodeUCS4_GetMax.
That doesn't count :-) It's only a trick to prevent external code from using the wrong Unicode APIs.
There still is a real function behind the renaming.
What is the scenario when may want to change what PyUnicodeUCS?_GetMax return and have extensions pick up the change without a recompile?
If an extensions uses the stable ABI, it will want to know whether the interpreter was built for UCS2 or UCS4 (even if it doesn't use the Unicode APIs directly).
UCS2 case will certainly never change since it is already 0xFFFF. Is it possible that USC4 will be expanded beyond 0x10FFFF?
Well, the Unicode Consortium decided to not go beyond 0x10FFFF, but then you never know... when they started out on the quest, 16 bits appeared more than enough, but they found out relatively quickly that the Asian scripts had enough code points to easily fill that space.
Once space is available, it tends to get used sooner or later :-)
Note that we can have both a macro and a function version. This is fairly standard practice in Python C-API.
Sure, but what for ?
Note that PyUnicode_FromOrdinal() is documented (in unicodeobject.h) as follows without a reference to PyUnicode_GetMax():
""" Create a Unicode Object from the given Unicode code point ordinal.
The ordinal must be in range(0x10000) on narrow Python builds (UCS2), and range(0x110000) on wide builds (UCS4). A ValueError is raised in case it is not. """
The actual implementation actually checks UCS4 range only.
if (ordinal < 0 || ordinal > 0x10ffff) { PyErr_SetString(PyExc_ValueError, "chr() arg not in range(0x110000)"); return NULL; }
This actually looks like a bug:
len(chr(0x10FFFF)) 2
(on a USC2 build.)
Yes, it's a documentation bug. I guess someone forgot to update the comment in unicodeobject.h after the change to have chr()/unichr() return a 2-char string instead of a 1-char string for non-BMP code points.
Also, I think PyUnicode_FromOrdinal() should take Py_UNICODE argument rather than int.
No, an ordinal is a number, not a typed value. We have PyUnicode_FromUnicode() to create strings from Py_UNICODE* arrays. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 16 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Tue, Nov 16, 2010 at 3:06 PM, M.-A. Lemburg <mal@egenix.com> wrote: ..
len(chr(0x10FFFF)) 2
(on a USC2 build.)
Yes, it's a documentation bug. I guess someone forgot to update the comment in unicodeobject.h after the change to have chr()/unichr() return a 2-char string instead of a 1-char string for non-BMP code points.
Same problem in reST doc for chr(i): """ chr(i) Return the string of one character whose Unicode codepoint is the integer i. For example, chr(97) returns the string 'a'. This is the inverse of ord(). The valid range for the argument depends how Python was configured – it may be either UCS2 [0..0xFFFF] or UCS4 [0..0x10FFFF]. ValueError will be raised if i is outside that range. """ http://docs.python.org/dev/py3k/library/functions.html?chr And in ord(c): """ ord(c) Given a string of length one, return an integer representing the Unicode code point of the character. For example, ord('a') returns the integer 97 and ord('\u2020') returns 8224. This is the inverse of chr(). If the argument length is not one, a TypeError will be raised. (If Python was built with UCS2 Unicode, then the character’s code point must be in the range [0..65535] inclusive; otherwise the string length is two!) """ http://docs.python.org/dev/py3k/library/functions.html#ord
participants (2)
-
Alexander Belopolsky -
M.-A. Lemburg