Py_UNICODE* string support
Hi, Please review my feature proposal to add Py_UNICODE* string support for better Windows interoperability: https://github.com/cython/cython/pull/191 This is motivated by my current work that involves calling lots of Windows APIs. If people are interested I can elaborate on some important points, like the choice of base type (Py_UNICODE vs wchar_t) or the nature of Py_UNICODE* literals or why this feature is necessary at all. Best regards, Nikita Nemkin
Nikita Nemkin, 03.03.2013 08:39:
Please review my feature proposal to add Py_UNICODE* string support for better Windows interoperability: https://github.com/cython/cython/pull/191
This is motivated by my current work that involves calling lots of Windows APIs.
If people are interested I can elaborate on some important points, like the choice of base type (Py_UNICODE vs wchar_t) or the nature of Py_UNICODE* literals or why this feature is necessary at all.
Are you aware that Py_UNICODE is deprecated as of Py3.3? http://docs.python.org/3.4/c-api/unicode.html Your changes look a bit excessive for supporting something that's inefficient in recent Python versions and basically "dead". Stefan
On Sun, 03 Mar 2013 13:52:49 +0600, Stefan Behnel <stefan_ml@behnel.de> wrote:
Are you aware that Py_UNICODE is deprecated as of Py3.3?
http://docs.python.org/3.4/c-api/unicode.html
Your changes look a bit excessive for supporting something that's inefficient in recent Python versions and basically "dead".
Yes, I'm well aware of Py3.3 changes, but consider this: 1. _All_ system APIs on Windows, old, new and in-between, use UTF-16 in the form of zero-terminated 2-byte wchar_t* strings (on Windows Py_UNICODE is _always_ aliased to wchar_t specifically for this reason). Whatever happens to Python internals, the need to interoperate with UTF-16 based platforms won't go away. 2. PY_UNICODE family of APIs remains the recommended way to interoperate with Windows. (So said the autor of PEP393 himself, I could find the relevant discussion in python-dev.) 3. It is not _that_ inefficient. Actually, it has the same efficiency as the UTF8-related APIs (which have to be used on UTF-8 platforms like most *nix systems). UTF8 allows sharing of ASCII buffer and has to convert USC2/UCS4, Py_UNICODE shares UCS2 buffer (assuming narrow build) and has to convert ASCII. One alternative to Py_UNICODE that I have rejected is using Python's wchar_t support. It's practicaly useless for these reasons: 1) wchar_t APIs do not exist in Py2 and have to be implemented for compatibility. 2) Implementing them brings in all the pain of nonportable wchar_t type (on *nix systems in general), whereas it's the primary users would target Windows, where (pretty horrible) wchar_t portability workarounds would be dead code. 3) wchar_t APIs do not offer a zero-copy option and do not manage the memory for us. The changes are some 50 lines of code, not counting the tests. I wouldn't call that excessive. And they mostly mirror existing code, no trickery of any kind. Inbuilt Py_UNICODE* support also means that the users would be shielded from 3.3 changes and Cython is free to optimize sting handling in the future. Believe me, nobody calls Py_UNICODE APIs because they want to, they just have to. Best regards, Nikita Nemkin
Nikita Nemkin, 03.03.2013 09:25:
On Sun, 03 Mar 2013 13:52:49 +0600, Stefan Behnel wrote:
Are you aware that Py_UNICODE is deprecated as of Py3.3?
http://docs.python.org/3.4/c-api/unicode.html
Your changes look a bit excessive for supporting something that's inefficient in recent Python versions and basically "dead".
Yes, I'm well aware of Py3.3 changes, but consider this:
1. _All_ system APIs on Windows, old, new and in-between, use UTF-16 in the form of zero-terminated 2-byte wchar_t* strings (on Windows Py_UNICODE is _always_ aliased to wchar_t specifically for this reason). Whatever happens to Python internals, the need to interoperate with UTF-16 based platforms won't go away.
Ok, fine with me. Your changes look fairly reasonable, especially for a first try. I have the following comments. 1) I would like to get rid of UnicodeConst. A Py_UNICODE* is not different from any other C array, except that it can coerce to and from Unicode strings. So the representation of a literal should be a (properly reference counted) Python Unicode object, and users would be allowed to cast them to <Py_UNICODE*>, just as we support it for <char*> and bytes. 2) non-BMP literals should be supported by representing them as normal Unicode strings and creating the Py_UNICODE representation at need (i.e. explicitly through a cast, at runtime). Py_UNICODE[] literals are simply not portable. 3) __Pyx_Py_UNICODE_strlen() is ok, but only for the special case that all we have is a Py_UNICODE*. As long as we are dealing with Unicode string objects, that won't be needed, so len() should be constant time in the normal case instead of linear time. 4) most of the changes in PyrexTypes.py and ExprNodes.py look ok. I would eventually like to see a couple of refactorings on these sections (because the special cases add up over time), but that's not required for this change. So, the basic idea would be to use Unicode strings and their (optional) internal representation as Py_UNICODE[] instead of making Py_UNICODE[] a first class data type. And then go from there and optimise certain things to use the unpacked array directly, so that users won't need to put explicit C-API calls into their code. Stefan
On Sun, 03 Mar 2013 15:32:36 +0600, Stefan Behnel <stefan_ml@behnel.de> wrote:
1) I would like to get rid of UnicodeConst. A Py_UNICODE* is not different from any other C array, except that it can coerce to and from Unicode strings. So the representation of a literal should be a (properly reference counted) Python Unicode object, and users would be allowed to cast them to <Py_UNICODE*>, just as we support it for <char*> and bytes.
I understand the idea. Since Python unicode literals are implicitly coercible to Py_UNICODE*, there appears to be no need for C-level Py_UNICODE[] literals. Indeed, client code will look exactly (!) the same whether they are supported or not. Except when it comes to nogil. (For example, native callbacks are almost guaranteed to be nogil.) Hiding Python operations in what appears to be pure C-level code will break users' assumptions. This is #1 reason why I went for C-level literals. #2 reason is efficiency on Py3.3. C-level literals don't need conversions and don't call any conversion APIs.
2) non-BMP literals should be supported by representing them as normal Unicode strings and creating the Py_UNICODE representation at need (i.e. explicitly through a cast, at runtime). Py_UNICODE[] literals are simply not portable.
Py_UNICODE[] literals can be made fully portable if non-BMP ones are wrapped like this: #ifdef Py_UNICODE_WIDE static const k_xxx[] = { <UTF-32 array without surrogates>, 0 }; #else static const k_xxx[] = { <UTF-16 array with surrogates>, 0 }; #endif Literals containing only BMP chars are already portable and don't need this wrapping.
3) __Pyx_Py_UNICODE_strlen() is ok, but only for the special case that all we have is a Py_UNICODE*. As long as we are dealing with Unicode string objects, that won't be needed, so len() should be constant time in the normal case instead of linear time.
len(Py_UNICODE*) simply mirrors len(char*). Its putpose is to provide platform-independent Py_UNICODE_strlen (which is Py3 only and deprecated in 3.3).
So, the basic idea would be to use Unicode strings and their (optional) internal representation as Py_UNICODE[] instead of making Py_UNICODE[] a first class data type. And then go from there and optimise certain things to use the unpacked array directly, so that users won't need to put explicit C-API calls into their code.
Please reconsider your decision wrt C-level literals. I believe that nogil code and a bit of efficiency (on 3.3) justify their existence. (char* literals do have C-level literals, Py_UNICODE* is in the same basket when it comes to Windows code). The code to support them is also small and well-contained. I've updated my pull request to fully support for non-BMP Py_UNICODE[] literals. If you are still not convinced, so be it, I'll drop C-level literal support. Best regards, Nikita Nemkin PS. I made a false claim in the previous mail. (Some of) Python's wchar_t APIs do exist in Py2. But they won't manage the memory automatically anyway.
Nikita Nemkin, 03.03.2013 14:40:
Please reconsider your decision wrt C-level literals. I believe that nogil code and a bit of efficiency (on 3.3) justify their existence. (char* literals do have C-level literals, Py_UNICODE* is in the same basket when it comes to Windows code). The code to support them is also small and well-contained. I've updated my pull request to fully support for non-BMP Py_UNICODE[] literals.
Ok, I think it's ok now. I can accept the special casing of Py_UNICODE literals, it actually adds a value. As one little nit-pick, may I ask you to rename the new name references to "unicode" into "py_unicode" in your code? For example, "is_unicode", "get_unicode_const", "unicode_const_index", etc. Given that Py_UNICODE is no longer the native equivalent of Python's unicode type in Py3.3, I'd like to avoid confusion in the code. The name "unicode" is much more likely to refer to the builtin Python type than to a native C type when it appears in Cython's sources. Stefan
Stefan Behnel, 03.03.2013 20:41:
Nikita Nemkin, 03.03.2013 14:40:
Please reconsider your decision wrt C-level literals. I believe that nogil code and a bit of efficiency (on 3.3) justify their existence. (char* literals do have C-level literals, Py_UNICODE* is in the same basket when it comes to Windows code). The code to support them is also small and well-contained. I've updated my pull request to fully support for non-BMP Py_UNICODE[] literals.
Ok, I think it's ok now. I can accept the special casing of Py_UNICODE literals, it actually adds a value.
As one little nit-pick, may I ask you to rename the new name references to "unicode" into "py_unicode" in your code? For example, "is_unicode", "get_unicode_const", "unicode_const_index", etc. Given that Py_UNICODE is no longer the native equivalent of Python's unicode type in Py3.3, I'd like to avoid confusion in the code. The name "unicode" is much more likely to refer to the builtin Python type than to a native C type when it appears in Cython's sources.
Oh, and yet another thing: could you write up some documentation for this in docs/src/tutorial/strings.rst ? Basically a Windows/wchar_t related section, that also warns about the inefficiency in Py3.3, so that users don't accidentally assume it's efficient for anything that needs to be portable. Stefan
On Mon, 04 Mar 2013 01:56:59 +0600, Stefan Behnel <stefan_ml@behnel.de> wrote:
As one little nit-pick, may I ask you to rename the new name references to "unicode" into "py_unicode" in your code? For example, "is_unicode", "get_unicode_const", "unicode_const_index", etc. Given that Py_UNICODE is no longer the native equivalent of Python's unicode type in Py3.3, I'd like to avoid confusion in the code. The name "unicode" is much more likely to refer to the builtin Python type than to a native C type when it appears in Cython's sources.
Actually, "py_unicode" is even more likely to be mistaken for Python-level unicode. There are already pairs of methods like get_string_const (C-level) + get_py_string_const (Py-level). I suggest one of "py_unicode_ptr", "py_unicode_str", "wstring", "wide_string", "ustring", "unicode_string" to unambiguously refer to Py_UNICODE* variables and constants. Take yout pick.
Oh, and yet another thing: could you write up some documentation for this in docs/src/tutorial/strings.rst ? Basically a Windows/wchar_t related section, that also warns about the inefficiency in Py3.3, so that users don't accidentally assume it's efficient for anything that needs to be portable.
Sure, I'm writing the docs now. Best regards, Nikita Nemkin
Nikita Nemkin, 04.03.2013 18:39:
On Mon, 04 Mar 2013 01:56:59 +0600, Stefan Behnel wrote:
As one little nit-pick, may I ask you to rename the new name references to "unicode" into "py_unicode" in your code? For example, "is_unicode", "get_unicode_const", "unicode_const_index", etc. Given that Py_UNICODE is no longer the native equivalent of Python's unicode type in Py3.3, I'd like to avoid confusion in the code. The name "unicode" is much more likely to refer to the builtin Python type than to a native C type when it appears in Cython's sources.
Actually, "py_unicode" is even more likely to be mistaken for Python-level unicode. There are already pairs of methods like get_string_const (C-level) + get_py_string_const (Py-level).
Agreed.
I suggest one of "py_unicode_ptr", "py_unicode_str", "wstring", "wide_string", "ustring", "unicode_string" to unambiguously refer to Py_UNICODE* variables and constants. Take yout pick.
I think "pyunicode_ptr" or even just "pyunicode" makes it quite clear what it's about and specifically that "pyunicode" is actually a type name, not a "py_something". Even "pyunicode_array" would work, although it might suggest that we know more at compile time than we do, such as the length. I'll let you choose between these three, although I'm leaning slightly towards an order of preference as they appear above.
Oh, and yet another thing: could you write up some documentation for this in docs/src/tutorial/strings.rst ? Basically a Windows/wchar_t related section, that also warns about the inefficiency in Py3.3, so that users don't accidentally assume it's efficient for anything that needs to be portable.
Sure, I'm writing the docs now.
Nice. Stefan
participants (2)
-
Nikita Nemkin -
Stefan Behnel