[Python-3000] How will unicode get used?
jimjjewett at gmail.com
Wed Sep 20 22:59:22 CEST 2006
On 9/20/06, Guido van Rossum <guido at python.org> wrote:
> On 9/20/06, Adam Olsen <rhamph at gmail.com> wrote:
> > On 9/20/06, Guido van Rossum <guido at python.org> wrote:
> > > Let me cut this short. The external string API in Py3k should not
> > > change or only very marginally so (like removing rarely used useless
> > > APIs or adding a few new conveniences).
> I thought we were discussing the Python API.
I don't think anyone has proposed much change to strings *as seen from
At most, there has been an implicit suggestion that the
bytes.decode().encode() dance be shortened.
> C code will likely have the same access to unicode objects as it has in 2.x.
Can C code still assume that
(1) the data buffer will always be available for any sort of
direct manipulation (including mutation)
(2) in a specific canonical encoding
(3) directly from the memory layout, without calling a "prepare"
or "recode" or "encode" method first.
Today, that canonical encoding is a compile-time choice, and any
specific choice causes integration hassles.
Unless the choice matches the system default for text, it also
requires many decode/encode round trips that might otherwise be
The proposed changes mostly boil down to removing the third
assumption, and agreeing that some implementations might delay the
decode-to-canonical-format until it was needed.
Rough Summary of new C API restrictions:
((PyStringObject *)string).ob_sval /* supported today */
PyString_AsString(string) /* already recommended */
((PyUnicodeObject *)string)->str /* supported today */
((PyUnicodeObject *)string)->defenc /* supported today */
PyUnicode_AsEncodedString(PyObject *unicode, /* already recommended */
const char *encoding,
const char *errors)
PyUnicode_AsAnyString(PyObject *unicode, /* new */
char **encoding, /* return the actual encoding */
const char *errors)
Also note that some macros would need to become functions. The most
PyUnicode_AS_DATA(string) /* supports mutation */
More information about the Python-3000