[Numpy-discussion] Unicode revisited

[Charles R Harris]

On Fri, Aug 3, 2012 at 7:03 PM, Travis Oliphant <travis at continuum.io> wrote:

> Hey all,
>
> Ondrej has been working hard with feedback from many others on improving
> Unicode support in NumPy (especially for Python 3.3).   Looking at what
> Python has done in Python 3.3 (PEP 393) and chatting on the Python issue
> tracker with the author of that PEP has made me wonder if we aren't "doing
> the wrong thing" in NumPy quite often.
>
> Basically, NumPy only supports UTF-32 in it's Unicode representation.
> All bytes in NumPy arrays should be either UTF-32LE or UTF-32BE.    This is
> all pretty easy to understand as long as you stick with NumPy arrays only.
>
> The difficulty starts when you start to interact with the unicode array
> scalar (which is the same data-structure exactly as a Python unicode object
> with a different type-name --- numpy.unicode_).    However, I overlooked
> the "encoding" argument to the standard "unicode" constructor which might
> have simplified what we are doing.    If I understand things correctly,
> now, all we need to do is to "decode" the UTF-32LE or UTF-32BE raw bytes in
> the array (depending on the dtype) into a unicode object.
>
> This is easily accomplished with  numpy.unicode_(<bytes object>,
> 'utf_32_be'  or 'utf_32_le').    There is also an "encoding" equivalent to
> go from the Python unicode object to the bytes representation in the NumPy
> array.   I think this is what we should be doing in most of the places and
> it should considerably simplify the Unicode code in NumPy --- eliminating
> possibly the ucsnarrow.c file.
>
> Am I missing something?
>
>
I can't comment on the rest, but I'd be happy to see the end of the
ucsnarrow.c file. It needs more work to be properly generalized and if
there is a way to avoid that, so much the better.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20120803/497d380b/attachment.html>