unicode data - accessing codepoints > FFFF on narrow python builts

vbr at email.cz vbr at email.cz
Wed Apr 18 05:37:56 EDT 2007


Hi all,
I'd like to ask about the usage of unicode data on a narrow python build.
Unicode string literals \N{name} work even without the (explicit) import of unicodedata and it correctly handles also the  "wider" unicodes planes - over FFFF

>>>  u"\N{LATIN SMALL LETTER E}"
u'e'
>>>  u"\N{GOTHIC LETTER AHSA}"
u'\U00010330'

The unicode data functions works analogous in the basic plane, but behave differently otherwise:

>>>  unicodedata.lookup("LATIN SMALL LETTER E")
u'e'
>>> unicodedata.lookup("GOTHIC LETTER AHSA")
u'\u0330'

(0001 gets trimmed)

Is it a bug in unicodedata, or is this the expected behaviour on a narrow build?

Another problem I have is to access the "characters" and their properties by the respective codepoints:
under FFFF it is possible, to use unichr(), which isn't valid for higher valules on a narrow build
It is possible to derive the codepoint from the surrogate pair, which would be usable also for wider codepoints.

Currently, I'm using a kind of parallel database for some unicode ranges above FFFF, but I don't think, this is the most effective way.
I actually found something similar at http: / / inamidst.com/phenny/modules/codepoint.py  using directly the UnicodeData.txt;

but I was wondering, If there is a simpler way for doing that; it seems obvious, that the data are present, if it could be used for constucting unicode literals.

Any hints are welcome,   thanks.

vbr



More information about the Python-list mailing list