[Python-Dev] Tcl and Unicode
Guido van Rossum
Sat, 07 Oct 2000 08:51:12 -0500
> Fix for next iteration of SF bug 115690 (Unicode headaches in IDLE). The
> parsing functions in support of auto-indent weren't expecting Unicode
> strings, but text.get() can now return them (although it remains muddy as
> to exactly when or why that can happen). Fixed that with a Big Hammer.
I apologize, I should have explained when text.get() returns Unicode:
Any string returned from Tcl/Tk that contains a byte with the 8th bit
set is translated from UTF-8 into Unicode, unless the translation
fails (in which case the original raw 8-bit string is returned as a
This *should* be correct because Tcl/Tk always uses UTF-8 internally.
(Even though it is "lenient" when receiving strings -- if a sequence
of characters has no valid Unicode representation, it appears to falls
back to Latin-1; I don't know the details of this algorithm.)
--Guido van Rossum (home page: http://www.python.org/~guido/)