[Tutor] more encoding confusion
jjcrump at myuw.net
Sun Aug 5 21:37:03 CEST 2007
On Sun, 5 Aug 2007, Kent Johnson wrote:
> Hmm...actually, isupper() works fine on unicode strings:
> In : s='H\303\211RON'.decode('utf-8')
> In : print 'H\303\211RON'
> In : s.isupper()
> Out: True
>> I modified uppers to include only the latin characters, and added the
>> apostrophe to catch placenames like L'ISLE.
> Then you are back to needing a regular expression I think.
Ah! I'm finally starting to get it. My problem wasn't with a regex to test
the line, my problem was with reading the file in as utf-8 to begin with.
When I take your advice and decode() right from the start using:
input = open('textfile', 'r')
text = input.readlines()
Then the regex problem does not even arise. Now I can use this instead:
for line in data:
as you point out, isupper() works just fine on unicode strings; it also
seems to consider the apostrophe uppercase as well because this catches
not only HÉRON, but L'ISLE as well.
Now my only glitch is that line.title() screws up placenames like STOKE
(BISHOP'S), turning it into Stoke (Bishop'S).
More information about the Tutor