[Patches] [ python-Patches-742290 ] unicode "support" for shlex.py

[SourceForge.net]

Patches item #742290, was opened at 2003-05-23 14:47
Message generated for change (Comment added) made by loewis
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=742290&group_id=5470

Category: Library (Lib)
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Just van Rossum (jvr)
Assigned to: Nobody/Anonymous (nobody)
Summary: unicode "support" for shlex.py

Initial Comment:
Due to shlex.py's use of cStringIO, it behaves badly when fed unicode 
strings. The attached patch fixes that by always using StringIO instead 
of cStringIO.

----------------------------------------------------------------------

>Comment By: Martin v. Löwis (loewis)
Date: 2003-05-24 14:32

Message:
Logged In: YES 
user_id=21627

To test whether a letter is a wordchar, you should check
whether it .isalnum() or equals '_'. Then you can do away
with self.wordchars, and it works the same for byte strings
and Unicode strings. Non-ASCII characters in byte strings
then work if locale.setlocale had been invoked.

----------------------------------------------------------------------

Comment By: Just van Rossum (jvr)
Date: 2003-05-23 14:57

Message:
Logged In: YES 
user_id=92689

Ugh, I take that back: it doesn't fix it, there's a gross snippet in shlex.py that 
makes it barf:

        if self.posix:
            self.wordchars += ('??·???ÂÊÁËÈÍÎÏÌÓÔ?ÒÚÛÙ??¯???¸???'
                               '¿¡¬????«»? ÀÃÕ????????÷ÿ??????')

Help. I'd love to fix this, but I'm not sure what would be correct (my intuition 
says to just yank the above snippet, but I'm sure that'll make _someone_ 
unhappy...).

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=742290&group_id=5470