[IPython-dev] The joys of Unicode... [was Re: weird ipython ansi codes]

[Fernando Perez]

[ Redirecting to the list so that anyone who really knows about
unicode can pitch in with words of wisdom.

context: I've been steadily learning about unicode and improving our
support, but I'm terribly worried that I'm still missing cases.  In
particular, we use shlex.shlex() to parse magic input, and I'm not
really sure that function is unicode-safe.  Perhaps it *is* unicode
safe but it must be used in a particular way, I'm just not sure.
]


On 3/30/07, Stefan van der Walt <stefan at sun.ac.za> wrote:
> Hrm, I wonder what caused this behaviour?

Could you check with current SVN please?  I'm still worried that we're
missing other problem spots with unicode handling.  In particular, see
the genutils changes in:

http://projects.scipy.org/ipython/ipython/changeset/2190

the only change there is a comment to flag the possible problem.  This
is called by parse_options:

http://projects.scipy.org/ipython/ipython/browser/ipython/trunk/IPython/Magic.py?rev=2190#L330

I've fixed some magics that do python-only arguments to first do
.encode('ascii'), but I'm worried that magics for which unicode input
makes sense will explode when arg_split() is called and shlex messes
up the original input...

I wish I had a good test case to understand this problem in its entirety...

Cheers,

f