[IPython-dev] Bug caused by commit 2ccc90d68ac01145ab97

Jörgen Stenarson jorgen.stenarson at bostream.nu
Fri Mar 4 15:00:32 EST 2011

Thomas Kluyver skrev 2011-03-03 21:37:
> On 3 March 2011 19:22, Jörgen Stenarson <jorgen.stenarson at bostream.nu
> <mailto:jorgen.stenarson at bostream.nu>> wrote:
>     yes it is possible to enter other character codes but you have to
>     use the unicode functions in the api to read them properly. It is
>     for the 8-bit api that the encoding is valid.
>   OK, I'm not entirely following what you mean here. If I did a =
> raw_input(), and entered a non-cp1252 character, what ends up in a? Does
> it mysteriously turn the output into a unicode object? Or does it
> produce some odd string of bytes?
>  > In pyreadline we are using the wide api and are returning unicode
> strings, however I have not tested that extensively so there may be bugs
> in that handling.
> OK, does that mean that we should add unicode strings to add_history,
> rather than byte strings? Only I think readline in Python 2 on Linux
> choked on unicode when it contained non-ascii characters. Or is there an
> encoding we should use instead of cp1252?
> Thomas


looking into it in more detail, pyreadline can return unicode strings 
but the call the hook that raw_input uses gets a regular string in 
return that has been encoded using .encode(pyreadline_codepage, u"replace").

I have attached a script and a screenshot of a session that demonstrates 
the difference between using raw_input and calling readline directly.

As things are now, using raw_input you should not receive any characters 
outside the current encoding because they will be converted to ? before 
being returned to raw_input.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: repl.png
Type: image/png
Size: 4196 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20110304/baa76356/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: repl.py
Type: application/x-python
Size: 123 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20110304/baa76356/attachment.bin>

More information about the IPython-dev mailing list