[ python-Bugs-1772788 ] chr(128) in u'only ascii' -> TypeError with misleading msg

[SourceForge.net]

Bugs item #1772788, was opened at 2007-08-13 01:54
Message generated for change (Comment added) made by laukpe
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1772788&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Pekka Laukkanen (laukpe)
Assigned to: Nobody/Anonymous (nobody)
Summary: chr(128) in u'only ascii' -> TypeError with misleading msg

Initial Comment:
A test using in format "chr(x) in <string>" raises a TypeError if "x" is in range 128-255 (i.e. non-ascii) and string is unicode. This happens even if the unicode string contains only ascii data as the example below demonstrates.

Python 2.5.1 (r251:54863, May  2 2007, 16:56:35) 
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> chr(127) in 'hello'
False
>>> chr(128) in 'hello'
False
>>> chr(127) in u'hi'
False
>>> chr(128) in u'hi'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'in <string>' requires string as left operand

This can cause pretty nasty and hard-to-debug bugs in code using "in <string>" format if e.g. user provided data is converted to unicode internally. Most other string operations work nicely between normal and unicode strings and I'd say simply returning False in this situation would be ok too. Issuing a warning similarly as below might be a good idea also.  

>>> chr(128) == u''
__main__:1: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal

Finally, the error message is somewhat misleading since the left operand is definitely a string.

>>> type(chr(128))
<type 'str'>

A real life example of code where this problem exist is telnetlib. I'll submit a separate bug about it as that problem can obviously be fixed in the library itself.


----------------------------------------------------------------------

>Comment By: Pekka Laukkanen (laukpe)
Date: 2007-08-21 17:03

Message:
Logged In: YES 
user_id=1379331
Originator: YES

Fredrik, you are obviously correct that most operations between normal and
unicode strings don't work if the normal string contains non-ascii data.

I still do think that a UnicodeWarning like you get from "chr(128) ==
u'foo'" would be nicer than an exception and prevent problems like the one
in telnetlib [1]. If an exception is raised I don't care too much about its
type but a better message would make debugging possible problems easier.

[1]
https://sourceforge.net/tracker/index.php?func=detail&aid=1772794&group_id=5470&atid=105470

----------------------------------------------------------------------

Comment By: Fredrik Lundh (effbot)
Date: 2007-08-21 11:48

Message:
Logged In: YES 
user_id=38376
Originator: NO

"Most other string operations work nicely between normal and unicode
strings"

Nope.  You *always* get errors if you mix Unicode with NON-ASCII data
(unless you've messed up the system's default encoding, which is a bad
thing to do if you care about portability).  Some examples:

>>> chr(128) + u"foo"
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0:
ordinal not in range(128)
>>> u"foo".find(chr(128))
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0:
ordinal not in range(128)

etc.  If there's a bug here, it's that you get a TypeError instead of a
ValueError subclass.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1772788&group_id=5470