[Python-Dev] decoding errors when comparing strings

[Guido van Rossum]

> > I see the exception as a useful warning that the program isn't
> > sufficiently Unicode aware to work correctly.  That's a *good* thing
> > in my book -- I'd rather raise an exception than silently fail.
> 
> I assume that means you're voting for alternative 3:
> 
>     "a third alternative would be to keep the exception, and make
>     the dictionary code exception proof."

Yes.

> because the following isn't exactly good behaviour:
> 
> >>> a = "„"
> >>> b = unicode(a, "iso-8859-1")
> >>> d = {}
> >>> d[a] = "a"
> >>> d[b] = "b"
> >>> len(d)
> UnicodeError: ASCII decoding error: ordinal not in range(128)
> >>> len(d)
> 2
> 
> (in other words, the dictionary implementation misbehaves if items
> with the same hash value cannot be successfully compared)

Good point.  This would happen if you used flakey class instances as
keys as well!

(Note that the exception really happens on the d[b] = "b" statement;
but because the dict implementation doesn't check for exceptions, it
gets reported too late.  We've seen this kind of bugs before in
Python.)

--Guido van Rossum (home page: http://www.pythonlabs.com/~guido/)