[issue10557] Malformed error message from float()

[Marc-Andre Lemburg]

Marc-Andre Lemburg <mal at egenix.com> added the comment:

Alexander Belopolsky wrote:
> 
> Alexander Belopolsky <belopolsky at users.sourceforge.net> added the comment:
> 
> After a bit of svn archeology, it does appear that Arabic-Indic digits' support was deliberate at least in the sense that the feature was tested for when the code was first committed. See r15000.

As I mentioned on python-dev (http://mail.python.org/pipermail/python-dev/2010-November/106077.html)
this support was added intentionally.

> The test migrated from file to file over the last 10 years, but it is still present in test_float.py:
> 
>         self.assertEqual(float(b"  \u0663.\u0661\u0664  ".decode('raw-unicode-escape')), 3.14)
> 
> (It should probably be now rewritten using a string literal.)
> 
> I am now attaching the patch (issue10557.diff) that fixes the bug without sacrificing non-ASCII digit support.
> If this approach is well-received, I would like to replace all calls to PyUnicode_EncodeDecimal() with the calls to the new _PyUnicode_EncodeDecimalUTF8() and deprecate Latin-1-oriented PyUnicode_EncodeDecimal().

It would be better to copy and iterate over the Unicode string first,
replacing any decimal code points with ASCII ones and then call the
UTF-8 encoder.

The code as it stands is very inefficient, since it will most likely
run the memcpy() part for every code point after the first non-ASCII
decimal one.

> For the future, I note that starting with Unicode 6.0.0, the Unicode Consortium promises that
> 
> """
> Characters with the property value Numeric_Type=de (Decimal) only occur in contiguous ranges of 10 characters, with ascending numeric values from 0 to 9 (Numeric_Value=0..9).
> """
> 
> This makes it very easy to check a numeric string does not contain a mix of digits from different scripts.

I'm not sure why you'd want to check for such ranges.

> I still believe that proper API should require explicit choice of language or locale before allowing digits other than 0-9 just as int() would not accept hexadecimal digits without explicit choice of base >= 16.  But this would be a subject of a feature request.

Since when do we require a locale or language to be specified when
using Unicode ?

The codecs, Unicode methods and other Unicode support features
happily work with all kinds of languages, mixed or not, without any
such specification.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10557>
_______________________________________