[ python-Bugs-1417699 ] float/atof have become locale aware

[SourceForge.net]

Bugs item #1417699, was opened at 2006-01-29 02:04
Message generated for change (Comment added) made by bernhard
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1417699&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.4
Status: Open
Resolution: None
Priority: 7
Submitted By: Bernhard Herzog (bernhard)
Assigned to: Nobody/Anonymous (nobody)
Summary: float/atof have become locale aware

Initial Comment:
The builtin float and the function string.atof have
become locale aware in Python 2.4:

Python 2.4.2 (#1, Nov 29 2005, 16:07:55) 
[GCC 2.95.4 20011002 (Debian prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for
more information.
>>> import locale
>>> import string
>>> locale.setlocale(locale.LC_ALL, "de_DE")
'de_DE'
>>> float("1,5")
1.5
>>> string.atof("1,5")
1.5


This doesn't match what's specified in pep 331:

        - float() and str() stay locale-unaware.

It also doesn't match the documentation for atof:

  Convert a string to a floating point number.
  The string must have the standard syntax for a
  floating point literal in Python, optionally
  preceded by a sign ("+" or "-"). Note that this
  behaves identical to the built-in function float()
  when passed a string.

The documentation for float() doesn't state which
format is accepted by float(), though.

The reason for this behavior is ultimately, that
PyOS_ascii_strtod accepts the locale specific
convention in addition to the "C"-locale/python
convention, even though the comment in the code
preceding its definition states:

  This function behaves like the standard strtod() 
  function does in the C locale.




----------------------------------------------------------------------

>Comment By: Bernhard Herzog (bernhard)
Date: 2006-06-10 21:18

Message:
Logged In: YES 
user_id=2369

gustavo, your patch seems to work fine.  However, it leads
to some test failures because the testsuite already tests
for the wrong behaviour.  I've uploaded a version of your
patch that in addtion to your changes contains modifications
to the test suite so that it tests for the new behavior and
also tests some more variations of floating point literals
accepted by float().

It would be nice if this patch could go into 2.5 before beta1

----------------------------------------------------------------------

Comment By: Gustavo J. A. M. Carneiro (gustavo)
Date: 2006-05-16 18:35

Message:
Logged In: YES 
user_id=908

It seems I can't upload files?!
I have put the patch to fix this problem, including unit
test, here: http://www.gnome.org/~gjc/python-1417699.diff

----------------------------------------------------------------------

Comment By: Gustavo J. A. M. Carneiro (gustavo)
Date: 2006-05-13 13:59

Message:
Logged In: YES 
user_id=908

I'm quite busy at this moment, but I'll send a patch soon,
promise.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2006-05-12 16:14

Message:
Logged In: YES 
user_id=21627

Unassigning myself - I don't plan to work on this anytime soon.

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2006-05-08 20:00

Message:
Logged In: YES 
user_id=849994

I grepped around and found PyOS_ascii_strtod in the
following places:

* stropmodule::atof
* cPickle::load_float (the writing of locale dependent
floats was fixed recently)
* floatobject::PyFloat_FromString
* complexobject::complex_subtype_from_string

In my opinion, all these should not tolerate localized
floats, so Gustavo, please ask the original author how to
achieve this.

----------------------------------------------------------------------

Comment By: Gustavo J. A. M. Carneiro (gustavo)
Date: 2006-05-04 11:47

Message:
Logged In: YES 
user_id=908

My comment is, is "PyOS_ascii_strtod accepting both forms"
something that is harmful and should be fixed?

I didn't exactly write the PyOS_ascii_strtod code; I only
integrated it with Python.  But if we really need to fix
this, I'll try to figure out how to make it not accept
floating points in locale aware format; if necessary I'll
ask help from  the original author of the code, Alexander
Larsson.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2006-05-04 06:42

Message:
Logged In: YES 
user_id=21627

It's a bug. As bernhard says, it originates from
PyOS_ascii_strtod accepting both forms, which in turn
happens because it just leaves the locale-specific decimal
point in the string, only replacing the . with the
locale-specific point. Then, the platform's strtod will
happily accept the locale-specific version. 

I'd like Gustavo Carneiro to comment.

----------------------------------------------------------------------

Comment By: Georg Brandl (birkenfeld)
Date: 2006-02-17 13:19

Message:
Logged In: YES 
user_id=1188172

Martin, you checked in the patch which is mentioned in PEP
331. Is this correct behavior?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1417699&group_id=5470