[issue25275] Documentation v/s behaviour mismatch wrt integer literals containing non-ASCII characters
New submission from Shreevatsa R: Summary: This is about int(u'१२३४') == 1234. At https://docs.python.org/2/library/functions.html and also https://docs.python.org/3/library/functions.html the documentation for class int(x=0) class int(x, base=10) says (respectively):
If x is not a number or if base is given, then x must be a string or Unicode object representing an integer literal in radix base.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in radix base.
If you follow the definition of "integer literal" into the reference (https://docs.python.org/2/reference/lexical_analysis.html#integers and https://docs.python.org/3/reference/lexical_analysis.html#integers respectively), the definitions ultimately involve nonzerodigit ::= "1"..."9" octdigit ::= "0"..."7" bindigit ::= "0" | "1" digit ::= "0"..."9" So it looks like whether the behaviour of int() conforms to its documentation hinges on what "representing" means. Apparently it is some definition under which u'१२३४' represents the integer literal 1234, but it would be great to either clarify the documentation of int() or change its behaviour. ---------- assignee: docs@python components: Documentation, Interpreter Core, Unicode messages: 251915 nosy: docs@python, ezio.melotti, haypo, shreevatsa priority: normal severity: normal status: open title: Documentation v/s behaviour mismatch wrt integer literals containing non-ASCII characters _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue25275> _______________________________________
R. David Murray added the comment: Apparently that documentation is simply wrong. The actual definition of what 'int' handles is *different* from what the parser handles. I think that difference must constitute a bug (not just a doc bug), but I'm not sure if it is something that we want to fix (changing the parser). I think the *operational* definition of int conversion for both is the same as for isdigit in python3 (https://docs.python.org/3/library/stdtypes.html#str.isdigit). (The python2 docs just say '8 bit strings may be locale dependent', which means the same thing but is less precise).
१२३४ File "<stdin>", line 1 १२३४ ^ SyntaxError: invalid character in identifier int('१२३४') 1234 '१२३४'.isdigit() True
The above behavior discrepancy doesn't apply to python2, since in python2 you can't use unicode in integer literals. So, this is a bit of a mess :(. The doc fix is simple: just replace the mention of integer literal with a link to isdigit, and fix the python2 isdigit docs to match python3's. ---------- nosy: +r.david.murray stage: -> needs patch type: -> behavior versions: +Python 2.7, Python 3.4, Python 3.5, Python 3.6 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue25275> _______________________________________
R. David Murray added the comment: I mean, in python2 you can use unicode in python code, only in strings, as opposed to python3 where unicode is valid in identifiers (but not integer literals, obviously). ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue25275> _______________________________________
R. David Murray added the comment: I mean, in python2 you can't use unicode in python code, only in strings, as opposed to python3 where unicode is valid in identifiers (but not integer literals, obviously). ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue25275> _______________________________________
Changes by R. David Murray <rdmurray@bitdance.com>: ---------- Removed message: http://bugs.python.org/msg251931 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue25275> _______________________________________
Shreevatsa R added the comment: Minor difference, but the relevant function for int() is not quite isdigit(), e.g.: >>> import unicodedata >>> s = u'\u2460' >>> unicodedata.name(s) 'CIRCLED DIGIT ONE' >>> print s ① >>> s.isdigit() True >>> s.isdecimal() False >>> int(s) Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'decimal' codec can't encode character u'\u2460' in position 0: invalid decimal Unicode string It seems to be isdecimal(), plus if there are other digits in the string then many leading and trailing space-like characters are also allowed (e.g. 5760 OGHAM SPACE MARK or 8195 EM SPACE or 12288 IDEOGRAPHIC SPACE: >>> 987 == int(u'\u3000\n 987\u1680\t') True ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue25275> _______________________________________
Shreevatsa R added the comment: About the mismatch: of course it's probably not a good idea to change the parser (so that simply typing १२३४ in Python 3 code is like typing 1234), but how about changing the behaviour of int()? Not sure whether anyone should be relying on int(u'१२३४') being 1234, given that it is not documented as such. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue25275> _______________________________________
R. David Murray added the comment: Good catch. Yes, it is already documented that Int ignores leading and trailing whitespace. But, even that isn't quite correct:
'A'.isdecimal() False int('A', 16) 10
I seem to vaguely recall a discussion somewhere in this tracker about what "should" count as digits for larger-than-decimal radii, but I don't remember the outcome. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue25275> _______________________________________
R. David Murray added the comment: No, we can't make it stop working for int, that would be a backward compatibility break. Doing so was discussed at one point and rejected (another issue somewhere in this tracker :) ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue25275> _______________________________________
Martin Panter added the comment: Related discussion and background in Issue 10581, although that report seems to be geared at extending the Unicode support even further (disallowing mixed scripts, allowing proper minus signs, full-width characters, Roman numerals, etc). The existing support is actually documented if you know where to look; see the sixth note under the table at <https://docs.python.org/dev/library/stdtypes.html#typesnumeric>. I agree that the references for each constructor should also document this as well. ---------- nosy: +martin.panter _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue25275> _______________________________________
participants (3)
-
Martin Panter -
R. David Murray
-
Shreevatsa R