On Thu, Dec 2, 2010 at 4:57 PM, Mark Dickinson email@example.com wrote: ..
(the decimal spec requires that non-European digits be accepted).
I think *requires* is too strong of a word to describe what the spec says. The decimal module documentation refers to two authorities:
1. IBM’s General Decimal Arithmetic Specification 2. IEEE standard 854-1987
The IEEE standards predates Unicode and unsurprisingly does not have anything related to the issue. the IBM's spec says the following in the Conversions section:
""" It is recommended that implementations also provide additional number formatting routines (including some which are locale-dependent), and if available should accept non-European decimal digits in strings. """ http://speleotrove.com/decimal/daconvs.html
This cannot possibly be interpreted as normative text. The emphasis is clearly on "formatting routines" with "non-European decimal digits" added as an afterthought. This recommendation can reasonably be interpreted as a requirement that conversion routines should accept what formatting routines can produce. In Python there are no formatting routines to produce non-European numerals, so there is no requirement to accept them in conversions.
I don't think decimal module should support non-European decimal digits. The only place where it can make some sense is in int() because here we have a fighting chance of producing a reasonable definition. The motivating use case is conversion of numerical data extracted from text using simple '\d+' regex matches.
Here is how I would do it:
1. String x of non-European decimal digits is only accepted in int(x), but not by int(x, 0) or int(x, 10). 2. If x contains one or more non-European digits, then
(a) all digits must be from the same block:
def basepoint(c): return ord(c) - unicodedata.digit(c) all(basepoint(c) == basepoint(x) for c in x) -> True
(b) and '+' or '-' sign is not alowed.
3. A character c is a digit if it matches '\d' regex. I think this means unicodedata.category(c) -> 'Nd'.
Condition 2(b) is important because there is no clear way to define what is acceptable as '+' or '-' using Unicode character properties and not all number systems even support local form of negation. (It is also YAGNI.)