How should we check numeric types at runtime in 2020?
TL;DR Try to make sense of this table: https://gist.github.com/ramalho/9f67fd245f424939c73e5c3bb21fa949 CONTEXT PEP 484 rejects the numeric tower (number.Number, number.Complex, number.Real etc.). The typing module now offers number-related SupportsX protocols which are runtime checkable, so I assumed some of these protocols could replace the numeric tower in practice. This is now more important than before, given the widespread use of NumPy with its dozens of numeric types. QUESTIONS What is the current best practice for testing numeric types at runtime, if the numeric tower is problematic? What use cases prompted the inclusion of the number-related SupportsX protocols as runtime checked ABCs? ISSUES WITH PROTOCOLS I wish I could forget about the numeric tower and use the SuportsX protocols, but I don't understand some of the results I'm getting with these protocols: 1) SupportsFloat issubclass(complex, typing.SupportsFloat) returns True but float(1+2j) raises TypeError: can't convert complex to float (the complex class does implement __float__, but I get that TypeError) 2) SupportsInt Same issue above: issubclass(complex, typing.SupportsInt) is True but int(1+2j) raises TypeError. In addition, issubclass(fractions.Fraction, typing.SupportsInt) returns False, but int(Fraction(7, 2) works, returns 3. 3) SupportsComplex Is issubclass(NT, typing.SupportsInt) is true ONLY for NT in [numpy.complex64, Decimal, and Fraction] but in fact all the numeric types from the stdlib and NumPy that I tried can be passed to complex() with no errors (as the first argument). THE TABLE AND SCRIPT TO BUILD IT I wrote a little script to create a table that shows these issues. See the table and script here if you are interested: https://gist.github.com/ramalho/9f67fd245f424939c73e5c3bb21fa949 The columns are concrete numeric types from the Python stdlib and NumPy. The rows represent three kinds of tests: 1) issubclass results against numbers ABCs Example: issubclass(number.Real, numpy.float16) 2) issubclass results against typing protocols Example: issubclass(typing.SupportsFloat, numpy.float16) 3) application of a built-in to a value built from a concrete type, given argument 1 Example 1: complex(float(1)) # result: (1+0j) with ComplexWarning Example 2: float(complex(1)) # no result, TypeError: can't convert complex to float Example 3: round(numpy.complex64(1)) # result: (1+0j) Cheers, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
Could you please post a corrected version of the script? Also, IMO the right way to test at runtime is still the numeric tower. The SupportsXxx classes are a crutch for static type checkers and their operational definition is "does the object have a __xxx__ method". The best way to discover at runtime whether something's an integer is still the numeric tower. (With one big caveat: I don't know whether numpy supports this. Last I looked there was scant mention of the numeric tower in the numpy docs, but IIRC there is code in numpy that uses it.) On Tue, Jun 16, 2020 at 8:18 PM Luciano Ramalho <luciano@ramalho.org> wrote:
TL;DR
Try to make sense of this table: https://gist.github.com/ramalho/9f67fd245f424939c73e5c3bb21fa949
CONTEXT
PEP 484 rejects the numeric tower (number.Number, number.Complex, number.Real etc.).
The typing module now offers number-related SupportsX protocols which are runtime checkable, so I assumed some of these protocols could replace the numeric tower in practice.
This is now more important than before, given the widespread use of NumPy with its dozens of numeric types.
QUESTIONS
What is the current best practice for testing numeric types at runtime, if the numeric tower is problematic?
What use cases prompted the inclusion of the number-related SupportsX protocols as runtime checked ABCs?
ISSUES WITH PROTOCOLS
I wish I could forget about the numeric tower and use the SuportsX protocols, but I don't understand some of the results I'm getting with these protocols:
1) SupportsFloat issubclass(complex, typing.SupportsFloat) returns True but float(1+2j) raises TypeError: can't convert complex to float (the complex class does implement __float__, but I get that TypeError)
2) SupportsInt Same issue above: issubclass(complex, typing.SupportsInt) is True but int(1+2j) raises TypeError. In addition, issubclass(fractions.Fraction, typing.SupportsInt) returns False, but int(Fraction(7, 2) works, returns 3.
3) SupportsComplex Is issubclass(NT, typing.SupportsInt) is true ONLY for NT in [numpy.complex64, Decimal, and Fraction] but in fact all the numeric types from the stdlib and NumPy that I tried can be passed to complex() with no errors (as the first argument).
THE TABLE AND SCRIPT TO BUILD IT
I wrote a little script to create a table that shows these issues. See the table and script here if you are interested:
https://gist.github.com/ramalho/9f67fd245f424939c73e5c3bb21fa949
The columns are concrete numeric types from the Python stdlib and NumPy.
The rows represent three kinds of tests:
1) issubclass results against numbers ABCs Example: issubclass(number.Real, numpy.float16) 2) issubclass results against typing protocols Example: issubclass(typing.SupportsFloat, numpy.float16) 3) application of a built-in to a value built from a concrete type, given argument 1 Example 1: complex(float(1)) # result: (1+0j) with ComplexWarning Example 2: float(complex(1)) # no result, TypeError: can't convert complex to float Example 3: round(numpy.complex64(1)) # result: (1+0j)
Cheers,
Luciano
-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Wed, Jun 17, 2020 at 5:31 PM Guido van Rossum <guido@python.org> wrote:
Could you please post a corrected version of the script?
Also, IMO the right way to test at runtime is still the numeric tower. The SupportsXxx classes are a crutch for static type checkers and their operational definition is "does the object have a __xxx__ method". The best way to discover at runtime whether something's an integer is still the numeric tower. (With one big caveat: I don't know whether numpy supports this. Last I looked there was scant mention of the numeric tower in the numpy docs, but IIRC there is code in numpy that uses it.)
NumPy's scalar types (e.g., float32 and int64) support the numeric tower. NumPy's array types don't, because they aren't substitutable for single numbers. But you can check dtypes with NumPy's own type hierarchy: https://numpy.org/devdocs/reference/arrays.scalars.html
On Tue, Jun 16, 2020 at 8:18 PM Luciano Ramalho <luciano@ramalho.org> wrote:
TL;DR
Try to make sense of this table: https://gist.github.com/ramalho/9f67fd245f424939c73e5c3bb21fa949
CONTEXT
PEP 484 rejects the numeric tower (number.Number, number.Complex, number.Real etc.).
The typing module now offers number-related SupportsX protocols which are runtime checkable, so I assumed some of these protocols could replace the numeric tower in practice.
This is now more important than before, given the widespread use of NumPy with its dozens of numeric types.
QUESTIONS
What is the current best practice for testing numeric types at runtime, if the numeric tower is problematic?
What use cases prompted the inclusion of the number-related SupportsX protocols as runtime checked ABCs?
ISSUES WITH PROTOCOLS
I wish I could forget about the numeric tower and use the SuportsX protocols, but I don't understand some of the results I'm getting with these protocols:
1) SupportsFloat issubclass(complex, typing.SupportsFloat) returns True but float(1+2j) raises TypeError: can't convert complex to float (the complex class does implement __float__, but I get that TypeError)
2) SupportsInt Same issue above: issubclass(complex, typing.SupportsInt) is True but int(1+2j) raises TypeError. In addition, issubclass(fractions.Fraction, typing.SupportsInt) returns False, but int(Fraction(7, 2) works, returns 3.
3) SupportsComplex Is issubclass(NT, typing.SupportsInt) is true ONLY for NT in [numpy.complex64, Decimal, and Fraction] but in fact all the numeric types from the stdlib and NumPy that I tried can be passed to complex() with no errors (as the first argument).
THE TABLE AND SCRIPT TO BUILD IT
I wrote a little script to create a table that shows these issues. See the table and script here if you are interested:
https://gist.github.com/ramalho/9f67fd245f424939c73e5c3bb21fa949
The columns are concrete numeric types from the Python stdlib and NumPy.
The rows represent three kinds of tests:
1) issubclass results against numbers ABCs Example: issubclass(number.Real, numpy.float16) 2) issubclass results against typing protocols Example: issubclass(typing.SupportsFloat, numpy.float16) 3) application of a built-in to a value built from a concrete type, given argument 1 Example 1: complex(float(1)) # result: (1+0j) with ComplexWarning Example 2: float(complex(1)) # no result, TypeError: can't convert complex to float Example 3: round(numpy.complex64(1)) # result: (1+0j)
Cheers,
Luciano
-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: shoyer@gmail.com
On Wed, Jun 17, 2020 at 6:33 PM Stephan Hoyer <shoyer@gmail.com> wrote:
On Wed, Jun 17, 2020 at 5:31 PM Guido van Rossum <guido@python.org> wrote:
Could you please post a corrected version of the script?
Also, IMO the right way to test at runtime is still the numeric tower. The SupportsXxx classes are a crutch for static type checkers and their operational definition is "does the object have a __xxx__ method". The best way to discover at runtime whether something's an integer is still the numeric tower. (With one big caveat: I don't know whether numpy supports this. Last I looked there was scant mention of the numeric tower in the numpy docs, but IIRC there is code in numpy that uses it.)
NumPy's scalar types (e.g., float32 and int64) support the numeric tower.
Thanks for confirming my old memory.
NumPy's array types don't, because they aren't substitutable for single numbers. But you can check dtypes with NumPy's own type hierarchy: https://numpy.org/devdocs/reference/arrays.scalars.html
Sure, because they aren't numbers! IIRC there's an edge case where numpy treats an array containing a single number as a number? Well that's probably fine. A bigger issue (for some) is that Decimal does not respect the numeric tower. But then, Decimal doesn't respect anything besides the IEEE standard. :-/ -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Wed, Jun 17, 2020 at 9:31 PM Guido van Rossum <guido@python.org> wrote:
Could you please post a corrected version of the script?
The script and table are correct and were correct when I posted them. Only parts of my description in my first e-mail had some errors when I wrongly reproduced some calls to issubclass(). I can't fix a message already sent, so I will write a blog post describing the problems more accurately. I apologize for the confusion.
Also, IMO the right way to test at runtime is still the numeric tower. The SupportsXxx classes are a crutch for static type checkers and their operational definition is "does the object have a __xxx__ method".
I understand that. The problem is that they are misleading, as the table shows. Here an attempt to use typing.SupportsFloat:
c = 1+0j type(c) <class 'complex'> isinstance(c, typing.SupportsFloat) True float(c) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: can't convert complex to float sys.version_info sys.version_info(major=3, minor=8, micro=0, releaselevel='final', serial=0)
The opposite problem also occurs—a protocol says "no" but reality says "yes":
x = 1.1 isinstance(x, typing.SupportsComplex) False complex(x) (1.1+0j)
Thanks for your attention. Best, Luciano
The best way to discover at runtime whether something's an integer is still the numeric tower. (With one big caveat: I don't know whether numpy supports this. Last I looked there was scant mention of the numeric tower in the numpy docs, but IIRC there is code in numpy that uses it.)
On Tue, Jun 16, 2020 at 8:18 PM Luciano Ramalho <luciano@ramalho.org> wrote:
TL;DR
Try to make sense of this table: https://gist.github.com/ramalho/9f67fd245f424939c73e5c3bb21fa949
CONTEXT
PEP 484 rejects the numeric tower (number.Number, number.Complex, number.Real etc.).
The typing module now offers number-related SupportsX protocols which are runtime checkable, so I assumed some of these protocols could replace the numeric tower in practice.
This is now more important than before, given the widespread use of NumPy with its dozens of numeric types.
QUESTIONS
What is the current best practice for testing numeric types at runtime, if the numeric tower is problematic?
What use cases prompted the inclusion of the number-related SupportsX protocols as runtime checked ABCs?
ISSUES WITH PROTOCOLS
I wish I could forget about the numeric tower and use the SuportsX protocols, but I don't understand some of the results I'm getting with these protocols:
1) SupportsFloat issubclass(complex, typing.SupportsFloat) returns True but float(1+2j) raises TypeError: can't convert complex to float (the complex class does implement __float__, but I get that TypeError)
2) SupportsInt Same issue above: issubclass(complex, typing.SupportsInt) is True but int(1+2j) raises TypeError. In addition, issubclass(fractions.Fraction, typing.SupportsInt) returns False, but int(Fraction(7, 2) works, returns 3.
3) SupportsComplex Is issubclass(NT, typing.SupportsInt) is true ONLY for NT in [numpy.complex64, Decimal, and Fraction] but in fact all the numeric types from the stdlib and NumPy that I tried can be passed to complex() with no errors (as the first argument).
THE TABLE AND SCRIPT TO BUILD IT
I wrote a little script to create a table that shows these issues. See the table and script here if you are interested:
https://gist.github.com/ramalho/9f67fd245f424939c73e5c3bb21fa949
The columns are concrete numeric types from the Python stdlib and NumPy.
The rows represent three kinds of tests:
1) issubclass results against numbers ABCs Example: issubclass(number.Real, numpy.float16) 2) issubclass results against typing protocols Example: issubclass(typing.SupportsFloat, numpy.float16) 3) application of a built-in to a value built from a concrete type, given argument 1 Example 1: complex(float(1)) # result: (1+0j) with ComplexWarning Example 2: float(complex(1)) # no result, TypeError: can't convert complex to float Example 3: round(numpy.complex64(1)) # result: (1+0j)
Cheers,
Luciano
-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) Pronouns: he/him (why is my pronoun here?)
-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
complex doesn't actually implement SupportsFloat or SupportsInt though, at least conceptually. SupportsFloat is essentially: ``` class SupportsFloat(Protocol): def __float__(self) -> float: ... ``` Meanwhile, complex is: ``` class complex: def __float__(self) -> NoReturn: ... ``` (complex.__float__ should be annotated with NoReturn <https://www.python.org/dev/peps/pep-0484/#the-noreturn-type> because it unconditionally raises an exception <https://github.com/python/cpython/blob/04fc4f2a46b2fd083639deb872c3a3037fdb47d6/Objects/complexobject.c#L679-L685> .) "A concrete type X is a subtype of protocol P if and only if X implements all protocol members of P with compatible types <https://www.python.org/dev/peps/pep-0544/#subtyping-relationships-with-other-types>." Since NoReturn is not compatible with float, complex isn't a subtype (doesn't implement) SupportsFloat. In fact, complex unsafely overlaps <https://www.python.org/dev/peps/pep-0544/#runtime-checkable-decorator-and-narrowing-types-by-isinstance> SupportsFloat, and `isinstance(1j, SupportsFloat)` should be rejected by type checkers per PEP 544. All of this holds true for SupportsInt as well, because complex.__int__ also always raises an exception. -- Teddy On Thu, Jun 18, 2020 at 8:45 AM Luciano Ramalho <luciano@ramalho.org> wrote:
On Wed, Jun 17, 2020 at 9:31 PM Guido van Rossum <guido@python.org> wrote:
Could you please post a corrected version of the script?
The script and table are correct and were correct when I posted them.
Only parts of my description in my first e-mail had some errors when I wrongly reproduced some calls to issubclass().
I can't fix a message already sent, so I will write a blog post describing the problems more accurately. I apologize for the confusion.
Also, IMO the right way to test at runtime is still the numeric tower. The SupportsXxx classes are a crutch for static type checkers and their operational definition is "does the object have a __xxx__ method".
I understand that. The problem is that they are misleading, as the table shows.
Here an attempt to use typing.SupportsFloat:
c = 1+0j type(c) <class 'complex'> isinstance(c, typing.SupportsFloat) True float(c) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: can't convert complex to float sys.version_info sys.version_info(major=3, minor=8, micro=0, releaselevel='final', serial=0)
The opposite problem also occurs—a protocol says "no" but reality says "yes":
x = 1.1 isinstance(x, typing.SupportsComplex) False complex(x) (1.1+0j)
Thanks for your attention.
Best,
Luciano
The best way to discover at runtime whether something's an integer is still the numeric tower. (With one big caveat: I don't know whether numpy supports this. Last I looked there was scant mention of the numeric tower in the numpy docs, but IIRC there is code in numpy that uses it.)
On Tue, Jun 16, 2020 at 8:18 PM Luciano Ramalho <luciano@ramalho.org> wrote:
TL;DR
Try to make sense of this table: https://gist.github.com/ramalho/9f67fd245f424939c73e5c3bb21fa949
CONTEXT
PEP 484 rejects the numeric tower (number.Number, number.Complex, number.Real etc.).
The typing module now offers number-related SupportsX protocols which are runtime checkable, so I assumed some of these protocols could replace the numeric tower in practice.
This is now more important than before, given the widespread use of NumPy with its dozens of numeric types.
QUESTIONS
What is the current best practice for testing numeric types at runtime, if the numeric tower is problematic?
What use cases prompted the inclusion of the number-related SupportsX protocols as runtime checked ABCs?
ISSUES WITH PROTOCOLS
I wish I could forget about the numeric tower and use the SuportsX protocols, but I don't understand some of the results I'm getting with these protocols:
1) SupportsFloat issubclass(complex, typing.SupportsFloat) returns True but float(1+2j) raises TypeError: can't convert complex to float (the complex class does implement __float__, but I get that TypeError)
2) SupportsInt Same issue above: issubclass(complex, typing.SupportsInt) is True but int(1+2j) raises TypeError. In addition, issubclass(fractions.Fraction, typing.SupportsInt) returns False, but int(Fraction(7, 2) works, returns 3.
3) SupportsComplex Is issubclass(NT, typing.SupportsInt) is true ONLY for NT in [numpy.complex64, Decimal, and Fraction] but in fact all the numeric types from the stdlib and NumPy that I tried can be passed to complex() with no errors (as the first argument).
THE TABLE AND SCRIPT TO BUILD IT
I wrote a little script to create a table that shows these issues. See the table and script here if you are interested:
https://gist.github.com/ramalho/9f67fd245f424939c73e5c3bb21fa949
The columns are concrete numeric types from the Python stdlib and NumPy.
The rows represent three kinds of tests:
1) issubclass results against numbers ABCs Example: issubclass(number.Real, numpy.float16) 2) issubclass results against typing protocols Example: issubclass(typing.SupportsFloat, numpy.float16) 3) application of a built-in to a value built from a concrete type, given argument 1 Example 1: complex(float(1)) # result: (1+0j) with ComplexWarning Example 2: float(complex(1)) # no result, TypeError: can't convert complex to float Example 3: round(numpy.complex64(1)) # result: (1+0j)
Cheers,
Luciano
-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) Pronouns: he/him (why is my pronoun here?)
-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: tsudol@google.com
On Thu, Jun 18, 2020 at 8:45 AM Luciano Ramalho <luciano@ramalho.org> wrote:
On Wed, Jun 17, 2020 at 9:31 PM Guido van Rossum <guido@python.org> wrote:
Could you please post a corrected version of the script?
The script and table are correct and were correct when I posted them.
Only parts of my description in my first e-mail had some errors when I wrongly reproduced some calls to issubclass().
I can't fix a message already sent, so I will write a blog post describing the problems more accurately. I apologize for the confusion.
Ah, sorry. Somehow Alex Martelli's post started a new thread in GMail and I didn't follow exactly what was going on; then I misread your reply as indicating that the script and table were wrong. My apologies!
Also, IMO the right way to test at runtime is still the numeric tower. The SupportsXxx classes are a crutch for static type checkers and their operational definition is "does the object have a __xxx__ method".
I understand that. The problem is that they are misleading, as the table shows.
Here an attempt to use typing.SupportsFloat:
c = 1+0j type(c) <class 'complex'> isinstance(c, typing.SupportsFloat) True float(c) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: can't convert complex to float sys.version_info sys.version_info(major=3, minor=8, micro=0, releaselevel='final', serial=0)
The opposite problem also occurs—a protocol says "no" but reality says "yes":
x = 1.1 isinstance(x, typing.SupportsComplex) False complex(x) (1.1+0j)
Thanks for your attention.
And thanks for the explanation -- your complaint is now much clearer. At *runtime* SupportsComplex and SupportsFloat and the others are predicated on "does it have a __complex__ / __float__ / etc. attribute." And float has no __complex__ method, but complex has a __float__ method, as well as an __int__ attribute -- both appear to be intended to give better error messages. However the runtime introspection implemented by typing's @runtime_checkable cannot introspect the true nature of these methods -- it just checks for their presence and returns True or False based on that. That's why SupportsFloat is a bad way to check for whether something "is" a float or even floatable. (Note that some strings are also floatable -- but SupportsFloat doesn't know.) As for why complex(x) works even though there's no x.__float__, that's because the complex() constructor accepts one or two floats as arguments and treats them as the real and imaginary parts. So I stick to my observation that the numeric tower is still the best way to check for a specific type of number, since all builtin types are registered as member of the appropriate level in the tower. What a type checker does is potentially different -- and Teddy Sudol's observation is correct that typeshed should be adjusted to set the return type of complex.__float__ and complex.__int__ to NoReturn. Somebody should probably send a PR their way. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
participants (4)
-
Guido van Rossum -
Luciano Ramalho -
Stephan Hoyer -
Teddy Sudol