Add additional special method lookups to math module
Currently the functions round(), math.ceil(), math.floor(), and math.trunc() all check for the existence of a special method (__round__, __ceil__, __floor__, and __trunc__). Would it be possible to enhance the math and cmath modules to check for the existence of a special method for (almost) functions? For example, math.sin(obj) would first check for obj.__sin__. Rationale I'm in the final stages of adding support for the MPFR (multiple-precision floating point) and MPC (multiple-precision complex) libraries to the next-generation of gmpy, currently known as gmpy2. If the special method checks are added to the math and cmath modules, then the new mpfr() and mpc() types can easily substitute for the existing float/complex types in code that uses the math or cmath module. Thoughts?
I like the idea, but there is a question of namespace size, particularly if there may be other meanings for some of the names. I would therefore prefer __math_sin__, __math_arctan__, etc. (Does it need to be even more specific than math?) -jJ On 11/9/11, Case Van Horsen <casevh@gmail.com> wrote:
On Wed, Nov 9, 2011 at 11:43 AM, Jim Jewett <jimjjewett@gmail.com> wrote:
Good idea. Creating a naming convention for functions that return real or complex values would be needed. math.sin(obj) would look for obj.__math_sin__. cmath.sin(obj) would look for obj.__cmath_sin__. gmpy2 supports integer, rational, real, and complex numbers. The integer, rational, and real objects would need both the __math_XXX__ and __cmath_XXX__ methods. The complex object would only support the __cmath_XXX__ methods. There is ambiguity with functions that take more than one argument: atan2, copysign, fmod, hypot, and pow. If only one of the arguments supports the special method, then that special method should be used. If both of the arguments supports the special method AND both arguments are of the same type, then the special method should be used. But what if both arguments are different types? I think it should try the special method associated with the first argument and if that return NotImplemented, then it should try the special method associated with the second argument. If the special methods ever return NotImplemented, then the normal math/cmath handling should make a final attempt. casevh
Case Van Horsen wrote:
Wouldn't duct typing the functions in math and cmath be equivalent and wouldn't slow down normal users? Howver, I'm not convinced that it's very nice changing the behaviour of built-in modules depending on the presence another module anyway, I'd just supply another module with the same api - people could always mess around with the import path if they really want to override the default. import math def mysin(x): xxxx math.sin = mysin Jeremy
On 2011-11-09, at 21:06 , Jeremy Sanders wrote:
This would mean custom numerical types wouldn't be drop-in compatible with existing numerical *code*, which I am under the impression is what Case Van Horsen wants (and is a desirable outcome). Furthermore, your second paragraph is not correct: Case does not propose "changing the behavior of built-in modules depending on the presence of another module", Case proposes adding *method hooks* to existing math and cmath functions. These would be protocols allowing custom numerical types to implement `math`/`cmath`'s operation in a sensible manner, as is already possible for four methods he mentions, as well as a number of other Python operations[0], including prominent numerical ones[1]. [0] http://docs.python.org/reference/datamodel.html [1] http://docs.python.org/reference/datamodel.html#emulating-numeric-types
Masklinn wrote:
I think it looks too much like hidden magic to me. Explicit is better than implicit. If I were doing this, I wouldn't change the current behaviour by default. I'd include a function which would change the built-in functions, if required. Jeremy
Masklinn wrote:
That makes no sense.
I think it does. I could write code relying on the math.XXX failing for types other than the built in numeric types, or for particular exceptions for particular calls. Also, if you override a standard module as a quick way to get existing code working, you can get lots of existing code paths which might fail in interesting ways, especially if there weren't unit tests for these cases. It would make more sense if the possibility was documented in the math documentation. If you go and change the behaviour of built-in functions just by importing a module it could lead to some weird bugs. I know monkey patching can do this currently, but doing it by default seems dangerous to me. Jeremy
On 11/9/11, Masklinn <masklinn@masklinn.net> wrote:
On 2011-11-09, at 22:10 , Jeremy Sanders wrote:
Masklinn wrote:
Several people have suggested that the standard library module not be changed, but that users be encouraged to monkeypatch instead. In other words, if I import funkymathtype, it should (as part of the import process) monkeypatch math (and cmath) to replace the builtin functions with something that handles its own types as well. Short of generic functions, I happen to think that is a bad idea, but it was suggested. And frankly, "import funkymathtype.math as math" probably is the right answer, in the short run, but that is still pretty fragile against other modules having imported math themselves. -jJ -jJ
On 10/11/11 15:17, Jim Jewett wrote:
The monkeypatching idea doesn't entirely avoid that problem either, since you would have to make sure the module doing the monkeypatching was imported before any other module imported stuff from math. -- Greg
On 9 November 2011 20:15, Masklinn <masklinn@masklinn.net> wrote:
I like the idea of being able to extend the math functions to allow user-defined numeric types to behave like built in ones. (In fact, if this were possible, complex could use it and a large proportion of the cmath module would be unneeded). However, I have to say that for me this is a purely theoretical issue - I've never had the need for the functionality myself in real code, and I doubt I ever will. Given the proliferation of special methods that would be needed, plus the runtime overhead of extra checking, I think the cost is likely too high. It's also worth noting the preamble text in the math module documentation - the functions there deliberately only handle floats, and are thin wrappers over the C standard library. To get any sort of generic behaviour, users need to use cmath, by design. So as stated the proposal is unlikely to get much support. Some alternative suggestions: 1. As already noted, you could include a gmpmath module with the relevant functions, and users could monkeypatch the math module if they wanted to do so. 2. To make such monkeypatching easier, publish a module that exposes a context manager to do the job: with monkeypatch(math, 'sin', gmpmath.sin): your code here... 3. Write a genericmath module that provides the "generic" versions you're proposing, and see how much use it gets - if it's sufficiently popular, you have a better case then for folding the functionality into at least cmath. 4. As Georg mentioned, numpy's ufuncs have a feature like this, so support that and people can use your types with numpy. That may be enough (depending on your expected user base). Overall, I think the idea of being able to use user-defined types interchangeably with built-in ones is a good one, but it's not something Python goes out of its way to support. If the idea of generic functions had taken off, then this is an area where they would have fit nicely - but once again, there wasn't much enthusiasm from the core developers for addressing the types of problem they were aimed at. Paul.
The big problem with generics in the past has been clearly articulating use cases. A genericmath and updated pprint could help a lot on that front. -- Nick Coghlan (via Gmail on Android, so likely to be more terse than usual) On Nov 10, 2011 6:52 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:
Jeremy Sanders wrote:
Wouldn't duct typing the functions in math and cmath be equivalent and wouldn't slow down normal users?
s/duct typing/monkey patching/ - I think I was thinking of animals and duck came into my mind rather than monkey. Jeremy
On 9 November 2011 06:17, Case Van Horsen <casevh@gmail.com> wrote:
I have been faced with a very similar situation recently. I am adding Python scripting capability to a Dynamic Geometry application called GeoGebra (www.geogebra.org). GeoGebra has its own numeric types. I have wrapped them in Python classes so that all arithmetic operations work correctly on them but it would be a big improvement if the standard analytic functions in the math module could work on them as well. So this would be a welcome addition for me (although, as GeoGebra is a Java application, I am using Jython, so I would have to wait a while to see this coming my way :). -- Arnaud
Case Van Horsen wrote:
I would not object to this. The only function I can honestly say I have had a concrete use-case for is math.sqrt. This comes up from time to time, e.g.: http://bytes.com/topic/python/answers/463861-how-overload-sqrt-module http://permalink.gmane.org/gmane.comp.python.general/694849 However, how far should we go? Does every function in the math module require a dunder method, e.g. __degrees__ ? What happens if we add more functions, say math.bessel? Do we really expect that all numeric types must support a __bessel__ method? I suspect that this proposal is actually bigger than it seems at first glance. We can: * Do nothing (the status quo). If you write a numeric type, you can support a small number of mathematical operations, such as + and math.floor, but not others, such as math.sqrt or math.sin. * Officially recommend that people monkey-patch the math module if they want to write a drop-in replacement for numeric types. I consider this unspeakable, but mention it for completeness since others have raised the possibility. * Add support for dunder methods in an ad hoc manner, when and as requested, without making any promises about any other functions. * Add support for dunder methods in a systematic way. This would require distinguishing between fundamental operations that should support dunder methods, and those that shouldn't (if any). This will probably need a PEP. * Instead of having to decide what operations should be supported ahead of time, perhaps there is a way for types to register themselves with the math module, e.g. say "I support sin, but not sinh". Somewhat akin to the way ABCs work, at least conceptually. One advantage of that may be that numeric classes won't have to use dunder methods for supporting the math module, e.g. MyNumber might register sin rather than __sin__. -- Steven
On 10/11/11 12:02, Steven D'Aprano wrote:
Another approach to all this would be to provide generic implementations of the math functions in terms of the basic arithmetic operations. While this wouldn't always be as fast as using type-specific methods, it would at least work with all number-like types and would extend easily to any new functions we might add without having to update every existing numeric type. It might even be possible to support type-specific accelerations of these generic implementations using a __polynomial__ method that takes a list of coefficients, or maybe a function for generating the coefficients. -- Greg
On Wed, Nov 9, 2011 at 3:02 PM, Steven D'Aprano <steve@pearwood.info> wrote:
It would be completely optional for a numeric type to support these methods. If they're not supported, the numeric type is converted to a float and then math.function proceeds as it currently does.
I hacked mathmodule.c FUNC1 macro to perform the lookup the many of math module functions. It was only about 15 lines of code (but it doesn't check if NotImplemented is returned.) Unfortunately, it increases the running time of math.sin, for example, by 15%. I need to look at the Identifier API to see if that helps the performance but I don't think 15% is a penalty everyone should pay.
Especially true if new nb_slots need to be created to avoid the performance impact. I was hoping the performance impact of the special method lookup was negligible so it could be a low impact change.
As long as they support __float__, they'll continue to work with the math module.
On 9 November 2011 23:02, Steven D'Aprano <steve@pearwood.info> wrote:
I haven't checked the details, but isn't this *exactly* how ABCs work? So math could first check for exact floats (for performance), then check for whether the object is an instance of the "TranscendentalMaths" ABC, and finally fall back to converting to float. All of this is ignoring the question of whether it's acceptable to change the documented contract of math to only work on actual floats, of course... Paul.
I like the idea, but there is a question of namespace size, particularly if there may be other meanings for some of the names. I would therefore prefer __math_sin__, __math_arctan__, etc. (Does it need to be even more specific than math?) -jJ On 11/9/11, Case Van Horsen <casevh@gmail.com> wrote:
On Wed, Nov 9, 2011 at 11:43 AM, Jim Jewett <jimjjewett@gmail.com> wrote:
Good idea. Creating a naming convention for functions that return real or complex values would be needed. math.sin(obj) would look for obj.__math_sin__. cmath.sin(obj) would look for obj.__cmath_sin__. gmpy2 supports integer, rational, real, and complex numbers. The integer, rational, and real objects would need both the __math_XXX__ and __cmath_XXX__ methods. The complex object would only support the __cmath_XXX__ methods. There is ambiguity with functions that take more than one argument: atan2, copysign, fmod, hypot, and pow. If only one of the arguments supports the special method, then that special method should be used. If both of the arguments supports the special method AND both arguments are of the same type, then the special method should be used. But what if both arguments are different types? I think it should try the special method associated with the first argument and if that return NotImplemented, then it should try the special method associated with the second argument. If the special methods ever return NotImplemented, then the normal math/cmath handling should make a final attempt. casevh
Case Van Horsen wrote:
Wouldn't duct typing the functions in math and cmath be equivalent and wouldn't slow down normal users? Howver, I'm not convinced that it's very nice changing the behaviour of built-in modules depending on the presence another module anyway, I'd just supply another module with the same api - people could always mess around with the import path if they really want to override the default. import math def mysin(x): xxxx math.sin = mysin Jeremy
On 2011-11-09, at 21:06 , Jeremy Sanders wrote:
This would mean custom numerical types wouldn't be drop-in compatible with existing numerical *code*, which I am under the impression is what Case Van Horsen wants (and is a desirable outcome). Furthermore, your second paragraph is not correct: Case does not propose "changing the behavior of built-in modules depending on the presence of another module", Case proposes adding *method hooks* to existing math and cmath functions. These would be protocols allowing custom numerical types to implement `math`/`cmath`'s operation in a sensible manner, as is already possible for four methods he mentions, as well as a number of other Python operations[0], including prominent numerical ones[1]. [0] http://docs.python.org/reference/datamodel.html [1] http://docs.python.org/reference/datamodel.html#emulating-numeric-types
Masklinn wrote:
I think it looks too much like hidden magic to me. Explicit is better than implicit. If I were doing this, I wouldn't change the current behaviour by default. I'd include a function which would change the built-in functions, if required. Jeremy
Masklinn wrote:
That makes no sense.
I think it does. I could write code relying on the math.XXX failing for types other than the built in numeric types, or for particular exceptions for particular calls. Also, if you override a standard module as a quick way to get existing code working, you can get lots of existing code paths which might fail in interesting ways, especially if there weren't unit tests for these cases. It would make more sense if the possibility was documented in the math documentation. If you go and change the behaviour of built-in functions just by importing a module it could lead to some weird bugs. I know monkey patching can do this currently, but doing it by default seems dangerous to me. Jeremy
On 11/9/11, Masklinn <masklinn@masklinn.net> wrote:
On 2011-11-09, at 22:10 , Jeremy Sanders wrote:
Masklinn wrote:
Several people have suggested that the standard library module not be changed, but that users be encouraged to monkeypatch instead. In other words, if I import funkymathtype, it should (as part of the import process) monkeypatch math (and cmath) to replace the builtin functions with something that handles its own types as well. Short of generic functions, I happen to think that is a bad idea, but it was suggested. And frankly, "import funkymathtype.math as math" probably is the right answer, in the short run, but that is still pretty fragile against other modules having imported math themselves. -jJ -jJ
On 10/11/11 15:17, Jim Jewett wrote:
The monkeypatching idea doesn't entirely avoid that problem either, since you would have to make sure the module doing the monkeypatching was imported before any other module imported stuff from math. -- Greg
On 9 November 2011 20:15, Masklinn <masklinn@masklinn.net> wrote:
I like the idea of being able to extend the math functions to allow user-defined numeric types to behave like built in ones. (In fact, if this were possible, complex could use it and a large proportion of the cmath module would be unneeded). However, I have to say that for me this is a purely theoretical issue - I've never had the need for the functionality myself in real code, and I doubt I ever will. Given the proliferation of special methods that would be needed, plus the runtime overhead of extra checking, I think the cost is likely too high. It's also worth noting the preamble text in the math module documentation - the functions there deliberately only handle floats, and are thin wrappers over the C standard library. To get any sort of generic behaviour, users need to use cmath, by design. So as stated the proposal is unlikely to get much support. Some alternative suggestions: 1. As already noted, you could include a gmpmath module with the relevant functions, and users could monkeypatch the math module if they wanted to do so. 2. To make such monkeypatching easier, publish a module that exposes a context manager to do the job: with monkeypatch(math, 'sin', gmpmath.sin): your code here... 3. Write a genericmath module that provides the "generic" versions you're proposing, and see how much use it gets - if it's sufficiently popular, you have a better case then for folding the functionality into at least cmath. 4. As Georg mentioned, numpy's ufuncs have a feature like this, so support that and people can use your types with numpy. That may be enough (depending on your expected user base). Overall, I think the idea of being able to use user-defined types interchangeably with built-in ones is a good one, but it's not something Python goes out of its way to support. If the idea of generic functions had taken off, then this is an area where they would have fit nicely - but once again, there wasn't much enthusiasm from the core developers for addressing the types of problem they were aimed at. Paul.
The big problem with generics in the past has been clearly articulating use cases. A genericmath and updated pprint could help a lot on that front. -- Nick Coghlan (via Gmail on Android, so likely to be more terse than usual) On Nov 10, 2011 6:52 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:
Jeremy Sanders wrote:
Wouldn't duct typing the functions in math and cmath be equivalent and wouldn't slow down normal users?
s/duct typing/monkey patching/ - I think I was thinking of animals and duck came into my mind rather than monkey. Jeremy
On 9 November 2011 06:17, Case Van Horsen <casevh@gmail.com> wrote:
I have been faced with a very similar situation recently. I am adding Python scripting capability to a Dynamic Geometry application called GeoGebra (www.geogebra.org). GeoGebra has its own numeric types. I have wrapped them in Python classes so that all arithmetic operations work correctly on them but it would be a big improvement if the standard analytic functions in the math module could work on them as well. So this would be a welcome addition for me (although, as GeoGebra is a Java application, I am using Jython, so I would have to wait a while to see this coming my way :). -- Arnaud
Case Van Horsen wrote:
I would not object to this. The only function I can honestly say I have had a concrete use-case for is math.sqrt. This comes up from time to time, e.g.: http://bytes.com/topic/python/answers/463861-how-overload-sqrt-module http://permalink.gmane.org/gmane.comp.python.general/694849 However, how far should we go? Does every function in the math module require a dunder method, e.g. __degrees__ ? What happens if we add more functions, say math.bessel? Do we really expect that all numeric types must support a __bessel__ method? I suspect that this proposal is actually bigger than it seems at first glance. We can: * Do nothing (the status quo). If you write a numeric type, you can support a small number of mathematical operations, such as + and math.floor, but not others, such as math.sqrt or math.sin. * Officially recommend that people monkey-patch the math module if they want to write a drop-in replacement for numeric types. I consider this unspeakable, but mention it for completeness since others have raised the possibility. * Add support for dunder methods in an ad hoc manner, when and as requested, without making any promises about any other functions. * Add support for dunder methods in a systematic way. This would require distinguishing between fundamental operations that should support dunder methods, and those that shouldn't (if any). This will probably need a PEP. * Instead of having to decide what operations should be supported ahead of time, perhaps there is a way for types to register themselves with the math module, e.g. say "I support sin, but not sinh". Somewhat akin to the way ABCs work, at least conceptually. One advantage of that may be that numeric classes won't have to use dunder methods for supporting the math module, e.g. MyNumber might register sin rather than __sin__. -- Steven
On 10/11/11 12:02, Steven D'Aprano wrote:
Another approach to all this would be to provide generic implementations of the math functions in terms of the basic arithmetic operations. While this wouldn't always be as fast as using type-specific methods, it would at least work with all number-like types and would extend easily to any new functions we might add without having to update every existing numeric type. It might even be possible to support type-specific accelerations of these generic implementations using a __polynomial__ method that takes a list of coefficients, or maybe a function for generating the coefficients. -- Greg
On Wed, Nov 9, 2011 at 3:02 PM, Steven D'Aprano <steve@pearwood.info> wrote:
It would be completely optional for a numeric type to support these methods. If they're not supported, the numeric type is converted to a float and then math.function proceeds as it currently does.
I hacked mathmodule.c FUNC1 macro to perform the lookup the many of math module functions. It was only about 15 lines of code (but it doesn't check if NotImplemented is returned.) Unfortunately, it increases the running time of math.sin, for example, by 15%. I need to look at the Identifier API to see if that helps the performance but I don't think 15% is a penalty everyone should pay.
Especially true if new nb_slots need to be created to avoid the performance impact. I was hoping the performance impact of the special method lookup was negligible so it could be a low impact change.
As long as they support __float__, they'll continue to work with the math module.
On 9 November 2011 23:02, Steven D'Aprano <steve@pearwood.info> wrote:
I haven't checked the details, but isn't this *exactly* how ABCs work? So math could first check for exact floats (for performance), then check for whether the object is an instance of the "TranscendentalMaths" ABC, and finally fall back to converting to float. All of this is ignoring the question of whether it's acceptable to change the documented contract of math to only work on actual floats, of course... Paul.
participants (12)
-
Antoine Pitrou -
Arnaud Delobelle -
Case Van Horsen -
Georg Brandl -
Greg Ewing -
Jeremy Sanders -
Jim Jewett -
Masklinn -
Nick Coghlan -
Paul Moore -
Steven D'Aprano -
Terry Reedy