Re: [Numpy-discussion] Integers to negative integer powers, time for a decision.
On Sun, Oct 9, 2016 at 12:59 PM, Stephan Hoyer <shoyer@gmail.com> wrote:
I agree with Sebastian and Nathaniel. I don't think we can deviating from the existing behavior (int ** int -> int) without breaking lots of existing code, and if we did, yes, we would need a new integer power function.
I think it's better to preserve the existing behavior when it gives sensible results, and error when it doesn't. Adding another function float_power for the case that is currently broken seems like the right way to go.
I actually suspect that the amount of code broken by int**int->float may be relatively small (though extremely annoying for those that it happens to, and it would definitely be good to have statistics). I mean, Numpy silently transitioned to int32+uint64->float64 not so long ago which broke my code, but the world didn’t end. If the primary argument against int**int->float seems to be the difficulty of managing the transition, with int**int->Error being the seen as the required yet *very* painful intermediate step for the large fraction of the int**int users who didn’t care if it was int or float (e.g. the output is likely to be cast to float in the next step anyway), and fail loudly for those users who need int**int->int, then if you are prepared to risk a less conservative transition (i.e. we think that latter group is small enough) you could skip the error on users and just throw a warning for a couple of releases, along the lines of: WARNING int**int -> int is going to be deprecated in favour of int**int->float in Numpy 1.16. To avoid seeing this message, either use “from numpy import __future_float_power__” or explicitly set the type of one of your inputs to float, or use the new ipower(x,y) function for integer powers. Peter
I still strongly favour ending up at int**int -> float, and like Peter's suggestion of raising a general warning rather than an exception for negative powers. -- Marten
On Tue, 11 Oct 2016, Peter Creasey wrote:
I agree with Sebastian and Nathaniel. I don't think we can deviating from the existing behavior (int ** int -> int) without breaking lots of existing code, and if we did, yes, we would need a new integer power function.
I think it's better to preserve the existing behavior when it gives sensible results, and error when it doesn't. Adding another function float_power for the case that is currently broken seems like the right way to go.
I actually suspect that the amount of code broken by int**int->float may be relatively small (though extremely annoying for those that it happens to, and it would definitely be good to have statistics). I mean, Numpy silently transitioned to int32+uint64->float64 not so long ago which broke my code, but the world didn’t end.
If the primary argument against int**int->float seems to be the difficulty of managing the transition, with int**int->Error being the seen as the required yet *very* painful intermediate step for the large fraction of the int**int users who didn’t care if it was int or float (e.g. the output is likely to be cast to float in the next step anyway), and fail loudly for those users who need int**int->int, then if you are prepared to risk a less conservative transition (i.e. we think that latter group is small enough) you could skip the error on users and just throw a warning for a couple of releases, along the lines of:
WARNING int**int -> int is going to be deprecated in favour of int**int->float in Numpy 1.16. To avoid seeing this message, either use “from numpy import __future_float_power__” or explicitly set the type of one of your inputs to float, or use the new ipower(x,y) function for integer powers.
Sorry for coming too late to the discussion and after PR "addressing" the issue by issuing an error was merged [1]. I got burnt by new behavior while trying to build fresh pandas release on Debian (we are freezing for release way too soon ;) ) -- some pandas tests failed since they rely on previous non-erroring behavior and we got numpy 1.12.0~b1 which included [1] in unstable/testing (candidate release) now. I quickly glanced over the discussion but I guess I have missed actual description of the problem being fixed here... what was it?? previous behavior, int**int->int made sense to me as it seemed to be consistent with casting Python's pow result to int, somewhat fulfilling desired promise for in-place operations and being inline with built-in pow results as far as I see it (up to casting). Current handling and error IMHO is going against rudimentary algebra, where numbers can be brought to negative power (integer or not). [1] https://github.com/numpy/numpy/pull/8231 -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
On Tue, Jan 3, 2017 at 9:00 AM, Yaroslav Halchenko <lists@onerussian.com> wrote:
Sorry for coming too late to the discussion and after PR "addressing" the issue by issuing an error was merged [1]. I got burnt by new behavior while trying to build fresh pandas release on Debian (we are freezing for release way too soon ;) ) -- some pandas tests failed since they rely on previous non-erroring behavior and we got numpy 1.12.0~b1 which included [1] in unstable/testing (candidate release) now.
I quickly glanced over the discussion but I guess I have missed actual description of the problem being fixed here... what was it??
previous behavior, int**int->int made sense to me as it seemed to be consistent with casting Python's pow result to int, somewhat fulfilling desired promise for in-place operations and being inline with built-in pow results as far as I see it (up to casting).
I believe this is exactly the behavior we preserved. Rather, we turned some cases that previously often gave wrong results (involving negative integer powers) into errors. The pandas test suite triggered this behavior, but not intentionally, and should be fixed in the next release: https://github.com/pandas-dev/pandas/pull/14498
On Tue, 03 Jan 2017, Stephan Hoyer wrote:
On Tue, Jan 3, 2017 at 9:00 AM, Yaroslav Halchenko <lists@onerussian.com> wrote:
Sorry for coming too late to the discussion and after PR "addressing" the issue by issuing an error was merged [1].A I got burnt by new behavior while trying to build fresh pandas release on Debian (we are freezing for release way too soon ;) ) -- some pandas tests failed since they rely on previous non-erroring behavior and we gotA numpy 1.12.0~b1 which included [1] in unstable/testing (candidate release) now.
I quickly glanced over the discussion but I guess I have missed actual description of the problem being fixed here...A what was it??
previous behavior, int**int->int made sense to me as it seemed to be consistent with casting Python's pow result to int, somewhat fulfilling desired promise for in-place operations and being inline with built-in pow results as far as I see it (up to casting).
I believe this is exactly the behavior we preserved. Rather, we turned some cases that previously often gave wrong results (involving negative integer powers) into errors.
hm... testing on current master (first result is from python's pow) $> python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))" ('numpy version: ', '1.13.0.dev0+02e2ea8') 0.25 Traceback (most recent call last): File "<string>", line 1, in <module> ValueError: Integers to negative integer powers are not allowed. testing on Debian's packaged beta $> python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))" ('numpy version: ', '1.12.0b1') 0.25 Traceback (most recent call last): File "<string>", line 1, in <module> ValueError: Integers to negative integer powers are not allowed. testing on stable debian box with elderly numpy, where it does behave sensibly: $> python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))" ('numpy version: ', '1.8.2') 0.25 0 what am I missing?
The pandas test suite triggered this behavior, but not intentionally, and should be fixed in the next release: https://github.com/pandas-dev/pandas/pull/14498
I don't think that was the full set of cases, e.g. (git)hopa/sid-i386:~exppsy/pandas[bf-i386] $> nosetests -s -v pandas/tests/test_expressions.py:TestExpressions.test_mixed_arithmetic_series test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions) ... ERROR ====================================================================== ERROR: test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 223, in test_mixed_arithmetic_series self.run_series(self.mixed2[col], self.mixed2[col], binary_comp=4) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 164, in run_series test_flex=False, **kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 93, in run_arithmetic_test expected = op(df, other) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 715, in wrapper result = wrap_results(safe_na_op(lvalues, rvalues)) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 676, in safe_na_op return na_op(lvalues, rvalues) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 652, in na_op raise_on_error=True, **eval_kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 210, in evaluate **eval_kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 63, in _evaluate_standard return op(a, b) ValueError: Integers to negative integer powers are not allowed. and being paranoid, I have rebuilt exact current master of pandas with master numpy in PYTHONPATH: (git)hopa:~exppsy/pandas[master]git $> PYTHONPATH=/home/yoh/proj/numpy nosetests -s -v pandas/tests/test_expressions.py:TestExpressions.test_mixed_arithmetic_series test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions) ... ERROR ====================================================================== ERROR: test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 223, in test_mixed_arithmetic_series self.run_series(self.mixed2[col], self.mixed2[col], binary_comp=4) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 164, in run_series test_flex=False, **kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 93, in run_arithmetic_test expected = op(df, other) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 715, in wrapper result = wrap_results(safe_na_op(lvalues, rvalues)) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 676, in safe_na_op return na_op(lvalues, rvalues) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 652, in na_op raise_on_error=True, **eval_kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 210, in evaluate **eval_kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 63, in _evaluate_standard return op(a, b) ValueError: Integers to negative integer powers are not allowed. ---------------------------------------------------------------------- Ran 1 test in 0.015s FAILED (errors=1) $> git describe --tags v0.19.0-303-gb957f6f $> PYTHONPATH=/home/yoh/proj/numpy python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))" ('numpy version: ', '1.13.0.dev0+02e2ea8') 0.25 Traceback (most recent call last): File "<string>", line 1, in <module> ValueError: Integers to negative integer powers are not allowed. -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
It's possible we should back off to just issuing a deprecation warning in 1.12? On Jan 3, 2017 1:47 PM, "Yaroslav Halchenko" <lists@onerussian.com> wrote:
On Tue, 03 Jan 2017, Stephan Hoyer wrote:
On Tue, Jan 3, 2017 at 9:00 AM, Yaroslav Halchenko < lists@onerussian.com> wrote:
Sorry for coming too late to the discussion and after PR
"addressing"
the issue by issuing an error was merged [1].A I got burnt by new behavior while trying to build fresh pandas release on Debian (we
are
freezing for release way too soon ;) ) -- some pandas tests failed
since
they rely on previous non-erroring behavior and we gotA numpy
1.12.0~b1
which included [1] in unstable/testing (candidate release) now.
I quickly glanced over the discussion but I guess I have missed actual description of the problem being fixed here...A what was
it??
previous behavior, int**int->int made sense to me as it seemed to be consistent with casting Python's pow result to int, somewhat
fulfilling
desired promise for in-place operations and being inline with
built-in
pow results as far as I see it (up to casting).
I believe this is exactly the behavior we preserved. Rather, we turned some cases that previously often gave wrong results (involving negative integer powers) into errors.
hm... testing on current master (first result is from python's pow)
$> python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))" ('numpy version: ', '1.13.0.dev0+02e2ea8') 0.25 Traceback (most recent call last): File "<string>", line 1, in <module> ValueError: Integers to negative integer powers are not allowed.
testing on Debian's packaged beta
$> python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))" ('numpy version: ', '1.12.0b1') 0.25 Traceback (most recent call last): File "<string>", line 1, in <module> ValueError: Integers to negative integer powers are not allowed.
testing on stable debian box with elderly numpy, where it does behave sensibly:
$> python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))" ('numpy version: ', '1.8.2') 0.25 0
what am I missing?
The pandas test suite triggered this behavior, but not intentionally, and should be fixed in the next release: https://github.com/pandas-dev/pandas/pull/14498
I don't think that was the full set of cases, e.g.
(git)hopa/sid-i386:~exppsy/pandas[bf-i386] $> nosetests -s -v pandas/tests/test_expressions. py:TestExpressions.test_mixed_arithmetic_series test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions) ... ERROR
====================================================================== ERROR: test_mixed_arithmetic_series (pandas.tests.test_ expressions.TestExpressions) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 223, in test_mixed_arithmetic_series self.run_series(self.mixed2[col], self.mixed2[col], binary_comp=4) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 164, in run_series test_flex=False, **kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 93, in run_arithmetic_test expected = op(df, other) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 715, in wrapper result = wrap_results(safe_na_op(lvalues, rvalues)) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 676, in safe_na_op return na_op(lvalues, rvalues) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 652, in na_op raise_on_error=True, **eval_kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 210, in evaluate **eval_kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 63, in _evaluate_standard return op(a, b) ValueError: Integers to negative integer powers are not allowed.
and being paranoid, I have rebuilt exact current master of pandas with master numpy in PYTHONPATH:
(git)hopa:~exppsy/pandas[master]git $> PYTHONPATH=/home/yoh/proj/numpy nosetests -s -v pandas/tests/test_expressions.py:TestExpressions.test_mixed_ arithmetic_series test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions) ... ERROR
====================================================================== ERROR: test_mixed_arithmetic_series (pandas.tests.test_ expressions.TestExpressions) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 223, in test_mixed_arithmetic_series self.run_series(self.mixed2[col], self.mixed2[col], binary_comp=4) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 164, in run_series test_flex=False, **kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 93, in run_arithmetic_test expected = op(df, other) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 715, in wrapper result = wrap_results(safe_na_op(lvalues, rvalues)) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 676, in safe_na_op return na_op(lvalues, rvalues) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 652, in na_op raise_on_error=True, **eval_kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 210, in evaluate **eval_kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 63, in _evaluate_standard return op(a, b) ValueError: Integers to negative integer powers are not allowed.
---------------------------------------------------------------------- Ran 1 test in 0.015s
FAILED (errors=1)
$> git describe --tags v0.19.0-303-gb957f6f
$> PYTHONPATH=/home/yoh/proj/numpy python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))"
('numpy version: ', '1.13.0.dev0+02e2ea8') 0.25 Traceback (most recent call last): File "<string>", line 1, in <module> ValueError: Integers to negative integer powers are not allowed.
-- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Jan 3, 2017 at 3:05 PM, Nathaniel Smith <njs@pobox.com> wrote:
It's possible we should back off to just issuing a deprecation warning in 1.12?
On Jan 3, 2017 1:47 PM, "Yaroslav Halchenko" <lists@onerussian.com> wrote:
hm... testing on current master (first result is from python's pow)
$> python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))" ('numpy version: ', '1.13.0.dev0+02e2ea8') 0.25 Traceback (most recent call last): File "<string>", line 1, in <module> ValueError: Integers to negative integer powers are not allowed.
testing on Debian's packaged beta
$> python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))" ('numpy version: ', '1.12.0b1') 0.25 Traceback (most recent call last): File "<string>", line 1, in <module> ValueError: Integers to negative integer powers are not allowed.
testing on stable debian box with elderly numpy, where it does behave sensibly:
$> python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))" ('numpy version: ', '1.8.2') 0.25 0
what am I missing?
2 ** -2 should be 0.25.
On old versions of NumPy, you see the the incorrect answer 0. We are now preferring to give an error rather than the wrong answer.
The pandas test suite triggered this behavior, but not intentionally, and
should be fixed in the next release: https://github.com/pandas-dev/pandas/pull/14498
I don't think that was the full set of cases, e.g.
(git)hopa/sid-i386:~exppsy/pandas[bf-i386] $> nosetests -s -v pandas/tests/test_expressions. py:TestExpressions.test_mixed_arithmetic_series test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions) ... ERROR
====================================================================== ERROR: test_mixed_arithmetic_series (pandas.tests.test_expressions .TestExpressions) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 223, in test_mixed_arithmetic_series self.run_series(self.mixed2[col], self.mixed2[col], binary_comp=4) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 164, in run_series test_flex=False, **kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 93, in run_arithmetic_test expected = op(df, other) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 715, in wrapper result = wrap_results(safe_na_op(lvalues, rvalues)) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 676, in safe_na_op return na_op(lvalues, rvalues) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 652, in na_op raise_on_error=True, **eval_kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 210, in evaluate **eval_kwargs) File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 63, in _evaluate_standard return op(a, b) ValueError: Integers to negative integer powers are not allowed.
Agreed, it looks like pandas still has this issue in the test suite. Nonetheless, I don't think this should be an issue for users -- pandas defines all handling of arithmetic to numpy.
On Tue, 03 Jan 2017, Stephan Hoyer wrote:
testing on stable debian box with elderly numpy, where it does behave sensibly:
$> python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))" ('numpy version: ', '1.8.2') 0.25 0
what am I missing?
2 ** -2 should be 0.25.
On old versions of NumPy, you see the the incorrect answer 0. We are now preferring to give an error rather than the wrong answer.
it is correct up to casting/truncating to an int for the desire to maintain the int data type -- the same as
int(0.25) 0 1/4 0
or even
np.arange(5)/4 array([0, 0, 0, 0, 1])
so it is IMHO more of a documented feature and I don't see why pow needs to get all so special. Sure thing, in the bring future, unless in-place operation is demanded I would have voted for consistent float output. -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
participants (5)
-
Marten van Kerkwijk -
Nathaniel Smith -
Peter Creasey -
Stephan Hoyer -
Yaroslav Halchenko