NEP 50: Promotion rules for Python scalars
Hi all,
I would like to share the first formal draft of
NEP 50: Promotion rules for Python scalars
with everyone. The full text can be found here:
https://numpy.org/neps/nep0050scalarpromotion.html
NEP 50 is an attempt to remove valuebased casting/promotion. We wish to replace it with clearer rules for the resulting dtype when mixing NumPy arrays and Python scalars. As a brief example, the proposal allows the following (unchanged):
>>> np.array([1, 2, 3], dtype=np.int8) + 100 np.array([101, 102, 103], dtype=np.int8)
While clearing up confusion caused by the valueinspecting behavior that we see sometimes, such as:
>>> np.array([1, 2, 3], dtype=np.int8) + 300 np.array([301, 302, 303], dtype=np.int16) # note the int16
Where 300 is too large to fit an ``int8``. As well as removing the special behavior of 0D arrays or NumPy scalars:
>>> res = np.array(1, dtype=np.int8) + 100 >>> res.dtype dtype('int64')
This is the continuation of a long discussion (see the "Discussion" section), including the poll I once posted: https://discuss.scientificpython.org/t/pollfuturenumpybehaviorwhenmixi...
I would be happy for any feadback, be it just editorial or fundamental discussion. There are many alternatives which I have tried to capture in the NEP. So lets discuss here, or on discuss:
https://discuss.scientificpython.org/t/nep50promotionrulesforpythonsc...
For smaller edits, don't hesitate to open a NumPy PR, or propose edits on my branch (you can use the edit button to create a PR):
https://github.com/seberg/numpy/blob/nep50/doc/neps/nep0050scalarpromotio...
An important part of moving forward will be assessing the real world impact. To start that process, I have created a branch as a draft PR (at this time):
https://github.com/numpy/numpy/pull/21626
It is missing some parts, but should allow preliminary testing. The main missing part is that the integer warnings and errors are less strict than proposed in the NEP. It would be invaluable to get a better idea to what extent existing code, especially enduser code, is affected by the proposed changes.
Thanks in advance for any input! This is a big, complicated proposal, but finding a way forward will hopefully clear up a source of confusion and inconsistencies that make both maintainers and users life harder.
Cheers,
Sebastian
On Wed, Jun 1, 2022 at 5:51 PM Sebastian Berg sebastian@sipsolutions.net wrote:
An important part of moving forward will be assessing the real world impact. To start that process, I have created a branch as a draft PR (at this time):
https://github.com/numpy/numpy/pull/21626
It is missing some parts, but should allow preliminary testing. The main missing part is that the integer warnings and errors are less strict than proposed in the NEP. It would be invaluable to get a better idea to what extent existing code, especially enduser code, is affected by the proposed changes.
Thanks Sebastian! For testing, did you already try with some of the usual suspects, or would it be helpful to use this branch on SciPy, Pandas, etc.? Also, do you expect it's useful to do platformspecific testing? I can imagine there's some Windowsspecific behavior; adapting a SciPy CI job to work from your branch is easy to do if that would be helpful.
Cheers, Ralf
On Wed, 20220601 at 20:23 +0200, Ralf Gommers wrote:
On Wed, Jun 1, 2022 at 5:51 PM Sebastian Berg sebastian@sipsolutions.net wrote:
An important part of moving forward will be assessing the real world impact. To start that process, I have created a branch as a draft PR (at this time):
https://github.com/numpy/numpy/pull/21626
It is missing some parts, but should allow preliminary testing. The main missing part is that the integer warnings and errors are less strict than proposed in the NEP. It would be invaluable to get a better idea to what extent existing code, especially enduser code, is affected by the proposed changes.
Thanks Sebastian! For testing, did you already try with some of the usual suspects, or would it be helpful to use this branch on SciPy, Pandas, etc.? Also, do you expect it's useful to do platformspecific testing? I can imagine there's some Windowsspecific behavior; adapting a SciPy CI job to work from your branch is easy to do if that would be helpful.
Yes, I have for SciPy. As noted in the PR, those look "mostly harmless" on first sight (not that it won't mean quite a bit of work, but I think it is manageable work). I would be more scared if there is a need to systematically vet all places where behavior (may have) changed.
For example, in NumPy:
np.median(np.float32([1, 2, 3, 4]))
did return a float64 before and will now return a float32. I assume because somewhere we write: `(np.float64(3) + np.float32(2)) / 2`.
There a few places that I suspect just need updated test or a bit of thought. And at least one or two that need to use the correct integer types (IIRC `scipy.io.idl` seems to be using some low precision or unsigned integer type internally and that leads to failures).
I thought pandas would fail much harder, but it seems only had a 150 200 failures (many probably clustered). One larger annoyance there is that one parametrized test runs into an infinite recursion which makes it run excruciatingly slow.
In any case, I believe that it would be far more helpful if those more familiar with the libraries have a look at the failures. Not only do they know better how much impact they have; it also helps to get a feel for how painful the transition will be.
One problem I see, is that I still expect that libraries are not the main issue. Using a SciPy integrator may end up with a float32 rather than a float64 result. In the SciPy test suite, that probably just means tweaking the test a bit. But that same change will also break someones script out there, somewhere. So the real affected persons (who may occasionally get less precise/breaking results) are likely the endusers rather than the libraries.
Cheers,
Sebastian
Cheers, Ralf _______________________________________________ NumPyDiscussion mailing list  numpydiscussion@python.org To unsubscribe send an email to numpydiscussionleave@python.org https://mail.python.org/mailman3/lists/numpydiscussion.python.org/ Member address: sebastian@sipsolutions.net
For example, in NumPy:
np.median(np.float32([1, 2, 3, 4]))
did return a float64 before and will now return a float32. I assume because somewhere we write: `(np.float64(3) + np.float32(2)) / 2`.
Sorry, I missed this part of the discussion — I know the discussion centered around Python literals being weak, but for NumPy dtypes, I thought the larger dtype would always win?
Indeed, reading the NEP I see:
Expression: array([1.], float32) + array(1., float64) Old result: array([2.], float32) New result: array([2.], float64)
which seems to contradict your statement above?
On Wed, 20220601 at 18:37 0500, Juan NunezIglesias wrote:
For example, in NumPy:
np.median(np.float32([1, 2, 3, 4]))
did return a float64 before and will now return a float32. I assume because somewhere we write: `(np.float64(3) + np.float32(2)) / 2`.
Sorry, I missed this part of the discussion — I know the discussion centered around Python literals being weak, but for NumPy dtypes, I thought the larger dtype would always win?
Good reading carefully enough to notice :)!
Sorry... my bad, the float64 is a typo. That should have read:
(float32(3) + float32(2)) / 2
Which does show the change in behavior as described/discussed. If there was a float64 involved, of course the result would be/remain float64.
 Sebastian
Indeed, reading the NEP I see:
Expression: array([1.], float32) + array(1., float64) Old result: array([2.], float32) New result: array([2.], float64)
which seems to contradict your statement above? _______________________________________________ NumPyDiscussion mailing list  numpydiscussion@python.org To unsubscribe send an email to numpydiscussionleave@python.org https://mail.python.org/mailman3/lists/numpydiscussion.python.org/ Member address: sebastian@sipsolutions.net
On Wed, 20220601 at 08:49 0700, Sebastian Berg wrote:
Hi all,
I would like to share the first formal draft of
NEP 50: Promotion rules for Python scalars
with everyone. The full text can be found here:
As a brief update on this, as noted in the NEP (Note at the end of the abstract), our nightly wheels can now be used to try out the first changes here. To use you have to install NumPy from the nightlies:
pip install i https://pypi.anaconda.org/scipywheelsnightly/simple numpy upgrade
(I added an upgrade in case you have a numpy version already). And then run Python e.g. with:
NPY_PROMOTION_STATE=weak ipython
Please see the NEP note for more information. As of now, especially the error when integers are too large is missing (this is added in an open PR).
It would be very interesting to hear whether your use case (e.g. scripts) is affected by the changes! Right now mainly scipy and sklearn were tried, but for this change I believe the impact on end users is much more important than that on libraries.
Cheers,
Sebastian
NEP 50 is an attempt to remove valuebased casting/promotion. We wish to replace it with clearer rules for the resulting dtype when mixing NumPy arrays and Python scalars. As a brief example, the proposal allows the following (unchanged):
>>> np.array([1, 2, 3], dtype=np.int8) + 100 np.array([101, 102, 103], dtype=np.int8)
While clearing up confusion caused by the valueinspecting behavior that we see sometimes, such as:
>>> np.array([1, 2, 3], dtype=np.int8) + 300 np.array([301, 302, 303], dtype=np.int16) # note the int16
Where 300 is too large to fit an ``int8``. As well as removing the special behavior of 0D arrays or NumPy scalars:
>>> res = np.array(1, dtype=np.int8) + 100 >>> res.dtype dtype('int64')
This is the continuation of a long discussion (see the "Discussion" section), including the poll I once posted: https://discuss.scientificpython.org/t/pollfuturenumpybehaviorwhenmixi...
I would be happy for any feadback, be it just editorial or fundamental discussion. There are many alternatives which I have tried to capture in the NEP. So lets discuss here, or on discuss:
https://discuss.scientificpython.org/t/nep50promotionrulesforpythonsc...
For smaller edits, don't hesitate to open a NumPy PR, or propose edits on my branch (you can use the edit button to create a PR):
https://github.com/seberg/numpy/blob/nep50/doc/neps/nep0050scalarpromotio...
An important part of moving forward will be assessing the real world impact. To start that process, I have created a branch as a draft PR (at this time):
https://github.com/numpy/numpy/pull/21626
It is missing some parts, but should allow preliminary testing. The main missing part is that the integer warnings and errors are less strict than proposed in the NEP. It would be invaluable to get a better idea to what extent existing code, especially enduser code, is affected by the proposed changes.
Thanks in advance for any input! This is a big, complicated proposal, but finding a way forward will hopefully clear up a source of confusion and inconsistencies that make both maintainers and users life harder.
Cheers,
Sebastian _______________________________________________ NumPyDiscussion mailing list  numpydiscussion@python.org To unsubscribe send an email to numpydiscussionleave@python.org https://mail.python.org/mailman3/lists/numpydiscussion.python.org/ Member address: sebastian@sipsolutions.net
participants (3)

Juan NunezIglesias

Ralf Gommers

Sebastian Berg