Hi, On Mon, Nov 12, 2012 at 1:11 PM, Nathaniel Smith <njs@pobox.com> wrote:

On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

Hi,

I wanted to check that everyone knows about and is happy with the scalar casting changes from 1.6.0.

Specifically, the rules for (array, scalar) casting have changed such that the resulting dtype depends on the _value_ of the scalar.

Mark W has documented these changes here:

http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html

Specifically, as of 1.6.0:

In [19]: arr = np.array([1.], dtype=np.float32)

In [20]: (arr + (2**16-1)).dtype Out[20]: dtype('float32')

In [21]: (arr + (2**16)).dtype Out[21]: dtype('float64')

In [25]: arr = np.array([1.], dtype=np.int8)

In [26]: (arr + 127).dtype Out[26]: dtype('int8')

In [27]: (arr + 128).dtype Out[27]: dtype('int16')

There's discussion about the changes here:

http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html

It seems to me that this change is hard to explain, and does what you want only some of the time, making it a false friend.

The old behaviour was that in these cases, the scalar was always cast to the type of the array, right? So np.array([1], dtype=np.int8) + 256 returned 1? Is that the behaviour you prefer?

Right. In that case of course, I'm getting something a bit nasty. But if you're working with int8, I think you expect to be careful of overflow. And you may well not want an automatic and maybe surprising upcast to int16.

I agree that the 1.6 behaviour is surprising and somewhat inconsistent. There are many places where you can get an overflow in numpy, and in all the other cases we just let the overflow happen. And in fact you can still get an overflow with arr + scalar operations, so this doesn't really fix anything.

Right - it's a half-fix, which seems to me worse than no fix.

I find the specific handling of unsigned -> signed and float32 -> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly representable as a float32, but it doesn't *overflow*, it just gives you 2.0**16... if I'm using float32 then I presumably don't care that much about exact representability, so it's surprising that numpy is working to enforce it, and definitely a separate decision from what to do about overflow.)

None of those threads seem to really get into the question of what the best behaviour here *is*, though.

Possibly the most defensible choice is to treat ufunc(arr, scalar) operations as performing an implicit cast of the scalar to arr's dtype, and using the standard implicit casting rules -- which I think means, raising an error if !can_cast(scalar, arr.dtype, casting="safe")

You mean: In [25]: arr = np.array([1.], dtype=np.int8) In [27]: arr + 128 ValueError - cannot safely cast 128 to array dtype int8? That would be a major change. If I really wanted to do that, would you then suggest I cast to an array? arr + np.array([128]) It would be very good to make a well-argued long-term decision, whatever the chosen outcome. Maybe this is the place for a partly retrospective NEP? Best, Matthew

2012/11/12 Nathaniel Smith <njs@pobox.com>

On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

Hi,

I wanted to check that everyone knows about and is happy with the scalar casting changes from 1.6.0.

Specifically, the rules for (array, scalar) casting have changed such that the resulting dtype depends on the _value_ of the scalar.

Mark W has documented these changes here:

http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules

http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html

...

http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html

...

Specifically, as of 1.6.0:

In [19]: arr = np.array([1.], dtype=np.float32)

In [20]: (arr + (2**16-1)).dtype Out[20]: dtype('float32')

In [21]: (arr + (2**16)).dtype Out[21]: dtype('float64')

In [25]: arr = np.array([1.], dtype=np.int8)

In [26]: (arr + 127).dtype Out[26]: dtype('int8')

In [27]: (arr + 128).dtype Out[27]: dtype('int16')

There's discussion about the changes here:

http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html

...

http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html

http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html

...

It seems to me that this change is hard to explain, and does what you want only some of the time, making it a false friend.

The old behaviour was that in these cases, the scalar was always cast to the type of the array, right? So np.array([1], dtype=np.int8) + 256 returned 1? Is that the behaviour you prefer?

I agree that the 1.6 behaviour is surprising and somewhat inconsistent. There are many places where you can get an overflow in numpy, and in all the other cases we just let the overflow happen. And in fact you can still get an overflow with arr + scalar operations, so this doesn't really fix anything.

I find the specific handling of unsigned -> signed and float32 -> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly representable as a float32, but it doesn't *overflow*, it just gives you 2.0**16... if I'm using float32 then I presumably don't care that much about exact representability, so it's surprising that numpy is working to enforce it, and definitely a separate decision from what to do about overflow.)

None of those threads seem to really get into the question of what the best behaviour here *is*, though.

Possibly the most defensible choice is to treat ufunc(arr, scalar) operations as performing an implicit cast of the scalar to arr's dtype, and using the standard implicit casting rules -- which I think means, raising an error if !can_cast(scalar, arr.dtype, casting="safe")

I like this suggestion. It may break some existing code, but I think it'd be for the best. The current behavior can be very confusing. -=- Olivier

On Monday, November 12, 2012, Benjamin Root wrote:

On Monday, November 12, 2012, Olivier Delalleau wrote:

...

2012/11/12 Nathaniel Smith <njs@pobox.com>

...

On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

Hi,

I wanted to check that everyone knows about and is happy with the scalar casting changes from 1.6.0.

Specifically, the rules for (array, scalar) casting have changed such that the resulting dtype depends on the _value_ of the scalar.

Mark W has documented these changes here:

http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules

http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html

...

http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html

...

Specifically, as of 1.6.0:

In [19]: arr = np.array([1.], dtype=np.float32)

In [20]: (arr + (2**16-1)).dtype Out[20]: dtype('float32')

In [21]: (arr + (2**16)).dtype Out[21]: dtype('float64')

In [25]: arr = np.array([1.], dtype=np.int8)

In [26]: (arr + 127).dtype Out[26]: dtype('int8')

In [27]: (arr + 128).dtype Out[27]: dtype('int16')

There's discussion about the changes here:

http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html

...

http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html

...

http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html

...

It seems to me that this change is hard to explain, and does what you want only some of the time, making it a false friend.

The old behaviour was that in these cases, the scalar was always cast to the type of the array, right? So np.array([1], dtype=np.int8) + 256 returned 1? Is that the behaviour you prefer?

I agree that the 1.6 behaviour is surprising and somewhat inconsistent. There are many places where you can get an overflow in numpy, and in all the other cases we just let the overflow happen. And in fact you can still get an overflow with arr + scalar operations, so this doesn't really fix anything.

I find the specific handling of unsigned -> signed and float32 -> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly representable as a float32, but it doesn't *overflow*, it just gives you 2.0**16... if I'm using float32 then I presumably don't care that much about exact representability, so it's surprising that numpy is working to enforce it, and definitely a separate decision from what to do about overflow.)

None of those threads seem to really get into the question of what the best behaviour here *is*, though.

Possibly the most defensible choice is to treat ufunc(arr, scalar) operations as performing an implicit cast of the scalar to arr's dtype, and using the standard implicit casting rules -- which I think means, raising an error if !can_cast(scalar, arr.dtype, casting="safe")

I like this suggestion. It may break some existing code, but I think it'd be for the best. The current behavior can be very confusing.

-=- Olivier

"break some existing code"

I really should set up an email filter for this phrase and have it send back an email automatically: "Are you nuts?!"

We just resolved an issue where the "safe" casting rule unexpectedly broke existing code with regards to unplaced operations. The solution was to warn about the change in the upcoming release and to throw errors in a later release. Playing around with fundemental things like this need to be done methodically and carefully.

Cheers! Ben Root

Stupid autocorrect: unplaced --> inplace

On Monday, November 12, 2012, Matthew Brett wrote:

Hi,

On Mon, Nov 12, 2012 at 8:15 PM, Benjamin Root <ben.root@ou.edu> wrote:

...

On Monday, November 12, 2012, Olivier Delalleau wrote:

...

2012/11/12 Nathaniel Smith <njs@pobox.com>

...

On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <

...

...

...

wrote:

...

Hi,

I wanted to check that everyone knows about and is happy with the scalar casting changes from 1.6.0.

Specifically, the rules for (array, scalar) casting have changed such that the resulting dtype depends on the _value_ of the scalar.

Mark W has documented these changes here:

http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules

http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html

...

http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html

...

Specifically, as of 1.6.0:

In [19]: arr = np.array([1.], dtype=np.float32)

In [20]: (arr + (2**16-1)).dtype Out[20]: dtype('float32')

In [21]: (arr + (2**16)).dtype Out[21]: dtype('float64')

In [25]: arr = np.array([1.], dtype=np.int8)

In [26]: (arr + 127).dtype Out[26]: dtype('int8')

In [27]: (arr + 128).dtype Out[27]: dtype('int16')

There's discussion about the changes here:

http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html

...

http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html

...

http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html

...

It seems to me that this change is hard to explain, and does what you want only some of the time, making it a false friend.

The old behaviour was that in these cases, the scalar was always cast to the type of the array, right? So np.array([1], dtype=np.int8) + 256 returned 1? Is that the behaviour you prefer?

I agree that the 1.6 behaviour is surprising and somewhat inconsistent. There are many places where you can get an overflow in numpy, and in all the other cases we just let the overflow happen. And in fact you can still get an overflow with arr + scalar operations, so this doesn't really fix anything.

I find the specific handling of unsigned -> signed and float32 -> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly representable as a float32, but it doesn't *overflow*, it just gives you 2.0**16... if I'm using float32 then I presumably don't care that much about exact representability, so it's surprising that numpy is working to enforce it, and definitely a separate decision from what to do about overflow.)

None of those threads seem to really get into the question of what the best behaviour here *is*, though.

Possibly the moWell, hold on though, I was asking earlier in the

matthew.brett@gmail.com> thread what we thought the behavior should be in 2.0 or maybe better put, sometime in the future.

If we know what we think the best answer is, and we think the best answer is worth shooting for, then we can try to think of sensible ways of getting there.

I guess that's what Nathaniel and Olivier were thinking of but they can correct me if I'm wrong...

Cheers,

Matthew

I am fine with migrating to better solutions (I have yet to decide on this current situation, though), but whatever change is adopted must go through a deprecation process, which was my point. Outright breaking of code as a first step is the wrong choice, and I was merely nipping it in the bud. Cheers! Ben Root

2012/11/12 Matthew Brett <matthew.brett@gmail.com>

Hi,

On Mon, Nov 12, 2012 at 8:15 PM, Benjamin Root <ben.root@ou.edu> wrote:

...

On Monday, November 12, 2012, Olivier Delalleau wrote:

...

2012/11/12 Nathaniel Smith <njs@pobox.com>

...

On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <

matthew.brett@gmail.com>

...

...

...

wrote:

...

Hi,

I wanted to check that everyone knows about and is happy with the scalar casting changes from 1.6.0.

Specifically, the rules for (array, scalar) casting have changed such that the resulting dtype depends on the _value_ of the scalar.

Mark W has documented these changes here:

http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules

http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html

...

http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html

...

Specifically, as of 1.6.0:

In [19]: arr = np.array([1.], dtype=np.float32)

In [20]: (arr + (2**16-1)).dtype Out[20]: dtype('float32')

In [21]: (arr + (2**16)).dtype Out[21]: dtype('float64')

In [25]: arr = np.array([1.], dtype=np.int8)

In [26]: (arr + 127).dtype Out[26]: dtype('int8')

In [27]: (arr + 128).dtype Out[27]: dtype('int16')

There's discussion about the changes here:

http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html

...

http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html

...

http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html

...

It seems to me that this change is hard to explain, and does what you want only some of the time, making it a false friend.

The old behaviour was that in these cases, the scalar was always cast to the type of the array, right? So np.array([1], dtype=np.int8) + 256 returned 1? Is that the behaviour you prefer?

I agree that the 1.6 behaviour is surprising and somewhat inconsistent. There are many places where you can get an overflow in numpy, and in all the other cases we just let the overflow happen. And in fact you can still get an overflow with arr + scalar operations, so this doesn't really fix anything.

I find the specific handling of unsigned -> signed and float32 -> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly representable as a float32, but it doesn't *overflow*, it just gives you 2.0**16... if I'm using float32 then I presumably don't care that much about exact representability, so it's surprising that numpy is working to enforce it, and definitely a separate decision from what to do about overflow.)

None of those threads seem to really get into the question of what the best behaviour here *is*, though.

Possibly the most defensible choice is to treat ufunc(arr, scalar) operations as performing an implicit cast of the scalar to arr's dtype, and using the standard implicit casting rules -- which I think means, raising an error if !can_cast(scalar, arr.dtype, casting="safe")

I like this suggestion. It may break some existing code, but I think it'd be for the best. The current behavior can be very confusing.

-=- Olivier

"break some existing code"

I really should set up an email filter for this phrase and have it send back an email automatically: "Are you nuts?!"

Well, hold on though, I was asking earlier in the thread what we thought the behavior should be in 2.0 or maybe better put, sometime in the future.

If we know what we think the best answer is, and we think the best answer is worth shooting for, then we can try to think of sensible ways of getting there.

I guess that's what Nathaniel and Olivier were thinking of but they can correct me if I'm wrong...

Cheers,

Matthew

This is indeed what I had in mind, thanks. I definitely agree a (long) period with a deprecation warning would be needed if this is changed. -=- Olivier

On Wed, Jan 2, 2013 at 11:24 AM, Nathaniel Smith <njs@pobox.com> wrote:

This discussion seems to have petered out without reaching consensus one way or another. This seems like an important issue, so I've opened a bug: https://github.com/numpy/numpy/issues/2878 Hopefully this way we'll at least not forget about it; also I tried to summarize the main issues there and would welcome comments.

Consensus in that bug report seems to be that for array/scalar operations like: np.array([1], dtype=np.int8) + 1000 # can't be represented as an int8! we should raise an error, rather than either silently upcasting the result (as in 1.6 and 1.7) or silently downcasting the scalar (as in 1.5 and earlier). The next question is how to handle the warning period, or if there should be a warning period, given that we've already silently changed the semantics of this operation, so raising a warning now is perhaps like noticing that the horses are gone and putting up a notice warning that we plan to close the barn door shortly. But then again, people who have already adjusted their code for 1.6 may appreciate such a warning. Or maybe no-one actually writes dumb things like int8-plus-1000 so it doesn't matter, but anyway I thought the list should have a heads-up :-) -n

On 01/04/2013 12:39 AM, Andrew Collette wrote:

...

Consensus in that bug report seems to be that for array/scalar operations like: np.array([1], dtype=np.int8) + 1000 # can't be represented as an int8! we should raise an error, rather than either silently upcasting the result (as in 1.6 and 1.7) or silently downcasting the scalar (as in 1.5 and earlier).

I have run into this a few times as a NumPy user, and I just wanted to comment that (in my opinion), having this case generate an error is the worst of both worlds. The reason people can't decide between rollover and promotion is because neither is objectively better. One

If neither is objectively better, I think that is a very good reason to kick it down to the user. "Explicit is better than implicit".

avoids memory inflation, and the other avoids losing precision. You just need to pick one and document it. Kicking the can down the road to the user, and making him/her explicitly test for this condition, is not a very good solution.

It's a good solution to encourage bug-free code. It may not be a good solution to avoid typing.

What does this mean in practical terms for NumPy users? I personally don't relish the choice of always using numpy.add, or always wrapping my additions in checks for ValueError.

I think you usually have a bug in your program when this happens, since either the dtype is wrong, or the value one is trying to store is wrong. I know that's true for myself, though I don't claim to know everybody elses usecases. Dag Sverre

On 4 Jan 2013 00:39, "Peter Cock" <p.j.a.cock@googlemail.com> wrote:

I agree with Dag rather than Andrew, "Explicit is better than implicit". i.e. What Nathaniel described earlier as the apparent consensus.

Since I've actually used NumPy arrays with specific low memory types, I thought I should comment about my use case if case it is helpful:

I've only used the low precision types like np.uint8 (unsigned) where I needed to limit my memory usage. In this case, the topology of a graph allowing multiple edges held as an integer adjacency matrix, A. I would calculate things like A^n for paths of length n, and also make changes to A directly (e.g. adding edges). So an overflow was always possible, and neither the old behaviour (type preserving but wrapping on overflow giving data corruption) nor the current behaviour (type promotion overriding my deliberate memory management) are nice. My preferences here would be for an exception, so I knew right away.

I don't think the changes we're talking about here will help your use case actually; this is only about the specific case where one of your operands, itself, cannot be cleanly cast to the types being used for the operation - it won't detect overflow in general. For that you want #593: https://github.com/numpy/numpy/issues/593 On another note, while you're here, perhaps I can tempt you into having a go at fixing #593? :-) -n

On Fri, Jan 4, 2013 at 12:39 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

...

Since I've actually used NumPy arrays with specific low memory types, I thought I should comment about my use case if case it is helpful:

I've only used the low precision types like np.uint8 (unsigned) where I needed to limit my memory usage. In this case, the topology of a graph allowing multiple edges held as an integer adjacency matrix, A. I would calculate things like A^n for paths of length n, and also make changes to A directly (e.g. adding edges). So an overflow was always possible, and neither the old behaviour (type preserving but wrapping on overflow giving data corruption) nor the current behaviour (type promotion overriding my deliberate memory management) are nice. My preferences here would be for an exception, so I knew right away.

The other use case which comes to mind is dealing with low level libraries and/or file formats, and here automagic type promotion would probably be unwelcome.

Regards,

Peter

Elsewhere on the thread, Nathaniel Smith <njs@pobox.com> wrote:

To be clear: we're only talking here about the case where you have a mix of a narrow dtype in an array and a scalar value that cannot be represented in that narrow dtype. If both sides are arrays then we continue to upcast as normal. So my impression is that this means very little in practical terms, because this is a rare and historically poorly supported situation.

But if this is something you're running into in practice then you may have a better idea than us about the practical effects. Do you have any examples where this has come up that you can share?

-n

Clarification appreciated - on closer inspection for my adjacency matrix example I would not fall over the issue in https://github.com/numpy/numpy/issues/2878

...

...

import numpy as np np.__version__ '1.6.1' A = np.zeros((100,100), np.uint8) # Matrix could be very big A[3,4] = 255 # Max value, setting up next step in example A[3,4] 255 A[3,4] += 1 # Silently overflows on NumPy 1.6 A[3,4] 0

To trigger the contentious behaviour I'd have to do something like this:

...

...

A = np.zeros((100,100), np.uint8) B = A + 256 B array([[256, 256, 256, ..., 256, 256, 256], [256, 256, 256, ..., 256, 256, 256], [256, 256, 256, ..., 256, 256, 256], ..., [256, 256, 256, ..., 256, 256, 256], [256, 256, 256, ..., 256, 256, 256], [256, 256, 256, ..., 256, 256, 256]], dtype=uint16)

I wasn't doing anything like that in my code though, just simple matrix multiplication and in situ element modification, for example A[i,j] += 1 to add an edge. I still agree that for https://github.com/numpy/numpy/issues/2878 an exception sounds sensible. Peter

2013/1/3 Andrew Collette <andrew.collette@gmail.com>:

Hi Dag,

...

If neither is objectively better, I think that is a very good reason to kick it down to the user. "Explicit is better than implicit".

I agree with you, up to a point. However, we are talking about an extremely common operation that I think most people (myself included) would not expect to raise an exception: namely, adding a number to an array.

...

It's a good solution to encourage bug-free code. It may not be a good solution to avoid typing.

Ha! But seriously, checking every time I make an addition? And in the current version of numpy it's not buggy code to add 128 to an int8 array; it's documented to give you an int16 with the result of the addition. Maybe it shouldn't, but that's what it does.

...

I think you usually have a bug in your program when this happens, since either the dtype is wrong, or the value one is trying to store is wrong. I know that's true for myself, though I don't claim to know everybody elses usecases.

I don't think it's unreasonable to add a number to an int16 array (or int32), and rely on specific, documented behavior if the number is outside the range. For example, IDL will clip the value. Up until 1.6, in NumPy it would roll over. Currently it upcasts.

I won't make the case for upcasting vs rollover again, as I think that's dealt with extensively in the threads linked in the bug. I am concerned about the tests I need to add wherever I might have a scalar, or the program blows up.

It occurs to me that, if I have "a = b + c" in my code, and "c" is sometimes a scalar and sometimes an array, I will get different behavior. If I have this right, if "c" is an array of larger dtype, including a 1-element array, it will upcast, if it's the same dtype, it will roll over regardless, but if it's a scalar and the result won't fit, it will raise ValueError.

By the way, how do I test for this? I can't test just the scalar because the proposed behavior (as I understand it) considers the result of the addition. Should I always compute amax (nanmax)? Do I need to try adding them and look for ValueError?

And things like this suddenly become dangerous:

try: some_function(myarray + something) except ValueError: print "Problem in some_function!"

Actually, the proposed behavior considers only the value of the scalar, not the result of the addition. So the correct way to do things with this proposal would be to be sure you don't add to an array a scalar value that can't fit in the array's dtype. In 1.6.1, you should make this check anyway, since otherwise your computation can be doing something completely different without telling you (and I doubt it's what you'd want): In [50]: np.array([2], dtype='int8') + 127 Out[50]: array([-127], dtype=int8) In [51]: np.array([2], dtype='int8') + 128 Out[51]: array([130], dtype=int16) If the decision is to always roll-over, the first thing to decide is whether this means the scalar is downcasted, or the output of the computation. It doesn't matter for +, but for instance for the "maximum" ufunc, I don't think it makes sense to perform the computation at higher precision then downcast the output, as you would otherwise have: np.maximum(np.ones(1, dtype='int8'), 128)) == [-128] So out of consistency (across ufuncs) I think it should always downcast the scalar (it has the advantage of being more efficient too, since you don't need to do an upcast to perform the computation). But then you're up for some nasty surprise if your scalar overflows and you didn't expect it. For instance the "maximum" example above would return [1], which may be expected... or not (maybe you wanted to obtain [128] instead?). Another solution is to forget about trying to be smart and always upcast the operation. That would be my 2nd preferred solution, but it would make it very annoying to deal with Python scalars (typically int64 / float64) that would be upcasting lots of things, potentially breaking a significant amount of existing code. So, personally, I don't see a straightforward solution without warning/error, that would be safe enough for programmers. -=- Olivier

Nathaniel asked:

...

But if this is something you're running into in practice then you may have a better idea than us about the practical effects. Do you have any examples where this has come up that you can share?

The only time I really ran into the 1.5/1.6 change was some old code ported from IDL which did odd things with the wrapping behavior. But what I'm really trying to get a handle on here is the proposed future behavior. I am coming to this from the perspective of both a user and a library developer (h5py) trying to work out what if anything I have to do when handling arrays and values I get from users.

Andrew

2013/1/3 Andrew Collette <andrew.collette@gmail.com>:

...

Another solution is to forget about trying to be smart and always upcast the operation. That would be my 2nd preferred solution, but it would make it very annoying to deal with Python scalars (typically int64 / float64) that would be upcasting lots of things, potentially breaking a significant amount of existing code.

So, personally, I don't see a straightforward solution without warning/error, that would be safe enough for programmers.

I guess what's really confusing me here is that I had assumed that this:

result = myarray + scalar

was equivalent to this:

result = myarray + numpy.array(scalar)

where the dtype of the converted scalar was chosen to be "just big enough" for it to fit. Then you proceed using the normal rules for array addition. Yes, you can have upcasting or rollover depending on the values involved, but you have that anyway with array addition; it's just how arrays work in NumPy.

A key difference is that with arrays, the dtype is not chosen "just big enough" for your data to fit. Either you set the dtype yourself, or you're using the default inferred dtype (int/float). In both cases you should know what to expect, and it doesn't depend on the actual numeric values (except for the auto int/float distinction).

Also, have I got this (proposed behavior) right?

array([127], dtype=int8) + 128 -> ValueError array([127], dtype=int8) + 127 -> -2

It seems like all this does is raise an error when the current rules would require upcasting, but still allows rollover for smaller values. What error condition, specifically, is the ValueError designed to tell me about? You can still get "unexpected" data (if you're not expecting rollover) with no exception.

The ValueError is here to warn you that the operation may not be doing what you want. The rollover for smaller values would be the documented (and thus hopefully expected) behavior. Taking the addition as an example may be misleading, as it makes it look like we could just "always rollover" to obtain consistent behavior, and programmers are to some extent used to integer rollover on this kind of operation. However, I gave examples with "maximum" that I believe show it's not that easy (this behavior would just appear "wrong"). Another example is with the integer division, where casting the scalar silently would result in array([-128], dtype=int8) // 128 -> [1] which is unlikely to be something someone would like to obtain. To summarize the goals of the proposal (in my mind): 1. Low cognitive load (simple and consistent across ufuncs). 2. Low risk of doing something unexpected. 3. Efficient by default. 4. Most existing (non buggy) code should not be affected. If we always do the silent cast, it will significantly break existing code relying on the 1.6 behavior, and increases the risk of doing something unexpected (bad on #2 & #4) If we always upcast, we may break existing code and lose efficiency (bad on #3 and #4). If we keep current behavior, we stay with something that's difficult to understand and has high risk of doing weird things (bad on #1 and #2). -=- Olivier

(sorry, no time for full reply, so for now just answering what I believe is the main point) 2013/1/4 Andrew Collette <andrew.collette@gmail.com>:

...

The ValueError is here to warn you that the operation may not be doing what you want. The rollover for smaller values would be the documented (and thus hopefully expected) behavior.

Right, but what confuses me is that the only thing this prevents is the current upcast behavior. Why is that so evil it should be replaced with an exception?

The evilness lies in the silent switch between the rollover and upcast behavior, as in the example I gave previously: In [50]: np.array([2], dtype='int8') + 127 Out[50]: array([-127], dtype=int8) In [51]: np.array([2], dtype='int8') + 128 Out[51]: array([130], dtype=int16) If the scalar is the user-supplied value, it's likely you actually want a fixed behavior (either rollover or upcast) regardless of the numeric value being provided. Looking at what other numeric libraries are doing is definitely a good suggestion. -=- Olivier

Hi, On Fri, Jan 4, 2013 at 4:01 PM, Andrew Collette <andrew.collette@gmail.com> wrote:

...

From a more basic perspective, I think that adding a number to an array should never raise an exception. I've not used any other language in which this behavior takes place. In C, you have rollover behavior, in IDL you roll over or clip, and in NumPy you either roll or upcast, depending on the version. IDL, etc. manage to handle things like max() or total() in a sensible (or at least defensible) fashion, and without raising an error.

That's a reasonable point. Looks like we lost consensus. What about returning to the 1.5 behavior instead? Best, Matthew

On Sat, Jan 5, 2013 at 12:32 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

Hi,

On Fri, Jan 4, 2013 at 4:54 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

Hi,

On Fri, Jan 4, 2013 at 4:01 PM, Andrew Collette <andrew.collette@gmail.com> wrote:

...

...

From a more basic perspective, I think that adding a number to an array should never raise an exception. I've not used any other language in which this behavior takes place. In C, you have rollover behavior, in IDL you roll over or clip, and in NumPy you either roll or upcast, depending on the version. IDL, etc. manage to handle things like max() or total() in a sensible (or at least defensible) fashion, and without raising an error.

That's a reasonable point.

Looks like we lost consensus.

What about returning to the 1.5 behavior instead?

If we do return to the 1.5 behavior, we would need to think about doing this in 1.7.

If there are a large number of 1.5.x and previous users who would upgrade to 1.7, leaving the 1.6 behavior in 1.7 will mean that they will get double the confusion:

1) The behavior has changed to something they weren't expecting 2) The behavior is going to change back very soon

I disagree. 1.7 is basically done, the 1.6 changes are out there already, and we still have work to do just to get consensus on how we want to handle this, plus implement the changes. Basically, the way I think about it in general is, you have the first release that contains some bug, and then you have the first release that doesn't contain it. Minimizing the amount of *time* between those releases is important. Minimizing the *number of releases* in between does not -- according to that logic, we shouldn't have released 1.6.1 and 1.6.2 until we were confident that we'd fixed *all* the bugs, because otherwise they might have misled people into upgrading too soon. Holding 1.7 back for this isn't going to get this change done or to users any faster; it's just going to hold back all the other changes in 1.7. I do think we ought to aim to shorten our release cycle drastically. Like release 1.8 within 2-3 months after 1.7. But let's talk about that after 1.7 is out. -n

Hi, On Sat, Jan 5, 2013 at 2:38 PM, Nathaniel Smith <njs@pobox.com> wrote:

On Sat, Jan 5, 2013 at 12:32 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

Hi,

On Fri, Jan 4, 2013 at 4:54 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

Hi,

On Fri, Jan 4, 2013 at 4:01 PM, Andrew Collette <andrew.collette@gmail.com> wrote:

...

...

From a more basic perspective, I think that adding a number to an array should never raise an exception. I've not used any other language in which this behavior takes place. In C, you have rollover behavior, in IDL you roll over or clip, and in NumPy you either roll or upcast, depending on the version. IDL, etc. manage to handle things like max() or total() in a sensible (or at least defensible) fashion, and without raising an error.

That's a reasonable point.

Looks like we lost consensus.

What about returning to the 1.5 behavior instead?

If we do return to the 1.5 behavior, we would need to think about doing this in 1.7.

If there are a large number of 1.5.x and previous users who would upgrade to 1.7, leaving the 1.6 behavior in 1.7 will mean that they will get double the confusion:

1) The behavior has changed to something they weren't expecting 2) The behavior is going to change back very soon

I disagree. 1.7 is basically done, the 1.6 changes are out there already, and we still have work to do just to get consensus on how we want to handle this, plus implement the changes.

Basically, the way I think about it in general is, you have the first release that contains some bug, and then you have the first release that doesn't contain it. Minimizing the amount of *time* between those releases is important. Minimizing the *number of releases* in between does not -- according to that logic, we shouldn't have released 1.6.1 and 1.6.2 until we were confident that we'd fixed *all* the bugs, because otherwise they might have misled people into upgrading too soon. Holding 1.7 back for this isn't going to get this change done or to users any faster; it's just going to hold back all the other changes in 1.7.

I do think we ought to aim to shorten our release cycle drastically. Like release 1.8 within 2-3 months after 1.7. But let's talk about that after 1.7 is out.

Yes, I was imagining that resolving this question would be rather quick, and therefore any delay to 1.7 would be very small, but if it takes more than a few days to come to a solution, it's possible there would not be net benefit. To Ralf - I think a 'bugfix only' metric doesn't help all that much in this case, because if we revert to 1.5 behavior, this could very reasonably be described as a bugfix. Cheers, Matthew

Hi, On Sat, Jan 5, 2013 at 4:16 PM, Nathaniel Smith <njs@pobox.com> wrote:

On 5 Jan 2013 15:59, "Matthew Brett" <matthew.brett@gmail.com> wrote:

...

Hi,

On Sat, Jan 5, 2013 at 2:38 PM, Nathaniel Smith <njs@pobox.com> wrote:

...

On Sat, Jan 5, 2013 at 12:32 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

Hi,

On Fri, Jan 4, 2013 at 4:54 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

Hi,

On Fri, Jan 4, 2013 at 4:01 PM, Andrew Collette <andrew.collette@gmail.com> wrote:

...

>From a more basic perspective, I think that adding a number to an array should never raise an exception. I've not used any other language in which this behavior takes place. In C, you have rollover behavior, in IDL you roll over or clip, and in NumPy you either roll or upcast, depending on the version. IDL, etc. manage to handle things like max() or total() in a sensible (or at least defensible) fashion, and without raising an error.

That's a reasonable point.

Looks like we lost consensus.

What about returning to the 1.5 behavior instead?

If we do return to the 1.5 behavior, we would need to think about doing this in 1.7.

If there are a large number of 1.5.x and previous users who would upgrade to 1.7, leaving the 1.6 behavior in 1.7 will mean that they will get double the confusion:

1) The behavior has changed to something they weren't expecting 2) The behavior is going to change back very soon

I disagree. 1.7 is basically done, the 1.6 changes are out there already, and we still have work to do just to get consensus on how we want to handle this, plus implement the changes.

Basically, the way I think about it in general is, you have the first release that contains some bug, and then you have the first release that doesn't contain it. Minimizing the amount of *time* between those releases is important. Minimizing the *number of releases* in between does not -- according to that logic, we shouldn't have released 1.6.1 and 1.6.2 until we were confident that we'd fixed *all* the bugs, because otherwise they might have misled people into upgrading too soon. Holding 1.7 back for this isn't going to get this change done or to users any faster; it's just going to hold back all the other changes in 1.7.

I do think we ought to aim to shorten our release cycle drastically. Like release 1.8 within 2-3 months after 1.7. But let's talk about that after 1.7 is out.

Yes, I was imagining that resolving this question would be rather quick, and therefore any delay to 1.7 would be very small, but if it takes more than a few days to come to a solution, it's possible there would not be net benefit.

To Ralf - I think a 'bugfix only' metric doesn't help all that much in this case, because if we revert to 1.5 behavior, this could very reasonably be described as a bugfix.

It's not just the time to make the change, it's the time to make sure that we haven't created any new unexpected problems in the process. 1.7's already gone through many weeks of stabilization and testing. Really at this point the criterion isn't really even bug fixes only, but release critical bugs and doc fixes only (and the only RC bugs left should be ones discovered through the beta/rc cycle).

OK, I understand. This must influence the decision on what to do about the scalar casting. Further from 1.5.x makes reverting to 1.5.x less attractive. The longer the 1.6.x changes have been in the wild, the stronger the argument for leaving things as they are. Best, Matthew

Hi,

In fact in 1.6 there is no assignment of a dtype to '1' which makes the way 1.6 handles it consistent with the array rules:

I guess I'm a little out of my depth here... what are the array rules?

# Ah-hah, it looks like '1' has a uint8 dtype: (np.ones(2, dtype=np.uint8) / np.ones(2, dtype=np.uint8)).dtype == np.uint8 (np.ones(2, dtype=np.uint8) / 1).dtype == np.uint8 # But wait! No it doesn't! (np.ones(2, dtype=np.int8) / np.ones(2, dtype=np.uint8)).dtype == np.int16 (np.ones(2, dtype=np.int8) / 1).dtype == np.int8 # Apparently in this case it has an int8 dtype instead. (np.ones(2, dtype=np.int8) / np.ones(2, dtype=np.int8)).dtype == np.int8

Yes, this is a good point... I hadn't thought about whether it should be unsigned or signed. In the case of something like "1", where it's ambiguous, couldn't we prefer the sign of the other participant in the addition?

interaction between these. But in 1.6, as soon as you have a uint8 array, suddenly all the other precisions might spring magically into being at any moment.

I can see how this would be really annoying for someone close to the max memory on their machine.

So options: If we require that new dtypes shouldn't be suddenly introduced then we have to pick from: 1) a / 300 silently rolls over the 300 before attempting the operation (1.5-style)

Were people really not happy with this behavior? My reading of this thread: http://thread.gmane.org/gmane.comp.python.numeric.general/47986 was that the change was, although not an accident, certainly unexpected for most people. I don't have a strong preference either way, but I'm interested in why we're so eager to keep the "corrected" behavior.

2) a / 300 upcasts to machine precision (use the same rules for arrays and scalars) 3) a / 300 gives an error (the proposal you don't like)

If we instead treat a Python scalar like 1 as having the smallest precision dtype that can hold its value, then we have to accept either uint8 + 1 -> uint16 or int8 + 1 -> int16

Is there any consistent way we could prefer the "signedness" of the other participant? That would lead to both uint8 +1 -> uint8 and int8 + 1 -> int8.

Or there's the current code, whose behaviour no-one actually understands. (And I mean that both figuratively -- it's clearly confusing enough that people won't be able to remember it well in practice -- and literally -- even we developers don't know what it will do without running it to see.)

I agree the current behavior is confusing. Regardless of the details of what to do, I suppose my main objection is that, to me, it's really unexpected that adding a number to an array could result in an exception. Andrew

2013/1/5 Nathaniel Smith <njs@pobox.com>:

On Fri, Jan 4, 2013 at 5:25 PM, Andrew Collette <andrew.collette@gmail.com> wrote:

...

I agree the current behavior is confusing. Regardless of the details of what to do, I suppose my main objection is that, to me, it's really unexpected that adding a number to an array could result in an exception.

I think the main objection to the 1.5 behaviour was that it violated "Errors should never pass silently." (from 'import this'). Granted there are tons of places where numpy violates this but this is the one we're thinking about right now...

Okay, here's another idea I'll throw out, maybe it's a good compromise:

1) We go back to the 1.5 behaviour.

2) If this produces a rollover/overflow/etc., we signal that using the standard mechanisms (whatever is configured via np.seterr). So by default things like np.maximum(np.array([1, 2, 3], dtype=uint8), 256) would succeed (and produce [1, 2, 3] with dtype uint8), but also issue a warning that 256 had rolled over to become 0. Alternatively those who want to be paranoid could call np.seterr(overflow="raise") and then it would be an error.

That'd work for me as well. Although I'm not sure about the name "overflow", it sounds generic enough that it may be associated to many different situations. If I want to have an error but only for this very specific scenario (an "unsafe" cast in a mixed scalar/array operation), would that be possible? Also, do we all agree that "float32 array + float64 scalar" should cast the scalar to float32 (thus resulting in a float32 array as output) without warning, even if the scalar can't be represented exactly in float32? -=- Olivier

2013/1/6 Nathaniel Smith <njs@pobox.com>:

On Mon, Jan 7, 2013 at 1:43 AM, Olivier Delalleau <shish@keba.be> wrote:

...

2013/1/5 Nathaniel Smith <njs@pobox.com>:

...

On Fri, Jan 4, 2013 at 5:25 PM, Andrew Collette <andrew.collette@gmail.com> wrote:

...

I agree the current behavior is confusing. Regardless of the details of what to do, I suppose my main objection is that, to me, it's really unexpected that adding a number to an array could result in an exception.

I think the main objection to the 1.5 behaviour was that it violated "Errors should never pass silently." (from 'import this'). Granted there are tons of places where numpy violates this but this is the one we're thinking about right now...

Okay, here's another idea I'll throw out, maybe it's a good compromise:

1) We go back to the 1.5 behaviour.

2) If this produces a rollover/overflow/etc., we signal that using the standard mechanisms (whatever is configured via np.seterr). So by default things like np.maximum(np.array([1, 2, 3], dtype=uint8), 256) would succeed (and produce [1, 2, 3] with dtype uint8), but also issue a warning that 256 had rolled over to become 0. Alternatively those who want to be paranoid could call np.seterr(overflow="raise") and then it would be an error.

That'd work for me as well. Although I'm not sure about the name "overflow", it sounds generic enough that it may be associated to many different situations. If I want to have an error but only for this very specific scenario (an "unsafe" cast in a mixed scalar/array operation), would that be possible?

I suggested "overflow" because that's how we signal rollover in general right now:

In [5]: np.int8(100) * np.int8(2) /home/njs/.user-python2.7-64bit/bin/ipython:1: RuntimeWarning: overflow encountered in byte_scalars #!/home/njs/.user-python2.7-64bit/bin/python Out[5]: -56

Two caveats on this: One, right now this is only implemented for scalars, not arrays -- which is bug #593 -- and two, I actually agree (?) that integer rollover and float overflow are different things we should probably add a new category to np.seterr() for integer rollover specifically.

But the proposal here is that we not add a specific category for "unsafe cast" (which we would then have to define!), but instead just signal it using the standard mechanisms for the particular kind of corruption that happened. (Which right now is overflow, and might become something else later.)

Hehe, I didn't even know there was supposed to be a warning for arrays... Ok. But I'm not convinced that re-using the "overflow" category is a good idea, because to me the overflow is typically associated to the result of an operation (when it goes beyond the dtype's supported range), while here the problem is with the unsafe cast an input (even if it makes no difference for addition, it does for some other ufuncs). I may also want to have different error settings for operation overflow vs. input overflow. It may just be me though... let's see what others think about it.

...

Also, do we all agree that "float32 array + float64 scalar" should cast the scalar to float32 (thus resulting in a float32 array as output) without warning, even if the scalar can't be represented exactly in float32?

I guess for consistency, if this proposal is adopted then a float64 which ends up getting cast to 'inf' or 0.0 should trigger an overflow or underflow warning respectively... e.g.:

In [12]: np.float64(1e300) Out[12]: 1.0000000000000001e+300

In [13]: np.float32(_12) Out[13]: inf

...but otherwise I think yes we agree.

Sounds good to me. -=- Olivier

Hi, On Fri, Jan 4, 2013 at 5:25 PM, Andrew Collette <andrew.collette@gmail.com> wrote:

I agree the current behavior is confusing. Regardless of the details of what to do, I suppose my main objection is that, to me, it's really unexpected that adding a number to an array could result in an exception.

I realized when I thought about it, that I did not have a clear idea of your exact use case. How does the user specify the thing to add, and why do you need to avoid an error in the case that adding would overflow the type? Would you mind giving an idiot-level explanation? Best, Matthew

Hi, On Mon, Jan 7, 2013 at 4:33 PM, Andrew Collette <andrew.collette@gmail.com> wrote:

Hi Matthew,

...

I realized when I thought about it, that I did not have a clear idea of your exact use case. How does the user specify the thing to add, and why do you need to avoid an error in the case that adding would overflow the type? Would you mind giving an idiot-level explanation?

There isn't a specific use case I had in mind... from a developer's perspective, what bothers me about the proposed behavior is that every use of "+" on user-generated input becomes a time bomb. Since h5py deals with user-generated files, I have to deal with all kinds of dtypes, including low-precision ones like int8/uint8. They come from user-supplied function and methods arguments, sure, but also from datasets in files; attributes; virtually everywhere.

I suppose what I'm really asking is that numpy provides (continues to provide) a default rule in this situation, as does every other scientific language I've used. One reason to avoid a ValueError in favor of default behavior (in addition to the large amount of work required to check every use of "+") is so there's an established behavior users know to expect.

For example, one feature we're thinking of implementing involves adding an offset to a dataset when it's read. Should we roll over? Upcast? It seems to me there's great value in being able to say "We do what numpy does." If numpy doesn't answer the question, everybody makes up their own rules. There are certainly cases where the answer is obvious to the application: you have a huge number of int8's and don't want to upcast. Or you don't want to lose precision. But if numpy provides a default rule, nobody is prevented from making careful choices based on their application's requirements, and there's the additional value of having an common, documented default behavior.

Just to be clear, you mean you might have something like this? def my_func('array_name', some_offset): arr = load_somehow('array_name') # dtype hitherto unknown return arr + some_offset ? And the problem is that it fails late? Is it really better that something bad happens for the addition than that it raises an error? You'll also often get an error when trying to add structured dtypes, but maybe you cant return these from a 'load'? Best, Matthew

Hi, On Mon, Jan 7, 2013 at 8:50 PM, Andrew Collette <andrew.collette@gmail.com> wrote:

Hi Matthew,

...

Just to be clear, you mean you might have something like this?

def my_func('array_name', some_offset): arr = load_somehow('array_name') # dtype hitherto unknown return arr + some_offset

? And the problem is that it fails late? Is it really better that something bad happens for the addition than that it raises an error?

You'll also often get an error when trying to add structured dtypes, but maybe you cant return these from a 'load'?

In this specific case I would like to just use "+" and say "We add your offset using the NumPy rules," which is a problem if there are no NumPy rules for addition in the specific case where some_offset happens to be a scalar and not an array, and also slightly larger than arr.dtype can hold. I personally prefer upcasting to some reasonable type big enough to hold some_offset, as I described earlier, although that's not crucial.

But I think we're getting a little caught up in the details of this example. My basic point is: yes, people should be careful to check dtypes, etc. where it's important to their application; but people who want to rely on some reasonable NumPy-supplied default behavior should be excused from doing so.

For myself, I find detailed examples helpful, because I find it difficult to think about more general rules without applying them to practical cases. In this case I think you'd probably agree it would be reasonable to raise an error - all other things being equal? Can you think of another practical case where it would be reasonably clear that it was the wrong thing to do? Cheers, Matthew

Hi, On Mon, Jan 7, 2013 at 9:18 PM, Andrew Collette <andrew.collette@gmail.com> wrote:

Hi Matthew,

...

In this case I think you'd probably agree it would be reasonable to raise an error - all other things being equal?

No, I don't agree. I want there to be some default semantics I can rely on. Preferably, I want it to do the same thing it would do if some_offset were an array with element-by-element offsets, which is the current behavior of numpy 1.6 if you assume a reasonable dtype for some_offset.

Ah - well - I only meant that raising an error in the example would be no more surprising than raising an error at the python prompt. Do you agree with that? I mean, if the user knew that:

...

...

np.array([1], dtype=np.int8) + 128

would raise an error, they'd probably expect your offset routine to do the same.

...

Can you think of another practical case where it would be reasonably clear that it was the wrong thing to do?

I consider "myarray + constant -> Error" clearly wrong no matter what the context. I've never seen it in any other analysis language I've used. But it's also possible that I'm alone in this... I haven't seen many other people here arguing against the change.

I agree it kind of feels funny, but that's why I wanted to ask you for some silly but specific example where the funniness would be more apparent. Cheers, Matthew

Hi, On Mon, Jan 7, 2013 at 10:03 PM, Andrew Collette <andrew.collette@gmail.com> wrote:

Hi Matthew,

...

Ah - well - I only meant that raising an error in the example would be no more surprising than raising an error at the python prompt. Do you agree with that? I mean, if the user knew that:

...

...

...

np.array([1], dtype=np.int8) + 128

would raise an error, they'd probably expect your offset routine to do the same.

I think they would be surprised in both cases, considering this works fine:

np.array([1], dtype=np.int8) + np.array([128])

...

I agree it kind of feels funny, but that's why I wanted to ask you for some silly but specific example where the funniness would be more apparent.

Here are a couple of examples I slapped together, specifically highlighting the value of the present (or similar) upcasting behavior. Granted, they are contrived and can all be fixed by conditional code, but this is my best effort at illustrating the "real-world" problems people may run into.

Note that there is no easy way for the user to force upcasting to avoid the error, unless e.g. an "upcast" keyword were added to these functions, or code added to inspect the data dtype and use numpy.add to simulate the current behavior.

def map_heights(self, dataset_name, heightmap): """ Correct altitudes by adding a custom heightmap

dataset_name: Name of HDF5 dataset containing altitude data heightmap: Corrections in meters. Must match shape of the dataset (or be a scalar). """ # TODO: scattered reports of errors when a constant heightmap value is used

return self.f[dataset_name][...] + heightmap

def perform_analysis(self, dataset_name, kernel_offset=128): """ Apply Frobnication analysis, using optional linear offset

dataset_name: Name of dataset in file kernel_offset: Optional sequencing parameter. Must be a power of 2 and at least 16 (default 128) """ # TODO: people report certain files frobnicate fine in IDL but not in Python...

import frob data = self.f[dataset_name][...] try: return frob.frobnicate(data + kernel_offset) except ValueError: raise AnalysisFailed("Invalid input data")

Thanks - I know it seems silly - but it is helpful. There are two separate issues though: 1) Is the upcasting behavior of 1.6 better than the overflow behavior of 1.5? 2) If the upcasting of 1.6 is bad, is it better to raise an error or silently overflow, as in 1.5? Taking 2) first, in this example:

return self.f[dataset_name][...] + heightmap

assuming it is not going to upcast, would you rather it overflow than raise an error? Why? The second seems more explicit and sensible to me. For 1) - of course the upcasting in 1.6 is only going to work some of the time. For example: In [2]: np.array([127], dtype=np.int8) * 1000 Out[2]: array([-4072], dtype=int16) So - you'll get something, but there's a reasonable chance you won't get what you were expecting. Of course that is true for 1.5 as well, but at least the rule there is simpler and so easier - in my opinion - to think about. Best, Matthew

Hi, On Mon, Jan 7, 2013 at 10:58 PM, Andrew Collette <andrew.collette@gmail.com> wrote:

Hi,

...

Taking 2) first, in this example:

...

return self.f[dataset_name][...] + heightmap

assuming it is not going to upcast, would you rather it overflow than raise an error? Why? The second seems more explicit and sensible to me.

Yes, I think this (the 1.5 overflow behavior) was a bit odd, if easy to understand.

...

For 1) - of course the upcasting in 1.6 is only going to work some of the time. For example:

In [2]: np.array([127], dtype=np.int8) * 1000 Out[2]: array([-4072], dtype=int16)

So - you'll get something, but there's a reasonable chance you won't get what you were expecting. Of course that is true for 1.5 as well, but at least the rule there is simpler and so easier - in my opinion - to think about.

Part of what my first example was trying to demonstrate was that the function author assumed arrays and scalars obeyed the same rules for addition.

For example, if data were int8 and heightmap were an int16 array with a max value of 32767, and the data had a max value in the same spot with e.g. 10, then the addition would overflow at that position, even with the int16 result. That's how array addition works in numpy, and as I understand it that's not slated to change.

But when we have a scalar of value 32767 (which fits in int16 but not int8), we are proposing instead to do nothing under the assumption that it's an error.

In summary: yes, there are some odd results, but they're consistent with the rules for addition elsewhere in numpy, and I would prefer that to treating this case as an error.

I think you are voting strongly for the current casting rules, because they make it less obvious to the user that scalars are different from arrays. Returning to the question of 1.5 behavior vs the error - I think you are saying you prefer the 1.5 silent-but-deadly approach to the error, but I think I still haven't grasped why. Maybe someone else can explain it? The holiday has not been good to my brain. Best, Matthew

2013/1/8 Andrew Collette <andrew.collette@gmail.com>:

Hi,

...

I think you are voting strongly for the current casting rules, because they make it less obvious to the user that scalars are different from arrays.

Maybe this is the source of my confusion... why should scalars be different from arrays? They should follow the same rules, as closely as possible. If a scalar value would fit in an int16, why not add it using the rules for an int16 array?

As I mentioned in another post, I also agree that it would make things simpler and safer to just yield the same result as if we were using a one-element array. My understanding of the motivation for the rule "scalars do not upcast arrays unless they are of a fundamentally different type" is that it avoids accidentally upcasting arrays in operations like "x + 1" (for instance if x is a float32 array, the upcast would yield a float64 result, and if x is an int16, it would yield int64), which may waste memory. I find it a useful feature, however I'm not sure it's worth the headaches it can lead to. However, my first reaction at the idea of dropping this rule altogether is that it would lead to a long and painful deprecation process. I may be wrong though, I really haven't thought about it much. -=- Olivier

On Tue, Jan 8, 2013 at 7:28 PM, Alan G Isaac <alan.isaac@gmail.com> wrote:

On 1/8/2013 1:48 PM, Olivier Delalleau wrote:

...

As I mentioned in another post, I also agree that it would make things simpler and safer to just yield the same result as if we were using a one-element array.

Yes! Anything else is going to drive people insane, especially new users.

New users don't use narrow-width dtypes... it's important to remember in this discussion that in numpy, non-standard dtypes only arise when users explicitly request them, so there's some expressed intention there that we want to try and respect. (As opposed to the type associated with Python manifest constants like the "2" in "2 * a", which probably no programmer looked at and thought "hmm, what I want here is 2-as-an-int64".) -n

On Tue, Jan 8, 2013 at 12:43 PM, Alan G Isaac <alan.isaac@gmail.com> wrote:

...

New users don't use narrow-width dtypes... it's important to remember

1. I think the first statement is wrong. Control over dtypes is a good reason for a new use to consider NumPy.

Absolutely.

Because NumPy supports broadcasting, it is natural for array-array operations and scalar-array operations to be consistent. I believe anything else will be too confusing.

Theoretically true -- but in practice, the problem arrises because it is easy to write literals with the standard python scalars, so one is very likely to want to do: arr = np.zeros((m,n), dtype=np.uint8) arr += 3 and not want an upcast. I don't think we want to require that to be spelled: arr += np.array(3, dtype=np.uint8) so that defines desired behaviour for array<->scalar. but what should this do? arr1 = np.zeros((m,n), dtype=np.uint8) arr2 = np.zeros((m,n), dtype=np.uint16) arr1 + arr2 or arr2 + arr1 upcast in both cases? use the type of the left operand? raise an exception? matching the array<-> scalar approach would mean always keeping the smallest type, which is unlikely to be what is wanted. Having it be dependent on order would be really ripe fro confusion. raising an exception might have been the best idea from the beginning. (though I wouldn't want that in the array<-> scalar case). So perhaps having a scalar array distinction, while quite impure, is the best compromise. NOTE: no matter how you slice it, at some point reducing operations produce something different (that can no longer be reduced), so I do think it would be nice for rank-zero arrays and scalars to be the same thing (in this regard and others) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 8 Jan 2013 17:24, "Andrew Collette" <andrew.collette@gmail.com> wrote:

Hi,

...

I think you are voting strongly for the current casting rules, because they make it less obvious to the user that scalars are different from arrays.

Maybe this is the source of my confusion... why should scalars be different from arrays? They should follow the same rules, as closely as possible. If a scalar value would fit in an int16, why not add it using the rules for an int16 array?

The problem is that rule for arrays - and for every other party of numpy in general - are that we *don't* pick types based on values. Numpy always uses input types to determine output types, not input values. # This value fits in an int8 In [5]: a = np.array([1]) # And yet... In [6]: a.dtype Out[6]: dtype('int64') In [7]: small = np.array([1], dtype=np.int8) # Computing 1 + 1 doesn't need a large integer... but we use one In [8]: (small + a).dtype Out[8]: dtype('int64') Python scalars have an unambiguous types: a Python 'int' is a C 'long', and a Python 'float' is a C 'double'. And these are the types that np.array() converts them to. So it's pretty unambiguous that "using the same rules for arrays and scalars" would mean, ignore the value of the scalar, and in expressions like np.array([1], dtype=np.int8) + 1 we should always upcast to int32/int64. The problem is that this makes working with narrow types very awkward for no real benefit, so everyone pretty much seems to want *some* kind of special case. These are both absolutely special cases: numarray through 1.5: in a binary operation, if one operand has ndim==0 and the other has ndim>0, ignore the width of the ndim==0 operand. 1.6, your proposal: in a binary operation, if one operand has ndim==0 and the other has ndim>0, downcast the ndim==0 item to the smallest width that is consistent with its value and the other operand's type. -n

On Tue, Jan 8, 2013 at 9:14 PM, Andrew Collette <andrew.collette@gmail.com> wrote:

Hi Nathaniel,

(Responding to both your emails)

...

The problem is that rule for arrays - and for every other party of numpy in general - are that we *don't* pick types based on values. Numpy always uses input types to determine output types, not input values.

Yes, of course... array operations are governed exclusively by their dtypes. It seems to me that, using the language of the bug report (2878), if we have this:

result = arr + scalar

I would argue that our job is, rather than to pick result.dtype, to pick scalar.dtype, and apply the normal rules for array operations.

Okay, but we already have unambiguous rules for picking scalar.dtype: you use whatever width the underlying type has, so it'd always be np.int_ or np.float64. Those are the normal rules for picking dtypes. I'm just trying to make clear that what you're arguing for is also a very special case, which also violates the rules numpy uses everywhere else. That doesn't mean we should rule it out ("Special cases aren't special enough to break the rules. / Although practicality beats purity."), but claiming that it is just "the normal rules" while everything else is a "special case" is rhetorically unhelpful.

...

So it's pretty unambiguous that "using the same rules for arrays and scalars" would mean, ignore the value of the scalar, and in expressions like np.array([1], dtype=np.int8) + 1 we should always upcast to int32/int64.

Ah, but that's my point: we already, in 1.6, ignore the intrinsic width of the scalar and effectively substitute one based on it's value:

...

...

...

a = np.array([1], dtype=int8) (a + 1).dtype dtype('int8') (a + 1000).dtype dtype('int16') (a + 90000).dtype dtype('int32') (a + 2**40).dtype dtype('int64')

Sure. But the only reason this is in 1.6 is that the person who made the change never mentioned it to anyone else, so it wasn't noticed until after 1.6 came out. If it had gone through proper review/mailing list discussion (like we're doing now) then it's very unlikely it would have gone in in its present form.

...

1.6, your proposal: in a binary operation, if one operand has ndim==0 and the other has ndim>0, downcast the ndim==0 item to the smallest width that is consistent with its value and the other operand's type.

Yes, exactly. I'm not trying to propose a completely new behavior: as I mentioned (although very far upthread), this is the mental model I had of how things worked in 1.6 already.

...

New users don't use narrow-width dtypes... it's important to remember in this discussion that in numpy, non-standard dtypes only arise when users explicitly request them, so there's some expressed intention there that we want to try and respect.

I would respectfully disagree. One example I cited was that when dealing with HDF5, it's very common to get int16's (and even int8's) when reading from a file because they are used to save disk space. All a new user has to do to get int8's from a file they got from someone else is:

...

...

...

data = some_hdf5_file['MyDataset'][...]

This is a general issue applying to data which is read from real-world external sources. For example, digitizers routinely represent their samples as int8's or int16's, and you apply a scale and offset to get a reading in volts.

This particular case is actually handled fine by 1.5, because int array + float scalar *does* upcast to float. It's width that's ignored (int8 versus int32), not the basic "kind" of data (int versus float). But overall this does sound like a problem -- but it's not a problem with the scalar/array rules, it's a problem with working with narrow width data in general. There's a good argument to be made that data files should be stored in compressed form, but read in in full-width form, exactly to avoid the problems that arise when trying to manipulate narrow-width representations. Suppose your scale and offset *were* integers, so that the "kind" casting rules didn't get invoked. Even if this were the case, then the rules you're arguing for would not actually solve your problem at all. It'd be very easy to have, say, scale=100, offset=100, both of which fit fine in an int8... but actually performing the scaling/offseting in an int8 would still be a terrible idea! The problem you're talking about is picking the correct width for an *operation*, and futzing about with picking the dtypes of *one input* to that operation is not going to help; it's like trying to ensure your house won't fall down by making sure the doors are really sturdy. -n

On 01/09/2013 06:22 PM, Chris Barker - NOAA Federal wrote:

On Wed, Jan 9, 2013 at 7:09 AM, Nathaniel Smith <njs@pobox.com> wrote:

...

...

This is a general issue applying to data which is read from real-world external sources. For example, digitizers routinely represent their samples as int8's or int16's, and you apply a scale and offset to get a reading in volts.

This particular case is actually handled fine by 1.5, because int array + float scalar *does* upcast to float. It's width that's ignored (int8 versus int32), not the basic "kind" of data (int versus float).

But overall this does sound like a problem -- but it's not a problem with the scalar/array rules, it's a problem with working with narrow width data in general.

Exactly -- this is key. details asside, we essentially have a choice between an approach that makes it easy to preserver your values -- upcasting liberally, or making it easy to preserve your dtype -- requiring users to specifically upcast where needed.

IIRC, our experience with earlier versions of numpy (and Numeric before that) is that all too often folks would choose a small dtype quite deliberately, then have it accidentally upcast for them -- this was determined to be not-so-good behavior.

I think the HDF (and also netcdf...) case is a special case -- the small dtype+scaling has been chosen deliberately by whoever created the data file (to save space), but we would want it generally opaque to the consumer of the file -- to me, that means the issue should be adressed by the file reading tools, not numpy. If your HDF5 reader chooses the the resulting dtype explicitly, it doesn't matter what numpy's defaults are. If the user wants to work with the raw, unscaled arrays, then they should know what they are doing.

+1. I think h5py should consider: File("my.h5")['int8_dset'].dtype == int64 File("my.h5", preserve_dtype=True)['int8_dset'].dtype == int8 Dag Sverre

On Thu, Jan 17, 2013 at 6:26 AM, Matthew Brett <matthew.brett@gmail.com> wrote:

I am starting to wonder if we should aim for making

* scalar and array casting rules the same; * Python int / float scalars become int32 / 64 or float64;

aren't they already? I'm not sure what you are proposing.

This has the benefit of being very easy to understand and explain. It makes dtypes predictable in the sense they don't depend on value.

That is key -- I don't think casting should ever depend on value.

Those wanting to maintain - say - float32 will need to cast scalars to float32.

Maybe the use-cases motivating the scalar casting rules - maintaining float32 precision in particular - can be dealt with by careful casting of scalars, throwing the burden onto the memory-conscious to maintain their dtypes.

IIRC this is how it worked "back in the day" (the Numeric day? -- and I'm pretty sure that in the long run it worked out badly. the core problem is that there are only python literals for a couple types, and it was oh so easy to do things like: my_arr = np,zeros(shape, dtype-float32) another_array = my_array * 4.0 and you'd suddenly get a float64 array. (of course, we already know all that..) I suppose this has the up side of being safe, and having scalar and array casting rules be the same is of course appealing, but you use a particular size dtype for a reason,and it's a real pain to maintain it. Casual users will use the defaults that match the Python types anyway. So in the in the spirit of "practicality beats purity" -- I"d like accidental upcasting to be hard to do. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Hi, On Fri, Jan 18, 2013 at 1:04 AM, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:

On Thu, Jan 17, 2013 at 6:26 AM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

I am starting to wonder if we should aim for making

* scalar and array casting rules the same; * Python int / float scalars become int32 / 64 or float64;

aren't they already? I'm not sure what you are proposing.

Sorry - yes that is what they are already, this sentence refers back to an earlier suggestion of mine on the thread, which I am discarding.

...

This has the benefit of being very easy to understand and explain. It makes dtypes predictable in the sense they don't depend on value.

That is key -- I don't think casting should ever depend on value.

...

Those wanting to maintain - say - float32 will need to cast scalars to float32.

Maybe the use-cases motivating the scalar casting rules - maintaining float32 precision in particular - can be dealt with by careful casting of scalars, throwing the burden onto the memory-conscious to maintain their dtypes.

IIRC this is how it worked "back in the day" (the Numeric day? -- and I'm pretty sure that in the long run it worked out badly. the core problem is that there are only python literals for a couple types, and it was oh so easy to do things like:

my_arr = np,zeros(shape, dtype-float32)

another_array = my_array * 4.0

and you'd suddenly get a float64 array. (of course, we already know all that..) I suppose this has the up side of being safe, and having scalar and array casting rules be the same is of course appealing, but you use a particular size dtype for a reason,and it's a real pain to maintain it.

Yes, I do understand that. The difference - as I understand it - is that back in the day, numeric did not have the the float32 etc scalars, so you could not do: another_array = my_array * np.float32(4.0) (please someone correct me if I'm wrong).

Casual users will use the defaults that match the Python types anyway.

I think what we are reading in this thread is that even experienced numpy users can find the scalar casting rules surprising, and that's a real problem, it seems to me. The person with a massive float32 array certainly should have the ability to control upcasting, but I think the default should be the least surprising thing, and that, it seems to me, is for the casting rules to be the same for arrays and scalars. In the very long term. Cheers, Matthew

On Thu, Jan 17, 2013 at 5:34 PM, Olivier Delalleau <shish@keba.be> wrote:

...

Yes, I do understand that. The difference - as I understand it - is that back in the day, numeric did not have the the float32 etc scalars, so you could not do:

another_array = my_array * np.float32(4.0)

(please someone correct me if I'm wrong).

correct, it didn't have any scalars, but you could (and had to) still do something like: another_array = my_array * np.array(4.0, dtype=np.float32) a bit more verbose, but the verbosity wasn't the key issue -- it was doing anything special at all.

...

...

Casual users will use the defaults that match the Python types anyway.

I think what we are reading in this thread is that even experienced numpy users can find the scalar casting rules surprising, and that's a real problem, it seems to me.

for sure -- but it's still relevant -- if you want non-default types, you need to understand the rules an be more careful.

...

The person with a massive float32 array certainly should have the ability to control upcasting, but I think the default should be the least surprising thing, and that, it seems to me, is for the casting rules to be the same for arrays and scalars. In the very long term.

"A foolish consistency is the hobgoblin of little minds" -- just kidding. But in all seriousness -- accidental upcasting really was a big old pain back in the day -- we are not making this up. We re using the term "least surprising", but I now I was often surprised that I had lost my nice compact array. The user will need to think about it no matter how you slice it.

Since most of the related issues seem to come from integer arrays, a middle-ground may be the following: - Integer-type arrays get upcasted by scalars as in usual array / array operations. - Float/Complex-type arrays don't get upcasted by scalars except when the scalar is complex and the array is float.

I'm not sure that integer arrays are any more of an an issue, and having integer types and float typed behave differently is really asking for trouble! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Fri, Jan 18, 2013 at 4:39 AM, Olivier Delalleau <shish@keba.be> wrote:

Le vendredi 18 janvier 2013, Chris Barker - NOAA Federal a écrit :

If you check again the examples in this thread exhibiting surprising / unexpected behavior, you'll notice most of them are with integers. The tricky thing about integers is that downcasting can dramatically change your result. With floats, not so much: you get approximation errors (usually what you want) and the occasional nan / inf creeping in (usally noticeable).

fair enough. However my core argument is that people use non-standard (usually smaller) dtypes for a reason, and it should be hard to accidentally up-cast. This is in contrast with the argument that accidental down-casting can produce incorrect results, and thus it should be hard to accidentally down-cast -- same argument whether the incorrect results are drastic or not.... It's really a question of which of these we think should be prioritized. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

2013/1/18 Matthew Brett <matthew.brett@gmail.com>:

Hi,

On Fri, Jan 18, 2013 at 7:58 PM, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:

...

On Fri, Jan 18, 2013 at 4:39 AM, Olivier Delalleau <shish@keba.be> wrote:

...

Le vendredi 18 janvier 2013, Chris Barker - NOAA Federal a écrit :

...

If you check again the examples in this thread exhibiting surprising / unexpected behavior, you'll notice most of them are with integers. The tricky thing about integers is that downcasting can dramatically change your result. With floats, not so much: you get approximation errors (usually what you want) and the occasional nan / inf creeping in (usally noticeable).

fair enough.

However my core argument is that people use non-standard (usually smaller) dtypes for a reason, and it should be hard to accidentally up-cast.

This is in contrast with the argument that accidental down-casting can produce incorrect results, and thus it should be hard to accidentally down-cast -- same argument whether the incorrect results are drastic or not....

It's really a question of which of these we think should be prioritized.

After thinking about it for a while, it seems to me Olivier's suggestion is a good one.

The rule becomes the following:

array + scalar casting is the same as array + array casting except array + scalar casting does not upcast floating point precision of the array.

Am I right (Chris, Perry?) that this deals with almost all your cases? Meaning that it is upcasting of floats that is the main problem, not upcasting of (u)ints?

This rule seems to me not very far from the current 1.6 behavior; it upcasts more - but the dtype is now predictable. It's easy to explain. It avoids the obvious errors that the 1.6 rules were trying to avoid. It doesn't seem too far to stretch to make a distinction between rules about range (ints) and rules about precision (float, complex).

What do you'all think?

Personally, I think the main issue with my suggestion is that it seems hard to go there from the current behavior -- without potentially breaking existing code in non-obvious ways. The main problematic case I foresee is the typical "small_int_array + 1", which would get upcasted while it wasn't the case before (neither in 1.5 nor in 1.6). That's why I think Nathaniel's proposal is more practical. -=- Olivier

I'd like to echo what Chris is saying. It was a big annoyance with Numeric to make it so hard to preserve the array type in ordinary expressions. Perry On Jan 17, 2013, at 8:04 PM, Chris Barker - NOAA Federal wrote:

On Thu, Jan 17, 2013 at 6:26 AM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

I am starting to wonder if we should aim for making

* scalar and array casting rules the same; * Python int / float scalars become int32 / 64 or float64;

aren't they already? I'm not sure what you are proposing.

...

This has the benefit of being very easy to understand and explain. It makes dtypes predictable in the sense they don't depend on value.

That is key -- I don't think casting should ever depend on value.

...

Those wanting to maintain - say - float32 will need to cast scalars to float32.

Maybe the use-cases motivating the scalar casting rules - maintaining float32 precision in particular - can be dealt with by careful casting of scalars, throwing the burden onto the memory-conscious to maintain their dtypes.

IIRC this is how it worked "back in the day" (the Numeric day? -- and I'm pretty sure that in the long run it worked out badly. the core problem is that there are only python literals for a couple types, and it was oh so easy to do things like:

my_arr = np,zeros(shape, dtype-float32)

another_array = my_array * 4.0

and you'd suddenly get a float64 array. (of course, we already know all that..) I suppose this has the up side of being safe, and having scalar and array casting rules be the same is of course appealing, but you use a particular size dtype for a reason,and it's a real pain to maintain it.

Casual users will use the defaults that match the Python types anyway.

So in the in the spirit of "practicality beats purity" -- I"d like accidental upcasting to be hard to do.

-Chris

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Tue, 2013-01-08 at 19:59 +0000, Nathaniel Smith wrote:

On 8 Jan 2013 17:24, "Andrew Collette" <andrew.collette@gmail.com> wrote:

...

Hi,

...

I think you are voting strongly for the current casting rules, because they make it less obvious to the user that scalars are different from arrays.

Maybe this is the source of my confusion... why should scalars be different from arrays? They should follow the same rules, as closely as possible. If a scalar value would fit in an int16, why not add it using the rules for an int16 array?

The problem is that rule for arrays - and for every other party of numpy in general - are that we *don't* pick types based on values. Numpy always uses input types to determine output types, not input values.

# This value fits in an int8 In [5]: a = np.array([1])

# And yet... In [6]: a.dtype Out[6]: dtype('int64')

In [7]: small = np.array([1], dtype=np.int8)

# Computing 1 + 1 doesn't need a large integer... but we use one In [8]: (small + a).dtype Out[8]: dtype('int64')

Python scalars have an unambiguous types: a Python 'int' is a C 'long', and a Python 'float' is a C 'double'. And these are the types that np.array() converts them to. So it's pretty unambiguous that "using the same rules for arrays and scalars" would mean, ignore the value of the scalar, and in expressions like np.array([1], dtype=np.int8) + 1 we should always upcast to int32/int64. The problem is that this makes working with narrow types very awkward for no real benefit, so everyone pretty much seems to want *some* kind of special case. These are both absolutely special cases:

numarray through 1.5: in a binary operation, if one operand has ndim==0 and the other has ndim>0, ignore the width of the ndim==0 operand.

1.6, your proposal: in a binary operation, if one operand has ndim==0 and the other has ndim>0, downcast the ndim==0 item to the smallest width that is consistent with its value and the other operand's type.

Well, that leaves the maybe not quite implausible proposal of saying that numpy scalars behave like arrays with ndim>0, but python scalars behave like they do in 1.6. to allow for easier working with narrow types. Sebastian

-n _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On 01/08/2013 06:20 PM, Andrew Collette wrote:

Hi,

...

I think you are voting strongly for the current casting rules, because they make it less obvious to the user that scalars are different from arrays.

Maybe this is the source of my confusion... why should scalars be different from arrays? They should follow the same rules, as closely

Scalars (as in, Python float/int) are inherently different because the user didn't specify a dtype. For an array, there was always *some* point where the user chose, explicitly or implicitly, a dtype.

as possible. If a scalar value would fit in an int16, why not add it using the rules for an int16 array?

So you are saying that, for an array x, you want x + random.randint(100000) to produce an array with a random dtype? So that after carefully testing that your code works, suddenly a different draw (or user input, or whatever) causes a different set of dtypes to ripple through your entire program? To me this is something that must be avoided at all costs. It's hard enough to reason about the code one writes without throwing in complete randomness (by which I mean, types determined by values). Dag Sverre

...

Returning to the question of 1.5 behavior vs the error - I think you are saying you prefer the 1.5 silent-but-deadly approach to the error, but I think I still haven't grasped why. Maybe someone else can explain it? The holiday has not been good to my brain.

In a strict choice between 1.5-behavior and errors, I'm not sure which one I would pick. I don't think either is particularly useful. Of course, other members of the community would likely have a different view, especially those who got used to the 1.5 behavior.

Andrew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Hi Dag,

So you are saying that, for an array x, you want

x + random.randint(100000)

to produce an array with a random dtype?

Under the proposed behavior, depending on the dtype of x and the value from random, this would sometimes add-with-rollover and sometimes raise ValueError. Under the 1.5 behavior, it would always add-with-rollover and preserve the type of x. Under the 1.6 behavior, it produces a range of dtypes, each of which is at least large enough to hold the random int. Personally, I prefer the third option, but I strongly prefer either the second or the third to the first. Andrew

Hi,

Keep in mind that in the third option (current 1.6 behavior) the dtype is large enough to hold the random number, but not necessarily to hold the result. So for instance if x is an int16 array with only positive values, the result of this addition may contain negative values (or not, depending on the number being drawn). That's the part I feel is flawed with this behavior, it is quite unpredictable.

Yes, certainly. But in either the proposed or 1.5 behavior, if the values in x are close to the limits of the type, this can happen also. Andrew

Hi Olivier,

...

Yes, certainly. But in either the proposed or 1.5 behavior, if the values in x are close to the limits of the type, this can happen also.

My previous email may not have been clear enough, so to be sure: in my above example, if the random number is 30000, then the result may contain negative values (int16). If the random number is 50000, then the result will only contain positive values (upcast to int32). Do you believe it is a good behavior?

Under the proposed behavior, if the random number is 30000, then you *still* may have negative values, and if it's 50000, you get ValueError. No, I don't think the behavior you outlined is particularly nice, but (1) it's consistent with array addition elsewhere, at least in my mind, and (2) I don't think that sometimes getting a ValueError is a big improvement. Although I still prefer automatic upcasting, this discussion is really making me see the value of a nice, simple rule like in 1.5. :) Andrew

Hi Dag,

But the default float dtype is double, and default integer dtype is at least int32.

So if you rely on NumPy-supplied default behaviour you are fine!

As I mentioned, this caught my interest because people routinely save data in HDF5 as int8 or int16 to save disk space. It's not at all unusual to end up with these precisions when you read from a file.

If you specify a smaller dtype for your arrays, you have some reason to do that.

In this case, the reason is that the person who gave me the file chose to store the data as e.g. int16. Good default semantics for things like addition make it easy to write generic code. Andrew