Mailman 3 Nice float -> integer conversion? - NumPy-Discussion

newer
Re: [Numpy-discussion] in the NA...

Nice float -> integer conversion?

Matthew Brett

11 Oct 2011 11 Oct '11

1:06 p.m.

Hi, Have I missed a fast way of doing nice float to integer conversion? By nice I mean, rounding to the nearest integer, converting NaN to 0, inf, -inf to the max and min of the integer range? The astype method and cast functions don't do what I need here: In [40]: np.array([1.6, np.nan, np.inf, -np.inf]).astype(np.int16) Out[40]: array([1, 0, 0, 0], dtype=int16) In [41]: np.cast[np.int16](np.array([1.6, np.nan, np.inf, -np.inf])) Out[41]: array([1, 0, 0, 0], dtype=int16) Have I missed something obvious? See y'all, Matthew

Show replies by thread

Derek Homeier

11 Oct 11 Oct

2:06 p.m.

On 11 Oct 2011, at 20:06, Matthew Brett wrote:

...

Have I missed a fast way of doing nice float to integer conversion?

By nice I mean, rounding to the nearest integer, converting NaN to 0, inf, -inf to the max and min of the integer range? The astype method and cast functions don't do what I need here:

In [40]: np.array([1.6, np.nan, np.inf, -np.inf]).astype(np.int16) Out[40]: array([1, 0, 0, 0], dtype=int16)

In [41]: np.cast[np.int16](np.array([1.6, np.nan, np.inf, -np.inf])) Out[41]: array([1, 0, 0, 0], dtype=int16)

Have I missed something obvious?

np.[a]round comes closer to what you wish (is there consensus that NaN should map to 0?), but not quite there, and it's not really consistent either! In [42]: c = np.zeros(4, np.int16) In [43]: d = np.zeros(4, np.int32) In [44]: np.around([1.6,np.nan,np.inf,-np.inf], out=c) Out[44]: array([2, 0, 0, 0], dtype=int16) In [45]: np.around([1.6,np.nan,np.inf,-np.inf], out=d) Out[45]: array([ 2, -2147483648, -2147483648, -2147483648], dtype=int32) Perhaps a starting point to harmonise this behaviour and get it closer to your expectations (it still would not be really nice having to define the output array first, I guess)... Cheers, Derek

josef.pktd＠gmail.com

2:18 p.m.

On Tue, Oct 11, 2011 at 3:06 PM, Derek Homeier wrote:

...

On 11 Oct 2011, at 20:06, Matthew Brett wrote:

...
Have I missed a fast way of doing nice float to integer conversion?

By nice I mean, rounding to the nearest integer, converting NaN to 0, inf, -inf to the max and min of the integer range? The astype method and cast functions don't do what I need here:

In [40]: np.array([1.6, np.nan, np.inf, -np.inf]).astype(np.int16) Out[40]: array([1, 0, 0, 0], dtype=int16)

In [41]: np.cast[np.int16](np.array([1.6, np.nan, np.inf, -np.inf])) Out[41]: array([1, 0, 0, 0], dtype=int16)

Have I missed something obvious?

np.[a]round comes closer to what you wish (is there consensus that NaN should map to 0?), but not quite there, and it's not really consistent either!

In [42]: c = np.zeros(4, np.int16) In [43]: d = np.zeros(4, np.int32) In [44]: np.around([1.6,np.nan,np.inf,-np.inf], out=c) Out[44]: array([2, 0, 0, 0], dtype=int16)

In [45]: np.around([1.6,np.nan,np.inf,-np.inf], out=d) Out[45]: array([ 2, -2147483648, -2147483648, -2147483648], dtype=int32)

Perhaps a starting point to harmonise this behaviour and get it closer to your expectations (it still would not be really nice having to define the output array first, I guess)...

what numpy is this?

...

...
...
np.array([1.6, np.nan, np.inf, -np.inf]).astype(np.int16) array([ 1, -32768, -32768, -32768], dtype=int16) np.__version__ '1.5.1' a = np.ones(4, np.int16) a[:]=np.array([1.6, np.nan, np.inf, -np.inf]) a array([ 1, -32768, -32768, -32768], dtype=int16)

I thought we get ValueError to avoid nan to zero bugs

...

...
...
a[2] = np.nan Traceback (most recent call last): File "", line 1, in <module> a[2] = np.nan ValueError: cannot convert float NaN to integer

Josef

...

Cheers, Derek

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Matthew Brett

2:32 p.m.

Hi, On Tue, Oct 11, 2011 at 3:06 PM, Derek Homeier wrote:

...

On 11 Oct 2011, at 20:06, Matthew Brett wrote:

...
Have I missed a fast way of doing nice float to integer conversion?

By nice I mean, rounding to the nearest integer, converting NaN to 0, inf, -inf to the max and min of the integer range? The astype method and cast functions don't do what I need here:

In [40]: np.array([1.6, np.nan, np.inf, -np.inf]).astype(np.int16) Out[40]: array([1, 0, 0, 0], dtype=int16)

In [41]: np.cast[np.int16](np.array([1.6, np.nan, np.inf, -np.inf])) Out[41]: array([1, 0, 0, 0], dtype=int16)

Have I missed something obvious?

np.[a]round comes closer to what you wish (is there consensus that NaN should map to 0?), but not quite there, and it's not really consistent either!

In [42]: c = np.zeros(4, np.int16) In [43]: d = np.zeros(4, np.int32) In [44]: np.around([1.6,np.nan,np.inf,-np.inf], out=c) Out[44]: array([2, 0, 0, 0], dtype=int16)

In [45]: np.around([1.6,np.nan,np.inf,-np.inf], out=d) Out[45]: array([ 2, -2147483648, -2147483648, -2147483648], dtype=int32)

Perhaps a starting point to harmonise this behaviour and get it closer to your expectations (it still would not be really nice having to define the output array first, I guess)...

Thanks - it hadn't occurred to me to try around with an output array - an interesting idea. But - isn't this different but just as bad? Best, Matthew

Derek Homeier

4:30 p.m.

On 11.10.2011, at 9:18PM, josef.pktd@gmail.com wrote:

...

...
In [42]: c = np.zeros(4, np.int16) In [43]: d = np.zeros(4, np.int32) In [44]: np.around([1.6,np.nan,np.inf,-np.inf], out=c) Out[44]: array([2, 0, 0, 0], dtype=int16)

In [45]: np.around([1.6,np.nan,np.inf,-np.inf], out=d) Out[45]: array([ 2, -2147483648, -2147483648, -2147483648], dtype=int32)

Perhaps a starting point to harmonise this behaviour and get it closer to your expectations (it still would not be really nice having to define the output array first, I guess)...

what numpy is this?

This was 1.6.1 I did suppress a RuntimeWarning that was raised on the first call, though: In [33]: np.around([1.67,np.nan,np.inf,-np.inf], decimals=1, out=d) /sw/lib/python2.7/site-packages/numpy/core/fromnumeric.py:37: RuntimeWarning: invalid value encountered in multiply result = getattr(asarray(obj),method)(*args, **kwds)

...

...
...
...
np.array([1.6, np.nan, np.inf, -np.inf]).astype(np.int16) array([ 1, -32768, -32768, -32768], dtype=int16) np.__version__ '1.5.1' a = np.ones(4, np.int16) a[:]=np.array([1.6, np.nan, np.inf, -np.inf]) a array([ 1, -32768, -32768, -32768], dtype=int16)

I thought we get ValueError to avoid nan to zero bugs

...
...
...
a[2] = np.nan Traceback (most recent call last): File "", line 1, in <module> a[2] = np.nan ValueError: cannot convert float NaN to integer

On master, an integer out raises a TypeError for any float input - not sure I'd consider that an improvement…

...

...
...
np.__version__ '2.0.0.dev-8f689df' np.around([1.6,-23.42, -13.98, 0.14], out=c) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/derek/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 2277, in around return _wrapit(a, 'round', decimals, out) File "/Users/derek/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 37, in _wrapit result = getattr(asarray(obj),method)(*args, **kwds) TypeError: ufunc 'rint' output (typecode 'd') could not be coerced to provided output parameter (typecode 'h') according to the casting rule “same_kind“

I thought the NaN might have been dealt with first, before casting to int, but that doesn't seem to be the case (on master, again):

...

...
...
np.around([1.6,np.nan,np.inf,-np.inf]) array([ 2., nan, inf, -inf]) np.around([1.6,np.nan,np.inf,-np.inf]).astype(np.int16) array([2, 0, 0, 0], dtype=int16) np.around([1.6,np.nan,np.inf,-np.inf]).astype(np.int32) array([ 2, -2147483648, -2147483648, -2147483648], dtype=int32)

Cheers, Derek

Matthew Brett

4:37 p.m.

Hi, On Tue, Oct 11, 2011 at 5:30 PM, Derek Homeier wrote:

...

On 11.10.2011, at 9:18PM, josef.pktd@gmail.com wrote:

...
...
In [42]: c = np.zeros(4, np.int16) In [43]: d = np.zeros(4, np.int32) In [44]: np.around([1.6,np.nan,np.inf,-np.inf], out=c) Out[44]: array([2, 0, 0, 0], dtype=int16)

In [45]: np.around([1.6,np.nan,np.inf,-np.inf], out=d) Out[45]: array([ 2, -2147483648, -2147483648, -2147483648], dtype=int32)

Perhaps a starting point to harmonise this behaviour and get it closer to your expectations (it still would not be really nice having to define the output array first, I guess)...

what numpy is this?

This was 1.6.1 I did suppress a RuntimeWarning that was raised on the first call, though: In [33]: np.around([1.67,np.nan,np.inf,-np.inf], decimals=1, out=d) /sw/lib/python2.7/site-packages/numpy/core/fromnumeric.py:37: RuntimeWarning: invalid value encountered in multiply result = getattr(asarray(obj),method)(*args, **kwds)

...
...
...
...
np.array([1.6, np.nan, np.inf, -np.inf]).astype(np.int16) array([ 1, -32768, -32768, -32768], dtype=int16) np.__version__ '1.5.1' a = np.ones(4, np.int16) a[:]=np.array([1.6, np.nan, np.inf, -np.inf]) a array([ 1, -32768, -32768, -32768], dtype=int16)

I thought we get ValueError to avoid nan to zero bugs

...
...
...
a[2] = np.nan Traceback (most recent call last): File "", line 1, in <module> a[2] = np.nan ValueError: cannot convert float NaN to integer

On master, an integer out raises a TypeError for any float input - not sure I'd consider that an improvement…

...
...
...
np.__version__ '2.0.0.dev-8f689df' np.around([1.6,-23.42, -13.98, 0.14], out=c) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/derek/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 2277, in around return _wrapit(a, 'round', decimals, out) File "/Users/derek/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 37, in _wrapit result = getattr(asarray(obj),method)(*args, **kwds) TypeError: ufunc 'rint' output (typecode 'd') could not be coerced to provided output parameter (typecode 'h') according to the casting rule “same_kind“

I thought the NaN might have been dealt with first, before casting to int, but that doesn't seem to be the case (on master, again):

...
...
...
np.around([1.6,np.nan,np.inf,-np.inf]) array([ 2., nan, inf, -inf]) np.around([1.6,np.nan,np.inf,-np.inf]).astype(np.int16) array([2, 0, 0, 0], dtype=int16) np.around([1.6,np.nan,np.inf,-np.inf]).astype(np.int32) array([ 2, -2147483648, -2147483648, -2147483648], dtype=int32)

Just to whet the appetite: In [85]: for t in np.sctypes['int'] + np.sctypes['uint']: ....: print np.array([np.nan], float).astype(t) ....: [0] [0] [-2147483648] [-2147483648] [-9223372036854775808] [0] [0] [2147483648] [2147483648] [9223372036854775808] In [89]: for t in np.sctypes['int'] + np.sctypes['uint']: ....: print np.around(np.array([np.nan], float), out=np.zeros(1, t)) ....: /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/fromnumeric.py:2278: RuntimeWarning: invalid value encountered in rint return round(decimals, out) [0] [0] [-2147483648] [-2147483648] [-9223372036854775808] [0] [0] [2147483648] [2147483648] [9223372036854775808] In [86]: np.__version__ Out[86]: '1.6.1' Maybe it would be good to have a np.nice_round function? See you, Matthew

Benjamin Root

6:32 p.m.

On Tue, Oct 11, 2011 at 2:06 PM, Derek Homeier < derek@astro.physik.uni-goettingen.de> wrote:

...

On 11 Oct 2011, at 20:06, Matthew Brett wrote:

...
Have I missed a fast way of doing nice float to integer conversion?

By nice I mean, rounding to the nearest integer, converting NaN to 0, inf, -inf to the max and min of the integer range? The astype method and cast functions don't do what I need here:

In [40]: np.array([1.6, np.nan, np.inf, -np.inf]).astype(np.int16) Out[40]: array([1, 0, 0, 0], dtype=int16)

In [41]: np.cast[np.int16](np.array([1.6, np.nan, np.inf, -np.inf])) Out[41]: array([1, 0, 0, 0], dtype=int16)

Have I missed something obvious?

np.[a]round comes closer to what you wish (is there consensus that NaN should map to 0?), but not quite there, and it's not really consistent either!

In a way, there is already consensus in the code. np.nan_to_num() by default converts nans to zero, and the infinities go to very large and very small. >>> np.set_printoptions(precision=8) >>> x = np.array([np.inf, -np.inf, np.nan, -128, 128]) >>> np.nan_to_num(x) array([ 1.79769313e+308, -1.79769313e+308, 0.00000000e+000, -1.28000000e+002, 1.28000000e+002]) Ben Root

Matthew Brett

15 Oct 15 Oct

2:20 p.m.

Hi, On Tue, Oct 11, 2011 at 7:32 PM, Benjamin Root wrote:

...

On Tue, Oct 11, 2011 at 2:06 PM, Derek Homeier wrote:

...
On 11 Oct 2011, at 20:06, Matthew Brett wrote:

...
Have I missed a fast way of doing nice float to integer conversion?

By nice I mean, rounding to the nearest integer, converting NaN to 0, inf, -inf to the max and min of the integer range? The astype method and cast functions don't do what I need here:

In [40]: np.array([1.6, np.nan, np.inf, -np.inf]).astype(np.int16) Out[40]: array([1, 0, 0, 0], dtype=int16)

In [41]: np.cast[np.int16](np.array([1.6, np.nan, np.inf, -np.inf])) Out[41]: array([1, 0, 0, 0], dtype=int16)

Have I missed something obvious?

np.[a]round comes closer to what you wish (is there consensus that NaN should map to 0?), but not quite there, and it's not really consistent either!

In a way, there is already consensus in the code. np.nan_to_num() by default converts nans to zero, and the infinities go to very large and very small.

>>> np.set_printoptions(precision=8) >>> x = np.array([np.inf, -np.inf, np.nan, -128, 128]) >>> np.nan_to_num(x) array([ 1.79769313e+308, -1.79769313e+308, 0.00000000e+000, -1.28000000e+002, 1.28000000e+002])

Right - but - we'd still need to round, and take care of the nasty issue of thresholding:

...

...
...
x = np.array([np.inf, -np.inf, np.nan, -128, 128]) x array([ inf, -inf, nan, -128., 128.]) nnx = np.nan_to_num(x) nnx

array([ 1.79769313e+308, -1.79769313e+308, 0.00000000e+000, -1.28000000e+002, 1.28000000e+002])

...

...
...
np.rint(nnx).astype(np.int8) array([ 0, 0, 0, -128, -128], dtype=int8)

So, I think nice_round would look something like: def nice_round(arr, out_type): in_type = arr.dtype.type mx = floor_exact(np.iinfo(out_type).max, in_type) mn = floor_exact(np.iinfo(out_type).max, in_type) nans = np.isnan(arr) out = np.rint(np.clip(arr, mn, mx)).astype(out_type) out[nans] = 0 return out with floor_exact being something like: https://github.com/matthew-brett/nibabel/blob/range-dtype-conversions/nibabe... See you, Matthew

Matthew Brett

1 Nov 1 Nov

11:02 p.m.

Hi, On Sat, Oct 15, 2011 at 12:20 PM, Matthew Brett wrote:

...

Hi,

On Tue, Oct 11, 2011 at 7:32 PM, Benjamin Root wrote:

...
On Tue, Oct 11, 2011 at 2:06 PM, Derek Homeier wrote:

...
On 11 Oct 2011, at 20:06, Matthew Brett wrote:

...
Have I missed a fast way of doing nice float to integer conversion?

By nice I mean, rounding to the nearest integer, converting NaN to 0, inf, -inf to the max and min of the integer range? The astype method and cast functions don't do what I need here:

In [40]: np.array([1.6, np.nan, np.inf, -np.inf]).astype(np.int16) Out[40]: array([1, 0, 0, 0], dtype=int16)

In [41]: np.cast[np.int16](np.array([1.6, np.nan, np.inf, -np.inf])) Out[41]: array([1, 0, 0, 0], dtype=int16)

Have I missed something obvious?

np.[a]round comes closer to what you wish (is there consensus that NaN should map to 0?), but not quite there, and it's not really consistent either!

In a way, there is already consensus in the code. np.nan_to_num() by default converts nans to zero, and the infinities go to very large and very small.

>>> np.set_printoptions(precision=8) >>> x = np.array([np.inf, -np.inf, np.nan, -128, 128]) >>> np.nan_to_num(x) array([ 1.79769313e+308, -1.79769313e+308, 0.00000000e+000, -1.28000000e+002, 1.28000000e+002])

Right - but - we'd still need to round, and take care of the nasty issue of thresholding:

...
...
...
x = np.array([np.inf, -np.inf, np.nan, -128, 128]) x array([ inf, -inf, nan, -128., 128.]) nnx = np.nan_to_num(x) nnx

array([ 1.79769313e+308, -1.79769313e+308, 0.00000000e+000, -1.28000000e+002, 1.28000000e+002])

...
...
...
np.rint(nnx).astype(np.int8) array([ 0, 0, 0, -128, -128], dtype=int8)

So, I think nice_round would look something like:

def nice_round(arr, out_type): in_type = arr.dtype.type mx = floor_exact(np.iinfo(out_type).max, in_type) mn = floor_exact(np.iinfo(out_type).max, in_type) nans = np.isnan(arr) out = np.rint(np.clip(arr, mn, mx)).astype(out_type) out[nans] = 0 return out

with floor_exact being something like:

https://github.com/matthew-brett/nibabel/blob/range-dtype-conversions/nibabe...

In case anyone is interested or for the sake of anyone later googling this thread - I made a working version of nice_round: https://github.com/matthew-brett/nibabel/blob/floating-stash/nibabel/casting... Docstring: def nice_round(arr, int_type, nan2zero=True, infmax=False): """ Round floating point array `arr` to type `int_type` Parameters ---------- arr : array-like Array of floating point type int_type : object Numpy integer type nan2zero : {True, False} Whether to convert NaN value to zero. Default is True. If False, and NaNs are present, raise CastingError infmax : {False, True} If True, set np.inf values in `arr` to be `int_type` integer maximum value, -np.inf as `int_type` integer minimum. If False, merely set infs to be numbers at or near the maximum / minumum number in `arr` that can be contained in `int_type`. Therefore False gives faster conversion at the expense of infs that are further from infinity. Returns ------- iarr : ndarray of type `int_type` Examples --------

...

...
...
nice_round([np.nan, np.inf, -np.inf, 1.1, 6.6], np.int16) array([ 0, 32767, -32768, 1, 7], dtype=int16)

It wasn't straightforward to find the right place to clip the array to stop overflow on casting, but I think it's working and tested now. See y'all, Matthew

4558

Age (days ago)

4580

Last active (days ago)

List overview

Download

8 comments

4 participants

participants (4)

Benjamin Root
Derek Homeier
josef.pktd＠gmail.com
Matthew Brett

Nice float -> integer conversion?

Benjamin Root

tags

participants (4)