add axis to results of reduction (mean, min, ...)
What's the best way of getting back the correct shape to be able to broadcast, mean, min,.. to the original array, that works for arbitrary dimension and axis? I thought I have seen some helper functions, but I don't find them anymore? Josef
a array([[1, 2, 3, 3, 0], [2, 2, 3, 2, 1]]) a-a.max(0) array([[-1, 0, 0, 0, -1], [ 0, 0, 0, -1, 0]]) a-a.max(1) Traceback (most recent call last): File "<pyshell#135>", line 1, in <module> a-a.max(1) ValueError: shape mismatch: objects cannot be broadcast to a single shape a-a.max(1)[:,None] array([[-2, -1, 0, 0, -3], [-1, -1, 0, -1, -2]])
On Thu, Aug 6, 2009 at 8:55 AM, <josef.pktd@gmail.com> wrote:
What's the best way of getting back the correct shape to be able to broadcast, mean, min,.. to the original array, that works for arbitrary dimension and axis?
I thought I have seen some helper functions, but I don't find them anymore?
Josef
a array([[1, 2, 3, 3, 0], [2, 2, 3, 2, 1]]) a-a.max(0) array([[-1, 0, 0, 0, -1], [ 0, 0, 0, -1, 0]]) a-a.max(1) Traceback (most recent call last): File "<pyshell#135>", line 1, in <module> a-a.max(1) ValueError: shape mismatch: objects cannot be broadcast to a single shape a-a.max(1)[:,None] array([[-2, -1, 0, 0, -3], [-1, -1, 0, -1, -2]])
Would this do it?
pylab.demean?? Type: function Base Class: <type 'function'> String Form: <function demean at 0x3c5c050> Namespace: Interactive File: /usr/lib/python2.6/dist-packages/matplotlib/mlab.py Definition: pylab.demean(x, axis=0) Source: def demean(x, axis=0): "Return x minus its mean along the specified axis" x = np.asarray(x) if axis: ind = [slice(None)] * axis ind.append(np.newaxis) return x - x.mean(axis)[ind] return x - x.mean(axis)
On Thu, Aug 6, 2009 at 11:03, Keith Goodman<kwgoodman@gmail.com> wrote:
On Thu, Aug 6, 2009 at 8:55 AM, <josef.pktd@gmail.com> wrote:
What's the best way of getting back the correct shape to be able to broadcast, mean, min,.. to the original array, that works for arbitrary dimension and axis?
I thought I have seen some helper functions, but I don't find them anymore?
Josef
a array([[1, 2, 3, 3, 0], [2, 2, 3, 2, 1]]) a-a.max(0) array([[-1, 0, 0, 0, -1], [ 0, 0, 0, -1, 0]]) a-a.max(1) Traceback (most recent call last): File "<pyshell#135>", line 1, in <module> a-a.max(1) ValueError: shape mismatch: objects cannot be broadcast to a single shape a-a.max(1)[:,None] array([[-2, -1, 0, 0, -3], [-1, -1, 0, -1, -2]])
Would this do it?
pylab.demean?? Type: function Base Class: <type 'function'> String Form: <function demean at 0x3c5c050> Namespace: Interactive File: /usr/lib/python2.6/dist-packages/matplotlib/mlab.py Definition: pylab.demean(x, axis=0) Source: def demean(x, axis=0): "Return x minus its mean along the specified axis" x = np.asarray(x) if axis: ind = [slice(None)] * axis ind.append(np.newaxis) return x - x.mean(axis)[ind] return x - x.mean(axis)
Ouch! That doesn't handle axis=-1. if axis != 0: ind = [slice(None)] * x.ndim ind[axis] = np.newaxis -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
thanksOn Thu, Aug 6, 2009 at 9:07 AM, Robert Kern<robert.kern@gmail.com> wrote:
On Thu, Aug 6, 2009 at 11:03, Keith Goodman<kwgoodman@gmail.com> wrote:
On Thu, Aug 6, 2009 at 8:55 AM, <josef.pktd@gmail.com> wrote:
What's the best way of getting back the correct shape to be able to broadcast, mean, min,.. to the original array, that works for arbitrary dimension and axis?
I thought I have seen some helper functions, but I don't find them anymore?
Josef
a array([[1, 2, 3, 3, 0], [2, 2, 3, 2, 1]]) a-a.max(0) array([[-1, 0, 0, 0, -1], [ 0, 0, 0, -1, 0]]) a-a.max(1) Traceback (most recent call last): File "<pyshell#135>", line 1, in <module> a-a.max(1) ValueError: shape mismatch: objects cannot be broadcast to a single shape a-a.max(1)[:,None] array([[-2, -1, 0, 0, -3], [-1, -1, 0, -1, -2]])
Would this do it?
pylab.demean?? Type: function Base Class: <type 'function'> String Form: <function demean at 0x3c5c050> Namespace: Interactive File: /usr/lib/python2.6/dist-packages/matplotlib/mlab.py Definition: pylab.demean(x, axis=0) Source: def demean(x, axis=0): "Return x minus its mean along the specified axis" x = np.asarray(x) if axis: ind = [slice(None)] * axis ind.append(np.newaxis) return x - x.mean(axis)[ind] return x - x.mean(axis)
Ouch! That doesn't handle axis=-1.
if axis != 0: ind = [slice(None)] * x.ndim ind[axis] = np.newaxis
Hey, didn't you warn us about the dangers of "if arr" the other day?
On Thu, Aug 6, 2009 at 11:15, Keith Goodman<kwgoodman@gmail.com> wrote:
thanksOn Thu, Aug 6, 2009 at 9:07 AM, Robert Kern<robert.kern@gmail.com> wrote:
On Thu, Aug 6, 2009 at 11:03, Keith Goodman<kwgoodman@gmail.com> wrote:
pylab.demean?? Type: function Base Class: <type 'function'> String Form: <function demean at 0x3c5c050> Namespace: Interactive File: /usr/lib/python2.6/dist-packages/matplotlib/mlab.py Definition: pylab.demean(x, axis=0) Source: def demean(x, axis=0): "Return x minus its mean along the specified axis" x = np.asarray(x) if axis: ind = [slice(None)] * axis ind.append(np.newaxis) return x - x.mean(axis)[ind] return x - x.mean(axis)
Ouch! That doesn't handle axis=-1.
if axis != 0: ind = [slice(None)] * x.ndim ind[axis] = np.newaxis
Hey, didn't you warn us about the dangers of "if arr" the other day?
Yes, but actually that wasn't quite the problem. "if axis:" would have been fine, if a bit obscure, as long as the body was fixed to not expect axis>0. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Thu, Aug 6, 2009 at 9:18 AM, Robert Kern<robert.kern@gmail.com> wrote:
On Thu, Aug 6, 2009 at 11:15, Keith Goodman<kwgoodman@gmail.com> wrote:
thanksOn Thu, Aug 6, 2009 at 9:07 AM, Robert Kern<robert.kern@gmail.com> wrote:
On Thu, Aug 6, 2009 at 11:03, Keith Goodman<kwgoodman@gmail.com> wrote:
pylab.demean?? Type: function Base Class: <type 'function'> String Form: <function demean at 0x3c5c050> Namespace: Interactive File: /usr/lib/python2.6/dist-packages/matplotlib/mlab.py Definition: pylab.demean(x, axis=0) Source: def demean(x, axis=0): "Return x minus its mean along the specified axis" x = np.asarray(x) if axis: ind = [slice(None)] * axis ind.append(np.newaxis) return x - x.mean(axis)[ind] return x - x.mean(axis)
Ouch! That doesn't handle axis=-1.
if axis != 0: ind = [slice(None)] * x.ndim ind[axis] = np.newaxis
Hey, didn't you warn us about the dangers of "if arr" the other day?
Yes, but actually that wasn't quite the problem. "if axis:" would have been fine, if a bit obscure, as long as the body was fixed to not expect axis>0.
Oh, of course. Thanks for pointing that out. Now I can fix a bug in my own code.
On Aug 6, 2009, at 12:22 PM, Keith Goodman wrote:
On Thu, Aug 6, 2009 at 9:18 AM, Robert Kern<robert.kern@gmail.com> wrote:
On Thu, Aug 6, 2009 at 11:15, Keith Goodman<kwgoodman@gmail.com> wrote:
thanksOn Thu, Aug 6, 2009 at 9:07 AM, Robert Kern<robert.kern@gmail.com
wrote: On Thu, Aug 6, 2009 at 11:03, Keith Goodman<kwgoodman@gmail.com> wrote:
> pylab.demean?? Type: function Base Class: <type 'function'> String Form: <function demean at 0x3c5c050> Namespace: Interactive File: /usr/lib/python2.6/dist-packages/matplotlib/ mlab.py Definition: pylab.demean(x, axis=0) Source: def demean(x, axis=0): "Return x minus its mean along the specified axis" x = np.asarray(x) if axis: ind = [slice(None)] * axis ind.append(np.newaxis) return x - x.mean(axis)[ind] return x - x.mean(axis)
FYI, there's a "anom" method for MaskedArrays that does the same thing as demean (if you can't /don't want to import mpl)
On Thu, Aug 6, 2009 at 12:07 PM, Robert Kern<robert.kern@gmail.com> wrote:
On Thu, Aug 6, 2009 at 11:03, Keith Goodman<kwgoodman@gmail.com> wrote:
On Thu, Aug 6, 2009 at 8:55 AM, <josef.pktd@gmail.com> wrote:
What's the best way of getting back the correct shape to be able to broadcast, mean, min,.. to the original array, that works for arbitrary dimension and axis?
I thought I have seen some helper functions, but I don't find them anymore?
Josef
a array([[1, 2, 3, 3, 0], [2, 2, 3, 2, 1]]) a-a.max(0) array([[-1, 0, 0, 0, -1], [ 0, 0, 0, -1, 0]]) a-a.max(1) Traceback (most recent call last): File "<pyshell#135>", line 1, in <module> a-a.max(1) ValueError: shape mismatch: objects cannot be broadcast to a single shape a-a.max(1)[:,None] array([[-2, -1, 0, 0, -3], [-1, -1, 0, -1, -2]])
Would this do it?
pylab.demean?? Type: function Base Class: <type 'function'> String Form: <function demean at 0x3c5c050> Namespace: Interactive File: /usr/lib/python2.6/dist-packages/matplotlib/mlab.py Definition: pylab.demean(x, axis=0) Source: def demean(x, axis=0): "Return x minus its mean along the specified axis" x = np.asarray(x) if axis: ind = [slice(None)] * axis ind.append(np.newaxis) return x - x.mean(axis)[ind] return x - x.mean(axis)
Ouch! That doesn't handle axis=-1.
if axis != 0: ind = [slice(None)] * x.ndim ind[axis] = np.newaxis
Thanks, that's it. I have seen implementation of helper functions similar to this in other packages, but I thought there is already something in numpy. I think this should be a simple helper function in numpy to avoid mistakes and complicated implementation like the one in stats.nanstd even if it's only a few lines. Josef
-- Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Thu, Aug 6, 2009 at 11:21, <josef.pktd@gmail.com> wrote:
On Thu, Aug 6, 2009 at 12:07 PM, Robert Kern<robert.kern@gmail.com> wrote:
if axis != 0: ind = [slice(None)] * x.ndim ind[axis] = np.newaxis
Thanks, that's it.
I have seen implementation of helper functions similar to this in other packages, but I thought there is already something in numpy. I think this should be a simple helper function in numpy to avoid mistakes and complicated implementation like the one in stats.nanstd even if it's only a few lines.
It would make a good contribution to numpy.lib.index_tricks. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Thu, Aug 6, 2009 at 9:07 AM, Robert Kern<robert.kern@gmail.com> wrote:
On Thu, Aug 6, 2009 at 11:03, Keith Goodman<kwgoodman@gmail.com> wrote:
On Thu, Aug 6, 2009 at 8:55 AM, <josef.pktd@gmail.com> wrote:
What's the best way of getting back the correct shape to be able to broadcast, mean, min,.. to the original array, that works for arbitrary dimension and axis?
I thought I have seen some helper functions, but I don't find them anymore?
Josef
a array([[1, 2, 3, 3, 0], [2, 2, 3, 2, 1]]) a-a.max(0) array([[-1, 0, 0, 0, -1], [ 0, 0, 0, -1, 0]]) a-a.max(1) Traceback (most recent call last): File "<pyshell#135>", line 1, in <module> a-a.max(1) ValueError: shape mismatch: objects cannot be broadcast to a single shape a-a.max(1)[:,None] array([[-2, -1, 0, 0, -3], [-1, -1, 0, -1, -2]])
Would this do it?
pylab.demean?? Type: function Base Class: <type 'function'> String Form: <function demean at 0x3c5c050> Namespace: Interactive File: /usr/lib/python2.6/dist-packages/matplotlib/mlab.py Definition: pylab.demean(x, axis=0) Source: def demean(x, axis=0): "Return x minus its mean along the specified axis" x = np.asarray(x) if axis: ind = [slice(None)] * axis ind.append(np.newaxis) return x - x.mean(axis)[ind] return x - x.mean(axis)
Ouch! That doesn't handle axis=-1.
if axis != 0: ind = [slice(None)] * x.ndim ind[axis] = np.newaxis
Ouch! That doesn't handle axis=None. if axis: ind = [slice(None)] * x.ndim ind[axis] = np.newaxis
On Mon, Aug 10, 2009 at 11:55 AM, Keith Goodman<kwgoodman@gmail.com> wrote:
On Thu, Aug 6, 2009 at 9:07 AM, Robert Kern<robert.kern@gmail.com> wrote:
On Thu, Aug 6, 2009 at 11:03, Keith Goodman<kwgoodman@gmail.com> wrote:
On Thu, Aug 6, 2009 at 8:55 AM, <josef.pktd@gmail.com> wrote:
What's the best way of getting back the correct shape to be able to broadcast, mean, min,.. to the original array, that works for arbitrary dimension and axis?
I thought I have seen some helper functions, but I don't find them anymore?
Josef
> a array([[1, 2, 3, 3, 0], [2, 2, 3, 2, 1]]) > a-a.max(0) array([[-1, 0, 0, 0, -1], [ 0, 0, 0, -1, 0]]) > a-a.max(1) Traceback (most recent call last): File "<pyshell#135>", line 1, in <module> a-a.max(1) ValueError: shape mismatch: objects cannot be broadcast to a single shape > a-a.max(1)[:,None] array([[-2, -1, 0, 0, -3], [-1, -1, 0, -1, -2]])
Would this do it?
pylab.demean?? Type: function Base Class: <type 'function'> String Form: <function demean at 0x3c5c050> Namespace: Interactive File: /usr/lib/python2.6/dist-packages/matplotlib/mlab.py Definition: pylab.demean(x, axis=0) Source: def demean(x, axis=0): "Return x minus its mean along the specified axis" x = np.asarray(x) if axis: ind = [slice(None)] * axis ind.append(np.newaxis) return x - x.mean(axis)[ind] return x - x.mean(axis)
Ouch! That doesn't handle axis=-1.
if axis != 0: ind = [slice(None)] * x.ndim ind[axis] = np.newaxis
Ouch! That doesn't handle axis=None.
if axis: ind = [slice(None)] * x.ndim ind[axis] = np.newaxis
that's why I used if axis != 0 and not axis is None: and included a testcase for None. (although my version looks a bit verbose but explicit) Josef
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Mon, Aug 10, 2009 at 9:10 AM, <josef.pktd@gmail.com> wrote:
On Mon, Aug 10, 2009 at 11:55 AM, Keith Goodman<kwgoodman@gmail.com> wrote:
On Thu, Aug 6, 2009 at 9:07 AM, Robert Kern<robert.kern@gmail.com> wrote:
On Thu, Aug 6, 2009 at 11:03, Keith Goodman<kwgoodman@gmail.com> wrote:
On Thu, Aug 6, 2009 at 8:55 AM, <josef.pktd@gmail.com> wrote:
What's the best way of getting back the correct shape to be able to broadcast, mean, min,.. to the original array, that works for arbitrary dimension and axis?
I thought I have seen some helper functions, but I don't find them anymore?
Josef
>> a array([[1, 2, 3, 3, 0], [2, 2, 3, 2, 1]]) >> a-a.max(0) array([[-1, 0, 0, 0, -1], [ 0, 0, 0, -1, 0]]) >> a-a.max(1) Traceback (most recent call last): File "<pyshell#135>", line 1, in <module> a-a.max(1) ValueError: shape mismatch: objects cannot be broadcast to a single shape >> a-a.max(1)[:,None] array([[-2, -1, 0, 0, -3], [-1, -1, 0, -1, -2]])
Would this do it?
pylab.demean?? Type: function Base Class: <type 'function'> String Form: <function demean at 0x3c5c050> Namespace: Interactive File: /usr/lib/python2.6/dist-packages/matplotlib/mlab.py Definition: pylab.demean(x, axis=0) Source: def demean(x, axis=0): "Return x minus its mean along the specified axis" x = np.asarray(x) if axis: ind = [slice(None)] * axis ind.append(np.newaxis) return x - x.mean(axis)[ind] return x - x.mean(axis)
Ouch! That doesn't handle axis=-1.
if axis != 0: ind = [slice(None)] * x.ndim ind[axis] = np.newaxis
Ouch! That doesn't handle axis=None.
if axis: ind = [slice(None)] * x.ndim ind[axis] = np.newaxis
that's why I used
if axis != 0 and not axis is None:
and included a testcase for None. (although my version looks a bit verbose but explicit)
I'm getting better. I'm only 3 days behind this time. Yeah, I caught it on a unit test too.
On Thu, Aug 6, 2009 at 9:55 AM, <josef.pktd@gmail.com> wrote:
What's the best way of getting back the correct shape to be able to broadcast, mean, min,.. to the original array, that works for arbitrary dimension and axis?
I thought I have seen some helper functions, but I don't find them anymore?
Adding a keyword to retain the number of dimensions has been mooted. It shouldn't be too difficult to implement and would allow things like:
scaled = a/a.max(1, reduce=0)
I could do that for 1.4 if folks are interested. Chuck
On Thu, Aug 6, 2009 at 9:58 AM, Charles R Harris<charlesr.harris@gmail.com> wrote:
On Thu, Aug 6, 2009 at 9:55 AM, <josef.pktd@gmail.com> wrote:
What's the best way of getting back the correct shape to be able to broadcast, mean, min,.. to the original array, that works for arbitrary dimension and axis?
I thought I have seen some helper functions, but I don't find them anymore?
Adding a keyword to retain the number of dimensions has been mooted. It shouldn't be too difficult to implement and would allow things like:
scaled = a/a.max(1, reduce=0)
I could do that for 1.4 if folks are interested.
I'd use that. It's better than what I usually do: scaled = a / a.max(1).reshape(-1,1)
On Thu, Aug 6, 2009 at 1:07 PM, Keith Goodman<kwgoodman@gmail.com> wrote:
On Thu, Aug 6, 2009 at 9:58 AM, Charles R Harris<charlesr.harris@gmail.com> wrote:
On Thu, Aug 6, 2009 at 9:55 AM, <josef.pktd@gmail.com> wrote:
What's the best way of getting back the correct shape to be able to broadcast, mean, min,.. to the original array, that works for arbitrary dimension and axis?
I thought I have seen some helper functions, but I don't find them anymore?
Adding a keyword to retain the number of dimensions has been mooted. It shouldn't be too difficult to implement and would allow things like:
scaled = a/a.max(1, reduce=0)
I could do that for 1.4 if folks are interested.
I'd use that. It's better than what I usually do:
scaled = a / a.max(1).reshape(-1,1) _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
added feature to numpy reduce function would be nice. but helper function is still useful for our own reduce operations something like this function with an awful name Josef import numpy as np def addreducedaxis(x,axis=None): '''adds axis so that results of reduce operation broadcast to original array Parameter --------- x : array n-dim array that is the result of reduce operation, e.g. mean, min axis : int axis that was removed in the reduce operation Return ------ y : array (n+1)dim array with additional axis ''' if axis != 0 and not axis is None: ind = [slice(None)] * (x.ndim+1) ind[axis] = np.newaxis return x[ind] else: return x a = np.array([[1,2,3,3,0],[2,2,3,2,1]]) a3 = np.dstack((a,a)) print np.all((a3-a3.mean(1)[:,None,:]) == (a3 - addreducedaxis(a3.mean(1),1))) print np.all((a3-a3.mean(-2)[:,None,:]) == (a3 - addreducedaxis(a3.mean(-2),-2))) print np.all((a3-a3.mean()) == (a3 - addreducedaxis(a3.mean()))) print np.all((a3.ravel()-a3.mean(None)) == (a3.ravel() - addreducedaxis(a3.mean(None),None))) print np.all((a3-a3.mean(None)) == (a3 - addreducedaxis(a3.mean(None),None))) #example usage from numpy.testing import assert_almost_equal for axis in [None,0,1,2]: m = a3.mean(axis) v = ((a3 - addreducedaxis(m,axis))**2).mean(axis) assert_almost_equal(v, np.var(a3,axis),15) #normalize array along one axis a3n = (a3 - addreducedaxis(np.mean(a3,1),1))/np.sqrt(addreducedaxis(np.var(a3,1),1)) print a3n.mean(1) print a3n.var(1)
Keith Goodman wrote:
On Thu, Aug 6, 2009 at 9:58 AM, Charles R Harris<charlesr.harris@gmail.com> wrote:
On Thu, Aug 6, 2009 at 9:55 AM, <josef.pktd@gmail.com> wrote:
What's the best way of getting back the correct shape to be able to broadcast, mean, min,.. to the original array, that works for arbitrary dimension and axis?
I thought I have seen some helper functions, but I don't find them anymore? Adding a keyword to retain the number of dimensions has been mooted. It shouldn't be too difficult to implement and would allow things like:
scaled = a/a.max(1, reduce=0) I could do that for 1.4 if folks are interested.
I'd use that. It's better than what I usually do:
scaled = a / a.max(1).reshape(-1,1)
To chime in after returning from holidays: I'd use that keyword a great deal. Would be more than welcome to me. I currently have loads of code numpy.newaxis-ing the results of min/max/mean operations...
participants (6)
-
Charles R Harris -
josef.pktd＠gmail.com -
Keith Goodman -
Pierre GM -
Robert Kern -
Vincent Schut