Slicing/selection in multiple dimensions simultaneously
The following seems to be a wart: is it expected? Set up a 10x10 array and some indexing arrays: a=arange(100) a.shape=(10,10) q=array([0,2,4,6,8]) r=array([0,5]) Suppose I want to extract only the "even" numbered rows from a - then print a[q,:] <works - output deleted> Every fifth column: print a[:,r] <works - output deleted> Only the even rows of every fifth column: print a[q,r] --------------------------------------------------------------------------- <type 'exceptions.ValueError'> Traceback (most recent call last) /.../.../.../<ipython console> in <module>() <type 'exceptions.ValueError'>: shape mismatch: objects cannot be broadcast to a single shape But, this works: print a[q,:][:,r] [[ 0 5] [20 25] [40 45] [60 65] [80 85]] So why does the a[q,r] form have problems? Thanks for your insights. Mike -- mike.ressler@alum.mit.edu
Mike Ressler wrote:
The following seems to be a wart: is it expected?
Set up a 10x10 array and some indexing arrays:
a=arange(100) a.shape=(10,10) q=array([0,2,4,6,8]) r=array([0,5])
Suppose I want to extract only the "even" numbered rows from a - then
print a[q,:]
<works - output deleted>
Every fifth column:
print a[:,r]
<works - output deleted>
Only the even rows of every fifth column:
print a[q,r]
--------------------------------------------------------------------------- <type 'exceptions.ValueError'> Traceback (most recent call last)
/.../.../.../<ipython console> in <module>()
<type 'exceptions.ValueError'>: shape mismatch: objects cannot be broadcast to a single shape
But, this works:
print a[q,:][:,r]
[[ 0 5] [20 25] [40 45] [60 65] [80 85]]
So why does the a[q,r] form have problems? Thanks for your insights.
It is intended that the form a[q,r] be the general case: q and r are broadcasted against each other to a single shape. The result of the indexing is an array of that broadcasted shape with elements found by using each pair of elements in the broadcasted q and r arrays as indices. There are operations you can express with this form that you couldn't if the behavior that you expected were the case whereas you can get the result you want relatively straightforwardly. In [6]: a[q[:,newaxis], r] Out[6]: array([[ 0, 5], [20, 25], [40, 45], [60, 65], [80, 85]]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On 9/11/07, Robert Kern <robert.kern@gmail.com> wrote:
Mike Ressler wrote:
The following seems to be a wart: is it expected?
Set up a 10x10 array and some indexing arrays:
a=arange(100) a.shape=(10,10) q=array([0,2,4,6,8]) r=array([0,5])
Suppose I want to extract only the "even" numbered rows from a - then
print a[q,:]
<works - output deleted>
Every fifth column:
print a[:,r]
<works - output deleted>
Only the even rows of every fifth column:
print a[q,r]
---------------------------------------------------------------------------
<type 'exceptions.ValueError'> Traceback (most recent call last)
/.../.../.../<ipython console> in <module>()
<type 'exceptions.ValueError'>: shape mismatch: objects cannot be broadcast to a single shape
But, this works:
print a[q,:][:,r]
[[ 0 5] [20 25] [40 45] [60 65] [80 85]]
So why does the a[q,r] form have problems? Thanks for your insights.
It is intended that the form a[q,r] be the general case: q and r are broadcasted against each other to a single shape. The result of the indexing is an array of that broadcasted shape with elements found by using each pair of elements in the broadcasted q and r arrays as indices.
There are operations you can express with this form that you couldn't if the behavior that you expected were the case whereas you can get the result you want relatively straightforwardly.
In [6]: a[q[:,newaxis], r] Out[6]: array([[ 0, 5], [20, 25], [40, 45], [60, 65], [80, 85]])
At the risk of making Robert grumpy: while it is true the form we ended up with is more general I've come to the conclusion that it was a bit of a mistake. In the spirit of making simple things simple and complex things possible, I suspect that having fancy-indexing do the obvious thing here[1] and delegating the more powerful but also more difficult to understand case to a function or method would have been overall more useful. Cases where the multidimensional features of fancy-indexing get used are messy enough that they don't benefit much from the conciseness of the indexing notation, at least in my experience. -- . __ . |-\ . . tim.hochberg@ieee.org [1] Just in case the 'obvious' thing isn't all that obvious: I mean restrict index-arrays to one dimension and have them simply select the given values along the axis. Hmmm. Without giving examples, which I have not time for right now, that's probably not any clearer than saying nothing. Ah well.
Timothy Hochberg wrote:
On 9/11/07, *Robert Kern* <robert.kern@gmail.com <mailto:robert.kern@gmail.com>> wrote:
Mike Ressler wrote: > The following seems to be a wart: is it expected? > > Set up a 10x10 array and some indexing arrays: > > a=arange(100) > a.shape=(10,10) > q=array([0,2,4,6,8]) > r=array([0,5]) > > Suppose I want to extract only the "even" numbered rows from a - then > > print a[q,:] > > <works - output deleted> > > Every fifth column: > > print a[:,r] > > <works - output deleted> > > Only the even rows of every fifth column: > > print a[q,r] > > ---------------------------------------------------------------------------
> <type 'exceptions.ValueError'> Traceback (most recent call last) > > /.../.../.../<ipython console> in <module>() > > <type 'exceptions.ValueError '>: shape mismatch: objects cannot be > broadcast to a single shape > > But, this works: > > print a[q,:][:,r] > > [[ 0 5] > [20 25] > [40 45] > [60 65] > [80 85]] > > So why does the a[q,r] form have problems? Thanks for your insights.
It is intended that the form a[q,r] be the general case: q and r are broadcasted against each other to a single shape. The result of the indexing is an array of that broadcasted shape with elements found by using each pair of elements in the broadcasted q and r arrays as indices.
There are operations you can express with this form that you couldn't if the behavior that you expected were the case whereas you can get the result you want relatively straightforwardly.
In [6]: a[q[:,newaxis], r] Out[6]: array([[ 0, 5], [20, 25], [40, 45], [60, 65], [80, 85]])
At the risk of making Robert grumpy: while it is true the form we ended up with is more general I've come to the conclusion that it was a bit of a mistake. In the spirit of making simple things simple and complex things possible, I suspect that having fancy-indexing do the obvious thing here[1] and delegating the more powerful but also more difficult to understand case to a function or method would have been overall more useful. Cases where the multidimensional features of fancy-indexing get used are messy enough that they don't benefit much from the conciseness of the indexing notation, at least in my experience.
This is a reasonable argument. It is reasonable enough that I intentionally made an ix_ function to do what you want. a[ix_(q,r)] does as originally expected if a bit more line-noise. -Travis
Am I right to assume that there is no way elegant way to interact with slices. i.e. Is there anyway to get a[ix_([2,3,6],:,[3,2])] to work? So that the dimension is completely specified? Or perhaps the only way to do this is via a[ix_([2,3,6],range(a.shape[1]),[3,2])] If anyone knows a better way? Thanks, Jonathan. On Tue, Sep 11, 2007 at 6:13 PM, Travis E. Oliphant <oliphant@enthought.com> wrote:
Timothy Hochberg wrote:
On 9/11/07, *Robert Kern* <robert.kern@gmail.com <mailto:robert.kern@gmail.com>> wrote:
Mike Ressler wrote: > The following seems to be a wart: is it expected? > > Set up a 10x10 array and some indexing arrays: > > a=arange(100) > a.shape=(10,10) > q=array([0,2,4,6,8]) > r=array([0,5]) > > Suppose I want to extract only the "even" numbered rows from a - then > > print a[q,:] > > <works - output deleted> > > Every fifth column: > > print a[:,r] > > <works - output deleted> > > Only the even rows of every fifth column: > > print a[q,r] > > ---------------------------------------------------------------------------
> <type 'exceptions.ValueError'> Traceback (most recent call last) > > /.../.../.../<ipython console> in <module>() > > <type 'exceptions.ValueError '>: shape mismatch: objects cannot be > broadcast to a single shape > > But, this works: > > print a[q,:][:,r] > > [[ 0 5] > [20 25] > [40 45] > [60 65] > [80 85]] > > So why does the a[q,r] form have problems? Thanks for your insights.
It is intended that the form a[q,r] be the general case: q and r are broadcasted against each other to a single shape. The result of the indexing is an array of that broadcasted shape with elements found by using each pair of elements in the broadcasted q and r arrays as indices.
There are operations you can express with this form that you couldn't if the behavior that you expected were the case whereas you can get the result you want relatively straightforwardly.
In [6]: a[q[:,newaxis], r] Out[6]: array([[ 0, 5], [20, 25], [40, 45], [60, 65], [80, 85]])
At the risk of making Robert grumpy: while it is true the form we ended up with is more general I've come to the conclusion that it was a bit of a mistake. In the spirit of making simple things simple and complex things possible, I suspect that having fancy-indexing do the obvious thing here[1] and delegating the more powerful but also more difficult to understand case to a function or method would have been overall more useful. Cases where the multidimensional features of fancy-indexing get used are messy enough that they don't benefit much from the conciseness of the indexing notation, at least in my experience.
This is a reasonable argument. It is reasonable enough that I intentionally made an ix_ function to do what you want.
a[ix_(q,r)]
does as originally expected if a bit more line-noise.
-Travis
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Hey Jon, On 26-Feb-09, at 10:00 PM, Jonathan Taylor wrote:
Am I right to assume that there is no way elegant way to interact with slices. i.e. Is there anyway to get
a[ix_([2,3,6],:,[3,2])]
to work? So that the dimension is completely specified? Or perhaps the only way to do this is via
a[ix_([2,3,6],range(a.shape[1]),[3,2])]
If anyone knows a better way?
a[[2,3,6],:,:][:,:,[3,2]] should do what you want. In [21]: a = randn(50,50,50) In [22]: all(a[ix_([2,3,6],range(a.shape[1]),[3,2])] == a[[2,3,6],:,:] [:,:,[3,2]]) Out[22]: True David
Hi Dave. This does seem like the only way to write this nicely. Unfortunately, I think this may be wasteful memory wise (in contrast to what the obvious matlab code would do) as it constructs an array with the whole first index intact at first. I think I will use it anyways though as I find the other notations pretty awkward. Jon. On Fri, Feb 27, 2009 at 11:54 PM, David Warde-Farley <dwf@cs.toronto.edu> wrote:
On 27-Feb-09, at 3:35 PM, David Warde-Farley wrote:
a[[2,3,6],:,:][:,:,[3,2]] should do what you want.
Slightly more elegantly (I always forget about this syntax):
a[[2,3,6], ...][..., [3,2]]
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
On 28-Feb-09, at 12:27 PM, Jonathan Taylor wrote:
This does seem like the only way to write this nicely. Unfortunately, I think this may be wasteful memory wise (in contrast to what the obvious matlab code would do) as it constructs an array with the whole first index intact at first.
True enough, though if I understand correctly, this is only a _view_ onto the original array, and nothing is immediately copied. So it does waste memory creating a view and then a view on the view, but I don't think it's proportional to the size of the returned array. Maybe Robert or someone else can confirm this. David
On Mon, Mar 2, 2009 at 04:19, David Warde-Farley <dwf@cs.toronto.edu> wrote:
On 28-Feb-09, at 12:27 PM, Jonathan Taylor wrote:
This does seem like the only way to write this nicely. Unfortunately, I think this may be wasteful memory wise (in contrast to what the obvious matlab code would do) as it constructs an array with the whole first index intact at first.
True enough, though if I understand correctly, this is only a _view_ onto the original array, and nothing is immediately copied. So it does waste memory creating a view and then a view on the view, but I don't think it's proportional to the size of the returned array.
a[[2,3,6], ...][..., [3,2]] You're doing fancy indexing, so there are copies both times. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On 2-Mar-09, at 12:25 PM, Robert Kern wrote:
a[[2,3,6], ...][..., [3,2]]
You're doing fancy indexing, so there are copies both times.
D'oh! So I guess the only way to avoid the second copy is to do what Jon initially suggested, i.e. a[ix_([2,3,6],range(a.shape[1]),[3,2])] ? I suppose xrange would be better than arange() or range() as it wouldn't create and destroy the list all at once. D
On Mon, Mar 2, 2009 at 13:10, David Warde-Farley <dwf@cs.toronto.edu> wrote:
On 2-Mar-09, at 12:25 PM, Robert Kern wrote:
a[[2,3,6], ...][..., [3,2]]
You're doing fancy indexing, so there are copies both times.
D'oh!
So I guess the only way to avoid the second copy is to do what Jon initially suggested, i.e. a[ix_([2,3,6],range(a.shape[1]),[3,2])] ?
I suppose xrange would be better than arange() or range() as it wouldn't create and destroy the list all at once.
I believe an array would be created from it, so arange() would be the "best" bet there. But really, the difference is so trivial. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Thu, Feb 26, 2009 at 21:00, Jonathan Taylor <jonathan.taylor@utoronto.ca> wrote:
Am I right to assume that there is no way elegant way to interact with slices. i.e. Is there anyway to get
a[ix_([2,3,6],:,[3,2])]
to work? So that the dimension is completely specified? Or perhaps the only way to do this is via
a[ix_([2,3,6],range(a.shape[1]),[3,2])]
If anyone knows a better way?
One could probably make ix_() take slice objects, too, to generate the correct arange() in the appropriate place. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Hi Robert 2009/2/27 Robert Kern <robert.kern@gmail.com>:
a[ix_([2,3,6],range(a.shape[1]),[3,2])]
If anyone knows a better way?
One could probably make ix_() take slice objects, too, to generate the correct arange() in the appropriate place.
I was wondering how one would implement this, since the ix_ function has no knowledge of the dimensions of "a". The best I could do was to allow a[ix_[[2,3,6], :3, [3, 2]] to work (see attached patch). Cheers Stéfan
On Tue, Mar 3, 2009 at 03:11, Stéfan van der Walt <stefan@sun.ac.za> wrote:
Hi Robert
2009/2/27 Robert Kern <robert.kern@gmail.com>:
a[ix_([2,3,6],range(a.shape[1]),[3,2])]
If anyone knows a better way?
One could probably make ix_() take slice objects, too, to generate the correct arange() in the appropriate place.
I was wondering how one would implement this, since the ix_ function has no knowledge of the dimensions of "a".
No, you're right. It doesn't work. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Tue, 03 Mar 2009 19:26:38 -0600, Robert Kern wrote:
On Tue, Mar 3, 2009 at 03:11, Stéfan van der Walt <stefan@sun.ac.za> wrote: [clip: ix_(...,:,...)] No, you're right. It doesn't work.
The only way to make it work seems to be to define a special shaped slice object that indexing understands... I wonder if these would be worth the trouble. -- Pauli Virtanen
Thanks, Robert, for the quick response. On 9/11/07, Robert Kern <robert.kern@gmail.com> wrote:
There are operations you can express with this form that you couldn't if the behavior that you expected were the case whereas you can get the result you want relatively straightforwardly.
In [6]: a[q[:,newaxis], r]
Ah, yes, of course. I forgot my broadcasting rules. However, I also thank Timothy Hochberg for the moral support of agreeing that the obvious way (a[q,r]) should be reasonably expected to work. Thanks again for your help - the routine is now working nicely. Mike -- mike.ressler@alum.mit.edu
participants (8)
-
David Warde-Farley -
Jonathan Taylor -
Mike Ressler -
Pauli Virtanen -
Robert Kern -
Stéfan van der Walt -
Timothy Hochberg -
Travis E. Oliphant