Hello,
I have an indexing problem which I suspect has a simple solution, but I've not been able to piece together various threads I've read on this list to solve.
I have an 80x1200x1200 nd.array of floats this_par. I have a 1200x1200 boolean array idx, and an 80-element float array pars. For each element of idx that is True, I wish to replace the corresponding 80x1x1 slice of this_par with the elements of pars.
I've tried lots of variations on the theme of
this_par[idx[np.newaxis, ...]] = pars[:, np.newaxis, np.newaxis]
but so far, no dice.
Any help greatly appreciated!
Thanks, Tim
--
Timothy W. Hilton PhD Candidate, Department of Meteorology The Pennsylvania State University 503 Walker Building, University Park, PA 16802 hilton@meteo.psu.edu
On Tue, Sep 21, 2010 at 6:20 PM, Timothy W. Hilton hilton@meteo.psu.edu wrote:
Hello,
I have an indexing problem which I suspect has a simple solution, but I've not been able to piece together various threads I've read on this list to solve.
I have an 80x1200x1200 nd.array of floats this_par. I have a 1200x1200 boolean array idx, and an 80-element float array pars. For each element of idx that is True, I wish to replace the corresponding 80x1x1 slice of this_par with the elements of pars.
I've tried lots of variations on the theme of
this_par[idx[np.newaxis, ...]] = pars[:, np.newaxis, np.newaxis]
but so far, no dice.
Any help greatly appreciated!
Thanks, Tim
This works, although I imagine it could be streamlined.
In [1]: this_par = N.ones((2,4,4)) In [2]: idx = N.random.random((4,4)) > 0.5 In [3]: pars = N.arange(2) - 10 In [4]: this_par[:,idx] = N.tile(pars, (idx.sum(), 1)).transpose() In [5]: idx Out[5] array([[ True, False, True, False], [False, False, True, True], [False, False, False, False], [False, False, False, False]], dtype=bool) In [6]: this_par Out[6]: array([[[-10., 1., -10., 1.], [ 1., 1., -10., -10.], [ 1., 1., 1., 1.], [ 1., 1., 1., 1.]], [[ -9., 1., -9., 1.], [ 1., 1., -9., -9.], [ 1., 1., 1., 1.], [ 1., 1., 1., 1.]]])
Brett
On Tue, Sep 21, 2010 at 7:54 PM, Brett Olsen brett.olsen@gmail.com wrote:
On Tue, Sep 21, 2010 at 6:20 PM, Timothy W. Hilton hilton@meteo.psu.edu wrote:
Hello,
I have an indexing problem which I suspect has a simple solution, but I've not been able to piece together various threads I've read on this list to solve.
I have an 80x1200x1200 nd.array of floats this_par. I have a 1200x1200 boolean array idx, and an 80-element float array pars. For each element of idx that is True, I wish to replace the corresponding 80x1x1 slice of this_par with the elements of pars.
I've tried lots of variations on the theme of
this_par[idx[np.newaxis, ...]] = pars[:, np.newaxis, np.newaxis]
but so far, no dice.
Any help greatly appreciated!
Thanks, Tim
This works, although I imagine it could be streamlined.
In [1]: this_par = N.ones((2,4,4)) In [2]: idx = N.random.random((4,4)) > 0.5 In [3]: pars = N.arange(2) - 10 In [4]: this_par[:,idx] = N.tile(pars, (idx.sum(), 1)).transpose() In [5]: idx Out[5] array([[ True, False, True, False], [False, False, True, True], [False, False, False, False], [False, False, False, False]], dtype=bool) In [6]: this_par Out[6]: array([[[-10., 1., -10., 1.], [ 1., 1., -10., -10.], [ 1., 1., 1., 1.], [ 1., 1., 1., 1.]], [[ -9., 1., -9., 1.], [ 1., 1., -9., -9.], [ 1., 1., 1., 1.], [ 1., 1., 1., 1.]]])
introspection works easier with an example
indexing with slice and boolean seems to flatten the boolean part, then we need only one newaxis ? seems to work
this_par[:,idx].shape # = N.tile(pars, (idx.sum(), 1)).transpose()
(2, 8)
idx
array([[ True, False, False, False], [ True, True, True, False], [ True, True, False, False], [ True, False, True, False]], dtype=bool)
this_par[:,idx] = pars[:,N.newaxis] this_par
array([[[-10., 1., 1., 1.], [-10., -10., -10., 1.], [-10., -10., 1., 1.], [-10., 1., -10., 1.]],
[[ -9., 1., 1., 1.], [ -9., -9., -9., 1.], [ -9., -9., 1., 1.], [ -9., 1., -9., 1.]]])
Josef
Brett _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Sep 21, 2010 at 6:20 PM, Timothy W. Hilton hilton@meteo.psu.eduwrote:
Hello,
I have an indexing problem which I suspect has a simple solution, but I've not been able to piece together various threads I've read on this list to solve.
I have an 80x1200x1200 nd.array of floats this_par. I have a 1200x1200 boolean array idx, and an 80-element float array pars. For each element of idx that is True, I wish to replace the corresponding 80x1x1 slice of this_par with the elements of pars.
I've tried lots of variations on the theme of
this_par[idx[np.newaxis, ...]] = pars[:, np.newaxis, np.newaxis]
but so far, no dice.
Any help greatly appreciated!
Thanks, Tim
This is a tough indexing question for me. I can visualize the arrays and the required selections in my mind, but hard to put them into application as in this case. Beyond 2D array operations indexing and broadcasting operations get somewhat complex to interpret at the first look and likewise later when I read and decipher my own code, not to mention if someone else wants to understand that same code.
What do you do with such big arrays? It is good to see new atmospheric sciences people in the lists. Our department is still swarming with IDL and Matlab users.
On 21 September 2010 19:20, Timothy W. Hilton hilton@meteo.psu.edu wrote:
I have an 80x1200x1200 nd.array of floats this_par. I have a 1200x1200 boolean array idx, and an 80-element float array pars. For each element of idx that is True, I wish to replace the corresponding 80x1x1 slice of this_par with the elements of pars.
I've tried lots of variations on the theme of
this_par[idx[np.newaxis, ...]] = pars[:, np.newaxis, np.newaxis]
but so far, no dice.
How about this?
In [1]: A = np.zeros((2,3,5))
In [2]: B = np.array([1,2])
In [3]: C = np.zeros((3,5), dtype=np.bool)
In [4]: C[1,1] = True
In [5]: C[2,3] = True
In [6]: C Out[6]: array([[False, False, False, False, False], [False, True, False, False, False], [False, False, False, True, False]], dtype=bool)
In [7]: A[:,C] = B[:,np.newaxis]
In [8]: A Out[8]: array([[[ 0., 0., 0., 0., 0.], [ 0., 1., 0., 0., 0.], [ 0., 0., 0., 1., 0.]],
[[ 0., 0., 0., 0., 0.], [ 0., 2., 0., 0., 0.], [ 0., 0., 0., 2., 0.]]])
The key is that indexing with C replaces the two axes C is indexing with only one; boolean indexing necessarily flattens the relevant axes. You can check this with (e.g.) A[:,C].shape.
Be careful with these "mixed" indexing modes (partly fancy indexing, partly slicing) as they can sometimes seem to reorder your axes for you:
In [16]: np.zeros((2,3,7))[:,np.ones(5,dtype=int),np.ones(5,dtype=int)].shape Out[16]: (2, 5)
In [17]: np.zeros((2,3,7))[np.ones(5,dtype=int),:,np.ones(5,dtype=int)].shape Out[17]: (5, 3)
In [18]: np.zeros((2,3,7))[np.ones(5,dtype=int),np.ones(5,dtype=int),:].shape Out[18]: (5, 7)
Anne
Hi Brett, Josef, Gökhan, and Anne,
Many thanks for the suggestions! This indexing problem was not as straight-forward as I had anticipated. I have it working now, thanks to your help.
Gökhan, I'm driving an ecosystem--atmosphere carbon dioxide flux model with MODIS measurements, so this_par and pars are parameter values that I need to combine with the measurements. The arrays represent time series of model parameters and MODIS reflectances and land surface products (enhanced vegetation index, land cover class, vegetation dynamics) at 1 km resolution for 1200 km by 1200 km "tiles".
There are lots of IDL and Matlab folks in my department too. I've been using R and, more recently, Scipy... Being able to test out code offline on my own machine without worrying about licenses (not only cost but also talking to license servers, availability of site licenses, etc.) is a big big help!
Along those lines, many thanks also to all who have put so much time and energy into developing python and Scipy. It's a great platform!
-Tim
On Wed, Sep 2010, 22 at 02:22:50PM -0400, Anne Archibald wrote:
On 21 September 2010 19:20, Timothy W. Hilton hilton@meteo.psu.edu wrote:
I have an 80x1200x1200 nd.array of floats this_par. I have a 1200x1200 boolean array idx, and an 80-element float array pars. For each element of idx that is True, I wish to replace the corresponding 80x1x1 slice of this_par with the elements of pars.
I've tried lots of variations on the theme of
this_par[idx[np.newaxis, ...]] = pars[:, np.newaxis, np.newaxis]
but so far, no dice.
How about this?
In [1]: A = np.zeros((2,3,5))
In [2]: B = np.array([1,2])
In [3]: C = np.zeros((3,5), dtype=np.bool)
In [4]: C[1,1] = True
In [5]: C[2,3] = True
In [6]: C Out[6]: array([[False, False, False, False, False], [False, True, False, False, False], [False, False, False, True, False]], dtype=bool)
In [7]: A[:,C] = B[:,np.newaxis]
In [8]: A Out[8]: array([[[ 0., 0., 0., 0., 0.], [ 0., 1., 0., 0., 0.], [ 0., 0., 0., 1., 0.]],
[[ 0., 0., 0., 0., 0.], [ 0., 2., 0., 0., 0.], [ 0., 0., 0., 2., 0.]]])
The key is that indexing with C replaces the two axes C is indexing with only one; boolean indexing necessarily flattens the relevant axes. You can check this with (e.g.) A[:,C].shape.
Be careful with these "mixed" indexing modes (partly fancy indexing, partly slicing) as they can sometimes seem to reorder your axes for you:
In [16]: np.zeros((2,3,7))[:,np.ones(5,dtype=int),np.ones(5,dtype=int)].shape Out[16]: (2, 5)
In [17]: np.zeros((2,3,7))[np.ones(5,dtype=int),:,np.ones(5,dtype=int)].shape Out[17]: (5, 3)
In [18]: np.zeros((2,3,7))[np.ones(5,dtype=int),np.ones(5,dtype=int),:].shape Out[18]: (5, 7)
Anne _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion