Best way to construct/slice 3dimensional ndarray from multiple 2d ndarrays?
Hi all, I have a method which builds a single 3d ndarray from several equaldimension 2d ndarrays, and another method which extracts the original 2d ndarrays back out from the 3d one. The way I'm doing this right now is pretty simple, e.g.: cube = np.asarray([arr1, arr2,...]) ... x = cube[0] I believe the way this is currently handled, is to use new memory locations first for the 3d array, and then later for the 2d slices. Does anyone know if there is a better way to handle this? Ideally, I would like to reuse the same memory locations instead of copying it anew each time. Also, when subclassing ndarray and calling obj = data.view(cls) for an ndarray "data", does this copy the data into the new object by value or reference? The method which extracts the 2d slice actually returns a subclass of ndarray created using the extracted data, so this is why I ask. Any insight or suggestions would be appreciated. Thanks! Keith
Right now you allocate new memory only when creating your 3d array. When you do "x = cube[0]" this creates a view that does not allocate more memory. If your 2d arrays were created independently, I don't think you can avoid this. If you have some control on the way your original 2D arrays are created, you can first initialize the 3d array with correct shape (or an upper bound on the number of 2d arrays), then use views on this 3d array ("x_i = cube[i]") to fill your 2D arrays in the same memory space. I can't help with your second question, sorry. = Olivier 2011/8/17 Keith Hughitt <keith.hughitt@gmail.com>
Hi all,
I have a method which builds a single 3d ndarray from several equaldimension 2d ndarrays, and another method which extracts the original 2d ndarrays back out from the 3d one.
The way I'm doing this right now is pretty simple, e.g.:
cube = np.asarray([arr1, arr2,...]) ... x = cube[0]
I believe the way this is currently handled, is to use new memory locations first for the 3d array, and then later for the 2d slices.
Does anyone know if there is a better way to handle this? Ideally, I would like to reuse the same memory locations instead of copying it anew each time.
Also, when subclassing ndarray and calling obj = data.view(cls) for an ndarray "data", does this copy the data into the new object by value or reference? The method which extracts the 2d slice actually returns a subclass of ndarray created using the extracted data, so this is why I ask.
Any insight or suggestions would be appreciated.
Thanks! Keith
_______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
The 2d arrays are read in using another library (PyFITS), so I probably won't be able to control that too much, otherwise that sounds like exactly what I need. I'm actually overriding the indexing operation so that the user gets back an ndarray subclass when they do "cube[0]": def __getitem__(self, key): """Overiding indexing operation""" if isinstance(key, int): data = np.ndarray.__getitem__(self, key) header = self._headers[key] for cls in BaseMap.__subclasses__(): if cls.is_datasource_for(header): return cls(data, header) raise UnrecognizedDataSouceError else: return np.ndarray.__getitem__(self, key) Which relates to the second part of the question I had about how the ndarray is handled when an instance of a ndarray subclass is created. Thanks for the suggestions! Keith On Wed, Aug 17, 2011 at 1:00 PM, Olivier Delalleau <shish@keba.be> wrote:
Right now you allocate new memory only when creating your 3d array. When you do "x = cube[0]" this creates a view that does not allocate more memory.
If your 2d arrays were created independently, I don't think you can avoid this. If you have some control on the way your original 2D arrays are created, you can first initialize the 3d array with correct shape (or an upper bound on the number of 2d arrays), then use views on this 3d array ("x_i = cube[i]") to fill your 2D arrays in the same memory space.
I can't help with your second question, sorry.
= Olivier
2011/8/17 Keith Hughitt <keith.hughitt@gmail.com>
Hi all,
I have a method which builds a single 3d ndarray from several equaldimension 2d ndarrays, and another method which extracts the original 2d ndarrays back out from the 3d one.
The way I'm doing this right now is pretty simple, e.g.:
cube = np.asarray([arr1, arr2,...]) ... x = cube[0]
I believe the way this is currently handled, is to use new memory locations first for the 3d array, and then later for the 2d slices.
Does anyone know if there is a better way to handle this? Ideally, I would like to reuse the same memory locations instead of copying it anew each time.
Also, when subclassing ndarray and calling obj = data.view(cls) for an ndarray "data", does this copy the data into the new object by value or reference? The method which extracts the 2d slice actually returns a subclass of ndarray created using the extracted data, so this is why I ask.
Any insight or suggestions would be appreciated.
Thanks! Keith
_______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
_______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
On Wed, Aug 17, 2011 at 9:04 AM, Keith Hughitt <keith.hughitt@gmail.com>wrote:
Also, when subclassing ndarray and calling obj = data.view(cls) for an ndarray "data", does this copy the data into the new object by value or reference? The method which extracts the 2d slice actually returns a subclass of ndarray created using the extracted data, so this is why I ask.
I think it should pass a reference  the following code suggests the subclass is sharing the same fundamental array object. You can use the .base attribute of the ndarray object to see if it is a view back to another ndarray object: import numpy as np class TestClass(np.ndarray): def __new__(cls, inp_array): return inp_array.view(cls) In [2]: x = np.ones(5) In [3]: obj = TestClass(x) In [4]: id(x), id(obj), id(obj.base) Out[4]: (23517648, 19708080, 23517648) In [5]: print x, obj [ 1. 1. 1. 1. 1.] [ 1. 1. 1. 1. 1.] In [6]: x[2] = 2 In [7]: print x, obj [ 1. 1. 2. 1. 1.] [ 1. 1. 2. 1. 1.] If you change the TestClass.__new__() to: "return np.array(inp_array).view(cls)" then you will make a copy of the input array instead, if that is needed. In that case, it looks like the .base attribute is a new ndarray, copied from the input array. Aronne [PS  also note that .base is set to None, if the ndarray is not a view into another ndarray; it turns out that None has a valid object number, which confused me at first  see id(None).]
Great! It looks like it is in fact working as desired: In [4]: cube.shape Out[4]: (5, 4096, 4096) In [5]: slice = cube[0] In [6]: cube[0,1000,1000] Out[6]: 618 In [7]: slice[1000,1000] Out[7]: 618 In [8]: slice[1000,1000] = 123 In [9]: cube[0, 1000,1000] Out[9]: 123 I didn't know about the .base attribute; that is really useful. Thank you both for the feedback. Keith On Wed, Aug 17, 2011 at 1:46 PM, Aronne Merrelli <aronne.merrelli@gmail.com>wrote:
On Wed, Aug 17, 2011 at 9:04 AM, Keith Hughitt <keith.hughitt@gmail.com>wrote:
Also, when subclassing ndarray and calling obj = data.view(cls) for an ndarray "data", does this copy the data into the new object by value or reference? The method which extracts the 2d slice actually returns a subclass of ndarray created using the extracted data, so this is why I ask.
I think it should pass a reference  the following code suggests the subclass is sharing the same fundamental array object. You can use the .base attribute of the ndarray object to see if it is a view back to another ndarray object:
import numpy as np class TestClass(np.ndarray): def __new__(cls, inp_array): return inp_array.view(cls)
In [2]: x = np.ones(5) In [3]: obj = TestClass(x) In [4]: id(x), id(obj), id(obj.base) Out[4]: (23517648, 19708080, 23517648) In [5]: print x, obj [ 1. 1. 1. 1. 1.] [ 1. 1. 1. 1. 1.] In [6]: x[2] = 2 In [7]: print x, obj [ 1. 1. 2. 1. 1.] [ 1. 1. 2. 1. 1.]
If you change the TestClass.__new__() to: "return np.array(inp_array).view(cls)" then you will make a copy of the input array instead, if that is needed. In that case, it looks like the .base attribute is a new ndarray, copied from the input array.
Aronne
[PS  also note that .base is set to None, if the ndarray is not a view into another ndarray; it turns out that None has a valid object number, which confused me at first  see id(None).]
_______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
participants (3)

Aronne Merrelli

Keith Hughitt

Olivier Delalleau