Hi,
I am using numpy with ipython from anaconda and I observe the following behavior:
Python 2.7.5 |Anaconda 1.7.0 (64-bit)| (default, Jun 28 2013, 22:10:09) Type "copyright", "credits" or "license" for more information.
IPython 1.0.0 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. Using matplotlib backend: Qt4Agg
In [1]: a= np.random.rand(500000,1000)
In [2]: a = a[:10000]
In [3]: c= np.random.rand(500000,1000)
In [4]:
After In[1] I have an extra 3.7 GB of memory used, but this memory is not released at In[2]. I thought there might be some clever memory management trick so I executted In[3] but that just added an extra 3.7GB of memorry without releasing anything.
Is that the right behavior in this case?
If yes then how do you release memorry by slicing away parts of an array? Can you give me a description of the numpy internals in this case?
Thank you very much for your time,
Georgios
On 28 October 2013 03:13, Georgios Exarchakis gexarcha1@gmail.com wrote:
If yes then how do you release memorry by slicing away parts of an array?
An array is a single Python object. In your example, there is always one reference pointing to the array (either the whole array or only a view), so the memory cannot be released.
In your case, a = a[:10000].copy() releases the memory by creating a new object and pointing a to this new one, so the big array gets unreferenced and thus, garbage collected. The price to pay is that, for some time, you have living in memory both the original array and your new one: if they were very big, you could run out of memory; and that you have to perform a memcopy (again, slow if you have lots of data).
/David.
Hi,
Along the line of what David said, I just looked at the flags :
a = np.arange(10)
a.flags [...] OWNDATA : True
a = a[:3]
a.flags [...] OWNDATA : False
Indeed, after a=a[:3], a is not the same Python object but still points to the data of the first object.
What I didn't find (by quick googling) is how to access the original array. Is it possible to access it (with Python code) ?
best, Pierre
On Mon, Oct 28, 2013 at 12:37 PM, Pierre Haessig pierre.haessig@crans.org wrote:
What I didn't find (by quick googling) is how to access the original array. Is it possible to access it (with Python code) ?
a.base
-- Robert Kern
Le 28/10/2013 13:40, Robert Kern a écrit :
What I didn't find (by quick googling) is how to access the original array. Is it possible to access it (with Python code) ?
a.base
Thanks! Is there a specific paragraph I missed in the user guide ?
I had googled "numpy access original array" and first result is http://docs.scipy.org/doc/numpy/user/basics.indexing.html In this document, it is mentionned several time that slicing yields "views of the original data", but the .base attribute is not mentionned. Should it be or is it out-of-scope of the Indexing guide ?
best, Pierre
On Mon, Oct 28, 2013 at 1:25 PM, Pierre Haessig pierre.haessig@crans.org wrote:
Le 28/10/2013 13:40, Robert Kern a écrit :
What I didn't find (by quick googling) is how to access the original array. Is it possible to access it (with Python code) ?
a.base
Thanks! Is there a specific paragraph I missed in the user guide ?
I had googled "numpy access original array" and first result is http://docs.scipy.org/doc/numpy/user/basics.indexing.html In this document, it is mentionned several time that slicing yields "views of the original data", but the .base attribute is not mentionned. Should it be or is it out-of-scope of the Indexing guide ?
I think it's out-of-scope for "Numpy basics", certainly. It's mostly an implementation detail. There isn't much that you can usefully do with it as a user.
http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.base.html
-- Robert Kern
On Mon, Oct 28, 2013 at 6:25 AM, Pierre Haessig pierre.haessig@crans.orgwrote:
a.base
In this document, it is mentionned several time that slicing yields "views of the original data", but the .base attribute is not mentionned. Should it be or is it out-of-scope of the Indexing guide ?
Indeed, that is not the least bit "basic" numpy usage -- the "basic way to do what you want is to keep the original array around in the first place -- i.e. give the slice a different name.
-Chris