[Numpy-discussion] ndarray.resize method and reference counting

[Jon Olav Vik]

Scott Sinclair <scott.sinclair.za <at> gmail.com> writes:

> >>> import numpy as np
> >>> x = np.eye(3)
> >>> x
> array([[ 1.,  0.,  0.],
>        [ 0.,  1.,  0.],
>        [ 0.,  0.,  1.]])
> >>> x.resize((5,5))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ValueError: cannot resize an array that has been referenced or is referencing
> another array in this way.  Use the resize function

I'm having the same problem, and have resigned to using "x = resize(x, 
newshape)" rather than "x.resize(newshape)" as suggested in the error message. 
Anything else becomes almost impossible to debug. Basically, you must never 
step into a function that does x.resize().

Part of the reason is explained here:
http://article.gmane.org/gmane.comp.python.numeric.general/5461

In my experience, reference counting behaves differently between:
* non-interactive running and simple IPython "run" of scripts (works as 
intended)
* interactive entering of commands in IPython (entering "x" at prompt "In [n]" 
defines an alias _n)
* debugging with "run -d" in Python (symbols in inner scopes seem to get extra 
references to them)

See the example below (using IPython 0.8.1/Python 2.5.2/Numpy 1.1.0/Linux) for 
illustration.

Hope this helps,
Jon Olav

====
In [10]: cat test.py
import numpy, sys

x = numpy.zeros(3)
print sys.getrefcount(x)
x.resize(2)
x
print sys.getrefcount(x)
x.resize(1)

def test():
    y = numpy.zeros(3)
    print sys.getrefcount(y)
    y.resize(2)
    y
    print sys.getrefcount(y)
    y.resize(1)

test()


In [11]: run test.py
2
2
2
2
=====

Non-interactively, getrefcount() returns 2 on all four occasions. Entering the 
same commands interactively, the refcount of x increases by 4 on entering "x" 
at the prompt. Inside a function call there is no problem, though.

=====
In [2]: import numpy, sys
In [3]: x = numpy.zeros(3)
In [4]: print sys.getrefcount(x)
2
In [5]: x.resize(2)
In [6]: x
Out[6]: array([ 0.,  0.])
In [7]: print sys.getrefcount(x)
6
In [8]: x.resize(1)
---------------------------------------------------------------------------
<type 'exceptions.ValueError'>: cannot resize an array that has been referenced 
or is referencing another array in this way.  Use the resize function

In [9]: def test():
   ...:         y = numpy.zeros(3)
   ...:         print sys.getrefcount(y)
   ...:         y.resize(2)
   ...:         y
   ...:         print sys.getrefcount(y)
   ...:         y.resize(1)
   ...:
In [10]: test()
2
2
=====

When debugging, however, the x.resize() works fine, while the y.resize() inside 
the test() function fails. (A workaround is to step over the function call, 
using "n" rather than "s" in line 19.)

====
In [5]: run -d test.py
ipdb> s
1---> 1 import numpy, sys
----> 3 x = numpy.zeros(3)
----> 4 print sys.getrefcount(x)
2

----> 5 x.resize(2)
----> 6 x
----> 7 print sys.getrefcount(x)
2

----> 8 x.resize(1)
---> 10 def test():

---> 19 test()
ipdb> s
---> 10 def test():
---> 11     y = numpy.zeros(3)
---> 12     print sys.getrefcount(y)
3

---> 13     y.resize(2)
ValueError: 'cannot resize an array that has been referenced or is referencing
\nanother array in this way.  Use the resize function'
=====