I've narrowed this down to a change between 1.0.4 and 1.1.0. Valgrind (of v1.3.0) shows the following result. The change was in setArrayFromSequence where PyArray_EnsureArray gets invoked in v1.1.0 where it did not in v1.0.4.
==10132== 4,474,768 (3,197,200 direct, 1,277,568 indirect) bytes in 39,965 blocks are definitely lost in loss record 36 of 36
==10132== at 0x4905B65: malloc (vg_replace_malloc.c:149)
==10132== by 0x7FFFDAA: array_alloc (arrayobject.c:7387)
==10132== by 0x8003465: PyArray_NewFromDescr (arrayobject.c:5900)
==10132== by 0x802434D: PyArray_EnsureArray (multiarraymodule.c:226)
==10132== by 0x8025766: setArrayFromSequence (arrayobject.c:7938)
==10132== by 0x80256B8: setArrayFromSequence (arrayobject.c:7957)
==10132== by 0x800F5FD: PyArray_FromAny (arrayobject.c:7984)
==10132== by 0x802CC21: PyArray_CheckFromAny (arrayobject.c:9530)
==10132== by 0x8037F64: _array_fromobject (multiarraymodule.c:6329)
==10132== by 0x4AC9501: PyEval_EvalFrameEx (ceval.c:3612)
==10132== by 0x4ACA894: PyEval_EvalCodeEx (ceval.c:2875)
==10132== by 0x4ACAA11: PyEval_EvalCode (ceval.c:514)
==10132== by 0x4AEC98B: PyRun_FileExFlags (pythonrun.c:1273)
==10132== by 0x4AED612: PyRun_SimpleFileExFlags (pythonrun.c:879)
==10132== by 0x4AF88C7: Py_Main (main.c:532)
==10132== by 0x38C821C3FA: (below main) (in /lib64/tls/libc-2.3.4.so)
==10132==
This simple example causes a massive leak in v1.3.0 which didn't exist in v1.0.1. What am I doing wrong? If I replacearr = [Array((2,2)), Array((2,2))]with
arr = [numpy.ndarray((2,2,)), numpy.ndarray((2,2))]then I don't have the leakimport numpyimport gc
class Array(numpy.ndarray):def __new__(subtype, shape, dtype=float, buffer=None, offset=0, strides=None, order=None, info=None):return numpy.ndarray.__new__(subtype, shape, dtype, buffer, offset, strides, order)def __array_finalize__(self, obj):print 'called array_finalize'if __name__=='__main__':arr = [Array((2,2)), Array((2,2))]nbytesAllocated = 0for i in xrange(1000000000):a = numpy.array(arr)nbytesAllocated += a.nbytesif i%1000 == 0:print 'allocted %s'%nbytesAllocatedgc.collect()
--
--jlm
--
--jlm