Memory leak when using Typed Memory View and np array of objects
Hi, I'm new to cython's typed memory view, and found some cases where the function that uses typed memory view has memory leaking. The leak happens when I pass a numpy array of objects, where each object itself is a numpy array.( You can get this 'weird' object from construction a pandas Series with a list of numpy arrays). Please see the following code snippet or use the attached code to reproduce the case. I appreciate any help and suggestions in advance! (I also posted on the cython-users google group. Apologize for the redundancy.) # BEGIN CONTENT OF test.pyx # this does not leak cpdef int do_nothing(arr): return 0 # this does leak cpdef int do_nothing_typed(double[:] arr): return 0 # this does leak cpdef int do_nothing_but_copy(arr): cdef double[:] _arr = arr return 0 # END CONTENT OF test.pyx # BEGIN CONTENT OF runtest.py # ... omit all the imports here def gc_obj_hist(): """ Returns a sorted map from type to the counts of in memory objects with the type """ hst = defaultdict(lambda: 0) for v in gc.get_objects(): hst[type(v)] += 1 l = sorted(hst.iteritems(), key=operator.itemgetter(1), reverse=True) return l # NOT LEAK def test1(n=10000): s = pd.Series([np.random.randn(10) for i in range(n)]) for i in range(n): do_nothing(s[i]) print "Top 5 object types after test 1: " + str(gc_obj_hist()[:5]) # LEAK def test2(n=10000): s = pd.Series([np.random.randn(10) for i in range(n)]) for i in range(n): do_nothing_typed(s[i]) print "Top 5 object types after test 2: " + str(gc_obj_hist()[:5]) # LEAK def test3(n=10000): s = pd.Series([np.random.randn(10) for i in range(n)]) for i in range(n): do_nothing_but_copy(s[i]) print "Top 5 object types after test 3: " + str(gc_obj_hist()[:5]) # NOT LEAK def test4(n=10000): s = pd.Series([np.random.randn(10) for i in range(n)]) for i in range(n): do_nothing_but_copy(np.array(s[i])) print "Top 5 object types after test 4: " + str(gc_obj_hist()[:5]) if __name__ == "__main__": n = 100000 test1(n) test2(n) test3(n) test4(n) # END CONTENT OF runtest.py Thanks, -jay
participants (1)
-
Haijie Gu