Thanks Andrew, that was a useful summary. I wish I had more time to work on optimizing numarray personally, but I don't. Instead I'll try to share what I know of the state of __del__/tp_dealloc so that people who want to work on it can come up with something better: 1. We need __del__/tp_dealloc. (This may be controversial but I hope not). Using the destructor makes the high level C-API cleaner. Getting rid of it means changing the C-API. __del__/tp_dealloc is used to transparently copy the contents of a working array back onto an ill-behaved (byteswapped, etc...) source array at extension function exit time. 2. There's a problem with the tp_dealloc I originally implemented which causes it to segfault for a ./configure'ed --with-pydebug Python. Looking at it today, it looks like it may be an exit-time garbage collection problem. There is no explicit garbage collection support in _numarray or _ndarray, so that may be the problem. 3. We're definitely not exploiting the "single underscore rule" yet. We use single underscores mostly to hide globals from module export. I don't think this is really critical, but that's the state of things. 4. Circular references should only be a problem for numerical arrays with "user introduced" cycles. numarray ObjectArrays have no __del__. I attached a patch against CVS that reinstates the old tp_dealloc; this shows where I left off in case someone has insight on how to fix it. I haven't tested it recently for a non-debug Python; I think it works. The patch segfaults after the C-API examples/selftest for debug Pythons: % python setup.py install --selftest Using EXTRA_COMPILE_ARGS = [] running install running build running build_py copying Lib/numinclude.py -> build/lib.linux-i686-2.4/numarray running build_ext running install_lib copying build/lib.linux-i686-2.4/numarray/numinclude.py -> /home/jmiller/work/lib/python2.4/site-packages/numarray byte-compiling /home/jmiller/work/lib/python2.4/site-packages/numarray/numinclude.py to numinclude.pyc running install_headers copying Include/numarray/numconfig.h -> /home/jmiller/work/include/python2.4/numarray running install_data Testing numarray 1.2a on Python (2, 4, 0, 'final', 0) numarray.numtest: ((0, 1231), (0, 1231)) numarray.ieeespecial: (0, 20) numarray.records: (0, 48) numarray.strings: (0, 186) numarray.memmap: (0, 82) numarray.objects: (0, 105) numarray.memorytest: (0, 16) numarray.examples.convolve: ((0, 20), (0, 20), (0, 20), (0, 20)) Segmentation fault (core dumped) That's all I can add for now. Regards, Todd On Mon, 2005-01-31 at 11:54 +1100, Andrew McNamara wrote:
This benchmark made me suspicious since I had already found it odd before that killing a numarray calculation with Ctrl-C nearly always gives a backtrace starting in __del__
Much of the python machinery may have been torn down when your __del__ method is called while the interpreter is exiting (I'm asuming you're talking about a script, rather than interactive mode). Code should be prepared for anything to fail - it's quite common for parts of __builtins__ to have been disassembled, etc.
The language reference has this to say:
http://python.org/doc/2.3.4/ref/customization.html#l2h-174
Warning: Due to the precarious circumstances under which __del__() methods are invoked, exceptions that occur during their execution are ignored, and a warning is printed to sys.stderr instead. Also, when __del__() is invoked in response to a module being deleted (e.g., when execution of the program is done), other globals referenced by the __del__() method may already have been deleted. For this reason, __del__() methods should do the absolute minimum needed to maintain external invariants. Starting with version 1.5, Python guarantees that globals whose name begins with a single underscore are deleted from their module before other globals are deleted; if no other references to such globals exist, this may help in assuring that imported modules are still available at the time when the __del__() method is called.
Another important caveat of classes with __del__ methods is mentioned in the library reference for the "gc" module:
http://python.org/doc/2.3.4/lib/module-gc.html
Objects that have __del__() methods and are part of a reference cycle cause the entire reference cycle to be uncollectable, including objects not necessarily in the cycle but reachable only from it. Python doesn't collect such cycles automatically because, in general, it isn't possible for Python to guess a safe order in which to run the __del__() methods.