[AstroPy] pyfits - memory leak in new_table / 'hello, i am new here'

[Martin Raue]

Hello all,

I am new here. I work in very high energy gamma-ray astronomy with HESS and CTA and currently developing some high level analysis tools for CTA using python (the classical numpy, scipy, matplotlib, pyfits).

I stumbled upon a problem with memory leakage in pyfits when using new_table.

Something similar has been reported here
http://physicsnlinux.wordpress.com/2011/03/28/pyfits-memory-leak-in-new_table/#more-103
and should be fixed in
http://trac6.assembla.com/pyfits/changeset/844

But I still have problems in 844 and 851. Unfortunately, I have problems installing the latest trunk on my system (some output attached).

Any suggestion and help would be very welcome!

Best wishes,
Martin

==============
Example script

import numpy as np
import pyfits
import gc

for i in range(20) :

    # create dummy data
    a = np.linspace(-1., 1., 1000000)
    b = a ** 3.

    a_col = pyfits.Column(name='ACOL', format='1E', array=a)
    b_col = pyfits.Column(name='BCOL', format='1E', array=b)

    coldefs_new = pyfits.ColDefs([a_col, b_col])

    newtable = pyfits.new_table(coldefs_new)

    # try to clean memory
    del newtable
    del coldefs_new
    gc.collect()

gc.collect()

=======================================================
setup.py output when trying to install the pyfits trunk

Something wrong in the installer modules ...

[uh2macastro04:~/tmp/trunk] $ python setup.py install
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.15.tar.gz
Extracting in /var/folders/NT/NTbrVxJ-Gw4ybwwyECKVcE+++TM/-Tmp-/tmp4nv73s
Now working in /var/folders/NT/NTbrVxJ-Gw4ybwwyECKVcE+++TM/-Tmp-/tmp4nv73s/distribute-0.6.15
Building a Distribute egg in /Users/mraue/tmp/trunk
/Users/mraue/tmp/trunk/distribute-0.6.15-py2.6.egg
zip_safe flag not set; analyzing archive contents...

Installed /private/var/folders/NT/NTbrVxJ-Gw4ybwwyECKVcE+++TM/-Tmp-/easy_install-NTCEXF/stsci.distutils-0.2dev-r13055/d2to1-0.1.5-py2.6.egg
zip_safe flag not set; analyzing archive contents...
stsci.__init__: module references __path__

Installed /Users/mraue/tmp/trunk/stsci.distutils-0.2dev_r13055-py2.6.egg
Searching for d2to1>=0.1.5
Reading http://pypi.python.org/simple/d2to1/
Best match: d2to1 0.1.5
Downloading http://pypi.python.org/packages/source/d/d2to1/d2to1-0.1.5.tar.gz#md5=34129f809b503449b283771b5c3f038b
Processing d2to1-0.1.5.tar.gz
Running d2to1-0.1.5/setup.py -q bdist_egg --dist-dir /var/folders/NT/NTbrVxJ-Gw4ybwwyECKVcE+++TM/-Tmp-/easy_install-ABs3GN/d2to1-0.1.5/egg-dist-tmp-oSIXxB
Traceback (most recent call last):
  File "setup.py", line 33, in <module>
    use_2to3=True
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/core.py", line 113, in setup
    _setup_distribution = dist = klass(attrs)
  File "build/bdist.macosx-10.3-fat/egg/setuptools/dist.py", line 221, in __init__
  File "build/bdist.macosx-10.3-fat/egg/setuptools/dist.py", line 245, in fetch_build_eggs
  File "build/bdist.macosx-10.3-fat/egg/pkg_resources.py", line 544, in resolve
  File "build/bdist.macosx-10.3-fat/egg/pkg_resources.py", line 786, in best_match
  File "build/bdist.macosx-10.3-fat/egg/pkg_resources.py", line 798, in obtain
  File "build/bdist.macosx-10.3-fat/egg/setuptools/dist.py", line 293, in fetch_build_egg
  File "build/bdist.macosx-10.3-fat/egg/setuptools/command/easy_install.py", line 584, in easy_install
  File "build/bdist.macosx-10.3-fat/egg/setuptools/command/easy_install.py", line 614, in install_item
  File "build/bdist.macosx-10.3-fat/egg/setuptools/command/easy_install.py", line 804, in install_eggs
  File "build/bdist.macosx-10.3-fat/egg/setuptools/command/easy_install.py", line 1081, in build_and_install
  File "build/bdist.macosx-10.3-fat/egg/setuptools/command/easy_install.py", line 1070, in run_setup
  File "build/bdist.macosx-10.3-fat/egg/setuptools/sandbox.py", line 29, in run_setup
  File "build/bdist.macosx-10.3-fat/egg/setuptools/sandbox.py", line 70, in run
  File "build/bdist.macosx-10.3-fat/egg/setuptools/sandbox.py", line 31, in <lambda>
  File "setup.py", line 21, in <module>
    except ImportError:
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/private/var/folders/NT/NTbrVxJ-Gw4ybwwyECKVcE+++TM/-Tmp-/easy_install-NTCEXF/stsci.distutils-0.2dev-r13055/d2to1-0.1.5-py2.6.egg/d2to1/core.py", line 111, in run_command
AttributeError: 'NoneType' object has no attribute 'info'

[uh2macastro04:~/tmp/trunk] $ python
Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin

-----------------------------------

Martin Raue

University of Hamburg
Institute for Experimental Physics
Luruper Chaussee 149
D-22761 Hamburg
Germany

Phone  : +49-40-8998-5216
Fax    : +49-40-8998-2170
E-Mail : martin.raue at desy.de