[AstroPy] pyfits - memory leak in new_table / 'hello, i am new here'

[Erik Bray]

On 06/15/2011 08:38 AM, Martin Raue wrote:
> Hi Stefan,
>
>> I tried your script with revision 851 and the memory leak is there.
>> Then I tried with trunk (currently 925) and there is no more memory leak.
>
> Thanks for checking!
>
>>
>> The problem you had with the installation happened to me as well, but it
>> worked for me with:
>>
>> sudo ./setup.py install
>>
>
> Unfortunately, that produces the same error (also r925). Erik Bray wrote me that he will have a look at the installer.
>
> In your case, it could be that your user and your superuser have different python version (and/or setups) installed or that /usr/bin/env points toward a different version of python then the python command.
>
> Cheers, m

Yes, the problem here is with a semi-experimental tool called d2to1 that 
I developed and am using in the packaging of pyfits.

(For those interested, its purpose is to make distutils2/packaging-style 
setup.cfg files work with setuptools/distribute).

That particular bug can only happen if you didn't already have d2to1 
installed.  I should have a fix for it checked in shortly.

As for the actual memory issue:  I confirmed that on trunk there is no 
memory leak.  But it is worth pointing out that pyfits.new_table() uses 
more memory than one might expect.  This is because it allocates a new 
array that has space for all the columns that will be added to it.  The 
original column arrays can't be use directly, because they have to be 
interleaved.

But if you're having trouble with memory usage when building tables, 
there are ways of dealing with this.  I can make specific suggestions if 
you're looking for ideas on how to go about it.

Erik