[AstroPy] PyFITS memory usage
embray at stsci.edu
Fri May 13 13:29:29 EDT 2011
On 05/13/2011 12:58 PM, G Jones wrote:
> I am seeing what looks like a memory leak when using PyFITS. It may not
> actually be a leak, but I am curious if anyone can explain the behavior.
> I am processing PSRFITS files which are ~40 GB and have a table with the
> data in it. Since the files are so large, I open them with memmap=True.
> I would expect the following simple script to have a constant memory
> footprint, but instead I see continuously growing memory usage.
> import pyfits
> hdul = pyfits.open('myfile.fits',mode="readonly", memmap=True)
> primary,secondary = hdul
> for k in range(100):
> x = secondary.data[k].field('DATA').astype('float32')
> I can see that the data is being read in from disk as it should be, but
> I don't expect the memory usage to increase, since x is constantly
> overwritten. The versions of programs I am using are:
> RHEL 5 x86_64 (2.6.18-194.32.1.el5)
> python 2.7.1
> numpy 2.0.0.dev
> pyfits 2.4.0
There is a known memory leak in PyFITS 2.4.0 (and possibly below). See
http://trac6.assembla.com/pyfits/ticket/49. This has been fixed in the
current development version, and the fix should be in the next release.
If you can patch your PyFITS installation, I have a patch for the memory
leak issue that applies cleanly to 2.4.0.
Though enough people have had this issue, that maybe we could release a
2.4.1 with this fix in the meantime? That might not be a bad idea.
More information about the AstroPy