[AstroPy] PyFITS Too many open files

[Erik Bray]

On 06/15/2013 06:42 PM, Nico Nell wrote:
> You can try running the python garbage collector after each file close with
> gc.collect().
>
> For the memmap issue consider submitting a bug report if you have time:
> https://github.com/spacetelescope/PyFITS/issues?state=open

Please don't :)  This isn't a bug.  I've seen this sort of thing before when 
people are opening lots of FITS files in a loop.  When you create references to 
the data with

     time = fits[1].data["TIME"]
     energy = fits[1].data["ENERGY"]

each one of those is opening file handles to read the data in.  I don't know 
what your code is doing with the data later in the loop, but my suggestion would 
be at the end of the loop to manually call

     del time
     del energy

Otherwise what ends up happening is that all your allowed file handles get used 
up before the Python garbage collector even gets a chance to run and close off 
unused file handles.

Erik

> On Sat, Jun 15, 2013 at 4:21 PM, Nicholas Earl <nchlsearl at gmail.com
> <mailto:nchlsearl at gmail.com>> wrote:
>
>     Thanks for the ideas! It seems that memmap=False solves the problem,
>     although I can see that it does begin using a large percentage of the memory
>     -- I can only imagine how much if I let it run on the entire set of
>     directories... Is there not any way to flush it all after I close the file?
>
>     I am using python 2.7.5, and PyFITs 3.1.2.
>
>     Thanks again, everyone, you're awesome.
>
>     Nicholas Earl
>     Graduate Research Assistant
>     San Diego State University
>
>
>     On Sat, Jun 15, 2013 at 11:01 AM, Nico Nell <nico.nell at gmail.com
>     <mailto:nico.nell at gmail.com>> wrote:
>
>         Sorry -- I should have read the blip of code better.
>
>         So, what version of python and pyfits are you using? How many files are
>         you opening?
>
>         What happens if you add 'memmap=False' in each of your pyfits open()
>         lines. The pyfits mmap stuff is relatively new so maybe there is an issue.
>
>         ~Nick
>
>
>         On Sat, Jun 15, 2013 at 11:29 AM, Nicholas Earl <nchlsearl at gmail.com
>         <mailto:nchlsearl at gmail.com>> wrote:
>
>             Sorry, but it seems I sent this back to only Nice Nell, and not the
>             entire list!
>
>             -----
>
>             Thanks for the reply. I was under the impression that the
>             with command automatically closed the file? Regardless, I went ahead
>             and wrote the script up like this
>
>             import pyfits
>
>             for filename in os.listdir(directory):
>             kep_id = filename[4:13]
>             fits = pyfits.open(directory + "/" + filename, ignore_missing_end=True)
>             time = fits[1].data["TIME"]
>             energy = fits[1].data["ENERGY"]
>             fits.close()
>
>             Unfortunately, I get the same error.
>
>             Nicholas Earl
>             Graduate Research Assistant
>             San Diego State University
>
>
>             On Fri, Jun 14, 2013 at 10:08 PM, Nico Nell <nico.nell at gmail.com
>             <mailto:nico.nell at gmail.com>> wrote:
>
>                 You need to close the files once you have retrieved the data...
>
>                 See here:
>                 http://pythonhosted.org/pyfits/api_docs/api_hdulists.html#pyfits.HDUList.close
>
>                 ~Nick
>
>
>                 On Fri, Jun 14, 2013 at 10:51 PM, Nicholas Earl
>                 <nchlsearl at gmail.com <mailto:nchlsearl at gmail.com>> wrote:
>
>                     Hello everyone,
>
>                     I'm having an issue with opening and retrieving information
>                     for a large amount of fits files. Mainly, I keep running
>                     into a mmap.error: [Errno 24] Too many open files error. The
>                     code I'm using is:
>
>                     import pyfits
>
>                     for filename in os.listdir(directory):
>                     kep_id = filename[4:13]
>
>                     with pyfits.open(directory + "/" + filename,
>                     ignore_missing_end=True) as fits:
>                     time = fits[1].data["TIME"]
>                     energy = fits[1].data["ENERGY"]
>
>                     Any ideas would be greatly appreciated!
>
>                     Nicholas Earl
>                     Graduate Research Assistant
>                     San Diego State University
>
>                     _______________________________________________
>                     AstroPy mailing list
>                     AstroPy at scipy.org <mailto:AstroPy at scipy.org>
>                     http://mail.scipy.org/mailman/listinfo/astropy
>
>
>
>
>
>
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>