[AstroPy] PyFITS Too many open files

[Nico Nell]

You can try running the python garbage collector after each file close with
gc.collect().

For the memmap issue consider submitting a bug report if you have time:
https://github.com/spacetelescope/PyFITS/issues?state=open

~Nick


On Sat, Jun 15, 2013 at 4:21 PM, Nicholas Earl <nchlsearl at gmail.com> wrote:

> Thanks for the ideas! It seems that memmap=False solves the problem,
> although I can see that it does begin using a large percentage of the
> memory -- I can only imagine how much if I let it run on the entire set of
> directories... Is there not any way to flush it all after I close the file?
>
> I am using python 2.7.5, and PyFITs 3.1.2.
>
> Thanks again, everyone, you're awesome.
>
> Nicholas Earl
> Graduate Research Assistant
> San Diego State University
>
>
> On Sat, Jun 15, 2013 at 11:01 AM, Nico Nell <nico.nell at gmail.com> wrote:
>
>> Sorry -- I should have read the blip of code better.
>>
>> So, what version of python and pyfits are you using? How many files are
>> you opening?
>>
>> What happens if you add 'memmap=False' in each of your pyfits open()
>> lines. The pyfits mmap stuff is relatively new so maybe there is an issue.
>>
>> ~Nick
>>
>>
>> On Sat, Jun 15, 2013 at 11:29 AM, Nicholas Earl <nchlsearl at gmail.com>wrote:
>>
>>> Sorry, but it seems I sent this back to only Nice Nell, and not the
>>> entire list!
>>>
>>> -----
>>>
>>> Thanks for the reply. I was under the impression that the with command
>>> automatically closed the file? Regardless, I went ahead and wrote the
>>> script up like this
>>>
>>> import pyfits
>>>
>>> for filename in os.listdir(directory):
>>>  kep_id = filename[4:13]
>>> fits = pyfits.open(directory + "/" + filename, ignore_missing_end=True)
>>>  time = fits[1].data["TIME"]
>>> energy = fits[1].data["ENERGY"]
>>>  fits.close()
>>>
>>> Unfortunately, I get the same error.
>>>
>>> Nicholas Earl
>>> Graduate Research Assistant
>>> San Diego State University
>>>
>>>
>>> On Fri, Jun 14, 2013 at 10:08 PM, Nico Nell <nico.nell at gmail.com> wrote:
>>>
>>>> You need to close the files once you have retrieved the data...
>>>>
>>>> See here:
>>>> http://pythonhosted.org/pyfits/api_docs/api_hdulists.html#pyfits.HDUList.close
>>>>
>>>> ~Nick
>>>>
>>>>
>>>> On Fri, Jun 14, 2013 at 10:51 PM, Nicholas Earl <nchlsearl at gmail.com>wrote:
>>>>
>>>>> Hello everyone,
>>>>>
>>>>> I'm having an issue with opening and retrieving information for a
>>>>> large amount of fits files. Mainly, I keep running into a mmap.error:
>>>>> [Errno 24] Too many open files error. The code I'm using is:
>>>>>
>>>>> import pyfits
>>>>>
>>>>> for filename in os.listdir(directory):
>>>>>  kep_id = filename[4:13]
>>>>>
>>>>> with pyfits.open(directory + "/" + filename, ignore_missing_end=True)
>>>>> as fits:
>>>>>  time = fits[1].data["TIME"]
>>>>> energy = fits[1].data["ENERGY"]
>>>>>
>>>>> Any ideas would be greatly appreciated!
>>>>>
>>>>> Nicholas Earl
>>>>> Graduate Research Assistant
>>>>> San Diego State University
>>>>>
>>>>> _______________________________________________
>>>>> AstroPy mailing list
>>>>> AstroPy at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/astropy
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20130615/5298fb40/attachment.html>