[AstroPy] "double free or corruption" crash: PyFITS + NumPy 1.6.0

[Russell Owen]

Just for completeness: the error I see on MacOS X 10.6 is:
Python(5086,0xa08a0540) malloc: *** error for object 0x22c66c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
but I've not yet tracked it down far enough to submit a ticket.

-- Russell

On Jul 5, 2011, at 4:15 PM, Russell Owen wrote:

> I don't see this on my MacOS X 10.6 box. I can load the file just fine.
> 
> However, I've been getting memory problems with numpy and Python 2.7 as well. I've reluctantly decided to go back to Python 2.6 to release applications that others can reliably use. I spent some time trying to reproduce the problem today and found one case that reliably causes numpy.subtract to segfault on MacOS X 10.5 (pyfits is not involved), but so far no luck on 10.6. I submitted a ticket.
> 
> Regards,
> 
> -- Russell
> 
> On Jul 3, 2011, at 1:02 PM, Tom Aldcroft wrote:
> 
>> I'm seeing some sort of memory or allocation error (glibc detected)
>> when trying to read a particular FITS file with the following setup:
>> 
>> OS: CentOS-5.6 (64-bit)
>> Python: 2.7.1.4 (ActiveState distribution)
>> NumPy: 1.6.0
>> PyFits: 2.4.0
>> 
>> If I downgrade to NumPy 1.5.0 and re-build everything then I do not
>> see this problem, while PyFITS 2.2.2 + NumPy 1.6.0 did show this
>> crash.  I've put the troublesome file on:
>> 
>>  http://hea-www.harvard.edu/~aldcroft/tmp/glibc_detected.fits.gz
>> 
>> This particular file is the output of the Chandra X-ray Center
>> automated processing which converts raw Chandra telemetry to "Level-0
>> engineering data".  I've read hundreds of thousands of such files with
>> Python 2.6, PyFits 2.2.2 and NumPy 1.3.0 without issue.  I recently
>> upgraded Python, PyFits and NumPy and ran into this problem.  Other
>> similar Level-0 files were read OK with the new versions, but not the
>> one in question.  I did a significant amount of regression testing
>> before doing this version upgrade and didn't see any differences, but
>> my testing didn't hit this particular file type.  For now I'm back at
>> NumPy 1.5.0.
>> 
>> Regards,
>> Tom Aldcroft
>> 
>> -----------------
>> 
>> ccosmos% python
>> ActivePython 2.7.1.4 (ActiveState Software Inc.) based on
>> Python 2.7.1 (r271:86832, Feb  7 2011, 11:30:54)
>> [GCC 4.0.2 20051125 (Red Hat 4.0.2-8)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import pyfits
>>>>> hdus = pyfits.open('glibc_detected.fits.gz')
>>>>> dat = hdus[1].data
>>>>> print dat
>> *** glibc detected ***
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python: double free
>> or corruption (!prev): 0x0000000017aea970 ***
>> ======= Backtrace: =========
>> /lib64/libc.so.6[0x390d07245f]
>> /lib64/libc.so.6(cfree+0x4b)[0x390d0728bb]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/multiarray.so[0x2b97362a94f2]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/umath.so[0x2b9736462061]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/umath.so[0x2b9736475b27]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/umath.so[0x2b973647739b]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x4fd4)[0x494954]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x4f7717]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x421f07]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x46e99f]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x68bf)[0x49623f]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5c52)[0x4955d2]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x4f7717]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_CallObjectWithKeywords+0x55)[0x48ed15]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/multiarray.so[0x2b97362873b4]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(_PyObject_Str+0x69)[0x449cc9]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Str+0x13)[0x449da3]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x449f7d]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyFile_WriteObject+0x7f)[0x42e04f]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x4b8d)[0x49450d]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCode+0x32)[0x4970e2]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyRun_InteractiveOneFlags+0x190)[0x4b9e10]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyRun_InteractiveLoopFlags+0x4e)[0x4ba01e]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyRun_AnyFileExFlags+0x4c)[0x4ba12c]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(Py_Main+0xa0e)[0x415bee]
>> /lib64/libc.so.6(__libc_start_main+0xf4)[0x390d01d994]
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x414df9]
>> ======= Memory map: ========
>> 00400000-0054b000 r-xp 00000000 08:04 106168389
>> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python
>> 0064b000-0068f000 rw-p 0014b000 08:04 106168389
>> /data/cosmos2/ska/arcAbort
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
> 
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy