[AstroPy] writing FITS files fast

Ivan Zolotukhin ivan.zolotukhin at gmail.com
Tue Apr 21 14:01:01 EDT 2015


Hi,

> fitsio does support 64-bit.

Indeed, I was probably using it wrong, but maybe its error message can
be improved not to cause a confusion in similar cases, even though my
use case is somewhat weird. Here's the minimal test case I had
problems with (tested in numpy 1.8.1 and 1.9.2, fitsio 0.9.7):

```
import numpy as np
from fitsio import FITS

fits = FITS('test.fits', 'rw')
arr = [1L, 2L]
nparr = np.array(arr)
nulls = np.equal(arr, None)
nparr[nulls] = -1 # or whatever your FITS null value is
#arr = nparr.astype('int64') # uncomment this line and comment the
next one to make it work
arr = np.array(nparr, dtype='int64')
fits.write({'test': arr})
fits.close()
```

This results in:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/home/user/<ipython-input-92-b693c45fa56d> in <module>()
----> 1 fits.write({'test': arr})

/usr/local/lib/python2.7/dist-packages/fitsio/fitslib.pyc in
write(self, data, units, extname, extver, compress, tile_dims, header,
names, table_type, **keys)
    446                              extname=extname, extver=extver,
header=header,
    447                              names=names,
--> 448                              table_type=table_type)
    449
    450

/usr/local/lib/python2.7/dist-packages/fitsio/fitslib.pyc in
write_table(self, data, table_type, names, formats, units, extname,
extver, header)
    714             self[-1]._update_info()
    715
--> 716         self[-1].write(data,names=names)
    717
    718     def create_table_hdu(self, data=None, dtype=None,

/usr/local/lib/python2.7/dist-packages/fitsio/fitslib.pyc in
write(self, data, **keys)
   1506                 firstrow=keys.get('firstrow',0)
   1507                 self._FITS.write_columns(self._ext+1,
nonobj_colnums, nonobj_arrays,
-> 1508                                          firstrow=firstrow+1)
   1509
   1510         # writing the object arrays always occurs the same way


TypeError: Unsupported numpy table datatype 9
```

I'll explain why I needed this seemingly excessive numpy operations. I
have a list of long integers as an input and I need to check it for
None values (even though the test case does not have it), that's why I
make numpy array of objects `nparr` first and set Nones in it to a
dedicated integer value (let me know if there's a way to avoid this in
case your input has missing values). Then I naively expected that
doing `np.array(nparr, dtype='int64')` casts my array from numpy array
of objects to the array of long integers. I still do not get this
numpy magics, but if I change the type casting to the line which is
commented above `arr = nparr.astype('int64')` the fitsio error is
gone. I initially thought this weird problem has something to do with
int64 because changing data type in question to int32 also makes this
error disappear.

As for performance, with fitsio I'm now bottlenecked by my SQL
database which is how I expect things to happen with complex data
models. This is still not the case for astropy 1.0.2, which
bottlenecks my FITS writing procedure. I can prepare a test case if
someone is interested.

--
With best regards,
 Ivan



More information about the AstroPy mailing list