[AstroPy] writing 2D string column to fits binary table

[Stephen Bailey]

On Tue, Jan 27, 2015 at 8:13 AM, Erik Bray <embray at stsci.edu> wrote:

> On 01/26/2015 08:50 PM, Stephen Bailey wrote:
> > astropy.io.fits and fitsio aficionados,
> >
> > I'm trying to write a FITS binary table that includes a column that is a
> 2D
> > array of strings.  Curiously, only files written by astropy.io.fits and
> read by
> > fitsio pass the test of actually reconstructing the input numpy array.
> >
> > astropy.io.fits loses the 2D information when reading back in and ends
> up with a
> > 1D array of null separated characters per row.
> >
> > fitsio.fits pads its file with spaces when reading back in.
> >
> > But the file written by astropy.io.fits and then read by fitsio gets me
> back
> > what I put in.
> >
> > Details:
> >
> > import numpy as np
> >
> > from astropy.io <http://astropy.io> import fits
> >
> > import fitsio
> >
> >
> > n = 10
> >
> > dtype = ([('ABC', 'S5', (3,)), ('X', int), ('Y', float)])
> >
> > data = np.zeros(n, dtype=dtype)
> >
> > data['X'] = np.arange(n)
> >
> > data['Y'] = np.arange(n)
> >
> > data['ABC'][:, 0] = 'a'
> >
> > data['ABC'][:, 1] = 'b'
> >
> > data['ABC'][:, 2] = 'c'
> >
> > data['ABC'][0] = ['x', 'y', 'z']
> >
> >
> > fits.writeto('apio.fits', data)
> >
> > fitsio.write('fio.fits', data)
> >
> >
> > astropy.io.fits complains:
> >
> >
> /Users/sbailey/anaconda/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py:782:
> > UserWarning: TDIM1 value (5,3) does not fit with the size of the array
> items
> > (5).  TDIM1 will be ignored.
> >
> >    actual_nitems, indx + 1))
>
> ... snip ...
>
> >
> > Feature?  Bug?  User error?  Other?
>
> Maybe a little of each, but mostly a bug, I think :)  The data is being
> written
> correctly--the code that generates that warning is just getting confused
> about
> axis order when it compares the generated TDIM1 keyword to the shape of the
> data.  In other words, cross talk between different parts of the code.
>
> This this:
>
> https://github.com/astropy/astropy/blob/master/astropy/io/fits/tests/test_table.py#L1704
>
> shows an example where that won't happen because the Column objects are all
> hand-written with proper values given for the TDIMn keywords.  But in
> principle
> non of that should be necessary and your example should just work (and it
> does
> in fact write the file correctly; it just shouldn't display that warning.
>
>  > In [*15*]: np.array(fits.getdata('apio.fits'))
>
> If you just write fits.getdata('apio.fits'), you see that the FITS_rec
> structure
> returned has the correct format.  I think the fact that converting it to a
> normal ndarray drops that formatting has to do with historical cruft, but I
> think the answer there, for now, is "don't do that" :)
>

Alas, I always prefer to get back to vanilla ndarray objects as quickly as
possible when doing I/O so that downstream algorithm code can rely on
ndarray only, not on the particular I/O format from whence it came.

>From your description, I think there are two separate astropy bug reports
brewing here:
1. cosmetic: astropy.io.fits.writeto() shouldn't issue that warning since
it actually is writing the file correctly
2. important: the conversion from the FITS_rec structure to the np.array is
not converting the chararray() portion correctly to maintain the 2D nature
of the string array.

Regards,

Stephen



>
> Erik
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150127/dbf7c774/attachment.html>