[AstroPy] writing 2D string column to fits binary table

[Erik Bray]

On 01/26/2015 08:50 PM, Stephen Bailey wrote:
> astropy.io.fits and fitsio aficionados,
>
> I'm trying to write a FITS binary table that includes a column that is a 2D
> array of strings.  Curiously, only files written by astropy.io.fits and read by
> fitsio pass the test of actually reconstructing the input numpy array.
>
> astropy.io.fits loses the 2D information when reading back in and ends up with a
> 1D array of null separated characters per row.
>
> fitsio.fits pads its file with spaces when reading back in.
>
> But the file written by astropy.io.fits and then read by fitsio gets me back
> what I put in.
>
> Details:
>
> import numpy as np
>
> from astropy.io <http://astropy.io> import fits
>
> import fitsio
>
>
> n = 10
>
> dtype = ([('ABC', 'S5', (3,)), ('X', int), ('Y', float)])
>
> data = np.zeros(n, dtype=dtype)
>
> data['X'] = np.arange(n)
>
> data['Y'] = np.arange(n)
>
> data['ABC'][:, 0] = 'a'
>
> data['ABC'][:, 1] = 'b'
>
> data['ABC'][:, 2] = 'c'
>
> data['ABC'][0] = ['x', 'y', 'z']
>
>
> fits.writeto('apio.fits', data)
>
> fitsio.write('fio.fits', data)
>
>
> astropy.io.fits complains:
>
> /Users/sbailey/anaconda/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py:782:
> UserWarning: TDIM1 value (5,3) does not fit with the size of the array items
> (5).  TDIM1 will be ignored.
>
>    actual_nitems, indx + 1))

... snip ...

>
> Feature?  Bug?  User error?  Other?

Maybe a little of each, but mostly a bug, I think :)  The data is being written 
correctly--the code that generates that warning is just getting confused about 
axis order when it compares the generated TDIM1 keyword to the shape of the 
data.  In other words, cross talk between different parts of the code.

This this: 
https://github.com/astropy/astropy/blob/master/astropy/io/fits/tests/test_table.py#L1704

shows an example where that won't happen because the Column objects are all 
hand-written with proper values given for the TDIMn keywords.  But in principle 
non of that should be necessary and your example should just work (and it does 
in fact write the file correctly; it just shouldn't display that warning.

 > In [*15*]: np.array(fits.getdata('apio.fits'))

If you just write fits.getdata('apio.fits'), you see that the FITS_rec structure 
returned has the correct format.  I think the fact that converting it to a 
normal ndarray drops that formatting has to do with historical cruft, but I 
think the answer there, for now, is "don't do that" :)

Erik