[AstroPy] Reading/writing tables

[Derek Homeier]

On 8 Dec 2020, at 9:08 am, yyang <yanbin.yang at obspm.fr> wrote:
> 
> Could it be linked to the your duplicated column named “R” ?  see  a screen shot enclosed.
> Cheers
> Yanbin
> <PastedGraphic-1.png>
> 
>> On 8 Dec 2020, at 08:58, Judith Irwin <irwinja at queensu.ca> wrote:
>> 
>> Could someone help with this problem? I want to read a fits table and write out a .csv file.  Here is what I'm doing:
>> 
>> >>> import astropy
>> >>> from astropy.table import Table
>> >>> from astropy.io import ascii
>> >>> t=Table.read('merged_0.3-7_newerr_src.fits')
>> >>> print t
>>       RA           DEC            RA_ERR           DEC_ERR            X       ... 797_OFFAXS_ANG WEIGHT_OFFAXS      WANG_ERR         POS_ERR    
>>      deg           deg             deg               deg             pix      ...     arcmin         arcmin          arcmin           arcmin   
>> ------------- ------------- ----------------- ----------------- ------------- ... -------------- -------------- ---------------- ----------------
>> 190.491776756 32.4809974381 7.81833679753e-06  4.8707159408e-06 4071.66367713 ...    2.958585048    2.958585048 0.00652461345853 0.00654797997354
>> 190.490783106 32.4956819235 2.04948640032e-05 3.86556461152e-05        4077.8 ...  2.07617011778  2.07617011778 0.00490486335967 0.00556319682508
>> (I'm only showing the first rows of the table)
>> >>> ascii.write(t, 'Xsources_new.csv', format='csv',overwrite=True)

It is not duplicated, but has itself a dtype or shape (2, ), i.e. the column represents a 2-dim array of shape (72, 2).

>>> t['SHAPE’, 'R’, 'ROTANG’, 'PSFRATIO']
<Table length=72>
  SHAPE                R [2]                ROTANG    
                        pix                  deg                  
  str10               float32              float32   
---------- ------------------------------ —————
ellipse             4.603549 .. 2.7749648  154.52849
ellipse             2.7087543 .. 1.141337   98.67701
ellipse             3.5766919 .. 2.232861  140.08878

That is in fact the problem, as structured columns like this are not supported by the (default) fast parser,
and in fact only incompletely supported in CSV format.
A quick workaround therefore would be to disable the fast parser:

>>> t.write('merged_0.3-7_newerr_src.csv', format='ascii.csv', fast_writer=False)

But this will represent your column ‘R’ as a string, so if you want to read the table back in from csv

>>> Table.read('merged_0.3-7_newerr_src.csv')['R']
<Column name='R' dtype='str30' length=72>
         4.603549 .. 2.7749648
         2.7087543 .. 1.141337

it will no longer be directly accessible as the original array.

To preserve them in a usable format, I suggest to change the table to a “flattened” column format
by replacing it with one column for each subcolumn, e.g.:

>>> t.add_columns([t['R'][:,0], t['R'][:,1]], indexes=t.index_column('R’) + np.zeros(2), names=['R0', 'R1'])
>>> t.remove_column('R')
>>> t.write('merged_0.3-7_newer_src.csv', format=‘ascii.csv’, overwrite=True)

I also recommend to upgrade your installation to a Python 3-based version, as Astropy 2.0.x is really old..;-)

But this issue as such still persists in the current version and is a bug, since the writer should
report a clearer error message (or automatically switch to the ‘fast_writer’=False version),
thanks for reporting this!

HTH
						Derek