[AstroPy] Reading/writing tables

Tue Dec 8 11:29:19 EST 2020

Thank you for this tip.  I haven't used CIAO before but may do so in the future.

Judith

________________________________
From: Doug Burke <dburke.gw at gmail.com>
Sent: December 8, 2020 11:17 AM
To: Judith Irwin <irwinja at queensu.ca>
Cc: Astronomical Python mailing list <astropy at python.org>
Subject: Re: [AstroPy] Reading/writing tables

Judith,

I know this isn't technically [*] a Python solution, but as this file was created with CIAO tools you can use the dmcopy tool to create a CSV file:

%  dmcopy merged_0.3-7_newerr_src.fits "x.csv[opt kernel=text,sep=',']"

although you may want to then remove the first line and fix the "header" line to change "#RA..." to '#RA,..." [in fact you probably want to delete that line too as it is unlikely to be much use due to the use of "vector" components like POS and the R column].

% cat x.csv
#TEXT/SIMPLE
#,RA,DEC,RA_ERR,DEC_ERR,POS(X,Y),X_ERR,Y_ERR,NPIXSOU,NET_COUNTS,NET_COUNTS_ERR,BKG_COUNTS,BKG_COUNTS_ERR,NET_RATE,NET_RATE_ERR,BKG_RATE,BKG_RATE_ERR,EXPTIME,EXPTIME_ERR,SRC_SIGNIFICANCE,PSF_SIZE,MULTI_CORREL_MAX,SHAPE,R[2],ROTANG,PSFRATIO,COMPONENT,CORRELATION_MAX(X_CORRELATION_MAX,Y_CORRELATION_MAX),FLUX_MAX(X_FLUX_MAX,Y_FLUX_MAX),SCALES(X_WAVELET_SCALE,Y_WAVELET_SCALE),797_OFFAXS_ANG,WEIGHT_OFFAXS,WANG_ERR,POS_ERR
190.4917767562, 32.48099743812, 7.8183367975271e-06, 4.8707159407968e-06, 4071.663677130, 3736.552690583, 4.8259735963256e-02, 3.5637592993680e-02, 145, 926.6190, 30.56142, 7.380970, 0.02443597, 0.00004269619, 1.40819088e-06, 3.40095852e-07, 1.12594922e-09, 2.17026160e+07, 177.4334, 240.5874, 1.670030, 0, ellipse, 4.603549, 2.774965, 154.5285, 0.7133945, 1, 4071.0, 3737.0, 4071.0, 3737.0, 1.414000, 1.414000, 2.958585048000, 2.958585048000, 6.5246134585293e-03, 6.5479799735354e-03
...

Doug

[*] in fact, it's about as far from a Python solution as you can get

On Tue, Dec 8, 2020 at 8:34 AM Judith Irwin <irwinja at queensu.ca<mailto:irwinja at queensu.ca>> wrote:
Brilliant thank you!  I was definitely looking for quick and easy and the following line worked:

t.write('mytest.csv', format='ascii.csv', fast_writer=False)

I'll retain the other advice for the future, too.

Judith

________________________________
From: AstroPy <astropy-bounces+irwinja=queensu.ca at python.org<mailto:queensu.ca at python.org>> on behalf of Peter Dzwig <pdzwig at summaventures.com<mailto:pdzwig at summaventures.com>>
Sent: December 8, 2020 7:38 AM
To: Astronomical Python mailing list <astropy at python.org<mailto:astropy at python.org>>; Derek Homeier <derek at astro.physik.uni-goettingen.de<mailto:derek at astro.physik.uni-goettingen.de>>
Subject: Re: [AstroPy] Reading/writing tables

An alternative might be to read in the relevant headers and the
appropriate data contained in the .fits file, then use numpy to create
the .csv with something like:

np.savetxt(outfile, array, delimiter=",")

Where array is the input data array and outfile is the name of the .csv
file you want to create.

For big files I find a strategy like this much more straightforward than
using tables! if I know  what the fits headers are.

You might also look at (perhaps temporarily) writing your [,2] array as
two columns.

Hope I have understood your problem and that this helps,

Peter Dzwig

On 08/12/2020 09:31, Derek Homeier wrote:
> On 8 Dec 2020, at 9:08 am, yyang <yanbin.yang at obspm.fr<mailto:yanbin.yang at obspm.fr>> wrote:
>>
>> Could it be linked to the your duplicated column named “R” ?  see  a screen shot enclosed.
>> Cheers
>> Yanbin
>> <PastedGraphic-1.png>
>>
>>> On 8 Dec 2020, at 08:58, Judith Irwin <irwinja at queensu.ca<mailto:irwinja at queensu.ca>> wrote:
>>>
>>> Could someone help with this problem? I want to read a fits table and write out a .csv file.  Here is what I'm doing:
>>>
>>>>>> import astropy
>>>>>> from astropy.table import Table
>>>>>> from astropy.io<https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fastropy.io%2F&data=04%7C01%7Cirwinja%40queensu.ca%7C4e34ae8864084e36d24c08d89b94d968%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C1%7C637430410933144230%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=GDUOEi%2Fm6j4uDtyXfGY2VRvmSou46R8Ww6yML%2BOet88%3D&reserved=0> import ascii
>>>>>> t=Table.read('merged_0.3-7_newerr_src.fits')
>>>>>> print t
>>>       RA           DEC            RA_ERR           DEC_ERR            X       ... 797_OFFAXS_ANG WEIGHT_OFFAXS      WANG_ERR         POS_ERR
>>>      deg           deg             deg               deg             pix      ...     arcmin         arcmin          arcmin           arcmin
>>> ------------- ------------- ----------------- ----------------- ------------- ... -------------- -------------- ---------------- ----------------
>>> 190.491776756 32.4809974381 7.81833679753e-06  4.8707159408e-06 4071.66367713 ...    2.958585048    2.958585048 0.00652461345853 0.00654797997354
>>> 190.490783106 32.4956819235 2.04948640032e-05 3.86556461152e-05        4077.8 ...  2.07617011778  2.07617011778 0.00490486335967 0.00556319682508
>>> (I'm only showing the first rows of the table)
>>>>>> ascii.write(t, 'Xsources_new.csv', format='csv',overwrite=True)
>
> It is not duplicated, but has itself a dtype or shape (2, ), i.e. the column represents a 2-dim array of shape (72, 2).
>
>>>> t['SHAPE’, 'R’, 'ROTANG’, 'PSFRATIO']
> <Table length=72>
>   SHAPE                R [2]                ROTANG
>                         pix                  deg
>   str10               float32              float32
> ---------- ------------------------------ —————
> ellipse             4.603549 .. 2.7749648  154.52849
> ellipse             2.7087543 .. 1.141337   98.67701
> ellipse             3.5766919 .. 2.232861  140.08878
>
> That is in fact the problem, as structured columns like this are not supported by the (default) fast parser,
> and in fact only incompletely supported in CSV format.
> A quick workaround therefore would be to disable the fast parser:
>
>>>> t.write('merged_0.3-7_newerr_src.csv', format='ascii.csv', fast_writer=False)
>
> But this will represent your column ‘R’ as a string, so if you want to read the table back in from csv
>
>>>> Table.read('merged_0.3-7_newerr_src.csv')['R']
> <Column name='R' dtype='str30' length=72>
>          4.603549 .. 2.7749648
>          2.7087543 .. 1.141337
>
> it will no longer be directly accessible as the original array.
>
> To preserve them in a usable format, I suggest to change the table to a “flattened” column format
> by replacing it with one column for each subcolumn, e.g.:
>
>>>> t.add_columns([t['R'][:,0], t['R'][:,1]], indexes=t.index_column('R’) + np.zeros(2), names=['R0', 'R1'])
>>>> t.remove_column('R')
>>>> t.write('merged_0.3-7_newer_src.csv', format=‘ascii.csv’, overwrite=True)
>
> I also recommend to upgrade your installation to a Python 3-based version, as Astropy 2.0.x is really old..;-)
>
> But this issue as such still persists in the current version and is a bug, since the writer should
> report a clearer error message (or automatically switch to the ‘fast_writer’=False version),
> thanks for reporting this!
>
> HTH
>                                                Derek
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at python.org<mailto:AstroPy at python.org>
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fastropy&data=04%7C01%7Cirwinja%40queensu.ca%7C88465cb05c28401b519a08d89b7792e2%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C637430285165593172%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SIGSQaGYFmR2IiiQGXCXZXR4J6d9a3kvctHSNAf%2FkwM%3D&reserved=0<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fastropy&data=04%7C01%7Cirwinja%40queensu.ca%7C4e34ae8864084e36d24c08d89b94d968%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C1%7C637430410933144230%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=AgRE1ZqNj9IjF3tCIC0doPIPal6qYNXOPlvMVa63Ilo%3D&reserved=0>
>

--

Dr. Peter Dzwig
_______________________________________________
AstroPy mailing list
AstroPy at python.org<mailto:AstroPy at python.org>
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fastropy&data=04%7C01%7Cirwinja%40queensu.ca%7C88465cb05c28401b519a08d89b7792e2%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C637430285165593172%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SIGSQaGYFmR2IiiQGXCXZXR4J6d9a3kvctHSNAf%2FkwM%3D&reserved=0<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fastropy&data=04%7C01%7Cirwinja%40queensu.ca%7C4e34ae8864084e36d24c08d89b94d968%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C1%7C637430410933154225%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pxLxNr7Sa0e%2FtlVt9xpZTlNPyA5AIzsVTmpALrlsiXE%3D&reserved=0>
_______________________________________________
AstroPy mailing list
AstroPy at python.org<mailto:AstroPy at python.org>
https://mail.python.org/mailman/listinfo/astropy<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fastropy&data=04%7C01%7Cirwinja%40queensu.ca%7C4e34ae8864084e36d24c08d89b94d968%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C1%7C637430410933164220%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=f5JIi7vRm3ANWiSOL20%2FK36XZ4bUGmzXi5YigocJSZE%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/astropy/attachments/20201208/87148d7d/attachment-0001.html>