[AstroPy] Reading/writing tables

Doug Burke dburke.gw at gmail.com
Tue Dec 8 11:17:55 EST 2020 

Judith,

I know this isn't technically [*] a Python solution, but as this file was
created with CIAO tools you can use the dmcopy tool to create a CSV file:

%  dmcopy merged_0.3-7_newerr_src.fits "x.csv[opt kernel=text,sep=',']"

although you may want to then remove the first line and fix the "header"
line to change "#RA..." to '#RA,..." [in fact you probably want to delete
that line too as it is unlikely to be much use due to the use of "vector"
components like POS and the R column].

% cat x.csv
#TEXT/SIMPLE
#,RA,DEC,RA_ERR,DEC_ERR,POS(X,Y),X_ERR,Y_ERR,NPIXSOU,NET_COUNTS,NET_COUNTS_ERR,BKG_COUNTS,BKG_COUNTS_ERR,NET_RATE,NET_RATE_ERR,BKG_RATE,BKG_RATE_ERR,EXPTIME,EXPTIME_ERR,SRC_SIGNIFICANCE,PSF_SIZE,MULTI_CORREL_MAX,SHAPE,R[2],ROTANG,PSFRATIO,COMPONENT,CORRELATION_MAX(X_CORRELATION_MAX,Y_CORRELATION_MAX),FLUX_MAX(X_FLUX_MAX,Y_FLUX_MAX),SCALES(X_WAVELET_SCALE,Y_WAVELET_SCALE),797_OFFAXS_ANG,WEIGHT_OFFAXS,WANG_ERR,POS_ERR
190.4917767562, 32.48099743812, 7.8183367975271e-06, 4.8707159407968e-06,
4071.663677130, 3736.552690583, 4.8259735963256e-02, 3.5637592993680e-02,
145, 926.6190, 30.56142, 7.380970, 0.02443597, 0.00004269619,
1.40819088e-06, 3.40095852e-07, 1.12594922e-09, 2.17026160e+07, 177.4334,
240.5874, 1.670030, 0, ellipse, 4.603549, 2.774965, 154.5285, 0.7133945, 1,
4071.0, 3737.0, 4071.0, 3737.0, 1.414000, 1.414000, 2.958585048000,
2.958585048000, 6.5246134585293e-03, 6.5479799735354e-03
...

Doug

[*] in fact, it's about as far from a Python solution as you can get


On Tue, Dec 8, 2020 at 8:34 AM Judith Irwin <irwinja at queensu.ca> wrote:

> Brilliant thank you!  I was definitely looking for quick and easy and the
> following line worked:
>
> t.write('mytest.csv', format='ascii.csv', fast_writer=False)
>
> I'll retain the other advice for the future, too.
>
> Judith
>
> ------------------------------
> *From:* AstroPy <astropy-bounces+irwinja=queensu.ca at python.org> on behalf
> of Peter Dzwig <pdzwig at summaventures.com>
> *Sent:* December 8, 2020 7:38 AM
> *To:* Astronomical Python mailing list <astropy at python.org>; Derek
> Homeier <derek at astro.physik.uni-goettingen.de>
> *Subject:* Re: [AstroPy] Reading/writing tables
>
> An alternative might be to read in the relevant headers and the
> appropriate data contained in the .fits file, then use numpy to create
> the .csv with something like:
>
> np.savetxt(outfile, array, delimiter=",")
>
> Where array is the input data array and outfile is the name of the .csv
> file you want to create.
>
> For big files I find a strategy like this much more straightforward than
> using tables! if I know  what the fits headers are.
>
> You might also look at (perhaps temporarily) writing your [,2] array as
> two columns.
>
> Hope I have understood your problem and that this helps,
>
> Peter Dzwig
>
> On 08/12/2020 09:31, Derek Homeier wrote:
> > On 8 Dec 2020, at 9:08 am, yyang <yanbin.yang at obspm.fr> wrote:
> >>
> >> Could it be linked to the your duplicated column named “R” ?  see  a
> screen shot enclosed.
> >> Cheers
> >> Yanbin
> >> <PastedGraphic-1.png>
> >>
> >>> On 8 Dec 2020, at 08:58, Judith Irwin <irwinja at queensu.ca> wrote:
> >>>
> >>> Could someone help with this problem? I want to read a fits table and
> write out a .csv file.  Here is what I'm doing:
> >>>
> >>>>>> import astropy
> >>>>>> from astropy.table import Table
> >>>>>> from astropy.io import ascii
> >>>>>> t=Table.read('merged_0.3-7_newerr_src.fits')
> >>>>>> print t
> >>>       RA           DEC            RA_ERR           DEC_ERR
> X       ... 797_OFFAXS_ANG WEIGHT_OFFAXS      WANG_ERR         POS_ERR
> >>>      deg           deg             deg               deg
> pix      ...     arcmin         arcmin          arcmin           arcmin
> >>> ------------- ------------- ----------------- -----------------
> ------------- ... -------------- -------------- ----------------
> ----------------
> >>> 190.491776756 32.4809974381 7.81833679753e-06  4.8707159408e-06
> 4071.66367713 ...    2.958585048    2.958585048 0.00652461345853
> 0.00654797997354
> >>> 190.490783106 32.4956819235 2.04948640032e-05 3.86556461152e-05
> 4077.8 ...  2.07617011778  2.07617011778 0.00490486335967 0.00556319682508
> >>> (I'm only showing the first rows of the table)
> >>>>>> ascii.write(t, 'Xsources_new.csv', format='csv',overwrite=True)
> >
> > It is not duplicated, but has itself a dtype or shape (2, ), i.e. the
> column represents a 2-dim array of shape (72, 2).
> >
> >>>> t['SHAPE’, 'R’, 'ROTANG’, 'PSFRATIO']
> > <Table length=72>
> >   SHAPE                R [2]                ROTANG
> >                         pix                  deg
> >   str10               float32              float32
> > ---------- ------------------------------ —————
> > ellipse             4.603549 .. 2.7749648  154.52849
> > ellipse             2.7087543 .. 1.141337   98.67701
> > ellipse             3.5766919 .. 2.232861  140.08878
> >
> > That is in fact the problem, as structured columns like this are not
> supported by the (default) fast parser,
> > and in fact only incompletely supported in CSV format.
> > A quick workaround therefore would be to disable the fast parser:
> >
> >>>> t.write('merged_0.3-7_newerr_src.csv', format='ascii.csv',
> fast_writer=False)
> >
> > But this will represent your column ‘R’ as a string, so if you want to
> read the table back in from csv
> >
> >>>> Table.read('merged_0.3-7_newerr_src.csv')['R']
> > <Column name='R' dtype='str30' length=72>
> >          4.603549 .. 2.7749648
> >          2.7087543 .. 1.141337
> >
> > it will no longer be directly accessible as the original array.
> >
> > To preserve them in a usable format, I suggest to change the table to a
> “flattened” column format
> > by replacing it with one column for each subcolumn, e.g.:
> >
> >>>> t.add_columns([t['R'][:,0], t['R'][:,1]], indexes=t.index_column('R’)
> + np.zeros(2), names=['R0', 'R1'])
> >>>> t.remove_column('R')
> >>>> t.write('merged_0.3-7_newer_src.csv', format=‘ascii.csv’,
> overwrite=True)
> >
> > I also recommend to upgrade your installation to a Python 3-based
> version, as Astropy 2.0.x is really old..;-)
> >
> > But this issue as such still persists in the current version and is a
> bug, since the writer should
> > report a clearer error message (or automatically switch to the
> ‘fast_writer’=False version),
> > thanks for reporting this!
> >
> > HTH
> >                                                Derek
> >
> > _______________________________________________
> > AstroPy mailing list
> > AstroPy at python.org
> >
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fastropy&data=04%7C01%7Cirwinja%40queensu.ca%7C88465cb05c28401b519a08d89b7792e2%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C637430285165593172%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SIGSQaGYFmR2IiiQGXCXZXR4J6d9a3kvctHSNAf%2FkwM%3D&reserved=0
> >
>
> --
>
> Dr. Peter Dzwig
> _______________________________________________
> AstroPy mailing list
> AstroPy at python.org
>
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fastropy&data=04%7C01%7Cirwinja%40queensu.ca%7C88465cb05c28401b519a08d89b7792e2%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C637430285165593172%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SIGSQaGYFmR2IiiQGXCXZXR4J6d9a3kvctHSNAf%2FkwM%3D&reserved=0
> _______________________________________________
> AstroPy mailing list
> AstroPy at python.org
> https://mail.python.org/mailman/listinfo/astropy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/astropy/attachments/20201208/f99183cb/attachment.html>

More information about the AstroPy mailing list