[AstroPy] Nested recarrays in FITS

[Arnon Sela]

Thank you.  It may be that HDF would be a good fit.

This is part of an effort to move ROTSE data from a proprietary format into
a format that is more shareable. The original thought was to use FITS as
other projects use it.

But it may be that we would need to move to more flexible format as HDF.

Thank you so much for the help. With this, I have already worked an example
with h5py.

Again, thank you!!!

On Mon, Dec 4, 2017 at 3:46 PM, Peter Teuben <teuben at astro.umd.edu> wrote:

>
>  another thought on this:
>
> I think the original question was also limited in not explaining why fits
> was needed. I could argue for pickle. Paul is right, HDF might be a better
> match, especially if you have to switch to another language, HDF has a more
> native match to that. But does it have to be persistent data? otherwise
> using a python-c/fortran interface is far more efficient.  (I believe HDF
> is actually more flexible than the F in FITS).
>
> You can't beat a native pickle:
>
>             import pickle
>             pickle.dump(datan_raw,open("test.dat","wb"))
>             ..
>             new_raw = pickle.load(open("test.dat", "rb"))
>
> So perhaps we could return the question and ask in what situation you need
> this data structure (for).
>
> - peter
>
>
> On 12/04/2017 10:08 PM, Paul Kuin wrote:
>
> I think that HDF does that for you. FIts is more flexible, but you have to
> do your own writes and retrievals. In the end you will be reinventing the
> wheel unless you check out how HDF does it, That's my opinion.
>
> Cheers,
>
>    Paul
>
> On Mon, Dec 4, 2017 at 9:02 PM, Arnon Sela <arnon.sela at gmail.com> wrote:
>
>> Dear Whom that Can Help,
>>
>> I have nested numpy recarray structure to be stored into Fits.
>> The following code is a just a test I used to build a nested structure
>> (data_for_fits variable in the last line of the code).
>>
>> Code start >>>>>>
>>
>> import numpy as np
>>
>> ''' The following two functions are adapted from:
>> adopted from https://stackoverflow.com/questions/32328889/numpy-structure
>> d-array-from-arbitrary-level-nested-dictionary
>> '''
>>
>> def mkdtype(d):
>>     ''' Creates dtype for nested dictionary with numpy based type objects
>>     '''
>>     result = []
>>     for k, v in d.items():
>>         if isinstance(v,np.ndarray):
>>             result.append((k, v.dtype, v.shape))
>>         else:
>>             result.append((k, mkdtype(v)))
>>     return np.dtype(result)
>>
>> def dict2recarray(data, rec=None):
>>     ''' Creates numpy.recarray from data (dict)
>>     '''
>>     def _dict2recarray(data, rec):
>>         if rec.dtype.names:
>>             for n in rec.dtype.names:
>>                 _dict2recarray(data[n], rec[n])
>>         else:
>>             rec[:] = data
>>         return rec
>>
>>     dtype = mkdtype(data)
>>     if rec is None:
>>         rec = np.zeros(dtype.shape, dtype)
>>
>>     return _dict2recarray(data, rec)
>>
>> datan_raw = {'DATA': {'D1': np.linspace( 0, 100, 8*4,).reshape(8, 4),
>>                       'D2': np.linspace( 0, 100, 10*5, ).reshape(10, 5),
>>                       'ND': {'D1': np.linspace( 0, 100, 10*5,
>> ).reshape(10, 5),
>>                              'D2': np.linspace( 0, 100, 8*4,).reshape(8,
>> 4), }}}
>>
>> dtype = mkdtype(datan_raw)
>> *data_for_fits* = dict2recarray(datan_raw)
>>
>>
>> >>>>>> Code ends
>>
>> I couldn't find documentation on how to build such a FITS structure
>> (nested recarrays).
>>
>> One option is to build sub-recarrays into different BIN tables with a
>> header that would correspond to a nested key in the recarray. But that
>> would require creating another function to reconstruct the recarray
>> structure after reading the BIN tables from the FITS file.
>>
>> The better option is to build FITS is such a manner that would retrieve
>> the structure correctly on FITS load().
>>
>> Thank you for your help,
>>
>> Best regards.
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at python.org
>> https://mail.python.org/mailman/listinfo/astropy
>>
>>
>
>
> --
>
> * * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * *
> Dr. N.P.M. Kuin      (n.kuin at ucl.ac.uk)
> phone +44-(0)1483 (prefix) -204111 (work)
> mobile +44(0)7908715953 <+44%207908%20715953>  skype ID: npkuin
> Mullard Space Science Laboratory  – University College London  –
> Holmbury St Mary – Dorking – Surrey RH5 6NT–  U.K.
>
>
> _______________________________________________
> AstroPy mailing listAstroPy at python.orghttps://mail.python.org/mailman/listinfo/astropy
>
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at python.org
> https://mail.python.org/mailman/listinfo/astropy
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20171204/ff9f60c2/attachment.html>