[AstroPy] VOTable problems

[Derek Homeier]

Hi Rick,
> 
> astropy.units.core.UnitScaleError: The column 'mass' could not be stored in FITS format because it has a scale '(1.0)' that is not recognized by the FITS standard. Either scale the data or change the units.
> 
> If one deletes the unit of the "mass" column, everything is ok.  If one uses a FITS-compatible unit like "solMass", everything is ok.  Thus, the problem is that the astropy implementation of converting a votable input into a FITS output won't accept non-FITS units.  The IVOA standard allows one to use any reasonable units and suggests that one follow the style of the FITS standard, but "jovMass" (Jovian masses, in the same syntax as "solMass" = Solar masses) isn't a FITS standard.  Generally, the IVOA standard should be the measure of things.  There are lots of tables that don't want to have to use wierd units like the FITS-acceptable "solMass/1047" or "1.8982e27*solMass”.
> 
the IVOA standard actually allows the user to invent any kind of units, reasonable or not.
Jovian or Jupiter masses, radii etc. per the VOUnit standard are not “known” units either,
but simply “unrecognised units” that a parser is expected to accept.
So transforming everything that the VOTable format would permit to FITS is a challenging task.
For FITS, the measure of things is the official standard accepted and published by the IAU, and
the principal philosophy of astropy’s fits module is to be forgiving on input, but strict with output: https://docs.astropy.org/en/latest/io/fits/usage/verification.html#verification
Further down it also states that any of these verification rules can be changed to the ‘warn’ or ‘silent’
level if explicitly requested, but that admittedly does not make life much easier, since none of these
options are accepted by the convenience functions for directly creating HDUs from tables.
As long as one has identifiable units, the easier solution will probably still be converting to a
standard-conforming unit as suggested in my first reply:

> tab[col] = tab[col].to(tab[col].unit.to_system(u.si)[0].bases[0])

Again, admittedly fairly awkward syntax (unless someone can suggest a more straightforward way
to code this), but least you don’t need to treat every unit individually. Though you might of course
prefer ‘Mjup’ to ‘kg’ in your example…
And it would certainly be nice to have an option for making `table_to_hdu` take care of this automatically.

> astropy should simply send a warning, saying that the FITS file created doesn't conform to the nominal FITS standard, but who cares, as long as the data has been stored.
> 
> The problem Doug mentioned has to do with strings:  note that above I used a FIELD with the following syntax
> 
> 	<FIELD ID="astring" arraysize="16" datatype="char" name="astring" ucd="meta.id;src" unit="">
> 
> Had I used the IVOA standard syntax
> 
> 	<FIELD ID="astring" arraysize="*" datatype="char" name="astring" ucd="meta.id;src" unit="">
> 
> (i.e. strings of varying length), I would have gotten the problem with string objects mentioned by Doug.  The astropy implementation doesn't implement the standard IVOA syntax in a fashion that leads to a correct translation into FITS.
> 
The astropy implementation should implement the standard in a way that is correctly reading VOTable files
into functional astropy Tables. Translation into FITS format is quite a different thing, especially with the two
following somewhat distinct standards. Acceptance of physical and nonphysical units is one example you’ve
pointed out; variable-length or -width table entries another that I think is very hard to accommodate in FITS.

> Yet another reason (other than the total lack of metadata storage) to work over the votable implementation.  A great task for a younger and better programmer than me....

If you could file an issue on GitHub with the specific problems arising with your data, that would certainly
help keeping it on the radar. Having someone able to devote a substantial part of their workforce to that
task would of course help as well, but at least for the FITS side some familiarity with the over two decades
of development history of PyFITS and astropy io.fits combined could probably also come in handy. ;-)

Cheers,
					Derek