NEP 42 and physical unit DType
NEP 42 mentions physical units as a possible use case for the new DType. Having worked on `unyt`, which is an ndarray subclass, and other unit system implementations based on the dispatch mechanism, I am quite familiar with the challenges. One challenge is integration with Matplotlib that has a unit conversion interface. This interface is invoked by a registry mapping from unit array type, such as `unyt_array`, and its conversion class. A custom DType will still result in an ndarray type - correct? If so, Matplotlib will attempt to perform its internal calculations eventually leading to an invalid operation when adding a physical quantity to a pure number. This happens today with `unyt_array` and some of the plotting functions that allow the subclass to leak through into the internals. Has this been discussed? I can imagine other libraries will have the same challenge.
On Fri, 2020-10-09 at 07:48 -0500, Lee Johnston wrote:
A custom DType will be attached to a normal NumPy array (or, hopefully a pandas dataframe, etc.). I am not sure if that is what you meant, but a custom DType will mean there is no array subclass involved (this is different from how pandas approaches extension dtypes). But, I am not sure that this makes much of a difference with respect to the problem.
It sounds unlikely that this would solve the issue immediately. Rather, we probably need to think about creating some kind of "protocol" to drop units, such as: arr.astype(Unitless) which would be a no-op for the NumPy numerical types. Or baking in something similar to make working with units easier. I would have hoped a bit that it is possible to avoid such issues e.g. by dividing by a step/difference, but I do not know the details. The most important question for me would be more whether there is something in the basic infrastructure to consider that cannot be done as an extension/protocol later (like the `arr.astype(Unitless)` cast, which should be a straight forward extension). Although, it would be interesting to figure out how a future Unit DType would best look in this regard! Cheers, Sebastian
Matplotlib should (internally) never try to do operations on user data before we do the unit conversion and my default position is that where we let that leak is a bug in mpl. I don't think bugs in Matplotlib should feed back to the dtype design. Tom On Fri, Oct 9, 2020 at 1:08 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
-- Thomas Caswell tcaswell@gmail.com
On Fri, 2020-10-09 at 07:48 -0500, Lee Johnston wrote:
A custom DType will be attached to a normal NumPy array (or, hopefully a pandas dataframe, etc.). I am not sure if that is what you meant, but a custom DType will mean there is no array subclass involved (this is different from how pandas approaches extension dtypes). But, I am not sure that this makes much of a difference with respect to the problem.
It sounds unlikely that this would solve the issue immediately. Rather, we probably need to think about creating some kind of "protocol" to drop units, such as: arr.astype(Unitless) which would be a no-op for the NumPy numerical types. Or baking in something similar to make working with units easier. I would have hoped a bit that it is possible to avoid such issues e.g. by dividing by a step/difference, but I do not know the details. The most important question for me would be more whether there is something in the basic infrastructure to consider that cannot be done as an extension/protocol later (like the `arr.astype(Unitless)` cast, which should be a straight forward extension). Although, it would be interesting to figure out how a future Unit DType would best look in this regard! Cheers, Sebastian
Matplotlib should (internally) never try to do operations on user data before we do the unit conversion and my default position is that where we let that leak is a bug in mpl. I don't think bugs in Matplotlib should feed back to the dtype design. Tom On Fri, Oct 9, 2020 at 1:08 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
-- Thomas Caswell tcaswell@gmail.com
participants (3)
-
Lee Johnston -
Sebastian Berg -
Thomas Caswell