"Backport" new DType API use to old DTypes
Hi all, I would like a project like `ml_dtypes` to be able to transition to the new DType API, I believe that is vital to make easy. If anyone has concerns around this, please chime in. IMO, we need to just do this to move forward. I.e. provide an API that allows using the new DType API all the way back with NumPy 2.0 (by compiling with NumPy 2.5 or vendoring some code). [1] The trick being that we can use the new DType API, so long the DType still is "legacy" and thus uses all the old code paths (unless the new API clearly replaced them). Without such "backport" there is a problem that any regression/bug means that we can't use the new API until the buggy NumPy version is dropped and since nothing is bug-free that can be years of waiting... The path is the following PR https://github.com/numpy/numpy/pull/31302 * Create a new DType registration slot, where the old defining struct is passed in. (We'll use less and less of it over time.) * We have a `static inline` function in NumPy 2.5 (which downstream can vendor to compile with older NumPy versions!). This static inline function does a bit of a crazy dance, but the end result is a working "old but new" DType (`ml_dtypes` tests pass). This gives a path, and a path mean we can actually start deprecating things soon. Even if some things are baked in slightly more after this (I think this is rather little in practice), having a path now is still better than not. Now is also a great time, because it is a good time to drop NumPy <2. Testing is a bit hard. The truly exhaustive tests is probably just indirectly `ml_dtypes` (which google tests exhaustively). (The `ml_dtypes` pass completely with this approach.) Cheers, Sebastian [1] There is exactly one corner-case I saw, and that is that we print: `np.array(..., dtype=repr(dtype))` for these DTypes, while legacy ones printed `dtype=dtype.name`, which is a bit shorter. (We can change that in NumPy 2.5+, but not on old versions.)
On Thu, Apr 23, 2026 at 3:40 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Hi all,
I would like a project like `ml_dtypes` to be able to transition to the new DType API, I believe that is vital to make easy. If anyone has concerns around this, please chime in. IMO, we need to just do this to move forward.
This seems like a good idea. The details are a bit complex it looks like, but if it works for the known users of the legacy API, then that should give enough confidence. It probably does need some integration test like building `ml_dtypes` against main, and then testing it against whatever the minimum version is that `ml_dtypes` supports. Cheers, Ralf
I.e. provide an API that allows using the new DType API all the way back with NumPy 2.0 (by compiling with NumPy 2.5 or vendoring some code). [1] The trick being that we can use the new DType API, so long the DType still is "legacy" and thus uses all the old code paths (unless the new API clearly replaced them).
Without such "backport" there is a problem that any regression/bug means that we can't use the new API until the buggy NumPy version is dropped and since nothing is bug-free that can be years of waiting...
The path is the following PR https://github.com/numpy/numpy/pull/31302 * Create a new DType registration slot, where the old defining struct is passed in. (We'll use less and less of it over time.) * We have a `static inline` function in NumPy 2.5 (which downstream can vendor to compile with older NumPy versions!). This static inline function does a bit of a crazy dance, but the end result is a working "old but new" DType (`ml_dtypes` tests pass).
This gives a path, and a path mean we can actually start deprecating things soon. Even if some things are baked in slightly more after this (I think this is rather little in practice), having a path now is still better than not. Now is also a great time, because it is a good time to drop NumPy <2.
Testing is a bit hard. The truly exhaustive tests is probably just indirectly `ml_dtypes` (which google tests exhaustively). (The `ml_dtypes` pass completely with this approach.)
Cheers,
Sebastian
[1] There is exactly one corner-case I saw, and that is that we print: `np.array(..., dtype=repr(dtype))` for these DTypes, while legacy ones printed `dtype=dtype.name`, which is a bit shorter. (We can change that in NumPy 2.5+, but not on old versions.) _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3//lists/numpy-discussion.python.org Member address: ralf.gommers@gmail.com
On Fri, 2026-04-24 at 10:54 +0200, Ralf Gommers via NumPy-Discussion wrote:
On Thu, Apr 23, 2026 at 3:40 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Hi all,
I would like a project like `ml_dtypes` to be able to transition to the new DType API, I believe that is vital to make easy. If anyone has concerns around this, please chime in. IMO, we need to just do this to move forward.
This seems like a good idea. The details are a bit complex it looks like, but if it works for the known users of the legacy API, then that should give enough confidence.
It probably does need some integration test like building `ml_dtypes` against main, and then testing it against whatever the minimum version is that `ml_dtypes` supports.
Yeah, I am honestly not yet sure what is useful here. `ml_dtypes` already tests with NumPy nightlies and compiles with the newest released NumPy available. Adding a test there that compiles with nightlies but runs with the oldest version is straight-forward. So `ml_dtypes` has better test coverage than what we can really hope to add into NumPy itself. I don't mind adding it too much, but it would be a first one for us and unless to run downstream tests (and maybe even build the project). (We definitely need to have some general tests for the new API of course.) - Sebastian
Cheers, Ralf
I.e. provide an API that allows using the new DType API all the way back with NumPy 2.0 (by compiling with NumPy 2.5 or vendoring some code). [1] The trick being that we can use the new DType API, so long the DType still is "legacy" and thus uses all the old code paths (unless the new API clearly replaced them).
Without such "backport" there is a problem that any regression/bug means that we can't use the new API until the buggy NumPy version is dropped and since nothing is bug-free that can be years of waiting...
The path is the following PR https://github.com/numpy/numpy/pull/31302 * Create a new DType registration slot, where the old defining struct is passed in. (We'll use less and less of it over time.) * We have a `static inline` function in NumPy 2.5 (which downstream can vendor to compile with older NumPy versions!). This static inline function does a bit of a crazy dance, but the end result is a working "old but new" DType (`ml_dtypes` tests pass).
This gives a path, and a path mean we can actually start deprecating things soon. Even if some things are baked in slightly more after this (I think this is rather little in practice), having a path now is still better than not. Now is also a great time, because it is a good time to drop NumPy <2.
Testing is a bit hard. The truly exhaustive tests is probably just indirectly `ml_dtypes` (which google tests exhaustively). (The `ml_dtypes` pass completely with this approach.)
Cheers,
Sebastian
[1] There is exactly one corner-case I saw, and that is that we print: `np.array(..., dtype=repr(dtype))` for these DTypes, while legacy ones printed `dtype=dtype.name`, which is a bit shorter. (We can change that in NumPy 2.5+, but not on old versions.) _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3//lists/numpy-discussion.python.org Member address: ralf.gommers@gmail.com
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3//lists/numpy-discussion.python.org Member address: sebastian@sipsolutions.net
participants (2)
-
Ralf Gommers -
Sebastian Berg