Numpy tools for getting the results of indexing operations
I'm trying to make a few different file backed array-like objects. However, I find myself struggling to get all the indexing operations right. Is there a collection of utilities that answer questions like: 1. Given a shape, and key that would be valid in numpy, what is the resulting shape. 2. Given a key, is it a fancy index key (I just don't want to support this use case). 3. Given a key, can you "expand" the of the key so that it matches my input shape? 4. Can you help me handle keys with `None` in them as an "np.newindex"? I feel like dask, zarr, xarray would have all had to use functions like these, I'm just wondering if there was any reconciliation of this kind of functionality since the development of various `__array__` NEPs. Best, Mark
I wrote the ndindex library to do exactly this sort of thing https://quansight-labs.github.io/ndindex/, namely the manipulation of NumPy index objects. Some of the things you mentioned aren't implemented yet (like checking if an index is an advanced index or not), but they are definitely in scope and I would love to add them. Aaron Meurer On Mon, Jan 30, 2023 at 9:28 PM Mark Harfouche <mark.harfouche@gmail.com> wrote:
I'm trying to make a few different file backed array-like objects.
However, I find myself struggling to get all the indexing operations right.
Is there a collection of utilities that answer questions like:
1. Given a shape, and key that would be valid in numpy, what is the resulting shape. 2. Given a key, is it a fancy index key (I just don't want to support this use case). 3. Given a key, can you "expand" the of the key so that it matches my input shape? 4. Can you help me handle keys with `None` in them as an "np.newindex"?
I feel like dask, zarr, xarray would have all had to use functions like these, I'm just wondering if there was any reconciliation of this kind of functionality since the development of various `__array__` NEPs.
Best,
Mark _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: asmeurer@gmail.com
+1 We are using ndindex for quite long time, and we have been impressed not only for how well it reproduces the NumPy indexing (bar exceptions like Aaron mentions), but also by the elegance of the API. Definitely a great complement to all libraries that has to handle n-dim data. Francesc On Tue, Jan 31, 2023 at 10:03 AM Aaron Meurer <asmeurer@gmail.com> wrote:
I wrote the ndindex library to do exactly this sort of thing https://quansight-labs.github.io/ndindex/, namely the manipulation of NumPy index objects. Some of the things you mentioned aren't implemented yet (like checking if an index is an advanced index or not), but they are definitely in scope and I would love to add them.
Aaron Meurer
On Mon, Jan 30, 2023 at 9:28 PM Mark Harfouche <mark.harfouche@gmail.com> wrote:
I'm trying to make a few different file backed array-like objects.
However, I find myself struggling to get all the indexing operations right.
Is there a collection of utilities that answer questions like:
1. Given a shape, and key that would be valid in numpy, what is the resulting shape. 2. Given a key, is it a fancy index key (I just don't want to support this use case). 3. Given a key, can you "expand" the of the key so that it matches my input shape? 4. Can you help me handle keys with `None` in them as an "np.newindex"?
I feel like dask, zarr, xarray would have all had to use functions like these, I'm just wondering if there was any reconciliation of this kind of functionality since the development of various `__array__` NEPs.
Best,
Mark _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: asmeurer@gmail.com
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: faltet@gmail.com
-- Francesc Alted
Thanks. I've gotten pretty far in what I need to do for my limited scope. But I'll definitely read the docs and add it to the things I should use upon feature expansion! Basically, ive found that in many cases, task managers add too much overhead for operations that should be straightforward. However, maybe i'm not grasping the true scope of all the intricacies. That said, being able to operate on the medata (shape, dtypes) is critical in any case. Best, Mark On Tue, Jan 31, 2023 at 4:17 AM Francesc Alted <faltet@gmail.com> wrote:
+1 We are using ndindex for quite long time, and we have been impressed not only for how well it reproduces the NumPy indexing (bar exceptions like Aaron mentions), but also by the elegance of the API. Definitely a great complement to all libraries that has to handle n-dim data.
Francesc
On Tue, Jan 31, 2023 at 10:03 AM Aaron Meurer <asmeurer@gmail.com> wrote:
I wrote the ndindex library to do exactly this sort of thing https://quansight-labs.github.io/ndindex/, namely the manipulation of NumPy index objects. Some of the things you mentioned aren't implemented yet (like checking if an index is an advanced index or not), but they are definitely in scope and I would love to add them.
Aaron Meurer
On Mon, Jan 30, 2023 at 9:28 PM Mark Harfouche <mark.harfouche@gmail.com> wrote:
I'm trying to make a few different file backed array-like objects.
However, I find myself struggling to get all the indexing operations right.
Is there a collection of utilities that answer questions like:
1. Given a shape, and key that would be valid in numpy, what is the resulting shape. 2. Given a key, is it a fancy index key (I just don't want to support this use case). 3. Given a key, can you "expand" the of the key so that it matches my input shape? 4. Can you help me handle keys with `None` in them as an "np.newindex"?
I feel like dask, zarr, xarray would have all had to use functions like these, I'm just wondering if there was any reconciliation of this kind of functionality since the development of various `__array__` NEPs.
Best,
Mark _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: asmeurer@gmail.com
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: faltet@gmail.com
-- Francesc Alted _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: mark.harfouche@gmail.com
participants (3)
-
Aaron Meurer -
Francesc Alted -
Mark Harfouche