Feature request: dot product along arbitrary axes
Currently there are lots of ways to compute dot products (dot, vdot, inner, tensordot, einsum...), but none of them are really convenient for the case of arrays of vectors, where one dimension (usually the last or the first) is the vector dimension. The simplest way to do this currently is `np.sum(a * b, axis=axis)`, but this makes vector algebra less readable without a wrapper function, and it's probably not optimized as much as matrix products. Another way to do it is by adding appropriate dimensions and using matmul, but that's arguably less readable and not obvious to do generically for arbitrary axes. I think either np.dot or np.vdot could easily be extended with an `axis` parameter that would convert it into a bulk vector operation, with the same semantics as `np.sum(a * b, axis=axis)`. It should also maybe have a `keep_dims` parameter, which is useful for preserving broadcasting. I submitted a corresponding issue at https://github.com/numpy/numpy/issues/21915
I don't understand. Both theretically and coding wise Matmul is the most readable thing that you can have within those options. That is in fact what the definition is. Can you give an example? On Mon, Jul 4, 2022, 04:49 <rmccampbell7@gmail.com> wrote:
Currently there are lots of ways to compute dot products (dot, vdot, inner, tensordot, einsum...), but none of them are really convenient for the case of arrays of vectors, where one dimension (usually the last or the first) is the vector dimension. The simplest way to do this currently is `np.sum(a * b, axis=axis)`, but this makes vector algebra less readable without a wrapper function, and it's probably not optimized as much as matrix products. Another way to do it is by adding appropriate dimensions and using matmul, but that's arguably less readable and not obvious to do generically for arbitrary axes. I think either np.dot or np.vdot could easily be extended with an `axis` parameter that would convert it into a bulk vector operation, with the same semantics as `np.sum(a * b, axis=axis)`. It should also maybe have a `keep_dims` parameter, which is useful for preserving broadcasting.
I submitted a corresponding issue at https://github.com/numpy/numpy/issues/21915 _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: ilhanpolat@gmail.com
Maybe I wasn't clear, I'm talking about the 1-dimensional vector product, but applied to N-D arrays of vectors. Certainly dot products can be realized as matrix products, and often are in mathematics for convenience, but matrices and vectors are not the same thing, theoretically or coding wise. If I have two (M, N, k) arrays a and b where k is the vector dimension, to dot product them using matrix notation I have to do: (a[:, :, np.newaxis, :] @ b[:, :, :, np.newaxis])[:, :, 0, 0] Which I certainly don't find readable (I always have to scratch my head a little bit to figure out whether the newaxis's are in the right places). If this is a common operation in larger expressions, then it basically has to be written as a separate function, which then someone reading the code may have to look at for the semantics. It also breaks down if you want to write generic vector functions that may be applied along different axes; then you need to do something like np.squeeze(np.expand_dims(a, axis=axis) @ np.expand_dims(b, axis=axis+1), (axis, axis+1)) (after normalizing the axis; if it's negative you'd need to do axis-1 and axis instead). Compare this to the simplicity, composability and consistency of: a.dot(b, axis=-1) * np.cross(c, d, axis=-1).dot(e, axis=-1) / np.linalg.norm(f, axis=-1) (the cross and norm operators already support an axis parameter)
The vecdot() function in the array API should be what you are looking for (note that the current implementation in numpy.array_api is incorrect, which I'm fixing at https://github.com/numpy/numpy/pull/21928). It works like dot() but it always applies the 1-D dot product case with broadcasting, and lets you specify the axis. We'd want this function to be added to the main numpy namespace as well. Aaron Meurer On Tue, Jul 5, 2022 at 5:39 PM <rmccampbell7@gmail.com> wrote:
Maybe I wasn't clear, I'm talking about the 1-dimensional vector product, but applied to N-D arrays of vectors. Certainly dot products can be realized as matrix products, and often are in mathematics for convenience, but matrices and vectors are not the same thing, theoretically or coding wise. If I have two (M, N, k) arrays a and b where k is the vector dimension, to dot product them using matrix notation I have to do:
(a[:, :, np.newaxis, :] @ b[:, :, :, np.newaxis])[:, :, 0, 0]
Which I certainly don't find readable (I always have to scratch my head a little bit to figure out whether the newaxis's are in the right places). If this is a common operation in larger expressions, then it basically has to be written as a separate function, which then someone reading the code may have to look at for the semantics. It also breaks down if you want to write generic vector functions that may be applied along different axes; then you need to do something like
np.squeeze(np.expand_dims(a, axis=axis) @ np.expand_dims(b, axis=axis+1), (axis, axis+1))
(after normalizing the axis; if it's negative you'd need to do axis-1 and axis instead).
Compare this to the simplicity, composability and consistency of:
a.dot(b, axis=-1) * np.cross(c, d, axis=-1).dot(e, axis=-1) / np.linalg.norm(f, axis=-1)
(the cross and norm operators already support an axis parameter) _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: asmeurer@gmail.com
On Tue, Jul 5, 2022 at 5:49 PM Aaron Meurer <asmeurer@gmail.com> wrote:
The vecdot() function in the array API should be what you are looking for (note that the current implementation in numpy.array_api is incorrect, which I'm fixing at https://github.com/numpy/numpy/pull/21928). It works like dot() but it always applies the 1-D dot product case with broadcasting, and lets you specify the axis. We'd want this function to be added to the main numpy namespace as well.
See https://data-apis.org/array-api/latest/API_specification/generated/signature... Aaron Meurer
Aaron Meurer
On Tue, Jul 5, 2022 at 5:39 PM <rmccampbell7@gmail.com> wrote:
Maybe I wasn't clear, I'm talking about the 1-dimensional vector product, but applied to N-D arrays of vectors. Certainly dot products can be realized as matrix products, and often are in mathematics for convenience, but matrices and vectors are not the same thing, theoretically or coding wise. If I have two (M, N, k) arrays a and b where k is the vector dimension, to dot product them using matrix notation I have to do:
(a[:, :, np.newaxis, :] @ b[:, :, :, np.newaxis])[:, :, 0, 0]
Which I certainly don't find readable (I always have to scratch my head a little bit to figure out whether the newaxis's are in the right places). If this is a common operation in larger expressions, then it basically has to be written as a separate function, which then someone reading the code may have to look at for the semantics. It also breaks down if you want to write generic vector functions that may be applied along different axes; then you need to do something like
np.squeeze(np.expand_dims(a, axis=axis) @ np.expand_dims(b, axis=axis+1), (axis, axis+1))
(after normalizing the axis; if it's negative you'd need to do axis-1 and axis instead).
Compare this to the simplicity, composability and consistency of:
a.dot(b, axis=-1) * np.cross(c, d, axis=-1).dot(e, axis=-1) / np.linalg.norm(f, axis=-1)
(the cross and norm operators already support an axis parameter) _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: asmeurer@gmail.com
I'm unaware of the context here, is this a specification for functions that it is hoped will eventually be made consistent across numpy/tensorflow/etc? If that's the idea then yeah, I'm all for it, but I would suggest also adding a keepdim parameter (as I mentioned above it helps with broadcasting, i.e. vec_array1.dot(vec_array2, keepdims=True) * vec_array3 would work as expected). But is there an active effort to incorporate these APIs back into numpy?
Oh nevermind, I see that this is added as an experimental module in the latest numpy version. It would be nice to not have to have another whole set of APIs, but on the other hand the numpy API is so messy and inconsistent that maybe it is a good thing :) But it does mean now we have at least 9 different functions/methods/operators that can compute dot products 😢 (not even including the other array_api functions...) I see that currently the vecdot function returns a 2x2 array from two 2x3 APIs, which matches np.inner but is not what I would expect. Does your fix make it instead return a 1-d length-2 array?
On Tue, Jul 5, 2022 at 9:10 PM <rmccampbell7@gmail.com> wrote:
Oh nevermind, I see that this is added as an experimental module in the latest numpy version. It would be nice to not have to have another whole set of APIs, but on the other hand the numpy API is so messy and inconsistent that maybe it is a good thing :) But it does mean now we have at least 9 different functions/methods/operators that can compute dot products 😢 (not even including the other array_api functions...)
The idea of the array API is to have a standard API across all Python array libraries. numpy.array_api is currently implemented as a fully conformant version of that API, but the plan is to eventually make NumPy itself conform as well (so vecdot should be added to numpy at some point).
I see that currently the vecdot function returns a 2x2 array from two 2x3 APIs, which matches np.inner but is not what I would expect. Does your fix make it instead return a 1-d length-2 array?
Yes. I incorrectly implemented vecdot using tensordot, but with my PR this would return a shape (2,) array (with the default axis=-1). Aaron Meurer
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: asmeurer@gmail.com
It might be just me, that @ product is way more readable than chaining different operators below that I don't find readable at all but anyways that's taste I guess. Also if you are going to do this, for a better performance code, you shouldn't bend the ops but you should wrangle the array to the correct type so that you end up straightforward array ops. Anyways, nevermind my noise if you are happy with it. On Wed, Jul 6, 2022 at 1:36 AM <rmccampbell7@gmail.com> wrote:
Maybe I wasn't clear, I'm talking about the 1-dimensional vector product, but applied to N-D arrays of vectors. Certainly dot products can be realized as matrix products, and often are in mathematics for convenience, but matrices and vectors are not the same thing, theoretically or coding wise. If I have two (M, N, k) arrays a and b where k is the vector dimension, to dot product them using matrix notation I have to do:
(a[:, :, np.newaxis, :] @ b[:, :, :, np.newaxis])[:, :, 0, 0]
Which I certainly don't find readable (I always have to scratch my head a little bit to figure out whether the newaxis's are in the right places). If this is a common operation in larger expressions, then it basically has to be written as a separate function, which then someone reading the code may have to look at for the semantics. It also breaks down if you want to write generic vector functions that may be applied along different axes; then you need to do something like
np.squeeze(np.expand_dims(a, axis=axis) @ np.expand_dims(b, axis=axis+1), (axis, axis+1))
(after normalizing the axis; if it's negative you'd need to do axis-1 and axis instead).
Compare this to the simplicity, composability and consistency of:
a.dot(b, axis=-1) * np.cross(c, d, axis=-1).dot(e, axis=-1) / np.linalg.norm(f, axis=-1)
(the cross and norm operators already support an axis parameter) _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: ilhanpolat@gmail.com
On Tue, 2022-07-05 at 23:36 +0000, rmccampbell7@gmail.com wrote:
Maybe I wasn't clear, I'm talking about the 1-dimensional vector product, but applied to N-D arrays of vectors. Certainly dot products can be realized as matrix products, and often are in mathematics for convenience, but matrices and vectors are not the same thing, theoretically or coding wise. If I have two (M, N, k) arrays a and b where k is the vector dimension, to dot product them using matrix notation I have to do:
(a[:, :, np.newaxis, :] @ b[:, :, :, np.newaxis])[:, :, 0, 0]
You can make it more readable for example with: res = a[..., np.newaxis, :] @ b[..., :, np.newaxis] res = res[..., 0, 0] (could remove the `:`). Maybe even more tricks like: rowmat = np.s_[..., np.newaxis, :] colmat = np.s_[..., :, np.newaxis] res = a[rowmat] @ a[colmat]
Which I certainly don't find readable (I always have to scratch my head a little bit to figure out whether the newaxis's are in the right places). If this is a common operation in larger expressions, then it basically has to be written as a separate function, which then someone reading the code may have to look at for the semantics. It also breaks down if you want to write generic vector functions that may be applied along different axes; then you need to do something like
I would suggest using `np.moveaxis` to implement a helper. Now of course there may be a point to put that helper into NumPy as `np.vecdot` (or similar), even if it is probably a 3 line function if implemented in terms of `matmul`. Cheers, Sebastian
np.squeeze(np.expand_dims(a, axis=axis) @ np.expand_dims(b, axis=axis+1), (axis, axis+1))
(after normalizing the axis; if it's negative you'd need to do axis-1 and axis instead).
Compare this to the simplicity, composability and consistency of:
a.dot(b, axis=-1) * np.cross(c, d, axis=-1).dot(e, axis=-1) / np.linalg.norm(f, axis=-1)
(the cross and norm operators already support an axis parameter) _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: sebastian@sipsolutions.net
Yes, if I am doing this more than once in some code I would make a helper. But it's much better I think to have a common function that people can learn and use consistently instead of having to roll their own functions all the time. Especially because numpy otherwise usually just works when you write the algorithm how you write it on paper. There shouldn't need to be too much thinking involved in the translation. Also yeah using fancy index tricks like `rowmat` and `colmat` simplifies the code a bit, but is still obscure and esoteric looking to somebody who's not intimately familiar with numpy indexing.
participants (4)
-
Aaron Meurer -
Ilhan Polat -
rmccampbell7＠gmail.com -
Sebastian Berg