![](https://secure.gravatar.com/avatar/ebcf345b98543c408d583ae438d87a31.jpg?s=120&d=mm&r=g)
I was updating some code that uses a custom array container built with the mixin library. Specifically, I was trying to eliminate some warnings due to the change to the __array__ interface to add a copy argument. In doing so, I discovered that, for two objects u, v in my container class, the code: u[…] = v performs a copy on v. Specifically, it calls __array__() with copy=True. This seems unnecessary and wasteful of memory. Can someone explain to me what is happening here? Thanks. — Daniel Israel XCP-4: Continuum Models and Numerical Algorithms dmi1@lanl.gov
![](https://secure.gravatar.com/avatar/ebcf345b98543c408d583ae438d87a31.jpg?s=120&d=mm&r=g)
Sure. I didn’t originally, because I thought it would require an entire custom array container, but the following trivial example actually shows the behavior in question: import numpy class MyThing(object): def __array__(self, dtype=None, copy=None): print(f"MyThing.__array__(dtype={dtype}, copy={copy})") return numpy.ones((5, 5)) u = numpy.zeros((5, 5)) v = MyThing() u[...] = v If you run this code, as part of the final assignment statement, the __array__ method is called for ‘v’ with copy=True. Why? — Daniel Israel XCP-4: Continuum Models and Numerical Algorithms dmi1@lanl.gov On Dec 25, 2024, at 3:23 PM, Steven Ellis <stevenalonzoellis@gmail.com> wrote: Hi David, New to the listserv, but, maybe you can provide a reproducible example? Steven On Wed, Dec 25, 2024, 2:19 PM Israel, Daniel M via NumPy-Discussion <numpy-discussion@python.org<mailto:numpy-discussion@python.org>> wrote: I was updating some code that uses a custom array container built with the mixin library. Specifically, I was trying to eliminate some warnings due to the change to the __array__ interface to add a copy argument. In doing so, I discovered that, for two objects u, v in my container class, the code: u[…] = v performs a copy on v. Specifically, it calls __array__() with copy=True. This seems unnecessary and wasteful of memory. Can someone explain to me what is happening here? Thanks. — Daniel Israel XCP-4: Continuum Models and Numerical Algorithms dmi1@lanl.gov<mailto:dmi1@lanl.gov> _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org<mailto:numpy-discussion@python.org> To unsubscribe send an email to numpy-discussion-leave@python.org<mailto:numpy-discussion-leave@python.org> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/<https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/numpy-discussion.python.org/__;!!Bt8fGhp8LhKGRg!A_N5Cns4800odNjjStN3QAH94TwUNBEkvr7lqW4-PnnUiGS_kckVyXKfS4QTR4YADAXXGOTbKZ-6RDYTC7knRCMj$> Member address: stevenalonzoellis@gmail.com<mailto:stevenalonzoellis@gmail.com>
![](https://secure.gravatar.com/avatar/b4f6d4f8b501cb05fd054944a166a121.jpg?s=120&d=mm&r=g)
That seems like a bug but not sure why it would happen. It needs to call `__array__`, but indeed of course not with `copy=True`. Would you open an issue on github? - Sebastian On Thu, 2024-12-26 at 03:46 +0000, Israel, Daniel M via NumPy- Discussion wrote:
![](https://secure.gravatar.com/avatar/0cf8ddb97a26d67e8f5807a961eb70ea.jpg?s=120&d=mm&r=g)
It looks like general assignment function `array_assign_subscript` calls `PyArray_CopyObject(view, op)` link <https://github.com/numpy/numpy/blob/a07c6c5f12eec1322d2487f0ed935c5c13b8d883...> which in turn always calls `PyArray_DiscoverDTypeAndShape` with `copy=1` and this is where `copy=True` takes place. We could add a `copy` parameter to `PyArray_CopyObject` or set `copy=-1` (to `None`) for the `PyArray_DiscoverDTypeAndShape` call. Mateusz On Thu, Dec 26, 2024 at 10:19 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:
![](https://secure.gravatar.com/avatar/697900d3a29858ea20cc109a2aee0af6.jpg?s=120&d=mm&r=g)
Seems to make sense to me? Or is the following a bug?
If you don't do a copy, then it is a view, right? And so, should modifying v[4] change u[4]? Relatedly, if these were object arrays, of mutable objects, should mutating u[4] point to the exact same instance as v[4] and mutating it in one array means that the other array "sees" the same changes? Sorry if I'm missing the point, just want to make sure we don't inadvertently change important implicit behavior. Ben Root On Thu, Dec 26, 2024 at 7:07 AM Mateusz Sokol <msokol@quansight.com> wrote:
![](https://secure.gravatar.com/avatar/764323a14e554c97ab74177e0bce51d4.jpg?s=120&d=mm&r=g)
On Thu, Dec 26, 2024 at 5:48 PM Benjamin Root via NumPy-Discussion < numpy-discussion@python.org> wrote:
No one is suggesting that behavior should change. The issue is whether or not an unnecessary, _additional_ copy gets made when converting a not-exactly-an-`ndarray` object to an `ndarray` object before doing the assignment (which always copies values over to the destination array). -- Robert Kern
![](https://secure.gravatar.com/avatar/ebcf345b98543c408d583ae438d87a31.jpg?s=120&d=mm&r=g)
Done. https://github.com/numpy/numpy/issues/28070 — Daniel Israel XCP-4: Continuum Models and Numerical Algorithms dmi1@lanl.gov On Dec 26, 2024, at 3:44 AM, Sebastian Berg <sebastian@sipsolutions.net> wrote: That seems like a bug but not sure why it would happen. It needs to call `__array__`, but indeed of course not with `copy=True`. Would you open an issue on github? - Sebastian On Thu, 2024-12-26 at 03:46 +0000, Israel, Daniel M via NumPy- Discussion wrote: Sure. I didn’t originally, because I thought it would require an entire custom array container, but the following trivial example actually shows the behavior in question: import numpy class MyThing(object): def __array__(self, dtype=None, copy=None): print(f"MyThing.__array__(dtype={dtype}, copy={copy})") return numpy.ones((5, 5)) u = numpy.zeros((5, 5)) v = MyThing() u[...] = v If you run this code, as part of the final assignment statement, the __array__ method is called for ‘v’ with copy=True. Why? — Daniel Israel XCP-4: Continuum Models and Numerical Algorithms dmi1@lanl.gov On Dec 25, 2024, at 3:23 PM, Steven Ellis <stevenalonzoellis@gmail.com> wrote: Hi David, New to the listserv, but, maybe you can provide a reproducible example? Steven On Wed, Dec 25, 2024, 2:19 PM Israel, Daniel M via NumPy-Discussion <numpy-discussion@python.org<mailto:numpy-discussion@python.org><mailto:numpy-discussion@python.org>> wrote: I was updating some code that uses a custom array container built with the mixin library. Specifically, I was trying to eliminate some warnings due to the change to the __array__ interface to add a copy argument. In doing so, I discovered that, for two objects u, v in my container class, the code: u[…] = v performs a copy on v. Specifically, it calls __array__() with copy=True. This seems unnecessary and wasteful of memory. Can someone explain to me what is happening here? Thanks. — Daniel Israel XCP-4: Continuum Models and Numerical Algorithms dmi1@lanl.gov<mailto:dmi1@lanl.gov><mailto:dmi1@lanl.gov> _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org<mailto:numpy-discussion@python.org><mailto:numpy-discussion@python.org> To unsubscribe send an email to numpy-discussion-leave@python.org<mailto:numpy-discussion-leave@python.org><mailto: numpy-discussion-leave@python.org<mailto:numpy-discussion-leave@python.org>> https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/numpy-dis... < https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/numpy-dis... Member address: stevenalonzoellis@gmail.com<mailto:stevenalonzoellis@gmail.com><mailto:stevenalonzoellis@gmail.com> _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org<mailto:numpy-discussion@python.org> To unsubscribe send an email to numpy-discussion-leave@python.org<mailto:numpy-discussion-leave@python.org> https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/numpy-dis... Member address: sebastian@sipsolutions.net<mailto:sebastian@sipsolutions.net>
![](https://secure.gravatar.com/avatar/ebcf345b98543c408d583ae438d87a31.jpg?s=120&d=mm&r=g)
Sure. I didn’t originally, because I thought it would require an entire custom array container, but the following trivial example actually shows the behavior in question: import numpy class MyThing(object): def __array__(self, dtype=None, copy=None): print(f"MyThing.__array__(dtype={dtype}, copy={copy})") return numpy.ones((5, 5)) u = numpy.zeros((5, 5)) v = MyThing() u[...] = v If you run this code, as part of the final assignment statement, the __array__ method is called for ‘v’ with copy=True. Why? — Daniel Israel XCP-4: Continuum Models and Numerical Algorithms dmi1@lanl.gov On Dec 25, 2024, at 3:23 PM, Steven Ellis <stevenalonzoellis@gmail.com> wrote: Hi David, New to the listserv, but, maybe you can provide a reproducible example? Steven On Wed, Dec 25, 2024, 2:19 PM Israel, Daniel M via NumPy-Discussion <numpy-discussion@python.org<mailto:numpy-discussion@python.org>> wrote: I was updating some code that uses a custom array container built with the mixin library. Specifically, I was trying to eliminate some warnings due to the change to the __array__ interface to add a copy argument. In doing so, I discovered that, for two objects u, v in my container class, the code: u[…] = v performs a copy on v. Specifically, it calls __array__() with copy=True. This seems unnecessary and wasteful of memory. Can someone explain to me what is happening here? Thanks. — Daniel Israel XCP-4: Continuum Models and Numerical Algorithms dmi1@lanl.gov<mailto:dmi1@lanl.gov> _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org<mailto:numpy-discussion@python.org> To unsubscribe send an email to numpy-discussion-leave@python.org<mailto:numpy-discussion-leave@python.org> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/<https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/numpy-discussion.python.org/__;!!Bt8fGhp8LhKGRg!A_N5Cns4800odNjjStN3QAH94TwUNBEkvr7lqW4-PnnUiGS_kckVyXKfS4QTR4YADAXXGOTbKZ-6RDYTC7knRCMj$> Member address: stevenalonzoellis@gmail.com<mailto:stevenalonzoellis@gmail.com>
![](https://secure.gravatar.com/avatar/b4f6d4f8b501cb05fd054944a166a121.jpg?s=120&d=mm&r=g)
That seems like a bug but not sure why it would happen. It needs to call `__array__`, but indeed of course not with `copy=True`. Would you open an issue on github? - Sebastian On Thu, 2024-12-26 at 03:46 +0000, Israel, Daniel M via NumPy- Discussion wrote:
![](https://secure.gravatar.com/avatar/0cf8ddb97a26d67e8f5807a961eb70ea.jpg?s=120&d=mm&r=g)
It looks like general assignment function `array_assign_subscript` calls `PyArray_CopyObject(view, op)` link <https://github.com/numpy/numpy/blob/a07c6c5f12eec1322d2487f0ed935c5c13b8d883...> which in turn always calls `PyArray_DiscoverDTypeAndShape` with `copy=1` and this is where `copy=True` takes place. We could add a `copy` parameter to `PyArray_CopyObject` or set `copy=-1` (to `None`) for the `PyArray_DiscoverDTypeAndShape` call. Mateusz On Thu, Dec 26, 2024 at 10:19 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:
![](https://secure.gravatar.com/avatar/697900d3a29858ea20cc109a2aee0af6.jpg?s=120&d=mm&r=g)
Seems to make sense to me? Or is the following a bug?
If you don't do a copy, then it is a view, right? And so, should modifying v[4] change u[4]? Relatedly, if these were object arrays, of mutable objects, should mutating u[4] point to the exact same instance as v[4] and mutating it in one array means that the other array "sees" the same changes? Sorry if I'm missing the point, just want to make sure we don't inadvertently change important implicit behavior. Ben Root On Thu, Dec 26, 2024 at 7:07 AM Mateusz Sokol <msokol@quansight.com> wrote:
![](https://secure.gravatar.com/avatar/764323a14e554c97ab74177e0bce51d4.jpg?s=120&d=mm&r=g)
On Thu, Dec 26, 2024 at 5:48 PM Benjamin Root via NumPy-Discussion < numpy-discussion@python.org> wrote:
No one is suggesting that behavior should change. The issue is whether or not an unnecessary, _additional_ copy gets made when converting a not-exactly-an-`ndarray` object to an `ndarray` object before doing the assignment (which always copies values over to the destination array). -- Robert Kern
![](https://secure.gravatar.com/avatar/ebcf345b98543c408d583ae438d87a31.jpg?s=120&d=mm&r=g)
Done. https://github.com/numpy/numpy/issues/28070 — Daniel Israel XCP-4: Continuum Models and Numerical Algorithms dmi1@lanl.gov On Dec 26, 2024, at 3:44 AM, Sebastian Berg <sebastian@sipsolutions.net> wrote: That seems like a bug but not sure why it would happen. It needs to call `__array__`, but indeed of course not with `copy=True`. Would you open an issue on github? - Sebastian On Thu, 2024-12-26 at 03:46 +0000, Israel, Daniel M via NumPy- Discussion wrote: Sure. I didn’t originally, because I thought it would require an entire custom array container, but the following trivial example actually shows the behavior in question: import numpy class MyThing(object): def __array__(self, dtype=None, copy=None): print(f"MyThing.__array__(dtype={dtype}, copy={copy})") return numpy.ones((5, 5)) u = numpy.zeros((5, 5)) v = MyThing() u[...] = v If you run this code, as part of the final assignment statement, the __array__ method is called for ‘v’ with copy=True. Why? — Daniel Israel XCP-4: Continuum Models and Numerical Algorithms dmi1@lanl.gov On Dec 25, 2024, at 3:23 PM, Steven Ellis <stevenalonzoellis@gmail.com> wrote: Hi David, New to the listserv, but, maybe you can provide a reproducible example? Steven On Wed, Dec 25, 2024, 2:19 PM Israel, Daniel M via NumPy-Discussion <numpy-discussion@python.org<mailto:numpy-discussion@python.org><mailto:numpy-discussion@python.org>> wrote: I was updating some code that uses a custom array container built with the mixin library. Specifically, I was trying to eliminate some warnings due to the change to the __array__ interface to add a copy argument. In doing so, I discovered that, for two objects u, v in my container class, the code: u[…] = v performs a copy on v. Specifically, it calls __array__() with copy=True. This seems unnecessary and wasteful of memory. Can someone explain to me what is happening here? Thanks. — Daniel Israel XCP-4: Continuum Models and Numerical Algorithms dmi1@lanl.gov<mailto:dmi1@lanl.gov><mailto:dmi1@lanl.gov> _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org<mailto:numpy-discussion@python.org><mailto:numpy-discussion@python.org> To unsubscribe send an email to numpy-discussion-leave@python.org<mailto:numpy-discussion-leave@python.org><mailto: numpy-discussion-leave@python.org<mailto:numpy-discussion-leave@python.org>> https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/numpy-dis... < https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/numpy-dis... Member address: stevenalonzoellis@gmail.com<mailto:stevenalonzoellis@gmail.com><mailto:stevenalonzoellis@gmail.com> _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org<mailto:numpy-discussion@python.org> To unsubscribe send an email to numpy-discussion-leave@python.org<mailto:numpy-discussion-leave@python.org> https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/numpy-dis... Member address: sebastian@sipsolutions.net<mailto:sebastian@sipsolutions.net>
participants (6)
-
Benjamin Root
-
Israel, Daniel M
-
Mateusz Sokol
-
Robert Kern
-
Sebastian Berg
-
Steven Ellis