Hi! Yes, I would advocate for a `force=` kwarg but personally I don't think it's explicit enough, but probably as explicit as can be given NumPy's API. Personally, I'd also raise a warning within PyData/Sparse and I hope it's in big bold letters in the docs in NumPy to be careful with this. Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: NumPy-Discussion <numpy-discussion-bounces+einstein.edison=gmail.com@python.org> on behalf of Ralf Gommers <ralf.gommers@gmail.com> Sent: Tuesday, April 28, 2020 11:51:13 AM To: Discussion of Numerical Python <numpy-discussion@python.org> Subject: Re: [Numpy-discussion] Proposal: add `force=` or `copy=` kwarg to `__array__` interface On Mon, Apr 27, 2020 at 12:10 AM Sebastian Berg <sebastian@sipsolutions.net<mailto:sebastian@sipsolutions.net>> wrote: On Sat, 2020-04-25 at 10:52 -0700, Stephan Hoyer wrote:
On Sat, Apr 25, 2020 at 10:40 AM Ralf Gommers <ralf.gommers@gmail.com<mailto:ralf.gommers@gmail.com>
wrote:
On Fri, Apr 24, 2020 at 12:35 PM Eric Wieser < wieser.eric+numpy@gmail.com<mailto:wieser.eric%2Bnumpy@gmail.com>> wrote:
Perhaps worth mentioning that we've discussed this sort of API before, in https://github.com/numpy/numpy/pull/11897.
Under that proposal, the api would be something like:
* `copy=True` - always copy, like it is today * `copy=False` - copy if needed, like it is today * `copy=np.never_copy` - never copy, throw an exception if not possible
There's a couple of issues I see with using `copy` for __array__: - copy is already weird (False doesn't mean no), and a [bool, some_obj_or_str] keyword isn't making that better - the behavior we're talking about can do more than copying, e.g. for PyTorch it would modify the autograd graph by adding detach(), and for sparse it's not just "make a copy" (which implies doubling memory use) but it densifies which can massively blow up the memory. - I'm -1 on adding things to the main namespace (never_copy) for something that can be handled differently (like a string, or a new keyword)
tl;dr a new `force` keyword would be better
I agree, “copy” is not a good description of this desired coercion behavior.
A new keyword argument like “force” would be much clearer.
That seems fine and practical. But, in the end it seems to me that the `force=` keyword, just means that some projects want to teach their users that: 1. `np.asarray()` can be expensive (and may always copy) 2. `np.asarray()` always loses type properties while others do not choose to teach about it. There seems very little or even no "promise" attached to either `force=True` or `force=False`. In the end, the question is whether sparse will actually want to implement `force=True` if the main reason we add is for library use. That's for PyData Sparse and scipy.sparse devs to answer. Maybe Hameer can answer for the former here. For SciPy that should be decided on the scipy-dev list, but my opinion would be: yes to adding __array__ that raises TypeError by default, and converts with `force=True`. There is no difference between a visualization library or numpy. In both cases the users memory will blow up just the same. As for PyTorch, is `.detach()` even a good reason? Maybe I am missing things, but:
torch.ones(10, requires_grad=True) + np.arange(10) # RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead.
So arguably, there is no type-safety concern due to `.detach()`. I'm not sure what the question is here; no one mentioned type-safety. The PyTorch maintainers have already said they're fine with adding a force keyword. There is an (obvious) general loss of type information that always occurs with an `np.asarray` call. But I do not see that creating any openings for bugs here, due to the wisdom of not allowing the above operation. In fact, it actually seems much worse for for xarray, or pandas. They do support the above operation and will potentially mess up if the arange was previously an xarray with a matching index, but different order. I am very much in favor of adding such things, but I still lack a bit of clarity as to whom we would be helping? See Juan's first email. I personally am ambivalent on this proposal, but if Juan and the Napari devs really want it, that's good enough for me. Cheers, Ralf If end-users will actually use `np.asarray(..., force=True)` over special methods, then great! But I am currently not sure the type- safety argument is all that big of a point. And the performance or memory-blowup argument remains true even for visualization libraries (where the array is purely input and never output as such). But yes, "never copy" is a somewhat different extension to `__array__` and `np.asarray`. It guarantees high speed and in-place behaviour which is useful for different settings. - Sebastian
Cheers, Ralf
I think the discussion stalled on the precise spelling of the third option.
`__array__` was not discussed there, but it seems like adding the `copy` argument to `__array__` would be a perfectly reasonable extension.
Eric
On Fri, 24 Apr 2020 at 03:00, Juan Nunez-Iglesias < jni@fastmail.com<mailto:jni@fastmail.com>> wrote:
Hi everyone,
One bit of expressivity we would miss is “copy if necessary, but
otherwise don’t bother”, but there are workarounds to this.
After a side discussion with Stéfan van der Walt, we came up with `allow_copy=True`, which would express to the downstream library that we don’t mind waiting, but that zero-copy would also be ok.
This sounds like the sort of thing that is use case driven. If enough projects want to use it, then I have no objections to adding the keyword. OTOH, we need to be careful about adding too many interoperability tricks as they complicate the code and makes it hard for folks to determine the best solution. Interoperability is a hot topic and we need to be careful not put too leave behind too many experiments in the NumPy code. Do you have any other ideas of how to achieve the same effect?
Personally, I don’t have any other ideas, but would be happy to hear some!
My view regarding API/experiment creep is that `__array__` is the oldest and most basic of all the interop tricks and that this can be safely maintained for future generations. Currently it only takes `dtype=` as a keyword argument, so it is a very lean API. I think this particular use case is very natural and I’ve encountered the reluctance to implicitly copy twice, so I expect it is reasonably common.
Regarding difficulty in determining the best solution, I would be happy to contribute to the dispatch basics guide together with the new kwarg. I agree that the protocols are getting quite numerous and I couldn’t find a single place that gathers all the best practices together. But, to reiterate my point: `__array__` is the simplest of these and I think this keyword is pretty safe to add.
For ease of discussion, here are the API options discussed so far, as well as a few extra that I don’t like but might trigger other ideas:
np.asarray(my_duck_array, allow_copy=True) # default is False, or None -> leave it to the duck array to decide np.asarray(my_duck_array, copy=True) # always copies, but, if supported by the duck array, defers to it for the copy np.asarray(my_duck_array, copy=‘allow’) # could take values ‘allow’, ‘force’, ’no’, True(=‘force’), False(=’no’) np.asarray(my_duck_array, force_copy=False, allow_copy=True) # separate concepts, but unclear what force_copy=True, allow_copy=False means! np.asarray(my_duck_array, force=True)
Juan. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org<mailto:NumPy-Discussion@python.org> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org<mailto:NumPy-Discussion@python.org> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org<mailto:NumPy-Discussion@python.org> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org<mailto:NumPy-Discussion@python.org> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org<mailto:NumPy-Discussion@python.org> https://mail.python.org/mailman/listinfo/numpy-discussion