[Numpy-discussion] Proposal: add `force=` or `copy=` kwarg to `__array__` interface

[Ralf Gommers]

On Fri, Apr 24, 2020 at 12:35 PM Eric Wieser <wieser.eric+numpy at gmail.com>
wrote:

> Perhaps worth mentioning that we've discussed this sort of API before, in
> https://github.com/numpy/numpy/pull/11897.
>
> Under that proposal, the api would be something like:
>
> * `copy=True` - always copy, like it is today
> * `copy=False` - copy if needed, like it is today
> * `copy=np.never_copy` - never copy, throw an exception if not possible
>

There's a couple of issues I see with using `copy` for __array__:
- copy is already weird (False doesn't mean no), and a [bool,
some_obj_or_str] keyword isn't making that better
- the behavior we're talking about can do more than copying, e.g. for
PyTorch it would modify the autograd graph by adding detach(), and for
sparse it's not just "make a copy" (which implies doubling memory use) but
it densifies which can massively blow up the memory.
- I'm -1 on adding things to the main namespace (never_copy) for something
that can be handled differently (like a string, or a new keyword)

tl;dr a new `force` keyword would be better

Cheers,
Ralf


> I think the discussion stalled on the precise spelling of the third option.
>
> `__array__` was not discussed there, but it seems like adding the `copy`
> argument to `__array__` would be a perfectly reasonable extension.
>
> Eric
>
> On Fri, 24 Apr 2020 at 03:00, Juan Nunez-Iglesias <jni at fastmail.com>
> wrote:
>
>> Hi everyone,
>>
>> One bit of expressivity we would miss is “copy if necessary, but
>>> otherwise don’t bother”, but there are workarounds to this.
>>>
>>
>> After a side discussion with Stéfan van der Walt, we came up with
>> `allow_copy=True`, which would express to the downstream library that we
>> don’t mind waiting, but that zero-copy would also be ok.
>>
>> This sounds like the sort of thing that is use case driven. If enough
>> projects want to use it, then I have no objections to adding the keyword.
>> OTOH, we need to be careful about adding too many interoperability tricks
>> as they complicate the code and makes it hard for folks to determine the
>> best solution. Interoperability is a hot topic and we need to be careful
>> not put too leave behind too many experiments in the NumPy code.  Do you
>> have any other ideas of how to achieve the same effect?
>>
>>
>> Personally, I don’t have any other ideas, but would be happy to hear some!
>>
>> My view regarding API/experiment creep is that `__array__` is the oldest
>> and most basic of all the interop tricks and that this can be safely
>> maintained for future generations. Currently it only takes `dtype=` as a
>> keyword argument, so it is a very lean API. I think this particular use
>> case is very natural and I’ve encountered the reluctance to implicitly copy
>> twice, so I expect it is reasonably common.
>>
>> Regarding difficulty in determining the best solution, I would be happy
>> to contribute to the dispatch basics guide together with the new kwarg. I
>> agree that the protocols are getting quite numerous and I couldn’t find a
>> single place that gathers all the best practices together. But, to
>> reiterate my point: `__array__` is the simplest of these and I think this
>> keyword is pretty safe to add.
>>
>> For ease of discussion, here are the API options discussed so far, as
>> well as a few extra that I don’t like but might trigger other ideas:
>>
>> np.asarray(my_duck_array, allow_copy=True)  # default is False, or None
>> -> leave it to the duck array to decide
>> np.asarray(my_duck_array, copy=True)  # always copies, but, if supported
>> by the duck array, defers to it for the copy
>> np.asarray(my_duck_array, copy=‘allow’)  # could take values ‘allow’,
>> ‘force’, ’no’, True(=‘force’), False(=’no’)
>> np.asarray(my_duck_array, force_copy=False, allow_copy=True)  # separate
>> concepts, but unclear what force_copy=True, allow_copy=False means!
>> np.asarray(my_duck_array, force=True)
>>
>> Juan.
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20200425/c7e35927/attachment.html>