[Numpy-discussion] Add guaranteed no-copy to array creation and reshape?

[Feng Yu]

Eric,

I agree these are questions that shall be answered. I think issues you
raised are toward functions always make a copy -- combining them with
NEVERCOPY does sound sane.

Your argument is that If the atom of the new behavior is per method, then
there is no need to worry about those functions that does not sound sane
when combined with NEVERCOPY.

Here is a proposal that may address your concerns:

1. The name of the flag does not need to be NEVERCOPY -- especially it
sounds insane combined with many methods.
    I think something like KEEPBASE may be easier to reason.
   The name KEEPBASE alwsy suggests it shall not be set on an object where
base is None; this feature may simplify the matter.

2. Methods that creates copy shall always create a copy, regardless of a
flag -- that's the definition of copy. (np.copy)
    KEEPBASE shall only affect methods that creates a new object object,
which may reference the provided base or create a new base.

3. Functions that are not a ndarray method, shall ignore the flag, unless
specified otherwise. (np.array)

4. arr + 0 when arr is KEEPBASE. It depends on if we think this triggered
np.add(), or ndarray.__add__. I think ndarray.__add__ is a shortcut to call
the ufunc np.add(). Thus the result of arr + 0 shall be the behavior on the
ufunc np.add, which shall ignore the flag.

Notice that in memory tight situations, users can already use inplace
operations to avoid copies. This verbose but highly controlled syntax may
be preferable than inferring an automated behavior that relies on KEEPBASE
of arguments.

5. I think on the operation level, the difference between
subclassing(flags) and function arguments would be:

array.view(keepbase=True).reshape(-1) vs array.reshape(-1, keepbase=True)

The gain is that no array method needs to be modified. The control can
still be at the level of the algorithm.



On Sat, Jan 12, 2019 at 11:22 PM Eric Wieser <wieser.eric+numpy at gmail.com>
wrote:

> I don’t think a NEVERCOPY entry in arr.flags would make much sense.
> Is this really a sensible limitation to put on how data gets used? Isn’t
> it up to the algorithm to decide whether to copy its data, not the original
> owner of the data?
>
> It also leads to some tricky questions of precedence - would np.array(arr,
> copy=True) respect the flag or the argument? How about np.array(arr)? Is arr
> + 0 considered a copy?
> By keeping it as a value passed in via a copy= kwarg, we don’t need to
> answer any of those questions.
>
> Eric
>
> On Thu, 10 Jan 2019 at 20:28 Ralf Gommers ralf.gommers at gmail.com
> <http://mailto:ralf.gommers@gmail.com> wrote:
>
> On Thu, Jan 10, 2019 at 11:21 AM Feng Yu <rainwoodman at gmail.com> wrote:
>>
>>> Hi Todd,
>>>
>>> I agree a flag is more suitable than classes.
>>>
>>> I would add another bonus of a flag than a function argument is to avoid
>>> massive contamination of function signatures for a global variation of
>>> behavior that affects many functions.
>>>
>>
>> I like this suggestion. Copy behavior fits very nicely with existing
>> flags (e.g. UPDATEIFCOPY, WRITEABLE) and avoids both namespace pollution
>> and complication docstrings.
>>
>> Ralf
>>
>>
>>> Yu
>>>
>>> On Wed, Jan 9, 2019 at 11:34 PM Todd <toddrjen at gmail.com> wrote:
>>>
>>>> On Mon, Jan 7, 2019, 14:22 Feng Yu <rainwoodman at gmail.com wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Was it ever brought up the possibility of a new array class
>>>>> (ndrefonly, ndview) that is strictly no copy?
>>>>>
>>>>> All operations on ndrefonly will return ndrefonly and if the operation
>>>>> cannot be completed without making a copy, it shall throw an error.
>>>>>
>>>>> On the implementation there are two choices if we use subclasses:
>>>>>
>>>>> - ndrefonly can be a subclass of ndarray. The pattern would be
>>>>> subclass limiting functionality of super, but ndrefonly is a ndarray.
>>>>> - ndarray as a subclass of ndarray. Subclass supplements functionality
>>>>> of super. : ndarray will not throw an error when a copy is necessary.
>>>>> However ndarray is not a ndarray.
>>>>>
>>>>> If we want to be wild they do not even need to be subclasses of each
>>>>> other, or maybe they shall both be subclasses of something more
>>>>> fundamental.
>>>>>
>>>>> - Yu
>>>>>
>>>>
>>>> I would prefer a flag for this.  Someone can make an array read-only by
>>>> setting `arr.flags.writable=False`.  So along those lines, we could have a
>>>> `arr.flags.copyable` flag that if set to `False` would result in an error
>>>> of any operation tried to copy the data.
>>>>
>>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
> ​
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190113/f117af8a/attachment-0001.html>