[Numpy-discussion] Rules for argument parsing/forwarding in __array_function__ and __array_ufunc__

[Sebastian Berg]

On Sat, 2020-12-05 at 22:05 -0800, Stephan Hoyer wrote:
> On Wed, Dec 2, 2020 at 3:39 PM Sebastian Berg <
> sebastian at sipsolutions.net>
> wrote:
> 
> > 1. If an argument is invalid in NumPy it is considered and error.
> >    For example:
> > 
> >        np.log(arr, my_weird_argument=True)
> > 
> >    is always an error even if the `__array_function__`
> > implementation
> >    of `arr` would support it.
> >    NEP 18 explicitly says that allowing forwarding could be done,
> > but
> >    will not be done at this time.
> > 
> 
> From my perspective, this is a good thing: it ensures that NumPy's
> API is
> only used for features that exist in NumPy. Otherwise I can imagine
> causing
> considerable confusion.
> 
> If you want to use my_weird_argument, you can call
> my_weird_library.log()
> instead.
> 
> 
> > 2. Arguments must only be forwarded if they are passed in:
> > 
> >        np.mean(cupy_array)
> > 
> >    ends up as `cupy.mean(cupy_array)` and not:
> > 
> >        cupy.mean(cupy_array, axis=None, dtype=None, out=None,
> >                  keepdims=False, where=True)
> > 
> >    meaning that CuPy does not need to implement all of those kwargs
> > and
> >    NumPy can add new ones without breaking anyones code.
> > 
> 
> My reasoning here was two-fold:
> 1. To avoid the unfortunate situation for functions like np.mean(),
> where
> NumPy jumps through considerable hoops to avoid passing extra
> arguments in
> an ad-hoc way to preserve backwards compatibility
> 2. To make it easy for a library to implement "incomplete" versions
> of
> NumPy's API, by simply omitting arguments.
> 
> The idea was that NumPy's API is open to partial implementations, but
> not
> extension.
> 

Indeed, changing this allows inlining in Python easier (because you
don't need to use `**kwargs` to be able to know which arguments were
not passed).
I guess the alternative is to force everyone to keep up, but you are of
course allowed to raise a NotImplementedError (which is actually nicer
for users, probably).

> 
> > 3. NumPy should not check the *validity* of the arguments. For
> > example:
> >    `np.add.reduce(xarray, axis="long")` should probably work in
> > xarray.
> >    (`xarray.DataArray` does not actually implement the above.)
> >    But a string cannot be used as an axis in NumPy.
> > 
> 
> I don't think libraries should be encouraged to abuse NumPy's API to
> mean
> something else. Certainly I would not use this in xarray :).
> 
> If we could check the validity of arguments cheaply, that would be
> fine by
> me. But I didn't think it was worth adding overhead to every function
> call.
> Perhaps type annotations could be relied on for these sorts of
> checks? I am
> pretty happy considering not checking the validity of arguments to be
> an
> implementation detail for now.

The current implementation of __array_function__ makes this hard,
because it assumes the call graph is:

    array_funciton_implementation(...):
        impl = find_implementation()
        # impl may be the default
        return impl()

Unlike for __array_ufunc__ where it is:

    ufunc.__call__(*args, **kwargs):
        if needs_to_defer(args, kwargs):
            return defered_result()

If you assume that NumPy is the main consumer, especially on the C-
side, validating the arguments (e.g. integer axis, NumPy dtypes) can
make things more comfortable.

"inlining" the dispatching as the second case, makes things quite a bit
more comfortable, but is not necessary.  However, it requires a small
change to the default __array_function__ (i.e. you have to change the
meaning to "defaul __array_function__" is the same as no array
function.)

Cheers,

Sebastian


> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <https://mail.python.org/pipermail/numpy-discussion/attachments/20201206/ef50ee2b/attachment.sig>