[Numpy-discussion] NEP: array API standard adoption (NEP 47)

Mon Mar 15 16:14:00 EDT 2021

On Thu, Mar 11, 2021 at 6:08 PM Sebastian Berg <sebastian at sipsolutions.net>
wrote:

> On Thu, 2021-03-11 at 12:37 +0100, Ralf Gommers wrote:
> > On Wed, Mar 10, 2021 at 6:41 PM Sebastian Berg <
> > sebastian at sipsolutions.net>
> > wrote:
> >
> > > Top Posting, to discuss post specific questions about NEP 47 and
> > > partially the start on implementing it in:
> > >
> > >     https://github.com/numpy/numpy/pull/18585
> > >
> > > There are probably many more that will crop up. But for me, each of
> > > these is a pretty major difficulty without a clear answer as of
> > > now.
> > >
> >
> > All great questions, that Sebastian. Let me reply to the questions
> > that
> > Aaron didn't reply to inline below.
> >
>
> To be clear, I do not expect complete answers to these questions right
> now.  (Although being unsure about some of them does make me slightly
> reluctant to merge the work-in-progress into NumPy proper as opposed to
> a separate repo.)
>
> Also, yes, most/all questions are hopefully are just trivialities to
> check of (or no more than seeds for thought).  Or even just a starting
> point for making NEP 47's "Usage and Impact" section more complete
> including them as either "example usage patterns" or "limitations".
>

Yes, those are always good to have more of.

>
> My second takeaway from the questions is that I have doubts the
> "minimal" version will pan out, it feels like many of the questions
> might disappear if you drop that part.

My impression is that a strictly compliant (or "minimal") version is *more*
useful than something that's a mix between portable and non-portable
functionality. The reason to add more than the minimum required
functionality would be that it's too hard to hide the numpy-specific
extras. E.g., if we'd do `np.array_api.int32 = np.int32` then that dtype
would have methods and behavior that's NumPy-specific. But it'd be hard to
hide, so we'd accept it.

It's maybe easier to discuss in a call, I've put it on the community
meeting agenda.

> So, from my current thinking,
> the minimal implementation may not be a good "NEP 47" implementation.
>
> That does _not_ mean that I think you should pause and reconsider or
> even worry about pleasing me with good answers!  Just continue under
> whatever assumption you prefer and if it turns out that "minimal" won't
> work for NEP 47: no harm done!  We need a "minimal implementation" in
> any case.
>

Yes, I agree.

> Cheers,
>
> Sebastian
>
>
>
> [1] If SciPy needs an additional NumPy code path to keep support
> `object` arrays or other dtypes – right now even complex –, then the
> reader needs to be aware of that to make a decision if NEP 47 will
> actually help for their library.
>

Clearly. This is why we'd like to have some WIP PRs for other libraries,
actual code to review will be more helpful than only a proposal.

> Will AstroPy have to reimplement `astropy.units.Quantity` to be
> "standard conform" (is that even possible!?) before it can easily adopt
> it any of its API that currently works with `astropy.units.Quantity`?
>

I'm not sure if the question is well-defined, so let me answer both cases:

1. If the APIs in question require units, then there's no other
array/tensor types that have unit support, so those APIs accept *only*
Quantity. Adopting the standard isn't possible.

2. If the units are unnecessary/optional, then Quantity is not special and
can be treated exactly the same as a `numpy.ndarray`. We don't intend to
make any changes to how ndarray subclasses work, so if ndarray works with
that API after adoption of the standard then Quantity works too.

Cheers,
Ralf

>
> >
> > > 1. I still need clarity how a library is supposed to use this
> > > namespace
> > > when the user passes in a NumPy array (mentioned before).  The user
> > > must get back a NumPy array after all.  Maybe that is just a
> > > decorator,
> > > but it seems important.
> > >
> >
> > I agree that it will be a common pattern that libraries will accept
> > all
> > standard-compliant array types plus numpy.ndarray. And the output
> > array
> > type should match the input type. In Aaron's implementation the new
> > array
> > object has a numpy.ndarray as private attribute, so that's the
> > instance
> > that should be returned. A decorator seems like a sensible way to
> > handle
> > that. Or a simple utility function, something like `return
> > correct_arraytype(out)`.
> >
> > Either way, that pattern should be added to NEP 47. I don't see a
> > fundamental problem here, we just need to find the nicest UX for it.
> >
> > 3. For all other function, the same problem applies. You don't
> > actually
> > > have anything to fix NumPy promotion rules.  You could bake your
> > > own
> > > cake here for numeric types, but I am not sure, you might also need
> > > NEP
> > > 43 in all its promotion power to pull it off.
> > >
> >
> > This is probably the single most difficult question implementation-
> > wise.
> > Note that there are only numerical dtypes (plus boolean), so dealing
> > with
> > string, datetime, object or third-party dtypes is a non-issue.
> >
> > 4. The PR makes no attempt at handling binary operators in any way
> > > aside from greedily coercing the other operand.
> > >
> >
> > Agreed. This is the same point as (3) I think - how to handle dtype
> > promotion is the main open question.
> >
> >
> > > 5. What happens with a mix of array-likes or even array subclasses
> > > like
> > > `astropy.quantity`?
> > >
> >
> > Array-likes (e.g. list) should raise an exception, the NEP clearly
> > says "do
> > not accept array_like dtypes". This is what every other array/tensor
> > library already does.
> >
> > Array subclasses should work as expected, assuming they're valid
> > subclasses
> > and not things like np.matrix. Using Mypy will help avoid writing
> > more
> > subclasses that break the Liskov substitution principle. More
> > comments in
> >
> https://numpy.org/neps/nep-0047-array-api-standard.html#the-asarray-asanyarray-pattern
> >
> > Mixing two different types of arrays into a single function call
> > should
> > raise an exception. A design goal is: enable writing functions
> > `somefunc(x1, x2)` that work for any type of array where `x1, x2`
> > come from
> > the same library = so they're either the same type, or two types for
> > which
> > the library itself knows how to mix them. If x1 and x2 are from
> > different
> > libraries, this will raise an exception.
> >
> > To be clear, it is not intended that `np.array_api.somefunc(x_cupy)`
> > works
> > - this will raise an exception.
> >
> > Cheers,
> > Ralf
> >
> >
> >
> > >
> > > I don't think we have to figure out everything up-front, but I do
> > > think
> > > there are a few very fundamental questions still open, at least for
> > > me
> > > personally.
> > >
> > > Cheers,
> > >
> > > Sebastian
> > >
> > >
> > >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/numpy-discussion/attachments/20210315/936ae9ae/attachment.html>