On Wed, Apr 24, 2019 at 9:56 PM Nathaniel Smith <njs@pobox.com> wrote:
When you say "numpy array specific" and
"__numpy_(nd)array_implementation__", that sounds to me like you're
trying to say "just step 3, skipping steps 1 and 2"? Step 3 is the one
that operates on ndarrays...

My thinking was that if we implement NumPy functions with duck typing (e.g., `np.stack()` in terms of  `.shape` + `np.concatenate()`), then step (3) could in some sense be the generic "array implementation", not only for NumPy arrays.
 
When we have some kind of __asduckarray__ coercion, then that will
complicate things too, because presumably we'll do something like

1. __array_function__ dispatch
2. __asduckarray__ coercion
3. __array_function__ dispatch again
4. ndarray coercion
5. [either "the implementation", or __array_function__ dispatch again,
depending on how you want to think about it]

I was thinking of something a little simpler: do __asduckarray__ rather than numpy.ndarray coercion inside the implementation of NumPy functions. Then making use of NumPy's implementations would be a matter of calling the NumPy implementation without ndarray coercion from side __array_function__.

e.g.,

class MyArray:
    def __duck_array__(self):
        return self
    def __array_function__(self, func, types, args, kwargs):
        ...
        if func in {np.stack, np.atleast_1d, ...}:
            # use NumPy's "duck typing" implementations for these functions
            return func.__duck_array_implementation__(*args, **kwargs)
        elif func == np.concatenate:
            # write my own version of np.concatenate
            ...

This would let you make use of duck typing in a controlled way if you use __array_function__. np.stack.__duck_array_implementation__ would look exactly like np.stack, except np.asanyarray() would be replaced by np.asduckarray().

The reason why we need the separate __duck_array_implementation__ and __numpy_array_implementation__/__skipping_array_function__ is because there are also use cases where you *don't* want to worry about how np.stack is implemented under the hood (i.e., in terms of np.concatenate), and want to go straight to the coercive numpy.ndarray implementation. This lets you avoid both the complexity and overhead associated with further dispatch checks.

I don't think we want repeated dispatching with __array_function__. That seems like a recipe for slow performance and confusion.