<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">On 10.09.19 05:32, Stephan Hoyer wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAEQ_Tvc_OWYUZbUXbc=tKhDtd2-S+2X=285hYGhLKjNusCv0VQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">On Mon, Sep 9, 2019 at 6:27 PM Ralf Gommers <<a
href="mailto:ralf.gommers@gmail.com" target="_blank"
moz-do-not-send="true">ralf.gommers@gmail.com</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">I think we've chosen to try the former -
dispatch on functions so we can reuse the NumPy API. It
could work out well, it could give some long-term
maintenance issues, time will tell. The question is now
if and how to plug the gap that __array_function__ left.
It's main limitation is "doesn't work for functions that
don't have an array-like input" - that left out ~10-20%
of functions. So now we have a proposal for a structural
solution to that last 10-20%. It seems logical to want
that gap plugged, rather than go back and say "we
shouldn't have gone for the first 80%, so let's go no
further".</div>
</div>
</blockquote>
<div><br>
</div>
<div>I'm excited about solving the remaining 10-20% of use
cases for flexible array dispatching, but the unumpy
interface suggested here (numpy.overridable) feels like a
redundant redo of __array_function__ and __array_ufunc__.<br>
</div>
<div><br>
</div>
<div>I would much rather continue to develop specialized
protocols for the remaining usecases. Summarizing those I've
seen in this thread, these include:</div>
<div>1. Overrides for customizing array creation and coercion.</div>
<div>2. Overrides to implement operations for new dtypes.</div>
<div>
<div>3. Overriding implementations of NumPy functions, e.g.,
FFT and ufuncs with MKL.</div>
</div>
<div>
<div><br>
</div>
<div>(1) could mostly be solved by adding np.duckarray() and
another function for duck array coercion. There is still
the matter of overriding np.zeros and the like, which
perhaps justifies another new protocol, but in my
experience the use-cases for truly an array from scratch
are quite rare.</div>
</div>
</div>
</div>
</blockquote>
<p>While they're rare for libraries like XArray; CuPy, Dask and
PyData/Sparse need these.<br>
</p>
<blockquote type="cite"
cite="mid:CAEQ_Tvc_OWYUZbUXbc=tKhDtd2-S+2X=285hYGhLKjNusCv0VQ@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div>(2) should be tackled as part of overhauling NumPy's
dtype system to better support user defined dtypes. But it
should definitely be in the form of specialized protocols,
e.g., which pass in preallocated arrays to into ufuncs for a
new dtype. By design, new dtypes should not be able to
customize the semantics of array *structure*.</div>
</div>
</div>
</blockquote>
We already have a split in the type system with e.g. Cython's
buffers, Numba's parallel type system. This is a different issue
altogether, e.g. allowing a unyt dtype to spawn a unyt array, rather
than forcing a re-write of unyt to cooperate with NumPy's new dtype
system.<br>
<blockquote type="cite"
cite="mid:CAEQ_Tvc_OWYUZbUXbc=tKhDtd2-S+2X=285hYGhLKjNusCv0VQ@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div>(3) could potentially motivate a new solution, but it
should exist *inside* of select existing NumPy
implementations, after checking for overrides with
__array_function__. If the only option NumPy provides for
overriding np.fft is to implement np.overrideable.fft, I
doubt that would suffice to convince MKL developers from
monkey patching it -- they already decided that a separate
namespace is not good enough for them.</div>
</div>
</div>
</blockquote>
<p>That has already been addressed by Ralf in another email. We're
proposing to merge that into NumPy proper.</p>
<p>Also, you're missing a few:</p>
<p>4. Having default implementations that allow overrides of a large
part of the API while defining only a small part. This holds for
e.g. transpose/concatenate.</p>
<p>5. Generation of Random numbers (overriding <tt>RandomState</tt>).
CuPy has its own implementation which would be nice to override.<br>
</p>
<blockquote type="cite"
cite="mid:CAEQ_Tvc_OWYUZbUXbc=tKhDtd2-S+2X=285hYGhLKjNusCv0VQ@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div>I also share Nathaniel's concern that the overrides in
unumpy are too powerful, by allowing for control from
arbitrary function arguments and even *non-local* control
(i.e., global variables) from context managers. This level
of flexibility can make code very hard to debug, especially
in larger codebases.</div>
</div>
</div>
</blockquote>
Backend switching needs global context, in any case. There isn't a
good way around that other than the class dundermethods outlined in
another thread, which would require rewrites of large amounts of
code.<br>
<blockquote type="cite"
cite="mid:CAEQ_Tvc_OWYUZbUXbc=tKhDtd2-S+2X=285hYGhLKjNusCv0VQ@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div>Best,</div>
<div>Stephan</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
NumPy-Discussion mailing list
<a class="moz-txt-link-abbreviated" href="mailto:NumPy-Discussion@python.org">NumPy-Discussion@python.org</a>
<a class="moz-txt-link-freetext" href="https://mail.python.org/mailman/listinfo/numpy-discussion">https://mail.python.org/mailman/listinfo/numpy-discussion</a>
</pre>
</blockquote>
<p><br>
</p>
</body>
</html>