Proposal to accept NEP-18, __array_function__ protocol
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level array functions": http://www.numpy.org/neps/nep-0018-array-function-protocol.html Since the last round of discussion, we added a new section on "Callable objects generated at runtime" clarifying that to handle such objects is out of scope for the initial proposal in the NEP. If there are no substantive objections within 7 days from this email, then the NEP will be accepted; see NEP 0 for more details. Cheers, Stpehan
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
I'm sorry, I've been trying to find the time to read what you ended up with, and haven't managed yet – could I get a few days extension? :-) -n On Wed, Aug 1, 2018 at 5:27 PM, Stephan Hoyer <shoyer@gmail.com> wrote:
-- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
Hey all, So I've finally read through NEP 18 (__array_function__). Sorry again for the delay! It's an impressive piece of work! Thanks to the many authors; there's clearly been a lot of thought put into this. # The trade-off between comprehensive APIs versus clean APIs At a high-level, what makes me nervous about this proposal is that it reminds me of a classic software design pattern that... I don't know a name for. You might call it the "structured monkeypatching" approach to extensibility. The pattern is: a project decides they want to allow some kind of extensions or addons or plugins or something, but defining a big structured API for this is too difficult. So they take their current API surface area and declare that that's the plugin API (plus some mechanism for plugins to hook in etc.). Is this pattern good? It's... hard to say. What generally happens is: 1. You get a very complete, powerful, flexible plugin API with minimal work. 2. This quickly leads to a rich system of powerful plugins, which drives quick uptake of the project, sometimes even driving out competitors. 3. The maintainers slowly realize that committing to such a large and unstructured API is horribly unwieldy and makes changes difficult. 4. The maintainers spend huge amounts of effort trying to crawl out from under the weight of their commitments, which mixed success. Examples: pytest, sphinx: For both of these projects, writing plugins is a miserable experience, and you never really know if they'll work with new releases or when composed with random other plugins. Both projects are absolutely the dominant players in their niche, far better than the competition, largely thanks to their rich plugin ecosystems. CPython: the C extension API is basically just... all of CPython's internals dumped into a header file. Without this numpy wouldn't exist. A key ingredient in Python's miraculous popularity. Also, at this point, possibly the largest millstone preventing further improvements in Python – this is why we can't have multicore support, JITs, etc. etc.; all the most ambitious discussions at the Python language summit the last few years have circled back to "...but we can't do that b/c it will break the C API". See also: https://mail.python.org/pipermail/python-dev/2018-July/154814.html Firefox: their original extension API was basically just "our UI is written in javascript, extension modules get to throw more javascript in the pot". One of Firefox's original USPs, and a key part of like... how Mozilla even exists instead of having gone out of business a decade ago. Eventually the extension API started blocking critical architectural changes (e.g. for better sandboxing), and they had to go through an *immensely* painful migration to a properly designed API, which tooks years and burned huge amounts of goodwill. So this is like... an extreme version of technical debt. You're making a deal with the devil for wealth and fame, and then eventually the bill becomes due. It's hard for me to say categorically that this is a bad idea – empirically, it can be very successful! But there are real trade-offs. And it makes me a bit nervous that Matt is the one proposing this, because I'm pretty sure if you asked him he'd say he's absolutely focused on how to get something working ASAP and has no plans to maintain numpy in the future. The other approach would be to incrementally add clean, well-defined dunder methods like __array_ufunc__, __array_concatenate__, etc. This way we end up putting some thought into each interface, making sure that it's something we can support, protecting downstream libraries from unnecessary complexity (e.g. they can implement __array_concatenate__ instead of hstack, vstack, row_stack, column_stack, ...), or avoiding adding new APIs entirely (e.g., by converting existing functions into ufuncs so __array_ufunc__ starts automagically working). And in the end we get a clean list of dunder methods that new array container implementations have to define. It's plausible to imagine a generic test suite for array containers. (I suspect that every library that tries to implement __array_function__ will end up with accidental behavioral differences, just because the numpy API is so vast and contains so many corner cases.) So the clean-well-defined-dunders approach has lots of upsides. The big downside is that this is a much longer road to go down. I am genuinely uncertain which of these approaches is better on net, or whether we should do both. But because I'm uncertain, I'm nervous about committing to the NEP 18 approach -- it feels risky. ## Can we mitigate that risk? One thing that helps is the way the proposal makes it all-or-nothing: if you have an __array_function__ method, then you are committing to reimplementing *all* the numpy API (or at least all the parts that you want to work at all). This is arguably a bad thing in the long run, because only large and well-resourced projects can realistically hope to implement __array_function__. But for now it does somewhat mitigate the risks, because the fewer users we have the easier it is to work with them to change course later. But that's probably not enough -- "don't worry, if we change it we'll only break large, important projects with lots of users" isn't actually *that* reassuring :-). The proposal also bills itself as an unstable, provisional experiment ("this protocol should be considered strictly experimental. We reserve the right to change the details of this protocol and how specific NumPy functions use it at any time in the future – even in otherwise bug-fix only releases of NumPy."). This mitigates a lot of risk! If we aren't committing to anything, then sure, why not experiment. But... this is wishful thinking. No matter what the NEP says, I simply don't believe that we'll actually go break dask, sparse arrays, xarray, and sklearn in a numpy point release. Or any numpy release. Nor should we. If we're serious about keeping this experimental – and I think that's an excellent idea for now! – then IMO we need to do something more to avoid getting trapped by backwards compatibility. My suggestion: at numpy import time, check for an envvar, like say NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1. If it's not set, then all the __array_function__ dispatches turn into no-ops. This lets interested downstream libraries and users try this out, but makes sure that we won't have a hundred thousand end users depending on it without realizing. Other advantages: - makes it easy for end-users to check how much overhead this adds (by running their code with it enabled vs disabled) - if/when we decide to commit to supporting it for real, we just remove the envvar. With this change, I'm overall +1 on the proposal. Without it, I... would like more convincing, at least :-). # Minor quibbles I don't really understand the 'types' frozenset. The NEP says "it will be used by most __array_function__ methods, which otherwise would need to extract this information themselves"... but they still need to extract the information themselves, because they still have to examine each object and figure out what type it is. And, simply creating a frozenset costs ~0.2 µs on my laptop, which is overhead that we can't possibly optimize later... -n On Wed, Aug 1, 2018 at 5:27 PM, Stephan Hoyer <shoyer@gmail.com> wrote:
-- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
Hi Nathaniel, Very well written summary, it provides a lot of perspective into the different ways that this could go wrong. Here is a little commentary.
Ah, yes. I’ve been an affectee of this in both instances. For example, there’s a bug that with doctests enabled, you can’t select tests to run, and with coverage enabled, the pydev debugger stops working. However, composition (at least) is better handled with this protocol. This is answered in more detail later on.
Ah, yes. I remember heated debates about this. A lot of good add-ons (as Mozilla calls them) were lost because of alienated developers or missing APIs.
Yes, this is the way I’d prefer to go as well, but the machinery required for converting something to a ufunc is rather complex. Take, for example, the three-argument np.where. In order to preserve full backward compatibility, we need to support structured arrays and strings… which is rather hard to do. Same with the np.identity ufunc that was proposed for casting to a given dtype. This isn’t necessarily an argument against using ufuncs, I’d actually take it as an argument for better documenting these sorts of things. In fact, I wanted to do both of these myself at one point, but I found the documentation for writing ufuncs insufficient for handling these corner cases.
It all comes down to how stable we consider the NumPy API to be. And I’d say pretty stable. Once a function is overridden, we generally do not let NumPy handle it at all. We generally tend to preserve backward compatibility in the API, and the API is all we’re exposing.
There was a proposal that we elected to leave out — Maybe this is a reason to put it back in. The proposal was to have a sentinel (np.NotImplementedButCoercible) that would take the regular route via coercion, with NotImplemented raising a TypeError. This way, someone could do these things incrementally. Also, another thing we could do (as Stephan Hoyer, I, Travis Oliphant, Tyler Reddy, Saul Shanabrook and a few others discussed in the SciPy 2018 sprints) is to go through the NumPy API, convert as much of them into ufuncs as possible, and identify a “base set” from the rest. Then duck array implementations would only need to implement this “base set” of operations along with __array_ufunc__. My idea is that we get started on identifying these things as soon as possible, and only allow this “base set” under __array_function__, and the rest should simply use these to create all NumPy functionality. I might take on identifying this “base set” of functionality in early September if enough people are interested. I might even go as far as to say that we shouldn’t allow any function under this protocol unless it’s in the "base set”, and rewrite the rest of NumPy to use the “base set”. It’s a big project for sure, but identifying the “base set”/ufunc-able functions shouldn’t take too long. The rewriting part might. The downside would be that some things (such as np.stack) could have better implementations if a custom one was allowed, rather than for example, error checking + concatenate + reshape or error-checking + introduce extra dimension + concatenate. I’m willing to live with this downside, personally, in favour of a cleaner API.
We also have to consider that this might hinder adoption. But I’m fine with that. Properly > Quickly, as long as it doesn’t take too long. I’m +0 on this until we properly hammer out this stuff, then we remove it and make this the default. However, I also realise that pydata/sparse is in the early stages, and can probably wait. Other duck array implementations such as Dask and XArray might need this soon-ish.
The rationale here is that most implementations would check if the types in the array are actually supported by their implementation. If not, they’d return NotImplemented. If it wasn’t done here, every input would need to do it individually, and this may take a lot of time. I do agree that it violates DRY a bit though… The types are already present in the passed-in arguments, and this can be inferred from those.
Hope that clarifies things! Best regards, Hameer Abbasi
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Mon, Aug 13, 2018 at 2:44 AM, Nathaniel Smith <njs@pobox.com> wrote:
Rereading this today I realized that it could come across like I have an issue with Matt specifically. I apologize to anyone who got that impression (esp. Matt!) -- that definitely wasn't my intent. Matt is awesome. I should stop writing these things at 2 am. What I should have said is: We have an unusual decision to make here, where there are two plausible approaches that both have significant upsides and downsides, and whose effects are going to be distributed in a complicated way across different parts of our community over time. So the big challenge is to figure out how to take all that into account and weigh the needs of different stakeholders against each other. One major argument for the __array_function__ approach is that it has an actual NEP, which happened because we have a contributor who took the lead on making it happen, and who's deeply involved in some of the target projects like dask and sparse, so can make sure that the proposal will work well for them. That's a huge advantage! But... it also makes me a *little* nervous, because when you have really talented and productive contributors like this it's easy to get swept up in their perspective. So I want to double-check that we're also thinking about the stakeholders who can't be as active in the discussion, like "numpy maintainers from the future". (And I mostly mean this as a "we should keep this in mind" kind of thing – like I said in my original post, I think moving forward implementing __array_function__ is a great idea; I just want to be cautious about getting more experience before committing.) -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/8749ec52cee260c4c1f67f2dec29d957.jpg?s=120&d=mm&r=g)
Hi Nathaniel, I appreciate the clarification. Thank you for that. For what it's worth, I think that you may overestimate my involvement in the writing of that NEP. I sat down with Stephan during a Numpy dev meeting and we hacked something together. Afterwards several other people poured their thoughts into the process. I'd like to think that my perspective helped to inform this NEP, but it wasn't, by far, the driving force. If anyone had a strongest hand in the writing process it would probably be Stephan, who I find generally has a more conservative and careful perspective than I do. That being said, I do think that Numpy would be wise to move quickly here. I think that the growing fragmentation that we see in array computing in Numpy (Tensorflow, Torch, Dask, Sparse, CuPy) is largely due to Numpy moving slowly in the past. There is, I think, a systemic problem slowly erupting now that I think the community to respond to quickly if it is possible to do so safely. I believe that Numpy should absolutely be willing to try something experimental, and then say "nope, that was a bad idea" and retract it if it doesn't work out well. I think that figuring out all of __array_concatenate__, __array_stack__, __array_foo__, etc. for each of the many cases will take too long to respond to in an effective timeframe. I believe that we simply don't move quickly enough that this piece-by-piece careful handling of the API will result in Numpy's API becoming a meaningful standard in the broader community in the near-future. That being said, I think that we *should* engage in this piece-by-piece discussion, and as we figure them out we should slowly encroach on __array_function__ and remove functionality from it, much as __array_ufunc__ is not included in it in the current NEP. Ideally we should get to exactly where you want to get to. I perceive the __array_function__ protocol as a sort of necessary stop-gap. All that being said, this is just my personal stance. I suspect that each of the authors of the NEP and others who engaged in its careful review have a different perspective, which should probably carry more weight than my own. Best, -matt On Mon, Aug 13, 2018 at 4:29 PM Nathaniel Smith <njs@pobox.com> wrote:
![](https://secure.gravatar.com/avatar/b4929294417e9ac44c17967baae75a36.jpg?s=120&d=mm&r=g)
Hi, Thanks Nathaniel for this thoughtful response. On Mon, Aug 13, 2018 at 10:44 AM, Nathaniel Smith <njs@pobox.com> wrote: ...
Does everyone agree that, if we had infinite time and resources, this would be the better solution? If we devoted all the resources of the current Numpy grant to taking this track, could we complete it in a reasonable time? Cheers, Matthew
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
More resources means (given NumPy’s consensus system), more people have to agree on the overall design, so in my mind, it might even be slower.
If we devoted all the resources of the current Numpy grant to taking this track, could we complete it in a reasonable time?
I somehow think just the design of all these different protocols, heck, even ironing all these different protocols out and ignoring implementation; would take an unreasonably long amount of time, as evidenced by this one NEP. I’m more in favour of using this one rather conservatively: Perhaps a mailing list consensus before actually adding a function to __array_function__, making sure it won’t hinder too much progress. I also differ with Nathaniel on one minor thing with his comparisons to Firefox, CPython, pytest and Sphinx: We’re not talking about monkey-patching NumPy internals, we’re just talking about monkey-patching the public API. Of course, this is still a cost and can still hinder development, but it’s definitely better than exposing all internals.
Best Regards, Hameer Abbasi
![](https://secure.gravatar.com/avatar/b4929294417e9ac44c17967baae75a36.jpg?s=120&d=mm&r=g)
Hi, On Wed, Aug 15, 2018 at 5:36 PM, Hameer Abbasi <einstein.edison@gmail.com> wrote:
I don't think that's likely. As far as I can see, past discussions have been slow because several people need to get deep down into the details in order to understand the problem, and then stay focused through the discussion. When there is no-one working on that full-time, it's easy for the discussion to drift into the background, and the shared understanding is lost. My suspicion is, to the extent that Matti and Tyler can devote time and energy to shepherding the discussion, these will become quicker and more productive. Cheers, Matthew
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
On Wed, Aug 15, 2018 at 10:25 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
If we devoted all the resources of the current Numpy grant to taking this track, could we complete it in a reasonable time?
I think it is further down the road than that. Determining a core set of functions would depend on feedback (need), as well be dependent on implementation details for other functions. I think a dependency list for common NumPy functions might be interesting. That said, I don't think the current proposal is orthogonal to this. I don't expect every NumPy function to make call through this API, and we should probably try to limit, and list, the functions that implement the proposed mechanism. Chuck
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
Nathaniel, Thanks for raising these thoughtful concerns. Your independent review of this proposal is greatly appreciated! See my responses inline below: On Mon, Aug 13, 2018 at 2:44 AM Nathaniel Smith <njs@pobox.com> wrote:
RE: accidental differences in behavior: I actually think that the __array_function__ approach is *less* prone to accidental differences in behavior, because we require implementing every function directly (or it raises an error). This avoids a classic subclassing problem that has plagued NumPy for years, where overriding the behavior of method A causes apparently unrelated method B to break, because it relied on method A internally. In NumPy, this constrained our implementation of np.median(), because it needed to call np.mean() in order for subclasses implementing units to work properly. There will certainly be accidental differences in behavior for third-party code that *uses* NumPy, but this is basically inevitable for any proposal to allow's NumPy's public API to be overloaded. It's also avoided by default by third-party libraries that follow the current best practice of casting all input arrays with np.asarray(). -------------- RE: a hypothetical simplified interface: The need to implement everything you want to use in NumPy's public API could certainly be onerous, but on the other hand there are a long list of projects that have already done this today -- and these are the projects that most need __array_function__. I'm sure there are cases were simplification would be warranted, but in particular I don't think __array_concatenate__ has significant advantages over simply implementing __array_function__ for np.concatenate. It's a slightly different way of spelling, but it basically does the same thing. The level of complexity to implement hstack, vstack, row_stack and column_stack in terms of np.concatenate is pretty minimal. __array_function__ implementors could easily copy and paste code from NumPy or use a third-party helpers library (like NDArrayOperatorsMixin) that provides such implementations. I also have other concerns about the "simplified API" approach beyond the difficulty of figuring it out, but those are already mentioned in the NEP: http://www.numpy.org/neps/nep-0018-array-function-protocol.html#implementati... But... this is wishful thinking. No matter what the NEP says, I simply
I agree, but to be clear, development for dask, sparse and xarray (and even broadly supported machine learning libraries like TensorFlow) still happens at a much faster pace than is currently the case for "core" projects in the SciPy stack like NumPy. It would not be a big deal to encounter breaking changes in a "major" NumPy release (i.e., 1.X -> 1.(X+1)). (Side note: sklearn doesn't directly implement any array types, so I don't think it would make use of __array_function__ in any way, except possibly to implement overloadable functions.)
- makes it easy for end-users to check how much overhead this adds (by
I'm slightly concerned that the cost of reading an environment variable with os.environ could exaggerate the performance cost of __array_function__. It takes about 1 microsecond to read an environment variable on my laptop, which is comparable to the full overhead of __array_function__. So we may want to switch to an explicit Python API instead, e.g., np.enable_experimental_array_function(). My bigger concern is when/how we decide to graduate __array_function__ from requiring an explicit opt-in. We don't need to make a final decision now, but it would be good to clear about what specifically we are waiting for. I see three types of likely scenarios for changing __array_function__: 1. We decide that the overloading the NumPy namespace in general is a bad idea, based on either performance or predictability consequences for third-party libraries. In this case, I imagine we would probably keep __array_function__, but revert to a separate namespace for explicitly overloaded functions, e.g., numpy.api. 2. We want to keep __array_function__, but need a breaking change to the interface (and we're really attached to keeping the name __array_function__). 3. We decide that specific functions should use a different interface (e.g., switch from __array_function__ to __array_ufunc__). (1) and (2) are the sort of major concerns that in my mind would warrant hiding a feature behind an experimental flag. For the most part, I expect (1) could be resolved relatively quickly by running benchmark suites after we have a working version of __array_function__. To be honest, I don't see either of these rollback scenarios as terribly likely, but the downside risk is large enough that we may want to protect ourselves for a major release or two (6-12 months). (3) will be a much longer process, likely to stretch out over years at the current pace of NumPy development. I don't think we'll want to keep an opt-in flag for this long of a period. Rather, we may want to accept a shorter deprecation cycle than usual. In most cases, I suspect we could incrementally switch to new overloads while preserving the __array_function__ overload for a release or two. I don't really understand the 'types' frozenset. The NEP says "it will
The most flexible alternative would be to just say that we provide an fixed-length iterable, and return a tuple object. (In my microbenchmarks, it's faster to make a tuple than a list or set.) In an early draft of the NEP, I proposed exactly this, but the speed difference seemed really marginal to me. I included 'types' in the interface because I really do think it's something that almost all __array_function__ implementations should use use. It preserves a nice separation of concerns between dispatching logic and implementations for a new type. At least as long as __array_function__ is experimental, I don't think we should be encouraging people to write functions that could return NotImplemented directly and to rely entirely on the NumPy interface. Many but not all implementations will need to look at argument types. This is only really essential for cases where mixed operations between NumPy arrays and another type are allowed. If you only implement the NumPy interface for MyArray objects, then in the usual Python style you wouldn't need isinstance checks. It's also important from an ecosystem perspective. If we don't make it easy to get type information, my guess is that many __array_function__ authors wouldn't bother to return NotImplemented for unexpected types, which means that __array_function__ will break in weird ways when used with objects from unrelated libraries. Cheers, Stephan
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
Ping to finish up this discussion so we can come to a conclusion. I'm in favor of the NEP, as I don't see it as orthogonal to Nathaniel's concerns. However, we might want to be selective as to which functions we expose via the `__array_function__` method. On Wed, Aug 15, 2018 at 10:45 AM, Stephan Hoyer <shoyer@gmail.com> wrote:
Here is Travis Oliphant's talk at PyBay <https://speakerdeck.com/teoliphant/ml-in-python>, where he talks about the proliferation of arrays and interfaces in the ML/AI ecosystem among other things. I think that we should definitely try to get NumPy out there as an option in the near future.
Chuck
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Mon, Aug 20, 2018 at 8:27 AM Charles R Harris <charlesr.harris@gmail.com> wrote:
Chunk -- thanks for bringing this back up. My proposal is to hide this feature behind an environment variable for now, or something morally equivalent to an environment variable if that's too slow (i.e., an explicit Python variable). I don't think Nathaniel's concerns are entirely warranted for the reasons I went into in my earlier reply, but I do really want to get this moving forward now in whatever way is necessary. We can figure out the rest down the road. Nathaniel -- are you OK with that?
Yes, there is an urgent need for this :). Cheers, Stephan
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Wed, Aug 15, 2018 at 9:45 AM, Stephan Hoyer <shoyer@gmail.com> wrote:
I don't think I follow... if B uses A internally, then overriding A shouldn't cause B to break, unless the overridden A is buggy. The median() case was different: it wasn't overriding A that caused B to break, that part worked fine. It was when we changed the implementation of B that we had problems. ...actually, this made me realize that I was uncertain about what exactly happened in that case. I just checked, and AFAICT with current astropy the call to mean() is unnecessary. I tried modifying np.median to remove the call to mean, and it still gave the same result for np.median([1, 2, 3] * u.m). I also checked np.percentile, and it seems to work fine on units-arrays if you make it call np.asanyarray instead of np.asarray. The issue is here if anyone else wants to refresh their memory: https://github.com/numpy/numpy/issues/3846 Reading the comments there it looks like we might have hacked np.median so it (a) uses np.asanyarray, and (b) always calls np.mean, and actually the 'asanyarray' change is what astropy needed and the 'mean' part was just a red herring all along! Whoops. And here I've been telling this story for years now...
And when we fix a bug in row_stack, this means we also have to fix it in all the copy-paste versions, which won't happen, so np.row_stack has different semantics on different objects, even if they started out matching. The NDArrayOperatorsMixin reduces the number of duplicate copies of the same code that need to be updated, but 2 copies is still a lot worse than 1 copy :-).
Yeah, there are definitely trade-offs. I don't have like, knock-down rebuttals to these or anything, but since I didn't comment on them before I might as well say a few words :-).
1. The details of how NumPy implements a high-level function in terms of overloaded functions now becomes an implicit part of NumPy’s public API. For example, refactoring stack to use np.block() instead of np.concatenate() internally would now become a breaking change.
The way I'm imagining this would work is, we guarantee not to take a function that used to be implemented in terms of overridable operations, and refactor it so it's implemented in terms of overridable operations. So long as people have correct implementations of __array_concatenate__ and __array_block__, they shouldn't care which one we use. In the interim period where we have __array_concatenate__ but there's no such thing as __array_block__, then that refactoring would indeed break things, so we shouldn't do that :-). But we could fix that by adding __array_block__.
2. Array libraries may prefer to implement high level functions differently than NumPy. For example, a library might prefer to implement a fundamental operations like mean() directly rather than relying on sum() followed by division. More generally, it’s not clear yet what exactly qualifies as core functionality, and figuring this out could be a large project.
True. And this is a very general problem... for example, the appropriate way to implement logistic regression is very different in-core versus out-of-core. You're never going to be able to take code written for ndarray, drop in an arbitrary new array object, and get optimal results in all cases -- that's just way too ambitious to hope for. There will be cases where reducing to operations like sum() and division is fine. There will be cases where you have a high-level operation like logistic regression, where reducing to sum() and division doesn't work, but reducing to slightly-higher-level operations like np.mean also doesn't work, because you need to redo the whole high-level operation. And then there will be cases where sum() and division are too low-level, but mean() is high-level enough to make the critical difference. It's that last one where it's important to be able to override mean() directly. Are there a lot of cases like this? For mean() in particular I doubt it. But then, mean() in particular is irrelevant here, because mean() is already directly overridable, regardless of __array_function__ :-). So really the question is about the larger landscape of numpy APIs: What traps are lurking in the underbrush that we don't know about? And yeah, the intuition behind the "simplified API" approach is that we need to do the work to clear out that underbrush, and the downside is exactly that that will be a lot of work and take time. So... I think this criticism is basically that restated?
3. We don’t yet have an overloading system for attributes and methods on array objects, e.g., for accessing .dtype and .shape. This should be the subject of a future NEP, but until then we should be reluctant to rely on these properties.
This one I don't understand. If you have a duck-array object, and you want to access its .dtype or .shape attributes, you just... write myobj.dtype or myobj.shape? That doesn't need a NEP though so I must be missing something :-).
They don't implement array types, but they do things like use sparse arrays internally, so from the user's point of view you could have some code that only uses numpy and sklearn, and then the new numpy release breaks sklearn (because it broke the sparse package that sklearn was using internally).
That's why I said "at numpy import time" :-). I was imagining we'd check it once at import, and then from then on it'd be stashed in some C global, so after that the overhead would just be a single predictable branch 'if (array_function_is_enabled) { ... }'.
If we do this, then libraries that want to use __array_function__ will just call it themselves at import time. The point of the env-var is that our policy is not to break end-users, so if we want an API to be provisional and experimental then it's end-users who need to be aware of that before using it. (This is also an advantage of checking the envvar only at import time: it means libraries can't easily just setenv() to enable the functionality behind users' backs.)
The motivation for keeping it provisional is that we'll know more after we have some implementation experience, so our future selves will be in a better position to make this decision. If I knew what I was waiting for, I might not need to wait :-). But yeah, to be clear, I'm totally OK with the possibility that we'll do this for a few releases and then look again and be like "eh... now that we have more experience, it looks like the original plan was fine after all, let's remove the envvar and document some kind of accelerated deprecation cycle".
This is much more of a detail as compared to the rest of the discussion, so I don't want to quibble too much about it. (Especially since if we keep things really-provisional, we can change our mind about the argument later :-).) Mostly I'm just confused, because there are lots of __dunder__ functions in Python (and NumPy), and none of them take a special 'types' argument... so what's special about __array_function__ that makes it necessary/worthwhile? Any implementation of, say, concatenate-via-array_function is going to involve iterating through all the arguments and looking at each of them to figure out what kind of object it is and how to handle it, right? That's true whether or not they've done a "pre-check" using the types set, so in theory it's just as easy to return NotImplemented at that point. But I guess your point in the last paragraph is that this means there will be lots of chances to mess up the NotImplemented-returning code in particular, especially since it's less likely to be tested than the happy path, which seems plausible. So basically the point of the types set is to let people factor out that little bit of lots of functions into one common place? I guess some careful devs might be unhappy with paying extra so that other lazier devs can get away with being lazy, but maybe it's a good tradeoff for us (esp. since as numpy devs, we'll be getting the bug reports regardless :-)). If that's the goal, then it does make me wonder if there might be a more direct way to accomplish it -- like, should we let classes define an __array_function_types__ attribute that numpy would check before even trying to dispatch to __array_function__? -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
I’m +0 on removing it, so mostly neutral, but slightly in favour. While I see the argument for having it, I also see it as a violation of DRY... The information is already available in relevant arguments. I doubt any people implementing this protocol are going to be lazy enough not to implement a type check. So far, we’ve been good on __array_ufunc__. The part of me that says it’s good to have it is the mostly “squeeze every bit of performance out” part. Best Regards Hameer Abbasi Sent from my iPhone
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith <njs@pobox.com> wrote:
Let me try another example with arrays with units. My understanding of the contract provided by unit implementations is their behavior should never deviate from NumPy unless an operation raises an error. (This is more explicit for arrays with units because they raise errors for operations with incompatible units, but practically speaking almost all duck arrays will have at least some unsupported operations in NumPy's giant API.) It is quite possible that NumPy functions could be (re)written in a way that is incompatible with some unit implementations but is perfectly valid for "full" duck arrays. We actually see this even within NumPy already -- for example, see this recent PR adding support for the datetime64 dtype to percentile: https://github.com/numpy/numpy/pull/11627 A lesser case of this are changes in NumPy causing performance issues for users of duck arrays, which is basically inevitable if we share implementations. I don't think it's possible to anticipate all of these cases, and I don't want NumPy to be unduly constrained in its internal design. I want our user support answer to be simple: if you care about performance for a particular array operations on your type of arrays, you should implement it yourself (i.e., with __array_function__). This definitely doesn't preclude the careful, systematic overriding approach. But I think we'll almost always want NumPy's external API to be overridable. And when we fix a bug in row_stack, this means we also have to fix it
I see your point, but in all seriousness if encounter a bug in np.row_stack at this point we might just call it a feature instead.
""we guarantee not to take a function that used to be implemented in terms of overridable operations, and refactor it so it's implemented in terms of overridable operations" Did you miss a "not" in here somewhere, e.g., "refactor it so it's NOT implemented"? If we ever tried to do something like this, I'm pretty sure that it just wouldn't happen -- unless we also change NumPy's extremely conservative approach to breaking third-party code. np.block() is much more complex to implement than np.concatenate(), and users would resist being forced to handle that complexity if they don't need it. (Example: TensorFlow has a concatenate function, but not block.)
mean() is not entirely hypothetical. TensorFlow and Eigen actually do implement mean separately from sum, though to be honest it's not entirely clear to me why: https://github.com/tensorflow/tensorflow/blob/1c1dad105a57bb13711492a8ba5ab9... https://eigen.tuxfamily.org/dox/unsupported/TensorFunctors_8h_source.html I do think this probably will come up with some frequency for other operations, but the bigger answer here really is consistency -- it allows projects and their users to have very clearly defined dependencies on NumPy's API. They don't need to worry about any implementation details from NumPy leaking into their override of a function.
We don't have np.asduckarray() yet or whatever we'll end up calling our proposed casting function from NEP 22, so we don't have a fully fleshed out mechanism for NumPy to declare "this object needs to support .shape and .dtype, or I'm going to cast it into something that does". More comments on the environment variable and the interface to come in my next email... Cheers, Stephan
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Tue, Aug 21, 2018 at 9:39 AM, Stephan Hoyer <shoyer@gmail.com> wrote:
From your mention of "full" duck arrays I guess you're thinking of
I clicked the link, but I don't see anything about units? Of course units are a tricky example to make inferences from, because they aren't a good fit for the duck array concept in general. (In terms of numpy's core semantics, data-with-units is a special dtype, not a special container type.) this distinction?: http://www.numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html#princip... You're right: if numpy changes the implementation of some high-level function to use protocol-A instead of protocol-B, and there's some partial-duck-array that only implements protocol-B, then it gets broken. Of course, in general __array_function__ has the same problem: if sklearn changes their implementation of some function to call numpy function A instead of numpy function B, and there's a partial-duck-array that only implements numpy function B, then sklearn is broken. I think that as duck arrays roll out, we're just going to have to get used to dealing with breakage like this sometimes. The advantage of __array_function__ is that we get to ignore these issues within numpy itself. The advantage of having focused-protocols is that they make it easier to implement full duck arrays, and they give us a vocabulary for talking about degrees of partiality. For example, with __array_concatenate__, a duck array either supports all the concatenation/stacking operations or none of them – so sklearn never has to worry that switching between np.row_stack and np.stack will cause issues.
NumPy (and Python in general) is never going to make everything 100% optimized all the time. Over and over we choose to accept small inefficiencies in order to improve maintainability. How big are these inefficiencies – 1% overhead, 10% overhead, 10x overhead...? Do they show up everywhere, or just for a few key functions? What's the maintenance cost of making NumPy's whole API overrideable, in terms of making it harder for us to evolve numpy? What about for users dealing with a proliferation of subtly incompatible implementations? You may be right that the tradeoffs work out so that every API needs to be individually overridable and the benefits are worth it, but we at least need to be asking these questions.
Yeah, you're right, row_stack is a bad example :-). But of course the point is that it's literally any bug-fix or added feature in numpy's public API. Here's a better, more concrete example: back in 2015, you added np.stack (PR #5605), which was a great new feature. Its implementation was entirely in terms of np.concatenate and other basic APIs like .ndim, asanyarray, etc. In the smallish-set-of-designed-protocols world, as soon as that's merged into numpy, you're done: it works on sparse arrays, dask arrays, tensorflow tensors, etc. People can use it as soon as they upgrade their numpy. In the __array_function__ world, merging into numpy is only the beginning: now you have to go make new PRs to sparse, dask, tensorflow, etc., get them merged, released, etc. Downstream projects may refuse to use it until it's supported in multiple projects that have their own release cycles, etc. Or another example: at a workshop a few years ago, Matti put up some of the source code to numpypy to demonstrate what it looked like. I immediately spotted a subtle bug, because I happened to know that it was one we'd found and fixed recently. (IIRC it was the thing where arr[...] should return a view of arr, not arr itself.) Of course indexing for duck arrays is its own mess that's somewhat orthogonal to __array_function__, but the basic point is that numpy has a lot of complex error-prone semantics, and we are still actively finding and fixing issues in numpy's own implementations.
Yeah, sorry.
I agree, we probably wouldn't do this particular change.
When you say "consistency" here that means: "they can be sure that when they disagree with the numpy devs about the semantics/implementation of a numpy API, then the numpy API will consistently act the way they want, not the way the numpy devs want". Right? This is a very double-edged sword :-).
That's true, but it's just as big a problem for NEP 18, because __array_function__ is never going to do much if you've already coerced the thing to an ndarray. Some kind of asduckarray solution is basically a prerequisite to any other duck array features. -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
Hi Nathaniel and Stephan, Since this conversation is getting a bit lengthy and I see a lot of repeated stuff, I’ll summarise the arguments for everyone’s benefit and then present my own viewpoints: Nathaniel: Undue maintenance burden on NumPy, since semantics have to match exactly Implementations of functions may change, which may break downstream library compatibility There may be time taken in merging this everywhere, so why not take time to define proper protocols? Hide this entire interface behind an environment variable, possibly to be removed later. Stephan: Semantics don’t have to match exactly, that isn’t the intent of most duck-arrays. This won’t happen given NumPy’s conservativeness. The protocols will just be copies of __array_function__, but less capable Provide an interface that only end-users may turn on. My viewpoints: I don’t think any Duck array implementers intend to copy semantics on that level. Dask, which is the most complete one, doesn’t have views, only copies. Many other semantics simply don’t match. The intent is to allow for code that expresses, well, intent (no pun intended) instead of relying heavily on semantics, but that can use arbitrary duck-array implementations instead of just ndarray. Most of the implementations in NumPy are pretty stable, and the only thing that’s likely to happen here is bug fixes. And we are free to fix bugs those; I doubt implementation-specific bugs will be copied. However, these first two points are for/against duck arrays in general, and not specific to this protocol, so IMO this discussion is completely orthogonal to this one. I agree with Stephan here: Defining a minimum API for NumPy that will complete duck arrays will produce a lot of functions in every case that cannot be overridden, as they simply cannot be expressed in terms of the protocols we have added so far. This will lead to more protocols being produced, and so on ad infinitum. We have to consider the burden that such a design would place on the maintainers of NumPy as well… I personally feel that the amount of such protocols we’ll so need are large enough that this line of action is more burdensome, rather than less. I prefer an approach with __array_function__ + mailing list ping before adding a function. May I propose an alternative that was already discussed, and one that I think everyone will be okay with: We put all overridable functions inside a new submodule, numpy.api, that will initially be a shallow-ish copy of the numpy module. I say ish because all modules inside NumPy will need to be shallow-copied as well. If we need to add __array_function__, we can always do that there. Normal users are using “regular” NumPy unless they know they’re using the API, but it is separately accessible. As for hiding it completely goes: We have to realise, the Python computation landscape is fragmenting. The slower we are, the more fragmented it will become. NumPy already isn’t “the standard” for machine learning. Regards, Hameer Abbasi
![](https://secure.gravatar.com/avatar/5f88830d19f9c83e2ddfd913496c5025.jpg?s=120&d=mm&r=g)
On Wed, Aug 22, 2018 at 4:22 AM Hameer Abbasi <einstein.edison@gmail.com> wrote:
May I propose an alternative that was already discussed, and one that I think everyone will be okay with:
That's a dangerous assumption on this list:) We put all overridable functions inside a new submodule, numpy.api, that
will initially be a shallow-ish copy of the numpy module.
This is not desirable. There are projects (e.g. statsmodels) that have added a .api submodule before. It's generally considered not a good idea, it's not very Pythonic. Everything one can import that doesn't have an underscore is normally part of the API of a package. In this particular case, I definitely prefer an envvar and relying on what is documented as part of __array_function__ rather than a new namespace. Cheers, Ralf I say ish because all modules inside NumPy will need to be shallow-copied
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Tue, Aug 21, 2018 at 6:47 PM Nathaniel Smith <njs@pobox.com> wrote:
To clarify: np.datetime64 arrays can be considered a variant of NumPy arrays that support units. Namely, they use a dtype for representing time units. I expect that the issues we've encountered with datetime64 will be indicative of some of the sort of issues that authors of unit-aware arrays will encounter.
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith <njs@pobox.com> wrote:
Indeed, I missed the "at numpy import time" bit :). In that case, I'm concerned that it isn't always possible to set environment variables once before importing NumPy. The environment variable solution works great if users have full control of their own Python binaries, but that isn't always the case today in this era of server-less infrastructure and online notebooks. One example offhand is Google's Colaboratory ( https://research.google.com/colaboratory), a web based Jupyter notebook. NumPy is always loaded when a notebook is opened, as you can check from inspecting sys.modules. Now, I work with the developers of Colaboratory, so we could probably figure out a work-around together, but I'm pretty sure this would also come up in the context of other tools. Another problem is unit testing. Does pytest use a separate Python process for running the tests in each file? I don't know and that feels like an implementation detail that I shouldn't have to know :). Yes, in principle I could use a subprocess in my __array_function__ for unit tests, but that would be really awkward.
I'm in complete agreement that only authors of end-user applications should invoke this option, but just because something is technically possible doesn't mean that people will actually do it or that we need to support that use case :). numpy.seterr() is a good example. It allows users to globally set how NumPy does error handling, but well written libraries still don't do that. TensorFlow has similar function tf.enable_eager_execution() for enabling "eager mode" that is also worth examining: https://www.tensorflow.org/api_docs/python/tf/enable_eager_execution To solve the testing issue, they wrote decorator for using with tests, run_in_graph_and_eager_modes(): https://www.tensorflow.org/api_docs/python/tf/contrib/eager/run_test_in_grap...
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Tue, Aug 21, 2018 at 6:12 PM, Stephan Hoyer <shoyer@gmail.com> wrote:
I mean, the idea of the envvar is to be a temporary measure enable devs to experiment with a provisional feature, while being awkward enough that people don't build lots of stuff assuming its there. It doesn't have to 100% supported in every environment.
Set the envvar before invoking pytest? For numpy itself we'll need to write a few awkward tests involving subprocesses to make sure the envvar parsing is working properly, but I don't think this is a big deal. As long as we only have 1-2 places that __array_function__ dispatch funnels through, we just need to make sure that they work properly with/without the envvar; no need to test every API separately. Or if it is an issue we can have some private API that's only available to the numpy test suite...
I didn't say "authors of end-user applications", I said "end-users" :-). That said, I dunno. My intuition is that if we have a function call like this then libraries that define __array_function__ will merrily call it in their package __init__ and it accomplishes nothing, but maybe I'm being too cynical and untrusting. -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Tue, Aug 21, 2018 at 6:57 PM Nathaniel Smith <njs@pobox.com> wrote:
My understanding of the idea of the envvar is to obtain informed consent from NumPy users, e.g., "I understand that this is a unsupported experimental feature that may be removed in the future without warning." It's pretty important for me personally that's it's possible to use this in a flexible set of environments, and in particular to have something that works in my preferred notebook environment. How else are we going to test this? Every limitation that we put into the experimental version of this feature decreases the likelihood that it gets used enough to know if it's even a viable solution. If it's too awkward, nobody's even going to bother testing it, and this whole effort will fall flat on its face.
These are mostly the same for NumPy, but I do disagree with you here. Ultimately we have to trust application developers to make the right choices for their tools. If they are willing to accept that maintenance burden of either (1) potentially being stuck on NumPy 1.16 forever or (2) needing to rewrite their code, that's their tradeoff to make. It's a little preposterous to force this decision onto end-users, who may not even know a tool is written in NumPy.
People can do lots of dumb things in Python (e.g., monkeypatching) -- the language doesn't stop them. Fortunately this mostly isn't a problem.
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
Hi everyone,
I might add that most duck array authors are highly unlikely to be newcomers to the Python space. We should just put a big warning there while enabling and that’ll be enough to scare away most devs from doing it by default.
Best Regards, Hameer Abbasi Sent from my iPhone
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Thu, Aug 23, 2018 at 9:02 AM, <einstein.edison@gmail.com> wrote:
That's a reasonable idea... a Big Obnoxious Warning(tm) when it's enabled, or on first use, would achieve a lot of the same purpose. E.g. if this_is_the_first_array_function_usage(): sys.stderr.write( "WARNING: this program uses NumPy's experimental '__array_function__' feature.\n" "It may change or be removed without warning, which might break this program.\n" "For details see http://www.numpy.org/neps/nep-0018-array-function-protocol.html\n" ) -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Fri, Aug 24, 2018 at 1:36 AM Hameer Abbasi <einstein.edison@gmail.com> wrote:
Issuing a FutureWarning seems roughly appropriate here. The Python 3.7 docs write: "Base category for warnings about deprecated features when those warnings are intended for end users of applications that are written in Python." Writing to sys.stderr directly is generally considered poor practice for a Python libraries. In my experience FutureWarning does a good job of satisfying the goals of being a "Big Obnoxious Warning" while still being silence-able and testable with standard tools.
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Fri, Aug 24, 2018 at 1:46 PM, Stephan Hoyer <shoyer@gmail.com> wrote:
Yeah, the reason warnings are normally recommended is because normally, you want to make it easy to silence. But this is the rare case where I didn't want to make it easy to silence, so I didn't suggest using a warning :-). Calling warnings.warn (or the C equivalent) is also very expensive, even if the warning ultimately isn't displayed. I guess we could do our own tracking of whether we've displayed the warning yet, and only even attempt to issue it once, but that partially defeats the purpose of using warnings in the first place. -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Fri, Aug 24, 2018 at 3:14 PM Nathaniel Smith <njs@pobox.com> wrote:
I thought the suggestion was to issue a warning when np.enable_experimental_array_function() is called. I agree that it's a non-starter to issue it every time an __array_function__ method is called -- warnings are way too slow for that. People can redirect stderr, so we're really not stopping anyone from silencing things by doing it in a non-standard way. We're just making it annoying and non-standard. Developers could even run Python in a subprocess and filter out all the warnings -- there's really nothing we can do to stop determined abusers of this feature. I get that you want to make this annoying and non-standard, but this is too extreme for me. Do you seriously imagine that we'll consider ourselves beholden in the future to users who didn't take us at our word?
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Fri, Aug 24, 2018 at 4:00 PM, Stephan Hoyer <shoyer@gmail.com> wrote:
If our protection against uninformed usage is a Big Obnoxious Warning(tm), then I was imagining that we could simplify by dropping enable_experimental_array_function entirely. Doesn't make a big difference either way though.
Let's break that question down into two parts: 1. if we do find ourselves in a situation where changing this would break lots of users, will we consider ourselves beholden to them? 2. is it plausible that we'll find ourselves in that situation? For the first question, I think the answer is... yes? We constantly bend over backwards to try to avoid breaking users. Our deprecation policy explicitly says that it doesn't matter what we say in the docs, the only thing that matters is whether a change breaks users. And to make things more complicated, it's easy to imagine scenarios where the people being broken aren't the ones who had a chance to read the docs – e.g. if a major package starts relying on __array_function__, then it's all *their* users who we'd be breaking, even though they had nothing to do with it. If any of {tensorflow/astropy/dask/sparse/sklearn/...} did start relying on __array_function__ for normal functionality, then *of course* that would come up in future discussions about changing __array_function__, and *of course* it would make us reluctant to do that. As it should, because breaking users is bad, we should try to avoid ending up in situations where that's what we have to do, even if we have a NEP to point to to justify it. But... maybe it's fine anyway, because this situation will never come up? Obviously I hope that our downstreams are all conscientious, and friendly, and take good care of their users, and would never create a situation like that. I'm sure XArray won't :-). But... people are busy, and distracted, and have pressures to get something shipped, and corners get cut. Companies *care* about what's right, but they mostly only *do* the minimum they have to. (Ask anyone who's tried to get funding for OSS...) Academics *care* about what's right, but they just don't have time to care much. So yeah... if there's a quick way to shut up the warning and make things work (or seem to work, temporarily), versus doing things right by talking to us, then I do think people might take the quick hack. The official Tensorflow wheels flat out lie about being manylinux compatible, and the Tensorflow team has never talked to anyone about how to fix this, they just upload them to PyPI and leave others get to deal with the fallout [1]. That may well be what's best for their users, I don't know. But stuff like this is normal, it happens all the time, and if someone does it with __array_function__ then we have no leverage. -n [1] https://github.com/tensorflow/tensorflow/issues/8802#issuecomment-401703703 -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/72f994ca072df3a3d2c3db8a137790fd.jpg?s=120&d=mm&r=g)
On 29/08/18 10:37, Nathaniel Smith wrote: trouble if we were proposing a broad change to something like indexing or the random number module (see those NEPs). If we break one of those major packages, it is on them to pin the version of NumPy they can work with. In my opinion very few end users will be implementing their own ndarray classes with `__array_function__`. While we will get issue reports, we can handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem and urge users to complain to those packages. Other than adding a warning, I am not sure what the concrete proposal is here. To not accept the NEP? Matti
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
One thing that might help here is nightly or continuous CI builds of the NumPy wheels. This would be good, as we could test it in CI, and fix it when it comes up. But I guess that’s another discussion. Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime.
![](https://secure.gravatar.com/avatar/8749ec52cee260c4c1f67f2dec29d957.jpg?s=120&d=mm&r=g)
1. if we do find ourselves in a situation where changing this would break lots of users, will we consider ourselves beholden to them?
I think that it would be useful for Numpy's continued evolution to develop the ability to include code on a provisional basis. Other projects do this and they just have big bold "Experimental" notes everywhere that a user might go to learn about the functionality (docs, docstrings, blogposts). Some users will definitely get burned, yes, but the alternative is to burn all other users a little bit by moving too slowly and not trying things out. This is different from how Numpy has operated in the past, but that might be ok.
2. is it plausible that we'll find ourselves in that situation?
Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime
Yup. Same for Dask. We're pretty accustomed to this. On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi <einstein.edison@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/851ff10fbb1363b7d6111ac60194cc1c.jpg?s=120&d=mm&r=g)
HI All, On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain, i.e., that it will continue to be possible to override, say, concatenate. It is not a big deal if the actual implementation changes (say, `__array_concatenate__` is introduced) and we need to do some version-dependent magic to ensure that our users do not notice the change; we did the same with `__array_ufunc__` - the wrap/prepare machinery will be gone only in the next version of astropy, when our minimum version for numpy reaches 1.13. One thing perhaps worth remembering that what really matters is not so much `__array_function__` but the set of functions that actually implement the override. It would seem unlikely this would be the complete numpy API on the first go; perhaps it is an idea to be somewhat specific about which part we start with? My vote would go towards array manipulation routines like concatenate. I would also suggest to avoid those functions for which ufunc implementations would seem quite possible (i.e., avoid things like median, etc.) All the best, Marten On Wed, Aug 29, 2018 at 8:31 AM Matthew Rocklin <mrocklin@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/8749ec52cee260c4c1f67f2dec29d957.jpg?s=120&d=mm&r=g)
My guess is that you wouldn't have this expectation if Numpy released this feature with explicit "Experimental" warnings attached to it. In that case astropy might just wait before adopting it until that label was dropped. Projects that were more risk-friendly would then take on the burden of trying it out, working out some kinks, and then hopefully in a version or two the "Experimental" label would be dropped and astropy would step in and adopt it more safely. This would give the ecosystem an opportunity to try something and modify it after having some experience without promising to support it in a fully backwards compatible way forever. On Wed, Aug 29, 2018 at 9:46 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/851ff10fbb1363b7d6111ac60194cc1c.jpg?s=120&d=mm&r=g)
On Wed, Aug 29, 2018 at 9:53 AM Matthew Rocklin <mrocklin@gmail.com> wrote:
I do think that even for an experimental feature one should be allowed to expect that there will continue to be a way provided by numpy to access the same functionality, i.e., once we allow people to do `np.concatenate(list-of-mimics)`, I think we should consider ourselves committed to providing some way to continue doing that - the experimental tag should only imply that we are not committed to the precise method with which that can be achieved. (Of course, I can be a bit more cavalier than most, since I am hopeful I can help ensure that some method will continue to be present; indeed, the same happened for __array_ufunc__ - I had already written a __numpy_ufunc__ implementation in Quantity and when that stalled in numpy, I picked up and finished the __array_ufunc__ code that Nathan had written.) -- Marten
![](https://secure.gravatar.com/avatar/49c5ca40df804ebf053e121a6104f405.jpg?s=120&d=mm&r=g)
Well, I guess I'll be proving Nathaniel right: I would *definitely* start using __array_function__ in astropy - not being able to concatenate Quantity and other instances which use it has been a long-standing pain.
That's fair enough, but I think Matt's point still stands. Any given project can weigh the benefits of early adoption against the risks, considering its need for a new feature and its resources available to deal with possible future changes. An "Experimental" designation gives numpy a little space to innovate and lets individual projects opt in. Without that space, numpy has to take on all the burden of continuity instead of sharing that burden with the community of projects that opt in to early adoption. Best, Dan Daniel B. Allan, Ph.D Associate Computational Scientist, Brookhaven National Lab (631) 344-3281 (no voicemail set up) ________________________________ From: NumPy-Discussion <numpy-discussion-bounces+dallan=bnl.gov@python.org> on behalf of Marten van Kerkwijk <m.h.vankerkwijk@gmail.com> Sent: Wednesday, August 29, 2018 10:43:55 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol On Wed, Aug 29, 2018 at 9:53 AM Matthew Rocklin <mrocklin@gmail.com<mailto:mrocklin@gmail.com>> wrote:
On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain,
My guess is that you wouldn't have this expectation if Numpy released this feature with explicit "Experimental" warnings attached to it. In that case astropy might just wait before adopting it until that label was dropped. Projects that were more risk-friendly would then take on the burden of trying it out, working out some kinks, and then hopefully in a version or two the "Experimental" label would be dropped and astropy would step in and adopt it more safely. Well, I guess I'll be proving Nathaniel right: I would *definitely* start using __array_function__ in astropy - not being able to concatenate Quantity and other instances which use it has been a long-standing pain. I do think that even for an experimental feature one should be allowed to expect that there will continue to be a way provided by numpy to access the same functionality, i.e., once we allow people to do `np.concatenate(list-of-mimics)`, I think we should consider ourselves committed to providing some way to continue doing that - the experimental tag should only imply that we are not committed to the precise method with which that can be achieved. (Of course, I can be a bit more cavalier than most, since I am hopeful I can help ensure that some method will continue to be present; indeed, the same happened for __array_ufunc__ - I had already written a __numpy_ufunc__ implementation in Quantity and when that stalled in numpy, I picked up and finished the __array_ufunc__ code that Nathan had written.) -- Marten
![](https://secure.gravatar.com/avatar/851ff10fbb1363b7d6111ac60194cc1c.jpg?s=120&d=mm&r=g)
Absolutely fine to have to deal with future chances - my main point is that by accepting the NEP, I think numpy is committing to provide some way to override whatever functions __array_function__ is introduced for, i.e., we cannot reasonably go back to not providing any way to override such a function (but by marking the feature experimental, we *can* change the way the override of any given function is implemented). -- Marten p.s. And, yes, do count me in for helping to back up that commitment!
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Wed, Aug 29, 2018 at 9:34 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
I'm not entirely sure that we're ready to make this commitment yet, but I think this ties into our decision about what sort of user opt-ins to require. I'd like to structure this decision in terms of the types of breaking changes we contemplate possibly making in this future with regards to this protocol: 1. Abandoning this approach of allowing overrides for NumPy's high level API entirely, in favor of requiring the use of a dedicated namespace like numpy.api. This might arise if we decide that the overhead of checking for __array_function__ attributes is too much, or if it's simply too confusing to allow NumPy functions to be overloaded in arbitrary ways. 2. Changing the general __array_function__ protocol in a breaking way, e.g., to eliminate the "types" argument. 3. Replacing __array_function__ with another override mechanism, either (a) in general (e.g., __array_function_v2__ without the "types" argument) or (b) for specific functions (e.g., switching to a more specific protocol like __array_ufunc__). 4. Removing the possibility of overriding specific functions altogether, because we want to require using a more generic interface (e.g., to require overriding np.stack implicitly via np.expand_dims/np.concatenate instead). Possible changes (1) or (2) would be good reasons for requiring end-users to make an intentional choice to enable __array_function__. Either of these changes would break everyone who uses the protocol. Concern about change (1) would be a particularly good reason to require an opt-in, because it's concerned about impact on users who don't use any new functionality. Possible changes (3) or (4) would be annoying to users, but in my mind would not justify requiring an explicit opt-in. Both could be achieved without immediately breaking users by making use of a standard deprecation cycle, i.e., by still calling __array_function__ but issuing a FutureWarning. The main reason for requiring the explicit opt-in would be if we don't want to need to bother with a deprecation cycle. To be clear, with the exception of change (3)(b), I don't think any of these are particularly likely or desirable outcomes. So in my personal opinion, I don't think we need the explicit opt-in. But I'm willing to make to go ahead with requiring the opt-in if that's what it takes to get everyone on board -- as long as we don't make the opt-in too onerous in a way that imposes hard limits on this protocol can be used. (I would be OK with an explicit Python API for enabling this feature that raises a FutureWarning when it's called, without any API for turning it off.) Best, Stephan
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Wed, Aug 29, 2018, 02:44 Matti Picus <matti.picus@gmail.com> wrote:
The proposal is just that while the NEP is considered experimental and provisional, we should use some kind of technical measures to discourage use in a non-experimental settings. We want to stay in control of when it escapes the lab, and docs alone, or trivially disableable messages, aren't a very effective way to do that. -n
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Wed, Aug 29, 2018 at 1:38 AM Nathaniel Smith <njs@pobox.com> wrote:
Yes, this is a good example of how library authors sometimes break rules imposed by their dependencies when not enforced by technical means. I'd like to think the TensorFlow team was making an intentional choice here and is willing to live with the consequence of their decision. Unfortunately right now many of those consequences are imposed on others, so this was certainly an inconsiderate choice. I don't know if they were aware of these costs that they are imposing on the rest of the Python ecosystem. I don't see any mention in PEP-513 about the consequences of lying about manylinux compatibility -- even though arguably such warnings should not be necessary. One lesson we might take from this is documentation is essential: we should have more prominent warnings in NEP-18 (given that it's also likely to serve as a form of user-facing docs) about how experimental this protocol is, and about the specific types of changes we anticipate in the future (e.g., see my other email). I also hope that Python packaging maintainers would not hesitate to break this abuse of the manylinux1 tag, e.g., if they wanted to add compatibility checks on wheels when uploaded to PyPI.
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
RE: the types argument On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith <njs@pobox.com> wrote:
What's special about __array_function__ is that it's a hook that lets you override an entire API through a single interface. Unlike protocols like __add__, implementers of __array_function__ don't know exactly which arguments could have implemented the operation.
It's also a pragmatic choice: libraries like dask.array and autograd.numpy have already implemented NumPy's API without overrides. These projects follow the current numpy convention: non-native array objects are coerced into native arrays (i.e., dask or autograd arrays). They don't do any type checking. I doubt there would be much appetite for writing alternative versions of these APIs that return NotImplemented instead -- especially while this feature remains experimental.
The only extra amount we pay extra is the price of converting these types into a Python data structure and passing them into the __array_function__ method call. We already had to collect them for __array_function__ itself to identify unique types to call -- so this is a pretty minimal extra cost.
This could potentially work, but now the __array_function__ protocol itself becomes more complex and out of sync with __array_ufunc__. It's a much smaller amount of additional complexity to add an additional passed argument.
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Thu, Aug 23, 2018 at 1:06 PM Hameer Abbasi <einstein.edison@gmail.com> wrote:
I don't follow -- can you please elaborate? If you don't want to do anything with the 'types' argument, you can simply ignore it. The problem of identifying whether arguments have valid types or not remains unchanged from the situation with __add__ or __array_ufunc__. 'types' just gives you another optional tool to help solve it.
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
On Fri, Aug 24, 2018 at 5:55 PM Stephan Hoyer <shoyer@gmail.com> wrote:
If we make specifying __array_function_types__ a mandatory part -- And such that it is a whitelist, the XArray or Dask would need to import sparse in order to specify that they accept mixing sparse arrays with native arrays (i.e. for adding sparse.SparseArray to __array_function_types__). Which is basically what I mean. It might be a 'soft' dependency, but there will be a dependency nonetheless.
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Fri, Aug 24, 2018, 09:07 Hameer Abbasi <einstein.edison@gmail.com> wrote:
Oh yeah, if we did this then we definitely wouldn't want to make it mandatory. Some `__array_function__` implementations might want to do checking another way, or support different types in different overloaded functions, or be able to handle arbitrary types. -n
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer <shoyer@gmail.com> wrote:
Skipping over all the discussion following this for brevity. I find myself in general agreement with Stephan and think we should go ahead and merge this. A few points: 1. I don't think we should have an opt in environment variable, just put `__array_function__` out there with the understanding that it might change with experience. I don't think scores, let alone thousands, of folks are going to rush to take advantage of this, most likely the developers of the proposal will be the early adopters. 2. I have a preference for implementing high level functions rather than low level, and fewer rather than more, with the choice driven by need. I think that will limit the amount of interdependence tangle and, in the long run, might serve to define an unofficial NumPy API. 3. NumPy cannot manage all the projects that make use of the new option to keep them in sync, we have neither the time nor experience to do so, and historic attempts along those lines tended to die in obscurity. To the extent that we do so, it comes down to 2. 4. I don't think this conflicts with Nathaniel's proposal, the main disagreement seems to over how to proceed and the selection of functions, see 2. 5. The `__array_function_types__` idea looks interesting. Chuck
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Sat, Sep 8, 2018 at 7:12 PM Charles R Harris <charlesr.harris@gmail.com> wrote:
I rolled back Chuck's merge of the "Acceptance" PR for this NEP because (1) it was not clear that if had reached consensus and (2) I still wanted to make a few changes based on this discussion. I have now drafted these revisions to the NEP to clarify its stance around backwards compatibility, and the type of the "types" argument: https://github.com/numpy/numpy/pull/11943 I have *not* included any form of the explicit "end-user opt-in" requested by Nathaniel. I don't think it is warranted based on the scope of anticipated future changes; see my earlier post [1] for details. Nobody has seriously suggested that we would release this functionality and later eliminate the ability to override NumPy functions entirely in the future. Of course, requiring an onerous explicit opt-in would be fine while this feature is initially under development, and until we are sure that checks for overriding arbitrary NumPy functions are fast enough to be generally viable. In particular, we should verify that __array_function__ does not meaningfully impact performance for major downstream components of the SciPy stack. But I think running standard benchmark suites (e.g., with ASV) and user testing of release candidates would be enough. If you have still have major objections to this version of proposal, please raise them unambiguously -- preferably with an formal veto [2]. Best, Stephan [1] https://mail.python.org/pipermail/numpy-discussion/2018-August/078669.html [2] https://docs.scipy.org/doc/numpy/dev/governance/governance.html
![](https://secure.gravatar.com/avatar/851ff10fbb1363b7d6111ac60194cc1c.jpg?s=120&d=mm&r=g)
Thanks for adding the clarifications. They read well to me, and I think make clear to a project like astropy what to expect (and I expect we'll be use it!). -- Marten
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Thu, Sep 13, 2018 at 10:30 AM Stephan Hoyer <shoyer@gmail.com> wrote:
Again, if anyone (yes, I'm looking at you, Nathaniel!) still has objections please speak up soon. If I don't hear anything, I'm going submit a PR to mark this proposal as "Accepted" tomorrow (one week after these latest revisions). As Chuck noted, "Tentatively accepted" or "Experimental" would probably be a better status, but that currently isn't one of our options for NEPs. In any case, I would love to move on from debate to implementation! As a side note, I recently noticed a CuPy pull request ( https://github.com/cupy/cupy/pull/1650 <https://github.com/cupy/cupy/pull/1650/files>) to add __array_function__. It's a testament to our proposed interface that the full PR is 9 lines of code without tests.
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Thu, Sep 13, 2018 at 10:30 AM, Stephan Hoyer <shoyer@gmail.com> wrote:
Okay, so this is a pretty substantial change! Before, the NEP's stance was "we might change anything, at any time, without any warning", which of course makes it easier to accept the NEP (since we can always back out), but was also so different from our normal rules that it seemed important to make sure people weren't using it without realizing. Now it actually makes a commitment: to not regress on what functions can be overloaded (though the details might change), and commits to an abbreviated-but-nonzero deprecation process when we change things. I get the impression that this is closer to what the authors were intending in the first place, so that's good! I would probably have kept the noisy warning and zero commitments for one release anyway, because IMO it's not a big deal and it rarely hurts to hedge bets and gather data. But on reflection, I think I am OK with this level of commitment if that's what y'all want to go for. (After all, it's not really any stronger than NEP 22's high-level plan.) So, +0. -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Thu, Sep 20, 2018 at 2:05 PM Stephan Hoyer <shoyer@gmail.com> wrote:
I have started implementing NEP-18 in a pull request: https://github.com/numpy/numpy/pull/12005 I propose a multi-step process for implementation: 1. Write a correctly working version of the core __array_function__ machinery in pure Python, adapted from the example implementation in the NEP. 2. Rewrite bottlenecks in C. 3. Implement overrides for various NumPy functions. I think the first step is mostly complete, but I'm particularly looking for help with the second and third steps, which should be able to happen incrementally and in parallel. Cheers, Stephan
![](https://secure.gravatar.com/avatar/7a7a675bc479a30c1389d4c64f612a95.jpg?s=120&d=mm&r=g)
Hi all, I have small question for clarification regarding some reducing numpy functions like np.sum, np.mean, etc, i.e. functions that now work on other array-like objects by checking if they have 'sum', 'mean', etc method and if so, "dispatch" to the method. Those functions are several times mentioned as examples in the text of the NEP, so although the NEP does not list the functions that will use the __array_function__ protocol explicitly, I assume that those functions will probably be in that list. But I wanted to check if that assumption is correct. Is it the intent that the functions that now dispatch to object methods will be updated to use the protocol instead? (probably keeping to method-dispatch as second try for backwards compatibility) Not something urgent, but I was just wondering that when reading the NEP, as this is something that would be useful for pandas (where we now need to do some gymnastics to deal with arguments passed by numpy to those methods that we don't support). Joris
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Wed, Sep 26, 2018 at 9:19 AM Joris Van den Bossche < jorisvandenbossche@gmail.com> wrote:
Hi Joris, Yes, this is my exact intent. Eventually, we might remove/deprecate automatically calling methods with the same name (due to how error prone this is), but that is a long way in the future if ever. Keep in mind that the current version of the NEP may not be a good fit for pandas, because it requires overriding *all* NumPy functions. See this section of the NEP for details: http://www.numpy.org/neps/nep-0018-array-function-protocol.html#coercion-to-... Cheers, Stephan
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
I'm sorry, I've been trying to find the time to read what you ended up with, and haven't managed yet – could I get a few days extension? :-) -n On Wed, Aug 1, 2018 at 5:27 PM, Stephan Hoyer <shoyer@gmail.com> wrote:
-- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
Hey all, So I've finally read through NEP 18 (__array_function__). Sorry again for the delay! It's an impressive piece of work! Thanks to the many authors; there's clearly been a lot of thought put into this. # The trade-off between comprehensive APIs versus clean APIs At a high-level, what makes me nervous about this proposal is that it reminds me of a classic software design pattern that... I don't know a name for. You might call it the "structured monkeypatching" approach to extensibility. The pattern is: a project decides they want to allow some kind of extensions or addons or plugins or something, but defining a big structured API for this is too difficult. So they take their current API surface area and declare that that's the plugin API (plus some mechanism for plugins to hook in etc.). Is this pattern good? It's... hard to say. What generally happens is: 1. You get a very complete, powerful, flexible plugin API with minimal work. 2. This quickly leads to a rich system of powerful plugins, which drives quick uptake of the project, sometimes even driving out competitors. 3. The maintainers slowly realize that committing to such a large and unstructured API is horribly unwieldy and makes changes difficult. 4. The maintainers spend huge amounts of effort trying to crawl out from under the weight of their commitments, which mixed success. Examples: pytest, sphinx: For both of these projects, writing plugins is a miserable experience, and you never really know if they'll work with new releases or when composed with random other plugins. Both projects are absolutely the dominant players in their niche, far better than the competition, largely thanks to their rich plugin ecosystems. CPython: the C extension API is basically just... all of CPython's internals dumped into a header file. Without this numpy wouldn't exist. A key ingredient in Python's miraculous popularity. Also, at this point, possibly the largest millstone preventing further improvements in Python – this is why we can't have multicore support, JITs, etc. etc.; all the most ambitious discussions at the Python language summit the last few years have circled back to "...but we can't do that b/c it will break the C API". See also: https://mail.python.org/pipermail/python-dev/2018-July/154814.html Firefox: their original extension API was basically just "our UI is written in javascript, extension modules get to throw more javascript in the pot". One of Firefox's original USPs, and a key part of like... how Mozilla even exists instead of having gone out of business a decade ago. Eventually the extension API started blocking critical architectural changes (e.g. for better sandboxing), and they had to go through an *immensely* painful migration to a properly designed API, which tooks years and burned huge amounts of goodwill. So this is like... an extreme version of technical debt. You're making a deal with the devil for wealth and fame, and then eventually the bill becomes due. It's hard for me to say categorically that this is a bad idea – empirically, it can be very successful! But there are real trade-offs. And it makes me a bit nervous that Matt is the one proposing this, because I'm pretty sure if you asked him he'd say he's absolutely focused on how to get something working ASAP and has no plans to maintain numpy in the future. The other approach would be to incrementally add clean, well-defined dunder methods like __array_ufunc__, __array_concatenate__, etc. This way we end up putting some thought into each interface, making sure that it's something we can support, protecting downstream libraries from unnecessary complexity (e.g. they can implement __array_concatenate__ instead of hstack, vstack, row_stack, column_stack, ...), or avoiding adding new APIs entirely (e.g., by converting existing functions into ufuncs so __array_ufunc__ starts automagically working). And in the end we get a clean list of dunder methods that new array container implementations have to define. It's plausible to imagine a generic test suite for array containers. (I suspect that every library that tries to implement __array_function__ will end up with accidental behavioral differences, just because the numpy API is so vast and contains so many corner cases.) So the clean-well-defined-dunders approach has lots of upsides. The big downside is that this is a much longer road to go down. I am genuinely uncertain which of these approaches is better on net, or whether we should do both. But because I'm uncertain, I'm nervous about committing to the NEP 18 approach -- it feels risky. ## Can we mitigate that risk? One thing that helps is the way the proposal makes it all-or-nothing: if you have an __array_function__ method, then you are committing to reimplementing *all* the numpy API (or at least all the parts that you want to work at all). This is arguably a bad thing in the long run, because only large and well-resourced projects can realistically hope to implement __array_function__. But for now it does somewhat mitigate the risks, because the fewer users we have the easier it is to work with them to change course later. But that's probably not enough -- "don't worry, if we change it we'll only break large, important projects with lots of users" isn't actually *that* reassuring :-). The proposal also bills itself as an unstable, provisional experiment ("this protocol should be considered strictly experimental. We reserve the right to change the details of this protocol and how specific NumPy functions use it at any time in the future – even in otherwise bug-fix only releases of NumPy."). This mitigates a lot of risk! If we aren't committing to anything, then sure, why not experiment. But... this is wishful thinking. No matter what the NEP says, I simply don't believe that we'll actually go break dask, sparse arrays, xarray, and sklearn in a numpy point release. Or any numpy release. Nor should we. If we're serious about keeping this experimental – and I think that's an excellent idea for now! – then IMO we need to do something more to avoid getting trapped by backwards compatibility. My suggestion: at numpy import time, check for an envvar, like say NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1. If it's not set, then all the __array_function__ dispatches turn into no-ops. This lets interested downstream libraries and users try this out, but makes sure that we won't have a hundred thousand end users depending on it without realizing. Other advantages: - makes it easy for end-users to check how much overhead this adds (by running their code with it enabled vs disabled) - if/when we decide to commit to supporting it for real, we just remove the envvar. With this change, I'm overall +1 on the proposal. Without it, I... would like more convincing, at least :-). # Minor quibbles I don't really understand the 'types' frozenset. The NEP says "it will be used by most __array_function__ methods, which otherwise would need to extract this information themselves"... but they still need to extract the information themselves, because they still have to examine each object and figure out what type it is. And, simply creating a frozenset costs ~0.2 µs on my laptop, which is overhead that we can't possibly optimize later... -n On Wed, Aug 1, 2018 at 5:27 PM, Stephan Hoyer <shoyer@gmail.com> wrote:
-- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
Hi Nathaniel, Very well written summary, it provides a lot of perspective into the different ways that this could go wrong. Here is a little commentary.
Ah, yes. I’ve been an affectee of this in both instances. For example, there’s a bug that with doctests enabled, you can’t select tests to run, and with coverage enabled, the pydev debugger stops working. However, composition (at least) is better handled with this protocol. This is answered in more detail later on.
Ah, yes. I remember heated debates about this. A lot of good add-ons (as Mozilla calls them) were lost because of alienated developers or missing APIs.
Yes, this is the way I’d prefer to go as well, but the machinery required for converting something to a ufunc is rather complex. Take, for example, the three-argument np.where. In order to preserve full backward compatibility, we need to support structured arrays and strings… which is rather hard to do. Same with the np.identity ufunc that was proposed for casting to a given dtype. This isn’t necessarily an argument against using ufuncs, I’d actually take it as an argument for better documenting these sorts of things. In fact, I wanted to do both of these myself at one point, but I found the documentation for writing ufuncs insufficient for handling these corner cases.
It all comes down to how stable we consider the NumPy API to be. And I’d say pretty stable. Once a function is overridden, we generally do not let NumPy handle it at all. We generally tend to preserve backward compatibility in the API, and the API is all we’re exposing.
There was a proposal that we elected to leave out — Maybe this is a reason to put it back in. The proposal was to have a sentinel (np.NotImplementedButCoercible) that would take the regular route via coercion, with NotImplemented raising a TypeError. This way, someone could do these things incrementally. Also, another thing we could do (as Stephan Hoyer, I, Travis Oliphant, Tyler Reddy, Saul Shanabrook and a few others discussed in the SciPy 2018 sprints) is to go through the NumPy API, convert as much of them into ufuncs as possible, and identify a “base set” from the rest. Then duck array implementations would only need to implement this “base set” of operations along with __array_ufunc__. My idea is that we get started on identifying these things as soon as possible, and only allow this “base set” under __array_function__, and the rest should simply use these to create all NumPy functionality. I might take on identifying this “base set” of functionality in early September if enough people are interested. I might even go as far as to say that we shouldn’t allow any function under this protocol unless it’s in the "base set”, and rewrite the rest of NumPy to use the “base set”. It’s a big project for sure, but identifying the “base set”/ufunc-able functions shouldn’t take too long. The rewriting part might. The downside would be that some things (such as np.stack) could have better implementations if a custom one was allowed, rather than for example, error checking + concatenate + reshape or error-checking + introduce extra dimension + concatenate. I’m willing to live with this downside, personally, in favour of a cleaner API.
We also have to consider that this might hinder adoption. But I’m fine with that. Properly > Quickly, as long as it doesn’t take too long. I’m +0 on this until we properly hammer out this stuff, then we remove it and make this the default. However, I also realise that pydata/sparse is in the early stages, and can probably wait. Other duck array implementations such as Dask and XArray might need this soon-ish.
The rationale here is that most implementations would check if the types in the array are actually supported by their implementation. If not, they’d return NotImplemented. If it wasn’t done here, every input would need to do it individually, and this may take a lot of time. I do agree that it violates DRY a bit though… The types are already present in the passed-in arguments, and this can be inferred from those.
Hope that clarifies things! Best regards, Hameer Abbasi
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Mon, Aug 13, 2018 at 2:44 AM, Nathaniel Smith <njs@pobox.com> wrote:
Rereading this today I realized that it could come across like I have an issue with Matt specifically. I apologize to anyone who got that impression (esp. Matt!) -- that definitely wasn't my intent. Matt is awesome. I should stop writing these things at 2 am. What I should have said is: We have an unusual decision to make here, where there are two plausible approaches that both have significant upsides and downsides, and whose effects are going to be distributed in a complicated way across different parts of our community over time. So the big challenge is to figure out how to take all that into account and weigh the needs of different stakeholders against each other. One major argument for the __array_function__ approach is that it has an actual NEP, which happened because we have a contributor who took the lead on making it happen, and who's deeply involved in some of the target projects like dask and sparse, so can make sure that the proposal will work well for them. That's a huge advantage! But... it also makes me a *little* nervous, because when you have really talented and productive contributors like this it's easy to get swept up in their perspective. So I want to double-check that we're also thinking about the stakeholders who can't be as active in the discussion, like "numpy maintainers from the future". (And I mostly mean this as a "we should keep this in mind" kind of thing – like I said in my original post, I think moving forward implementing __array_function__ is a great idea; I just want to be cautious about getting more experience before committing.) -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/8749ec52cee260c4c1f67f2dec29d957.jpg?s=120&d=mm&r=g)
Hi Nathaniel, I appreciate the clarification. Thank you for that. For what it's worth, I think that you may overestimate my involvement in the writing of that NEP. I sat down with Stephan during a Numpy dev meeting and we hacked something together. Afterwards several other people poured their thoughts into the process. I'd like to think that my perspective helped to inform this NEP, but it wasn't, by far, the driving force. If anyone had a strongest hand in the writing process it would probably be Stephan, who I find generally has a more conservative and careful perspective than I do. That being said, I do think that Numpy would be wise to move quickly here. I think that the growing fragmentation that we see in array computing in Numpy (Tensorflow, Torch, Dask, Sparse, CuPy) is largely due to Numpy moving slowly in the past. There is, I think, a systemic problem slowly erupting now that I think the community to respond to quickly if it is possible to do so safely. I believe that Numpy should absolutely be willing to try something experimental, and then say "nope, that was a bad idea" and retract it if it doesn't work out well. I think that figuring out all of __array_concatenate__, __array_stack__, __array_foo__, etc. for each of the many cases will take too long to respond to in an effective timeframe. I believe that we simply don't move quickly enough that this piece-by-piece careful handling of the API will result in Numpy's API becoming a meaningful standard in the broader community in the near-future. That being said, I think that we *should* engage in this piece-by-piece discussion, and as we figure them out we should slowly encroach on __array_function__ and remove functionality from it, much as __array_ufunc__ is not included in it in the current NEP. Ideally we should get to exactly where you want to get to. I perceive the __array_function__ protocol as a sort of necessary stop-gap. All that being said, this is just my personal stance. I suspect that each of the authors of the NEP and others who engaged in its careful review have a different perspective, which should probably carry more weight than my own. Best, -matt On Mon, Aug 13, 2018 at 4:29 PM Nathaniel Smith <njs@pobox.com> wrote:
![](https://secure.gravatar.com/avatar/b4929294417e9ac44c17967baae75a36.jpg?s=120&d=mm&r=g)
Hi, Thanks Nathaniel for this thoughtful response. On Mon, Aug 13, 2018 at 10:44 AM, Nathaniel Smith <njs@pobox.com> wrote: ...
Does everyone agree that, if we had infinite time and resources, this would be the better solution? If we devoted all the resources of the current Numpy grant to taking this track, could we complete it in a reasonable time? Cheers, Matthew
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
More resources means (given NumPy’s consensus system), more people have to agree on the overall design, so in my mind, it might even be slower.
If we devoted all the resources of the current Numpy grant to taking this track, could we complete it in a reasonable time?
I somehow think just the design of all these different protocols, heck, even ironing all these different protocols out and ignoring implementation; would take an unreasonably long amount of time, as evidenced by this one NEP. I’m more in favour of using this one rather conservatively: Perhaps a mailing list consensus before actually adding a function to __array_function__, making sure it won’t hinder too much progress. I also differ with Nathaniel on one minor thing with his comparisons to Firefox, CPython, pytest and Sphinx: We’re not talking about monkey-patching NumPy internals, we’re just talking about monkey-patching the public API. Of course, this is still a cost and can still hinder development, but it’s definitely better than exposing all internals.
Best Regards, Hameer Abbasi
![](https://secure.gravatar.com/avatar/b4929294417e9ac44c17967baae75a36.jpg?s=120&d=mm&r=g)
Hi, On Wed, Aug 15, 2018 at 5:36 PM, Hameer Abbasi <einstein.edison@gmail.com> wrote:
I don't think that's likely. As far as I can see, past discussions have been slow because several people need to get deep down into the details in order to understand the problem, and then stay focused through the discussion. When there is no-one working on that full-time, it's easy for the discussion to drift into the background, and the shared understanding is lost. My suspicion is, to the extent that Matti and Tyler can devote time and energy to shepherding the discussion, these will become quicker and more productive. Cheers, Matthew
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
On Wed, Aug 15, 2018 at 10:25 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
If we devoted all the resources of the current Numpy grant to taking this track, could we complete it in a reasonable time?
I think it is further down the road than that. Determining a core set of functions would depend on feedback (need), as well be dependent on implementation details for other functions. I think a dependency list for common NumPy functions might be interesting. That said, I don't think the current proposal is orthogonal to this. I don't expect every NumPy function to make call through this API, and we should probably try to limit, and list, the functions that implement the proposed mechanism. Chuck
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
Nathaniel, Thanks for raising these thoughtful concerns. Your independent review of this proposal is greatly appreciated! See my responses inline below: On Mon, Aug 13, 2018 at 2:44 AM Nathaniel Smith <njs@pobox.com> wrote:
RE: accidental differences in behavior: I actually think that the __array_function__ approach is *less* prone to accidental differences in behavior, because we require implementing every function directly (or it raises an error). This avoids a classic subclassing problem that has plagued NumPy for years, where overriding the behavior of method A causes apparently unrelated method B to break, because it relied on method A internally. In NumPy, this constrained our implementation of np.median(), because it needed to call np.mean() in order for subclasses implementing units to work properly. There will certainly be accidental differences in behavior for third-party code that *uses* NumPy, but this is basically inevitable for any proposal to allow's NumPy's public API to be overloaded. It's also avoided by default by third-party libraries that follow the current best practice of casting all input arrays with np.asarray(). -------------- RE: a hypothetical simplified interface: The need to implement everything you want to use in NumPy's public API could certainly be onerous, but on the other hand there are a long list of projects that have already done this today -- and these are the projects that most need __array_function__. I'm sure there are cases were simplification would be warranted, but in particular I don't think __array_concatenate__ has significant advantages over simply implementing __array_function__ for np.concatenate. It's a slightly different way of spelling, but it basically does the same thing. The level of complexity to implement hstack, vstack, row_stack and column_stack in terms of np.concatenate is pretty minimal. __array_function__ implementors could easily copy and paste code from NumPy or use a third-party helpers library (like NDArrayOperatorsMixin) that provides such implementations. I also have other concerns about the "simplified API" approach beyond the difficulty of figuring it out, but those are already mentioned in the NEP: http://www.numpy.org/neps/nep-0018-array-function-protocol.html#implementati... But... this is wishful thinking. No matter what the NEP says, I simply
I agree, but to be clear, development for dask, sparse and xarray (and even broadly supported machine learning libraries like TensorFlow) still happens at a much faster pace than is currently the case for "core" projects in the SciPy stack like NumPy. It would not be a big deal to encounter breaking changes in a "major" NumPy release (i.e., 1.X -> 1.(X+1)). (Side note: sklearn doesn't directly implement any array types, so I don't think it would make use of __array_function__ in any way, except possibly to implement overloadable functions.)
- makes it easy for end-users to check how much overhead this adds (by
I'm slightly concerned that the cost of reading an environment variable with os.environ could exaggerate the performance cost of __array_function__. It takes about 1 microsecond to read an environment variable on my laptop, which is comparable to the full overhead of __array_function__. So we may want to switch to an explicit Python API instead, e.g., np.enable_experimental_array_function(). My bigger concern is when/how we decide to graduate __array_function__ from requiring an explicit opt-in. We don't need to make a final decision now, but it would be good to clear about what specifically we are waiting for. I see three types of likely scenarios for changing __array_function__: 1. We decide that the overloading the NumPy namespace in general is a bad idea, based on either performance or predictability consequences for third-party libraries. In this case, I imagine we would probably keep __array_function__, but revert to a separate namespace for explicitly overloaded functions, e.g., numpy.api. 2. We want to keep __array_function__, but need a breaking change to the interface (and we're really attached to keeping the name __array_function__). 3. We decide that specific functions should use a different interface (e.g., switch from __array_function__ to __array_ufunc__). (1) and (2) are the sort of major concerns that in my mind would warrant hiding a feature behind an experimental flag. For the most part, I expect (1) could be resolved relatively quickly by running benchmark suites after we have a working version of __array_function__. To be honest, I don't see either of these rollback scenarios as terribly likely, but the downside risk is large enough that we may want to protect ourselves for a major release or two (6-12 months). (3) will be a much longer process, likely to stretch out over years at the current pace of NumPy development. I don't think we'll want to keep an opt-in flag for this long of a period. Rather, we may want to accept a shorter deprecation cycle than usual. In most cases, I suspect we could incrementally switch to new overloads while preserving the __array_function__ overload for a release or two. I don't really understand the 'types' frozenset. The NEP says "it will
The most flexible alternative would be to just say that we provide an fixed-length iterable, and return a tuple object. (In my microbenchmarks, it's faster to make a tuple than a list or set.) In an early draft of the NEP, I proposed exactly this, but the speed difference seemed really marginal to me. I included 'types' in the interface because I really do think it's something that almost all __array_function__ implementations should use use. It preserves a nice separation of concerns between dispatching logic and implementations for a new type. At least as long as __array_function__ is experimental, I don't think we should be encouraging people to write functions that could return NotImplemented directly and to rely entirely on the NumPy interface. Many but not all implementations will need to look at argument types. This is only really essential for cases where mixed operations between NumPy arrays and another type are allowed. If you only implement the NumPy interface for MyArray objects, then in the usual Python style you wouldn't need isinstance checks. It's also important from an ecosystem perspective. If we don't make it easy to get type information, my guess is that many __array_function__ authors wouldn't bother to return NotImplemented for unexpected types, which means that __array_function__ will break in weird ways when used with objects from unrelated libraries. Cheers, Stephan
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
Ping to finish up this discussion so we can come to a conclusion. I'm in favor of the NEP, as I don't see it as orthogonal to Nathaniel's concerns. However, we might want to be selective as to which functions we expose via the `__array_function__` method. On Wed, Aug 15, 2018 at 10:45 AM, Stephan Hoyer <shoyer@gmail.com> wrote:
Here is Travis Oliphant's talk at PyBay <https://speakerdeck.com/teoliphant/ml-in-python>, where he talks about the proliferation of arrays and interfaces in the ML/AI ecosystem among other things. I think that we should definitely try to get NumPy out there as an option in the near future.
Chuck
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Mon, Aug 20, 2018 at 8:27 AM Charles R Harris <charlesr.harris@gmail.com> wrote:
Chunk -- thanks for bringing this back up. My proposal is to hide this feature behind an environment variable for now, or something morally equivalent to an environment variable if that's too slow (i.e., an explicit Python variable). I don't think Nathaniel's concerns are entirely warranted for the reasons I went into in my earlier reply, but I do really want to get this moving forward now in whatever way is necessary. We can figure out the rest down the road. Nathaniel -- are you OK with that?
Yes, there is an urgent need for this :). Cheers, Stephan
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Wed, Aug 15, 2018 at 9:45 AM, Stephan Hoyer <shoyer@gmail.com> wrote:
I don't think I follow... if B uses A internally, then overriding A shouldn't cause B to break, unless the overridden A is buggy. The median() case was different: it wasn't overriding A that caused B to break, that part worked fine. It was when we changed the implementation of B that we had problems. ...actually, this made me realize that I was uncertain about what exactly happened in that case. I just checked, and AFAICT with current astropy the call to mean() is unnecessary. I tried modifying np.median to remove the call to mean, and it still gave the same result for np.median([1, 2, 3] * u.m). I also checked np.percentile, and it seems to work fine on units-arrays if you make it call np.asanyarray instead of np.asarray. The issue is here if anyone else wants to refresh their memory: https://github.com/numpy/numpy/issues/3846 Reading the comments there it looks like we might have hacked np.median so it (a) uses np.asanyarray, and (b) always calls np.mean, and actually the 'asanyarray' change is what astropy needed and the 'mean' part was just a red herring all along! Whoops. And here I've been telling this story for years now...
And when we fix a bug in row_stack, this means we also have to fix it in all the copy-paste versions, which won't happen, so np.row_stack has different semantics on different objects, even if they started out matching. The NDArrayOperatorsMixin reduces the number of duplicate copies of the same code that need to be updated, but 2 copies is still a lot worse than 1 copy :-).
Yeah, there are definitely trade-offs. I don't have like, knock-down rebuttals to these or anything, but since I didn't comment on them before I might as well say a few words :-).
1. The details of how NumPy implements a high-level function in terms of overloaded functions now becomes an implicit part of NumPy’s public API. For example, refactoring stack to use np.block() instead of np.concatenate() internally would now become a breaking change.
The way I'm imagining this would work is, we guarantee not to take a function that used to be implemented in terms of overridable operations, and refactor it so it's implemented in terms of overridable operations. So long as people have correct implementations of __array_concatenate__ and __array_block__, they shouldn't care which one we use. In the interim period where we have __array_concatenate__ but there's no such thing as __array_block__, then that refactoring would indeed break things, so we shouldn't do that :-). But we could fix that by adding __array_block__.
2. Array libraries may prefer to implement high level functions differently than NumPy. For example, a library might prefer to implement a fundamental operations like mean() directly rather than relying on sum() followed by division. More generally, it’s not clear yet what exactly qualifies as core functionality, and figuring this out could be a large project.
True. And this is a very general problem... for example, the appropriate way to implement logistic regression is very different in-core versus out-of-core. You're never going to be able to take code written for ndarray, drop in an arbitrary new array object, and get optimal results in all cases -- that's just way too ambitious to hope for. There will be cases where reducing to operations like sum() and division is fine. There will be cases where you have a high-level operation like logistic regression, where reducing to sum() and division doesn't work, but reducing to slightly-higher-level operations like np.mean also doesn't work, because you need to redo the whole high-level operation. And then there will be cases where sum() and division are too low-level, but mean() is high-level enough to make the critical difference. It's that last one where it's important to be able to override mean() directly. Are there a lot of cases like this? For mean() in particular I doubt it. But then, mean() in particular is irrelevant here, because mean() is already directly overridable, regardless of __array_function__ :-). So really the question is about the larger landscape of numpy APIs: What traps are lurking in the underbrush that we don't know about? And yeah, the intuition behind the "simplified API" approach is that we need to do the work to clear out that underbrush, and the downside is exactly that that will be a lot of work and take time. So... I think this criticism is basically that restated?
3. We don’t yet have an overloading system for attributes and methods on array objects, e.g., for accessing .dtype and .shape. This should be the subject of a future NEP, but until then we should be reluctant to rely on these properties.
This one I don't understand. If you have a duck-array object, and you want to access its .dtype or .shape attributes, you just... write myobj.dtype or myobj.shape? That doesn't need a NEP though so I must be missing something :-).
They don't implement array types, but they do things like use sparse arrays internally, so from the user's point of view you could have some code that only uses numpy and sklearn, and then the new numpy release breaks sklearn (because it broke the sparse package that sklearn was using internally).
That's why I said "at numpy import time" :-). I was imagining we'd check it once at import, and then from then on it'd be stashed in some C global, so after that the overhead would just be a single predictable branch 'if (array_function_is_enabled) { ... }'.
If we do this, then libraries that want to use __array_function__ will just call it themselves at import time. The point of the env-var is that our policy is not to break end-users, so if we want an API to be provisional and experimental then it's end-users who need to be aware of that before using it. (This is also an advantage of checking the envvar only at import time: it means libraries can't easily just setenv() to enable the functionality behind users' backs.)
The motivation for keeping it provisional is that we'll know more after we have some implementation experience, so our future selves will be in a better position to make this decision. If I knew what I was waiting for, I might not need to wait :-). But yeah, to be clear, I'm totally OK with the possibility that we'll do this for a few releases and then look again and be like "eh... now that we have more experience, it looks like the original plan was fine after all, let's remove the envvar and document some kind of accelerated deprecation cycle".
This is much more of a detail as compared to the rest of the discussion, so I don't want to quibble too much about it. (Especially since if we keep things really-provisional, we can change our mind about the argument later :-).) Mostly I'm just confused, because there are lots of __dunder__ functions in Python (and NumPy), and none of them take a special 'types' argument... so what's special about __array_function__ that makes it necessary/worthwhile? Any implementation of, say, concatenate-via-array_function is going to involve iterating through all the arguments and looking at each of them to figure out what kind of object it is and how to handle it, right? That's true whether or not they've done a "pre-check" using the types set, so in theory it's just as easy to return NotImplemented at that point. But I guess your point in the last paragraph is that this means there will be lots of chances to mess up the NotImplemented-returning code in particular, especially since it's less likely to be tested than the happy path, which seems plausible. So basically the point of the types set is to let people factor out that little bit of lots of functions into one common place? I guess some careful devs might be unhappy with paying extra so that other lazier devs can get away with being lazy, but maybe it's a good tradeoff for us (esp. since as numpy devs, we'll be getting the bug reports regardless :-)). If that's the goal, then it does make me wonder if there might be a more direct way to accomplish it -- like, should we let classes define an __array_function_types__ attribute that numpy would check before even trying to dispatch to __array_function__? -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
I’m +0 on removing it, so mostly neutral, but slightly in favour. While I see the argument for having it, I also see it as a violation of DRY... The information is already available in relevant arguments. I doubt any people implementing this protocol are going to be lazy enough not to implement a type check. So far, we’ve been good on __array_ufunc__. The part of me that says it’s good to have it is the mostly “squeeze every bit of performance out” part. Best Regards Hameer Abbasi Sent from my iPhone
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith <njs@pobox.com> wrote:
Let me try another example with arrays with units. My understanding of the contract provided by unit implementations is their behavior should never deviate from NumPy unless an operation raises an error. (This is more explicit for arrays with units because they raise errors for operations with incompatible units, but practically speaking almost all duck arrays will have at least some unsupported operations in NumPy's giant API.) It is quite possible that NumPy functions could be (re)written in a way that is incompatible with some unit implementations but is perfectly valid for "full" duck arrays. We actually see this even within NumPy already -- for example, see this recent PR adding support for the datetime64 dtype to percentile: https://github.com/numpy/numpy/pull/11627 A lesser case of this are changes in NumPy causing performance issues for users of duck arrays, which is basically inevitable if we share implementations. I don't think it's possible to anticipate all of these cases, and I don't want NumPy to be unduly constrained in its internal design. I want our user support answer to be simple: if you care about performance for a particular array operations on your type of arrays, you should implement it yourself (i.e., with __array_function__). This definitely doesn't preclude the careful, systematic overriding approach. But I think we'll almost always want NumPy's external API to be overridable. And when we fix a bug in row_stack, this means we also have to fix it
I see your point, but in all seriousness if encounter a bug in np.row_stack at this point we might just call it a feature instead.
""we guarantee not to take a function that used to be implemented in terms of overridable operations, and refactor it so it's implemented in terms of overridable operations" Did you miss a "not" in here somewhere, e.g., "refactor it so it's NOT implemented"? If we ever tried to do something like this, I'm pretty sure that it just wouldn't happen -- unless we also change NumPy's extremely conservative approach to breaking third-party code. np.block() is much more complex to implement than np.concatenate(), and users would resist being forced to handle that complexity if they don't need it. (Example: TensorFlow has a concatenate function, but not block.)
mean() is not entirely hypothetical. TensorFlow and Eigen actually do implement mean separately from sum, though to be honest it's not entirely clear to me why: https://github.com/tensorflow/tensorflow/blob/1c1dad105a57bb13711492a8ba5ab9... https://eigen.tuxfamily.org/dox/unsupported/TensorFunctors_8h_source.html I do think this probably will come up with some frequency for other operations, but the bigger answer here really is consistency -- it allows projects and their users to have very clearly defined dependencies on NumPy's API. They don't need to worry about any implementation details from NumPy leaking into their override of a function.
We don't have np.asduckarray() yet or whatever we'll end up calling our proposed casting function from NEP 22, so we don't have a fully fleshed out mechanism for NumPy to declare "this object needs to support .shape and .dtype, or I'm going to cast it into something that does". More comments on the environment variable and the interface to come in my next email... Cheers, Stephan
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Tue, Aug 21, 2018 at 9:39 AM, Stephan Hoyer <shoyer@gmail.com> wrote:
From your mention of "full" duck arrays I guess you're thinking of
I clicked the link, but I don't see anything about units? Of course units are a tricky example to make inferences from, because they aren't a good fit for the duck array concept in general. (In terms of numpy's core semantics, data-with-units is a special dtype, not a special container type.) this distinction?: http://www.numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html#princip... You're right: if numpy changes the implementation of some high-level function to use protocol-A instead of protocol-B, and there's some partial-duck-array that only implements protocol-B, then it gets broken. Of course, in general __array_function__ has the same problem: if sklearn changes their implementation of some function to call numpy function A instead of numpy function B, and there's a partial-duck-array that only implements numpy function B, then sklearn is broken. I think that as duck arrays roll out, we're just going to have to get used to dealing with breakage like this sometimes. The advantage of __array_function__ is that we get to ignore these issues within numpy itself. The advantage of having focused-protocols is that they make it easier to implement full duck arrays, and they give us a vocabulary for talking about degrees of partiality. For example, with __array_concatenate__, a duck array either supports all the concatenation/stacking operations or none of them – so sklearn never has to worry that switching between np.row_stack and np.stack will cause issues.
NumPy (and Python in general) is never going to make everything 100% optimized all the time. Over and over we choose to accept small inefficiencies in order to improve maintainability. How big are these inefficiencies – 1% overhead, 10% overhead, 10x overhead...? Do they show up everywhere, or just for a few key functions? What's the maintenance cost of making NumPy's whole API overrideable, in terms of making it harder for us to evolve numpy? What about for users dealing with a proliferation of subtly incompatible implementations? You may be right that the tradeoffs work out so that every API needs to be individually overridable and the benefits are worth it, but we at least need to be asking these questions.
Yeah, you're right, row_stack is a bad example :-). But of course the point is that it's literally any bug-fix or added feature in numpy's public API. Here's a better, more concrete example: back in 2015, you added np.stack (PR #5605), which was a great new feature. Its implementation was entirely in terms of np.concatenate and other basic APIs like .ndim, asanyarray, etc. In the smallish-set-of-designed-protocols world, as soon as that's merged into numpy, you're done: it works on sparse arrays, dask arrays, tensorflow tensors, etc. People can use it as soon as they upgrade their numpy. In the __array_function__ world, merging into numpy is only the beginning: now you have to go make new PRs to sparse, dask, tensorflow, etc., get them merged, released, etc. Downstream projects may refuse to use it until it's supported in multiple projects that have their own release cycles, etc. Or another example: at a workshop a few years ago, Matti put up some of the source code to numpypy to demonstrate what it looked like. I immediately spotted a subtle bug, because I happened to know that it was one we'd found and fixed recently. (IIRC it was the thing where arr[...] should return a view of arr, not arr itself.) Of course indexing for duck arrays is its own mess that's somewhat orthogonal to __array_function__, but the basic point is that numpy has a lot of complex error-prone semantics, and we are still actively finding and fixing issues in numpy's own implementations.
Yeah, sorry.
I agree, we probably wouldn't do this particular change.
When you say "consistency" here that means: "they can be sure that when they disagree with the numpy devs about the semantics/implementation of a numpy API, then the numpy API will consistently act the way they want, not the way the numpy devs want". Right? This is a very double-edged sword :-).
That's true, but it's just as big a problem for NEP 18, because __array_function__ is never going to do much if you've already coerced the thing to an ndarray. Some kind of asduckarray solution is basically a prerequisite to any other duck array features. -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
Hi Nathaniel and Stephan, Since this conversation is getting a bit lengthy and I see a lot of repeated stuff, I’ll summarise the arguments for everyone’s benefit and then present my own viewpoints: Nathaniel: Undue maintenance burden on NumPy, since semantics have to match exactly Implementations of functions may change, which may break downstream library compatibility There may be time taken in merging this everywhere, so why not take time to define proper protocols? Hide this entire interface behind an environment variable, possibly to be removed later. Stephan: Semantics don’t have to match exactly, that isn’t the intent of most duck-arrays. This won’t happen given NumPy’s conservativeness. The protocols will just be copies of __array_function__, but less capable Provide an interface that only end-users may turn on. My viewpoints: I don’t think any Duck array implementers intend to copy semantics on that level. Dask, which is the most complete one, doesn’t have views, only copies. Many other semantics simply don’t match. The intent is to allow for code that expresses, well, intent (no pun intended) instead of relying heavily on semantics, but that can use arbitrary duck-array implementations instead of just ndarray. Most of the implementations in NumPy are pretty stable, and the only thing that’s likely to happen here is bug fixes. And we are free to fix bugs those; I doubt implementation-specific bugs will be copied. However, these first two points are for/against duck arrays in general, and not specific to this protocol, so IMO this discussion is completely orthogonal to this one. I agree with Stephan here: Defining a minimum API for NumPy that will complete duck arrays will produce a lot of functions in every case that cannot be overridden, as they simply cannot be expressed in terms of the protocols we have added so far. This will lead to more protocols being produced, and so on ad infinitum. We have to consider the burden that such a design would place on the maintainers of NumPy as well… I personally feel that the amount of such protocols we’ll so need are large enough that this line of action is more burdensome, rather than less. I prefer an approach with __array_function__ + mailing list ping before adding a function. May I propose an alternative that was already discussed, and one that I think everyone will be okay with: We put all overridable functions inside a new submodule, numpy.api, that will initially be a shallow-ish copy of the numpy module. I say ish because all modules inside NumPy will need to be shallow-copied as well. If we need to add __array_function__, we can always do that there. Normal users are using “regular” NumPy unless they know they’re using the API, but it is separately accessible. As for hiding it completely goes: We have to realise, the Python computation landscape is fragmenting. The slower we are, the more fragmented it will become. NumPy already isn’t “the standard” for machine learning. Regards, Hameer Abbasi
![](https://secure.gravatar.com/avatar/5f88830d19f9c83e2ddfd913496c5025.jpg?s=120&d=mm&r=g)
On Wed, Aug 22, 2018 at 4:22 AM Hameer Abbasi <einstein.edison@gmail.com> wrote:
May I propose an alternative that was already discussed, and one that I think everyone will be okay with:
That's a dangerous assumption on this list:) We put all overridable functions inside a new submodule, numpy.api, that
will initially be a shallow-ish copy of the numpy module.
This is not desirable. There are projects (e.g. statsmodels) that have added a .api submodule before. It's generally considered not a good idea, it's not very Pythonic. Everything one can import that doesn't have an underscore is normally part of the API of a package. In this particular case, I definitely prefer an envvar and relying on what is documented as part of __array_function__ rather than a new namespace. Cheers, Ralf I say ish because all modules inside NumPy will need to be shallow-copied
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Tue, Aug 21, 2018 at 6:47 PM Nathaniel Smith <njs@pobox.com> wrote:
To clarify: np.datetime64 arrays can be considered a variant of NumPy arrays that support units. Namely, they use a dtype for representing time units. I expect that the issues we've encountered with datetime64 will be indicative of some of the sort of issues that authors of unit-aware arrays will encounter.
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith <njs@pobox.com> wrote:
Indeed, I missed the "at numpy import time" bit :). In that case, I'm concerned that it isn't always possible to set environment variables once before importing NumPy. The environment variable solution works great if users have full control of their own Python binaries, but that isn't always the case today in this era of server-less infrastructure and online notebooks. One example offhand is Google's Colaboratory ( https://research.google.com/colaboratory), a web based Jupyter notebook. NumPy is always loaded when a notebook is opened, as you can check from inspecting sys.modules. Now, I work with the developers of Colaboratory, so we could probably figure out a work-around together, but I'm pretty sure this would also come up in the context of other tools. Another problem is unit testing. Does pytest use a separate Python process for running the tests in each file? I don't know and that feels like an implementation detail that I shouldn't have to know :). Yes, in principle I could use a subprocess in my __array_function__ for unit tests, but that would be really awkward.
I'm in complete agreement that only authors of end-user applications should invoke this option, but just because something is technically possible doesn't mean that people will actually do it or that we need to support that use case :). numpy.seterr() is a good example. It allows users to globally set how NumPy does error handling, but well written libraries still don't do that. TensorFlow has similar function tf.enable_eager_execution() for enabling "eager mode" that is also worth examining: https://www.tensorflow.org/api_docs/python/tf/enable_eager_execution To solve the testing issue, they wrote decorator for using with tests, run_in_graph_and_eager_modes(): https://www.tensorflow.org/api_docs/python/tf/contrib/eager/run_test_in_grap...
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Tue, Aug 21, 2018 at 6:12 PM, Stephan Hoyer <shoyer@gmail.com> wrote:
I mean, the idea of the envvar is to be a temporary measure enable devs to experiment with a provisional feature, while being awkward enough that people don't build lots of stuff assuming its there. It doesn't have to 100% supported in every environment.
Set the envvar before invoking pytest? For numpy itself we'll need to write a few awkward tests involving subprocesses to make sure the envvar parsing is working properly, but I don't think this is a big deal. As long as we only have 1-2 places that __array_function__ dispatch funnels through, we just need to make sure that they work properly with/without the envvar; no need to test every API separately. Or if it is an issue we can have some private API that's only available to the numpy test suite...
I didn't say "authors of end-user applications", I said "end-users" :-). That said, I dunno. My intuition is that if we have a function call like this then libraries that define __array_function__ will merrily call it in their package __init__ and it accomplishes nothing, but maybe I'm being too cynical and untrusting. -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Tue, Aug 21, 2018 at 6:57 PM Nathaniel Smith <njs@pobox.com> wrote:
My understanding of the idea of the envvar is to obtain informed consent from NumPy users, e.g., "I understand that this is a unsupported experimental feature that may be removed in the future without warning." It's pretty important for me personally that's it's possible to use this in a flexible set of environments, and in particular to have something that works in my preferred notebook environment. How else are we going to test this? Every limitation that we put into the experimental version of this feature decreases the likelihood that it gets used enough to know if it's even a viable solution. If it's too awkward, nobody's even going to bother testing it, and this whole effort will fall flat on its face.
These are mostly the same for NumPy, but I do disagree with you here. Ultimately we have to trust application developers to make the right choices for their tools. If they are willing to accept that maintenance burden of either (1) potentially being stuck on NumPy 1.16 forever or (2) needing to rewrite their code, that's their tradeoff to make. It's a little preposterous to force this decision onto end-users, who may not even know a tool is written in NumPy.
People can do lots of dumb things in Python (e.g., monkeypatching) -- the language doesn't stop them. Fortunately this mostly isn't a problem.
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
Hi everyone,
I might add that most duck array authors are highly unlikely to be newcomers to the Python space. We should just put a big warning there while enabling and that’ll be enough to scare away most devs from doing it by default.
Best Regards, Hameer Abbasi Sent from my iPhone
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Thu, Aug 23, 2018 at 9:02 AM, <einstein.edison@gmail.com> wrote:
That's a reasonable idea... a Big Obnoxious Warning(tm) when it's enabled, or on first use, would achieve a lot of the same purpose. E.g. if this_is_the_first_array_function_usage(): sys.stderr.write( "WARNING: this program uses NumPy's experimental '__array_function__' feature.\n" "It may change or be removed without warning, which might break this program.\n" "For details see http://www.numpy.org/neps/nep-0018-array-function-protocol.html\n" ) -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Fri, Aug 24, 2018 at 1:36 AM Hameer Abbasi <einstein.edison@gmail.com> wrote:
Issuing a FutureWarning seems roughly appropriate here. The Python 3.7 docs write: "Base category for warnings about deprecated features when those warnings are intended for end users of applications that are written in Python." Writing to sys.stderr directly is generally considered poor practice for a Python libraries. In my experience FutureWarning does a good job of satisfying the goals of being a "Big Obnoxious Warning" while still being silence-able and testable with standard tools.
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Fri, Aug 24, 2018 at 1:46 PM, Stephan Hoyer <shoyer@gmail.com> wrote:
Yeah, the reason warnings are normally recommended is because normally, you want to make it easy to silence. But this is the rare case where I didn't want to make it easy to silence, so I didn't suggest using a warning :-). Calling warnings.warn (or the C equivalent) is also very expensive, even if the warning ultimately isn't displayed. I guess we could do our own tracking of whether we've displayed the warning yet, and only even attempt to issue it once, but that partially defeats the purpose of using warnings in the first place. -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Fri, Aug 24, 2018 at 3:14 PM Nathaniel Smith <njs@pobox.com> wrote:
I thought the suggestion was to issue a warning when np.enable_experimental_array_function() is called. I agree that it's a non-starter to issue it every time an __array_function__ method is called -- warnings are way too slow for that. People can redirect stderr, so we're really not stopping anyone from silencing things by doing it in a non-standard way. We're just making it annoying and non-standard. Developers could even run Python in a subprocess and filter out all the warnings -- there's really nothing we can do to stop determined abusers of this feature. I get that you want to make this annoying and non-standard, but this is too extreme for me. Do you seriously imagine that we'll consider ourselves beholden in the future to users who didn't take us at our word?
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Fri, Aug 24, 2018 at 4:00 PM, Stephan Hoyer <shoyer@gmail.com> wrote:
If our protection against uninformed usage is a Big Obnoxious Warning(tm), then I was imagining that we could simplify by dropping enable_experimental_array_function entirely. Doesn't make a big difference either way though.
Let's break that question down into two parts: 1. if we do find ourselves in a situation where changing this would break lots of users, will we consider ourselves beholden to them? 2. is it plausible that we'll find ourselves in that situation? For the first question, I think the answer is... yes? We constantly bend over backwards to try to avoid breaking users. Our deprecation policy explicitly says that it doesn't matter what we say in the docs, the only thing that matters is whether a change breaks users. And to make things more complicated, it's easy to imagine scenarios where the people being broken aren't the ones who had a chance to read the docs – e.g. if a major package starts relying on __array_function__, then it's all *their* users who we'd be breaking, even though they had nothing to do with it. If any of {tensorflow/astropy/dask/sparse/sklearn/...} did start relying on __array_function__ for normal functionality, then *of course* that would come up in future discussions about changing __array_function__, and *of course* it would make us reluctant to do that. As it should, because breaking users is bad, we should try to avoid ending up in situations where that's what we have to do, even if we have a NEP to point to to justify it. But... maybe it's fine anyway, because this situation will never come up? Obviously I hope that our downstreams are all conscientious, and friendly, and take good care of their users, and would never create a situation like that. I'm sure XArray won't :-). But... people are busy, and distracted, and have pressures to get something shipped, and corners get cut. Companies *care* about what's right, but they mostly only *do* the minimum they have to. (Ask anyone who's tried to get funding for OSS...) Academics *care* about what's right, but they just don't have time to care much. So yeah... if there's a quick way to shut up the warning and make things work (or seem to work, temporarily), versus doing things right by talking to us, then I do think people might take the quick hack. The official Tensorflow wheels flat out lie about being manylinux compatible, and the Tensorflow team has never talked to anyone about how to fix this, they just upload them to PyPI and leave others get to deal with the fallout [1]. That may well be what's best for their users, I don't know. But stuff like this is normal, it happens all the time, and if someone does it with __array_function__ then we have no leverage. -n [1] https://github.com/tensorflow/tensorflow/issues/8802#issuecomment-401703703 -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/72f994ca072df3a3d2c3db8a137790fd.jpg?s=120&d=mm&r=g)
On 29/08/18 10:37, Nathaniel Smith wrote: trouble if we were proposing a broad change to something like indexing or the random number module (see those NEPs). If we break one of those major packages, it is on them to pin the version of NumPy they can work with. In my opinion very few end users will be implementing their own ndarray classes with `__array_function__`. While we will get issue reports, we can handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem and urge users to complain to those packages. Other than adding a warning, I am not sure what the concrete proposal is here. To not accept the NEP? Matti
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
One thing that might help here is nightly or continuous CI builds of the NumPy wheels. This would be good, as we could test it in CI, and fix it when it comes up. But I guess that’s another discussion. Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime.
![](https://secure.gravatar.com/avatar/8749ec52cee260c4c1f67f2dec29d957.jpg?s=120&d=mm&r=g)
1. if we do find ourselves in a situation where changing this would break lots of users, will we consider ourselves beholden to them?
I think that it would be useful for Numpy's continued evolution to develop the ability to include code on a provisional basis. Other projects do this and they just have big bold "Experimental" notes everywhere that a user might go to learn about the functionality (docs, docstrings, blogposts). Some users will definitely get burned, yes, but the alternative is to burn all other users a little bit by moving too slowly and not trying things out. This is different from how Numpy has operated in the past, but that might be ok.
2. is it plausible that we'll find ourselves in that situation?
Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime
Yup. Same for Dask. We're pretty accustomed to this. On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi <einstein.edison@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/851ff10fbb1363b7d6111ac60194cc1c.jpg?s=120&d=mm&r=g)
HI All, On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain, i.e., that it will continue to be possible to override, say, concatenate. It is not a big deal if the actual implementation changes (say, `__array_concatenate__` is introduced) and we need to do some version-dependent magic to ensure that our users do not notice the change; we did the same with `__array_ufunc__` - the wrap/prepare machinery will be gone only in the next version of astropy, when our minimum version for numpy reaches 1.13. One thing perhaps worth remembering that what really matters is not so much `__array_function__` but the set of functions that actually implement the override. It would seem unlikely this would be the complete numpy API on the first go; perhaps it is an idea to be somewhat specific about which part we start with? My vote would go towards array manipulation routines like concatenate. I would also suggest to avoid those functions for which ufunc implementations would seem quite possible (i.e., avoid things like median, etc.) All the best, Marten On Wed, Aug 29, 2018 at 8:31 AM Matthew Rocklin <mrocklin@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/8749ec52cee260c4c1f67f2dec29d957.jpg?s=120&d=mm&r=g)
My guess is that you wouldn't have this expectation if Numpy released this feature with explicit "Experimental" warnings attached to it. In that case astropy might just wait before adopting it until that label was dropped. Projects that were more risk-friendly would then take on the burden of trying it out, working out some kinks, and then hopefully in a version or two the "Experimental" label would be dropped and astropy would step in and adopt it more safely. This would give the ecosystem an opportunity to try something and modify it after having some experience without promising to support it in a fully backwards compatible way forever. On Wed, Aug 29, 2018 at 9:46 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/851ff10fbb1363b7d6111ac60194cc1c.jpg?s=120&d=mm&r=g)
On Wed, Aug 29, 2018 at 9:53 AM Matthew Rocklin <mrocklin@gmail.com> wrote:
I do think that even for an experimental feature one should be allowed to expect that there will continue to be a way provided by numpy to access the same functionality, i.e., once we allow people to do `np.concatenate(list-of-mimics)`, I think we should consider ourselves committed to providing some way to continue doing that - the experimental tag should only imply that we are not committed to the precise method with which that can be achieved. (Of course, I can be a bit more cavalier than most, since I am hopeful I can help ensure that some method will continue to be present; indeed, the same happened for __array_ufunc__ - I had already written a __numpy_ufunc__ implementation in Quantity and when that stalled in numpy, I picked up and finished the __array_ufunc__ code that Nathan had written.) -- Marten
![](https://secure.gravatar.com/avatar/49c5ca40df804ebf053e121a6104f405.jpg?s=120&d=mm&r=g)
Well, I guess I'll be proving Nathaniel right: I would *definitely* start using __array_function__ in astropy - not being able to concatenate Quantity and other instances which use it has been a long-standing pain.
That's fair enough, but I think Matt's point still stands. Any given project can weigh the benefits of early adoption against the risks, considering its need for a new feature and its resources available to deal with possible future changes. An "Experimental" designation gives numpy a little space to innovate and lets individual projects opt in. Without that space, numpy has to take on all the burden of continuity instead of sharing that burden with the community of projects that opt in to early adoption. Best, Dan Daniel B. Allan, Ph.D Associate Computational Scientist, Brookhaven National Lab (631) 344-3281 (no voicemail set up) ________________________________ From: NumPy-Discussion <numpy-discussion-bounces+dallan=bnl.gov@python.org> on behalf of Marten van Kerkwijk <m.h.vankerkwijk@gmail.com> Sent: Wednesday, August 29, 2018 10:43:55 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol On Wed, Aug 29, 2018 at 9:53 AM Matthew Rocklin <mrocklin@gmail.com<mailto:mrocklin@gmail.com>> wrote:
On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain,
My guess is that you wouldn't have this expectation if Numpy released this feature with explicit "Experimental" warnings attached to it. In that case astropy might just wait before adopting it until that label was dropped. Projects that were more risk-friendly would then take on the burden of trying it out, working out some kinks, and then hopefully in a version or two the "Experimental" label would be dropped and astropy would step in and adopt it more safely. Well, I guess I'll be proving Nathaniel right: I would *definitely* start using __array_function__ in astropy - not being able to concatenate Quantity and other instances which use it has been a long-standing pain. I do think that even for an experimental feature one should be allowed to expect that there will continue to be a way provided by numpy to access the same functionality, i.e., once we allow people to do `np.concatenate(list-of-mimics)`, I think we should consider ourselves committed to providing some way to continue doing that - the experimental tag should only imply that we are not committed to the precise method with which that can be achieved. (Of course, I can be a bit more cavalier than most, since I am hopeful I can help ensure that some method will continue to be present; indeed, the same happened for __array_ufunc__ - I had already written a __numpy_ufunc__ implementation in Quantity and when that stalled in numpy, I picked up and finished the __array_ufunc__ code that Nathan had written.) -- Marten
![](https://secure.gravatar.com/avatar/851ff10fbb1363b7d6111ac60194cc1c.jpg?s=120&d=mm&r=g)
Absolutely fine to have to deal with future chances - my main point is that by accepting the NEP, I think numpy is committing to provide some way to override whatever functions __array_function__ is introduced for, i.e., we cannot reasonably go back to not providing any way to override such a function (but by marking the feature experimental, we *can* change the way the override of any given function is implemented). -- Marten p.s. And, yes, do count me in for helping to back up that commitment!
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Wed, Aug 29, 2018 at 9:34 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
I'm not entirely sure that we're ready to make this commitment yet, but I think this ties into our decision about what sort of user opt-ins to require. I'd like to structure this decision in terms of the types of breaking changes we contemplate possibly making in this future with regards to this protocol: 1. Abandoning this approach of allowing overrides for NumPy's high level API entirely, in favor of requiring the use of a dedicated namespace like numpy.api. This might arise if we decide that the overhead of checking for __array_function__ attributes is too much, or if it's simply too confusing to allow NumPy functions to be overloaded in arbitrary ways. 2. Changing the general __array_function__ protocol in a breaking way, e.g., to eliminate the "types" argument. 3. Replacing __array_function__ with another override mechanism, either (a) in general (e.g., __array_function_v2__ without the "types" argument) or (b) for specific functions (e.g., switching to a more specific protocol like __array_ufunc__). 4. Removing the possibility of overriding specific functions altogether, because we want to require using a more generic interface (e.g., to require overriding np.stack implicitly via np.expand_dims/np.concatenate instead). Possible changes (1) or (2) would be good reasons for requiring end-users to make an intentional choice to enable __array_function__. Either of these changes would break everyone who uses the protocol. Concern about change (1) would be a particularly good reason to require an opt-in, because it's concerned about impact on users who don't use any new functionality. Possible changes (3) or (4) would be annoying to users, but in my mind would not justify requiring an explicit opt-in. Both could be achieved without immediately breaking users by making use of a standard deprecation cycle, i.e., by still calling __array_function__ but issuing a FutureWarning. The main reason for requiring the explicit opt-in would be if we don't want to need to bother with a deprecation cycle. To be clear, with the exception of change (3)(b), I don't think any of these are particularly likely or desirable outcomes. So in my personal opinion, I don't think we need the explicit opt-in. But I'm willing to make to go ahead with requiring the opt-in if that's what it takes to get everyone on board -- as long as we don't make the opt-in too onerous in a way that imposes hard limits on this protocol can be used. (I would be OK with an explicit Python API for enabling this feature that raises a FutureWarning when it's called, without any API for turning it off.) Best, Stephan
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Wed, Aug 29, 2018, 02:44 Matti Picus <matti.picus@gmail.com> wrote:
The proposal is just that while the NEP is considered experimental and provisional, we should use some kind of technical measures to discourage use in a non-experimental settings. We want to stay in control of when it escapes the lab, and docs alone, or trivially disableable messages, aren't a very effective way to do that. -n
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Wed, Aug 29, 2018 at 1:38 AM Nathaniel Smith <njs@pobox.com> wrote:
Yes, this is a good example of how library authors sometimes break rules imposed by their dependencies when not enforced by technical means. I'd like to think the TensorFlow team was making an intentional choice here and is willing to live with the consequence of their decision. Unfortunately right now many of those consequences are imposed on others, so this was certainly an inconsiderate choice. I don't know if they were aware of these costs that they are imposing on the rest of the Python ecosystem. I don't see any mention in PEP-513 about the consequences of lying about manylinux compatibility -- even though arguably such warnings should not be necessary. One lesson we might take from this is documentation is essential: we should have more prominent warnings in NEP-18 (given that it's also likely to serve as a form of user-facing docs) about how experimental this protocol is, and about the specific types of changes we anticipate in the future (e.g., see my other email). I also hope that Python packaging maintainers would not hesitate to break this abuse of the manylinux1 tag, e.g., if they wanted to add compatibility checks on wheels when uploaded to PyPI.
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
RE: the types argument On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith <njs@pobox.com> wrote:
What's special about __array_function__ is that it's a hook that lets you override an entire API through a single interface. Unlike protocols like __add__, implementers of __array_function__ don't know exactly which arguments could have implemented the operation.
It's also a pragmatic choice: libraries like dask.array and autograd.numpy have already implemented NumPy's API without overrides. These projects follow the current numpy convention: non-native array objects are coerced into native arrays (i.e., dask or autograd arrays). They don't do any type checking. I doubt there would be much appetite for writing alternative versions of these APIs that return NotImplemented instead -- especially while this feature remains experimental.
The only extra amount we pay extra is the price of converting these types into a Python data structure and passing them into the __array_function__ method call. We already had to collect them for __array_function__ itself to identify unique types to call -- so this is a pretty minimal extra cost.
This could potentially work, but now the __array_function__ protocol itself becomes more complex and out of sync with __array_ufunc__. It's a much smaller amount of additional complexity to add an additional passed argument.
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Thu, Aug 23, 2018 at 1:06 PM Hameer Abbasi <einstein.edison@gmail.com> wrote:
I don't follow -- can you please elaborate? If you don't want to do anything with the 'types' argument, you can simply ignore it. The problem of identifying whether arguments have valid types or not remains unchanged from the situation with __add__ or __array_ufunc__. 'types' just gives you another optional tool to help solve it.
![](https://secure.gravatar.com/avatar/1198e2d145718c841565712312e04227.jpg?s=120&d=mm&r=g)
On Fri, Aug 24, 2018 at 5:55 PM Stephan Hoyer <shoyer@gmail.com> wrote:
If we make specifying __array_function_types__ a mandatory part -- And such that it is a whitelist, the XArray or Dask would need to import sparse in order to specify that they accept mixing sparse arrays with native arrays (i.e. for adding sparse.SparseArray to __array_function_types__). Which is basically what I mean. It might be a 'soft' dependency, but there will be a dependency nonetheless.
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Fri, Aug 24, 2018, 09:07 Hameer Abbasi <einstein.edison@gmail.com> wrote:
Oh yeah, if we did this then we definitely wouldn't want to make it mandatory. Some `__array_function__` implementations might want to do checking another way, or support different types in different overloaded functions, or be able to handle arbitrary types. -n
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer <shoyer@gmail.com> wrote:
Skipping over all the discussion following this for brevity. I find myself in general agreement with Stephan and think we should go ahead and merge this. A few points: 1. I don't think we should have an opt in environment variable, just put `__array_function__` out there with the understanding that it might change with experience. I don't think scores, let alone thousands, of folks are going to rush to take advantage of this, most likely the developers of the proposal will be the early adopters. 2. I have a preference for implementing high level functions rather than low level, and fewer rather than more, with the choice driven by need. I think that will limit the amount of interdependence tangle and, in the long run, might serve to define an unofficial NumPy API. 3. NumPy cannot manage all the projects that make use of the new option to keep them in sync, we have neither the time nor experience to do so, and historic attempts along those lines tended to die in obscurity. To the extent that we do so, it comes down to 2. 4. I don't think this conflicts with Nathaniel's proposal, the main disagreement seems to over how to proceed and the selection of functions, see 2. 5. The `__array_function_types__` idea looks interesting. Chuck
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Sat, Sep 8, 2018 at 7:12 PM Charles R Harris <charlesr.harris@gmail.com> wrote:
I rolled back Chuck's merge of the "Acceptance" PR for this NEP because (1) it was not clear that if had reached consensus and (2) I still wanted to make a few changes based on this discussion. I have now drafted these revisions to the NEP to clarify its stance around backwards compatibility, and the type of the "types" argument: https://github.com/numpy/numpy/pull/11943 I have *not* included any form of the explicit "end-user opt-in" requested by Nathaniel. I don't think it is warranted based on the scope of anticipated future changes; see my earlier post [1] for details. Nobody has seriously suggested that we would release this functionality and later eliminate the ability to override NumPy functions entirely in the future. Of course, requiring an onerous explicit opt-in would be fine while this feature is initially under development, and until we are sure that checks for overriding arbitrary NumPy functions are fast enough to be generally viable. In particular, we should verify that __array_function__ does not meaningfully impact performance for major downstream components of the SciPy stack. But I think running standard benchmark suites (e.g., with ASV) and user testing of release candidates would be enough. If you have still have major objections to this version of proposal, please raise them unambiguously -- preferably with an formal veto [2]. Best, Stephan [1] https://mail.python.org/pipermail/numpy-discussion/2018-August/078669.html [2] https://docs.scipy.org/doc/numpy/dev/governance/governance.html
![](https://secure.gravatar.com/avatar/851ff10fbb1363b7d6111ac60194cc1c.jpg?s=120&d=mm&r=g)
Thanks for adding the clarifications. They read well to me, and I think make clear to a project like astropy what to expect (and I expect we'll be use it!). -- Marten
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Thu, Sep 13, 2018 at 10:30 AM Stephan Hoyer <shoyer@gmail.com> wrote:
Again, if anyone (yes, I'm looking at you, Nathaniel!) still has objections please speak up soon. If I don't hear anything, I'm going submit a PR to mark this proposal as "Accepted" tomorrow (one week after these latest revisions). As Chuck noted, "Tentatively accepted" or "Experimental" would probably be a better status, but that currently isn't one of our options for NEPs. In any case, I would love to move on from debate to implementation! As a side note, I recently noticed a CuPy pull request ( https://github.com/cupy/cupy/pull/1650 <https://github.com/cupy/cupy/pull/1650/files>) to add __array_function__. It's a testament to our proposed interface that the full PR is 9 lines of code without tests.
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Thu, Sep 13, 2018 at 10:30 AM, Stephan Hoyer <shoyer@gmail.com> wrote:
Okay, so this is a pretty substantial change! Before, the NEP's stance was "we might change anything, at any time, without any warning", which of course makes it easier to accept the NEP (since we can always back out), but was also so different from our normal rules that it seemed important to make sure people weren't using it without realizing. Now it actually makes a commitment: to not regress on what functions can be overloaded (though the details might change), and commits to an abbreviated-but-nonzero deprecation process when we change things. I get the impression that this is closer to what the authors were intending in the first place, so that's good! I would probably have kept the noisy warning and zero commitments for one release anyway, because IMO it's not a big deal and it rarely hurts to hedge bets and gather data. But on reflection, I think I am OK with this level of commitment if that's what y'all want to go for. (After all, it's not really any stronger than NEP 22's high-level plan.) So, +0. -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Thu, Sep 20, 2018 at 2:05 PM Stephan Hoyer <shoyer@gmail.com> wrote:
I have started implementing NEP-18 in a pull request: https://github.com/numpy/numpy/pull/12005 I propose a multi-step process for implementation: 1. Write a correctly working version of the core __array_function__ machinery in pure Python, adapted from the example implementation in the NEP. 2. Rewrite bottlenecks in C. 3. Implement overrides for various NumPy functions. I think the first step is mostly complete, but I'm particularly looking for help with the second and third steps, which should be able to happen incrementally and in parallel. Cheers, Stephan
![](https://secure.gravatar.com/avatar/7a7a675bc479a30c1389d4c64f612a95.jpg?s=120&d=mm&r=g)
Hi all, I have small question for clarification regarding some reducing numpy functions like np.sum, np.mean, etc, i.e. functions that now work on other array-like objects by checking if they have 'sum', 'mean', etc method and if so, "dispatch" to the method. Those functions are several times mentioned as examples in the text of the NEP, so although the NEP does not list the functions that will use the __array_function__ protocol explicitly, I assume that those functions will probably be in that list. But I wanted to check if that assumption is correct. Is it the intent that the functions that now dispatch to object methods will be updated to use the protocol instead? (probably keeping to method-dispatch as second try for backwards compatibility) Not something urgent, but I was just wondering that when reading the NEP, as this is something that would be useful for pandas (where we now need to do some gymnastics to deal with arguments passed by numpy to those methods that we don't support). Joris
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Wed, Sep 26, 2018 at 9:19 AM Joris Van den Bossche < jorisvandenbossche@gmail.com> wrote:
Hi Joris, Yes, this is my exact intent. Eventually, we might remove/deprecate automatically calling methods with the same name (due to how error prone this is), but that is a long way in the future if ever. Keep in mind that the current version of the NEP may not be a good fit for pandas, because it requires overriding *all* NumPy functions. See this section of the NEP for details: http://www.numpy.org/neps/nep-0018-array-function-protocol.html#coercion-to-... Cheers, Stephan
participants (12)
-
Allan, Daniel
-
Charles R Harris
-
einstein.edison@gmail.com
-
Hameer Abbasi
-
Joris Van den Bossche
-
Marten van Kerkwijk
-
Matthew Brett
-
Matthew Rocklin
-
Matti Picus
-
Nathaniel Smith
-
Ralf Gommers
-
Stephan Hoyer