draft NEP for breaking ufunc ABI in a controlled way
Hi all, Here's a first draft NEP for comments. -- Synopsis ======== Improving numpy's dtype system requires that ufunc loops start having access to details of the specific dtype instance they are acting on: e.g. an implementation of np.equal for strings needs access to the dtype object in order to know what "n" to pass to strncmp. Similar issues arise with variable length strings, missing values, categorical data, unit support, datetime with timezone support, etc. -- this is a major blocker for improving numpy. Unfortunately, the current ufunc inner loop function signature makes it very difficult to provide this information. We might be able to wedge it in there, but it'd be ugly. The other option would be to change the signature. What would happen if we did this? For most common uses of the C API/ABI, we could do this easily while maintaining backwards compatibility. But there are also some rarely-used parts of the API/ABI that would be prohibitively difficult to preserve. In addition, there are other potential changes to ufuncs on the horizon (e.g. extensions of gufuncs to allow them to be used more generally), and the current API exposure is so massive that any such changes will be difficult to make in a fully compatible way. This NEP thus considers the possibility of closing down the ufunc API to a minimal, maintainable subset of the current API. To better understand the consequences of this potential change, I performed an exhaustive analysis of all the code on Github, Bitbucket, and Fedora, among others. The results make me highly confident that of all the publically available projects in the world, the only ones which touch the problematic parts of the ufunc API are: Numba, dynd-python, and `gulinalg <https://github.com/ContinuumIO/gulinalg>`_ (with the latter's exposure being trivial). Given this, I propose that for 1.11 we: 1) go ahead and hide/disable the problematic parts of the ABI/API, 2) coordinate with the known affected projects to minimize disruption to their users (which is made easier since they are all projects that are almost exclusively distributed via conda, which enforces strict NumPy ABI versioning), 3) publicize these changes widely so as to give any private code that might be affected a chance to speak up or adapt, and 4) leave the "ABI version tag" as it is, so as not to force rebuilds of the vast majority of projects that will be unaffected by these changes. This NEP defers the question of exactly what the improved API should be, since there's no point in trying to nail down the details until we've decided whether it's even possible to change. Details ======= The problem ----------- Currently, a ufunc inner loop implementation is called via the following function prototype:: typedef void (*PyUFuncGenericFunction) (char **args, npy_intp *dimensions, npy_intp *strides, void *innerloopdata); Here ``args`` is an array of pointers to 1-d buffers of input/output data, ``dimensions`` is a pointer to the number of entries in these buffers, ``strides`` is an array of integers giving the strides for each input/output array, and ``innerloopdata`` is an arbitrary void* supplied by whoever registered the ufunc loop. (For gufuncs, extra shape and stride information about the core dimensions also gets packed into the ends of these arrays in a somewhat complicated way.) There are 4 key items that define a NumPy array: data, shape, strides, dtype. Notice that this function only gets access to 3 of them. Our goal is to fix that. For example, a better signature would be:: typedef void (*PyUFuncGenericFunction_NEW) (char **data, npy_intp *shapes, npy_intp *strides, PyArray_Descr *dtypes, /* NEW */ void *innerloopdata); (In practice I suspect we might want to make some more changes as well, like upgrading gufunc core shape/strides to proper arguments instead of tacking it onto the existing arrays, and adding an "escape valve" void* reserved for future extensions. But working out such details is outside the scope of this NEP; the above will do for illustration.) The goal of this NEP is to clear the ground so that we can start supporting ufunc inner loops that take dtype arguments, and make other enhancements to ufunc functionality going forward. Proposal -------- Currently, the public API/ABI for ufuncs consists of the functions:: PyUFunc_GenericFunction PyUFunc_FromFuncAndData PyUFunc_FromFuncAndDataAndSignature PyUFunc_RegisterLoopForDescr PyUFunc_RegisterLoopForType PyUFunc_ReplaceLoopBySignature PyUFunc_SetUsesArraysAsData together with direct access to PyUFuncObject's internal fields:: typedef struct { PyObject_HEAD int nin, nout, nargs; int identity; PyUFuncGenericFunction *functions; void **data; int ntypes; int check_return; const char *name; char *types; const char *doc; void *ptr; PyObject *obj; PyObject *userloops; int core_enabled; int core_num_dim_ix; int *core_num_dims; int *core_dim_ixs; int *core_offsets; char *core_signature; PyUFunc_TypeResolutionFunc *type_resolver; PyUFunc_LegacyInnerLoopSelectionFunc *legacy_inner_loop_selector; PyUFunc_InnerLoopSelectionFunc *inner_loop_selector; PyUFunc_MaskedInnerLoopSelectionFunc *masked_inner_loop_selector; npy_uint32 *op_flags; npy_uint32 iter_flags; } PyUFuncObject; Obviously almost any future changes to how ufuncs work internally will involve touching some part of this public API/ABI. Concretely, the proposal here is that we avoid this by disabling the following functions (i.e., any attempt to call them should simply raise a ``NotImplementedError``):: PyUFunc_ReplaceLoopBySignature PyUFunc_SetUsesArraysAsData and that we reduce the publicly visible portion of PyUFuncObject down to:: typedef struct { PyObject_HEAD int nin, nout, nargs; } PyUFuncObject; Data on current API/ABI usage ----------------------------- In order to assess how much code would be affected by this proposal, I used a combination of Github search and Searchcode.com to trawl through the majority of all publicly available open source code. Neither search tool provides a fine-grained enough query language to directly tell us what we want to know, so I instead followed the strategy of first, casting a wide net: picking a set of search terms that are likely to catch all possibly-broken code (together with many false positives), and second, using automated tools to sift out the false positives and see what remained. Altogether, I reviewed 4464 search results. The tool I wrote to do this is `available on github <https://github.com/njsmith/codetrawl>`_, and so is `the analysis code itself <https://github.com/njsmith/ufunc-abi-analysis>`_. Uses of PyUFuncObject internals ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There are no functions in the public API which return ``PyUFuncObject*`` values directly, so any code that access PyUFuncObject fields will have to mention that token in the course of defining a variable, performing a cast, setting up a typedef, etc. Therefore, I searched Github for all files written in C, C++, Objective C, Python, or Cython, which mentioned either "PyUFuncObject AND types" or "PyUFuncObject AND NOT types". (This is to work around limitations on how many results Github search is willing to return to a single query.) In addition, I searched for ``PyUFuncObject`` on searchcode.com. The full report on these searches is available here: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/pyufuncobject-r... The following were screened out as non-problems: - Copies of NumPy itself (an astonishing number of people have checked in copies of it to their own source tree) - NumPy forks / precursors / etc. (e.g. Numeric also had a type called PyUFuncObject, the "bohrium" project has a fork of numpy 1.6, etc.) - Cython-generated boilerplate used to generate the "object has changed size" warning (which we `unconditionally filter out anyway <https://github.com/numpy/numpy/blob/master/numpy/__init__.py#L226>`_) - Lots of calls to ``PyUFunc_RegisterLoopForType`` and friends, which require casting the first argument to ``PyUFuncObject*`` - Misc. other unproblematic stuff (like Cython header declarations that never get used) There were also several cases that actually referenced PyUFuncObject internal fields: - The "rational" dtype from numpy-dtypes, which is used in a few projects, accesses ``ufunc->nargs`` as a safety check, but does not touch any other fields (`see here <https://github.com/numpy/numpy-dtypes/blob/c0175a6b1c5aa89b4520b29487f06d0e200e2a03/npytypes/rational/rational.c#L1140-L1151>`_). - Numba: does some rather elaborate things to support the definition of on-the-fly JITted ufuncs. These seem to be clear deficiencies in the ufunc API (e.g., there's no good way to control the lifespan of the array of function pointers passed to ``PyUFunc_FromFuncAndData``), so we should work with them to provide the API they need to do this in a maintainable way. Some of the relevant code: https://github.com/numba/numba/tree/master/numba/npyufunc https://github.com/numba/numba/blob/98752647a95ac6c9d480e81ca5c8afcfa3ddfd18... - dynd-python: Contains some code that attempts to extract the inner loops from a numpy ufunc object and wrap them into the dynd 'ufunc' equivalent: https://github.com/libdynd/dynd-python/blob/c06f8fc4e72257abac589faf76f10df8... - gulinalg: I'm not sure if anyone is still using this code since most of it was merged into numpy itself, but it's not a big deal in any case: all it contains is a `debugging function <https://github.com/ContinuumIO/gulinalg/blob/2ef365c48427c026dab4f45dc6f8b1b9af184460/gulinalg/src/gulinalg.c.src#L527-L550>`_ that dumps some internal fields from the PyUFuncObject. If you look, though, all calls to this function are already commented out :-). The full report is available here: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/pyufuncobject-r... In the course of this analysis, it was also noted that the standard Cython pxd files contain a wrapper for ufunc objects:: cdef class ufunc [object PyUFuncObject]: ... which means that Cython code can access internal struct fields via an object of type ``ufunc``, and thus escape our string-based search above. Therefore I also examined all Cython files on Github or searchcode.com that matched the query ``ufunc``, and searched for any lines matching any of the following regular expressions:: cdef\W+ufunc catches: 'cdef ufunc fn' cdef\W+.*\.\W*ufunc catches: 'cdef np.ufunc fn' <.*ufunc\W*> catches: '(<ufunc> fn).nargs', '(< np.ufunc > fn).nargs' cdef.*\(.*ufunc catches: 'cdef doit(np.ufunc fn, ...):' (I considered parsing the actual source and analysing it that way, but decided I was too lazy. This could be done if anyone is worried that the above regexes might miss things though.) There were zero files that contained matches for any of the above regexes: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/ufunc-cython-re... Uses of PyUFunc_ReplaceLoopBySignature ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Applying the same screening as above, the only code that was found that used this function is also in Numba: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/PyUFunc_Replace... Uses of PyUFunc_SetUsesArraysAsData ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Aside from being semi-broken since 1.7 (it never got implemented for "masked" ufunc loops, i.e. those that use where=), there appear to be zero uses of this functionality either inside or outside NumPy: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/PyUFunc_SetUses... Rationale --------- **Rationale for preserving the remaining API functions**:: PyUFunc_GenericFunction PyUFunc_FromFuncAndData PyUFunc_FromFuncAndDataAndSignature PyUFunc_RegisterLoopForDescr PyUFunc_RegisterLoopForType In addition to being widely used, these functions can easily be preserved even if we change how ufuncs work internally, because they only ingest loop function pointers, they never return them. So they can easily be modified to wrap whatever loop function(s) they receive inside an adapter function that calls them at the appropriate time, and then register that adapter function using whatever API we add in the future. **Rationale for preserving the particular fields that are preserved**: Preserving ``nargs`` lets us avoid a little bit of breakage with the random dtype, and it doesn't seem like preserving ``nin``, ``nout``, ``nargs`` fields will produce any undue burden on future changes to ufunc internals; even if we were to introduce variadic ufuncs we could always just stick a -1 in the appropriate fields or whatever. **Rationale for removing PyUFunc_ReplaceLoopBySignature**: this function *returns* the PyUFunc_GenericFunction that was replaced; if we stop representing all loops using the legacy PyUFunc_GenericFunction type, then this will not be possible to do in the future. **Rationale for removing PyUFunc_SetUsesArraysAsData**: If set as the ``innerloopdata`` on a ufunc loop, then this function acts as a sentinel value, and causes the ``innerloopdata`` to instead be set to a pointer to the passed-in PyArrayObjects. In principle we could preserve this function, but it has a number of deficiencies: - No-one appears to use it. - It's been buggy for several releases and no-one noticed. - AFAIK the only reason it was included in the first place is that it provides a backdoor for ufunc loops to get access to the dtypes -- but we are planning to fix this in a better way. - It can't be shimmed as easily as the loop registration functions, because we don't anticipate that the new-and-improved ufunc loop functions will *get* access to the array objects, only to the dtypes; so this would have to remain cluttering up the core dispatch path indefinitely. - We have good reason for *not* wanting to get ufunc loops get access to the actual array objects, because one of the goals on our roadmap is exactly to enable the use of ufuncs on non-ndarray objects. Giving ufuncs access to dtypes alone creates a clean boundary here: it guarantees that ufunc loops can work equally on all duck-array objects (so long as they have a dtype), and enforces the invariant that anything which affects the interpretation of data values should be attached to the dtype, not to the array object. Rejected alternatives --------------------- **Do nothing**: there's no way we'll ever be able to touch ufuncs at all if we don't hide those fields sooner or later. While any amount of breakage is regrettable, the costs of cleaning things up now are less than the costs of never improving numpy's APIs. **Somehow sneak the dtype information in via ``void *innerloopdata``**: This might let us preserve the signature of PyUFunc_GenericFunction, and thus preserve PyUFunc_ReplaceLoopBySignature. But we'd still have the problem of leaving way too much internal state exposed, and it's not even clear how this would work, given that we actually do want to preserve the use of ``innerloopdata`` for actual per-loop data. (This is where the PyUFunc_SetUsesArraysAsData hack fails.) -- Nathaniel J. Smith -- http://vorpus.org
Hi Nathaniel, On Sun, 20 Sep 2015 21:13:30 -0700 Nathaniel Smith <njs@pobox.com> wrote:
Thanks for a detailed and clear explanation of the proposed changes. As far as Numba is concerned, making changes is ok for us provided Numpy provides APIs to do what we want. Regards Antoine.
Hi Antoine, On Mon, Sep 21, 2015 at 2:44 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Good to hear, thanks! Any interest in designing those new APIs that will do what you want? :-) A no-brainer is that PyUFuncObject should just take responsibility for managing the memory of its own internal arrays instead of assuming that they'll always be statically allocated and forcing elaborate workarounds when they're not, but there is a lot of complicated stuff going on in numba/npyufunc/_internal.c... I am even wondering whether we should go ahead and reify a first-class "ufunc loop" object, so it can have its own refcounting. -n -- Nathaniel J. Smith -- http://vorpus.org
We have the PyArrayObject vs PyArrayObject_fields definition in ndarraytypes.h that is used to enforce access to the members through inline functions rather than directly, which seems to me like the right way to go: don't leave stones unturned, hide everything and provide PyUFunc_NIN, PyUFunc_NOUT and friends to handle those too. On Sun, Sep 20, 2015 at 9:13 PM, Nathaniel Smith <njs@pobox.com> wrote:
-- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.
On Mon, Sep 21, 2015 at 7:29 AM, Jaime Fernández del Río <jaime.frio@gmail.com> wrote:
The PyArrayObject vs PyArrayObject_fields distinction is only enabled if a downstream library explicitly requests it with #define NPY_NO_DEPRECATED_API, though -- the idea is that the changes in this NEP would be enabled unconditionally, even for old code. So the reason nin/nout/nargs are left exposed in this proposal is that there's some existing code out there that would break (until updated) if we hid them, and not much benefit to breaking it. If we're fine with breaking that code then we could just hide them unconditionally too. The only code I found in the wild that would be affected is the "rational" user-defined dtype, which would be trivially fixable since the only thing it does with ufunc->nargs is a quick consistency check: https://github.com/numpy/numpy-dtypes/blob/c0175a6b1c5aa89b4520b29487f06d0e2... Also it's not 100% clear right now whether we even want to keep supporting the old user-defined dtype API that this particular code is based around. But if this code uses ufunc->nargs then perhaps other code does too? I'm open to opinions -- I doubt it matters that much either way. I just want to make sure that we can hide the other stuff :-). When it comes to evolving these APIs in general: one unfortunate thing about the PyArrayObject changes in 1.7 is that because they were implemented using *inline* functions (/macros) they haven't affected the a*B*i exposure at all, even in code that has upgraded to the new calling conventions. While user code no longer *names* the internal fields directly, we still have to implement exactly the same fields and put them in exactly the same place in memory or else break ABI. And the other unfortunate thing is that we don't really have a mechanism for saying "okay, we're dropping support for the old way of doing things in 1.xx" -- in particular the current NPY_NO_DEPRECATED_API mechanism doesn't give us any way to detect and error out if someone tries to use an old version of the APIs, so ABI breaks still mean segfaults. I'm thinking that if/when we figure out how to implement the "sliding window" API/ABI idea that we talked about at SciPy, then that will give us a strategy for cleanly transitioning to a world with a maintainable API+ABI and it becomes worth sitting down and making up a set of setters/getters for the attributes that we want to make public in a maintainable way. But until then our only real options are either hard breaks or nothing, so unless we want to do a hard break there's not much point talking about it. -n
-- Nathaniel J. Smith -- http://vorpus.org
On Mon, Sep 21, 2015 at 10:23 PM, Nathaniel Smith <njs@pobox.com> wrote:
One thing we might consider along the way is separating numpy.multiarray and friends into an actual library plus a module. That way the new numpy api would be exposed in the library rather than by importing an array of pointers from the module. the a*B*i exposure at all, even in code that has upgraded to the new
<snip> Chuck
On Tue, Sep 22, 2015 at 3:43 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
Agreed. This would help the cythonizing process as well (on which I will try to write more about in a separate thread later). David
That sounds like a very good idea. I know that one of the original motivations for the odd import mechanism of NumPy was the AIX platform and it's lack of a shared library. I can't imagine that is still actually a problem. A simpler, library-based mechanism would be a welcome change from my perspective. -Travis On Tue, Sep 22, 2015 at 5:43 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
-- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io
On Tue, Sep 22, 2015 at 3:43 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:
On Mon, Sep 21, 2015 at 10:23 PM, Nathaniel Smith <njs@pobox.com> wrote:
[...]
I'm not sure whether we'll be able to pull this off at the technical level? Partly because anything involving cross-platform linker behavior is a recipe for unpleasantness, but mostly because doing sliding-window API/ABI tracking requires that we have some way to check which of multiple APIs a given third-party package is requesting, and provide a nice error if the one they want isn't available, and I'm not certain how to accomplish that via the regular linker. But sure, something to look into when we reach that point :-) -n -- Nathaniel J. Smith -- http://vorpus.org
On Tue, Sep 22, 2015 at 10:19 PM, Nathaniel Smith <njs@pobox.com> wrote:
I'd recommend the Henry Ford approach, "Any customer can have a car painted any color that he wants so long as it is *black*". Essentially, an ABI break split between a backward compatible layer on top, and a bare metal layer below, with the latter recommended. We would still need to solve the 'hide the structure" problem, but that is probably unavoidable whatever approach we take. In any case, it might be worthwhile making a list of the functions such a library would expose. I'm not sure how big a problem linking would be, likely Windows would continue to be the largest source of problems if we go the shared library route. Chuck
Hi, This e-mail is an attempt at proposing an API to solve Numba's needs. Attribute access ---------------- int PyUFunc_Nin(PyUFuncObject *) Replaces ufunc->nin. int PyUFunc_Nout(PyUFuncObject *) Replaces ufunc->nout. int PyUFunc_Nargs(PyUFuncObject *) Replaces ufunc->nargs. PyObject *PyUFunc_Name(PyUFuncObject *) Replaces ufunc->name, returns a unicode object. (alternative: return a const char *) For introspection, the following would be nice too: int PyUFunc_Identity(PyFuncObject *) Replaces ufunc->identity. const char *PyUFunc_Signature(PyUFuncObject *, int i) Gives a pointer to the types of the i'th signature. (equivalent today to &ufunc->ntypes[i * ufunc->nargs]) Lifetime control ---------------- PyObject *PyUFunc_SetObject(PyUFuncObject *, PyObject *) Sets the ufunc's "object" to the given object. The object has no special semantics except that it is DECREF'ed when the ufunc is deallocated (this is today's ufunc->obj). The DECREF should happen only after the ufunc has accessed any internal resources (since the DECREF could deallocate some of those resources). PyObject *PyUFunc_GetObject(PyUFuncObject *) Return the ufunc's current "object". Loop registration ----------------- int PyUFunc_RegisterLoopForSignature( PyUFuncObject* ufunc, PyUFuncGenericFunction function, int *arg_types, void *data, PyObject *obj) Register a loop implementation for the given arg_types (built-in types, presumably). This either appends the loop to the types and functions array (reallocating it if necessary), or replaces an existing one with the same signature. A copy of arg_types is done, such that the caller does not have to manage its lifetime. The optional "PyObject *obj" is an object which gets DECREF'ed when the loop is relinquished (for example when the ufunc is destroyed, or when the loop gets replaced with another by calling this function again). I cannot say I'm 100% sure this is sufficient, but this seems it should cover our current needs. Note this is a minimal proposal. For example, Numpy could instead decide to pass and return all argument types as PyArray_Descr pointers rather than raw integers, and that would probably work for us too. Regards Antoine.
On Tue, Sep 22, 2015 at 7:57 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Hi,
This e-mail is an attempt at proposing an API to solve Numba's needs.
Thanks!
These all seem trivially supportable going forward.
Hmm, I can imagine cases where we might want to change how this works. (E.g. if np.dot were a ufunc then the existing identity settings wouldn't work very well... and I have some vague memory that there might already some delicate code in a few places because of difficulties in defining "zero" and "one" for arbitrary dtypes.)
I assume the 'i' part isn't actually interesting here (since there's no longer any parallel vector of function pointers accessible), and the high-level semantics that you're looking for are "please give me the set of signatures that have a loop defined"? [Edit: Also, see the discussion below about integer type pointers. The consequences here are that we can certainly provide an operation like this, but if we do then we might be abandoning it in a few releases (e.g. it might start telling you about only a subset of defined signatures). So can you expand a bit on what you mean by "would be nice" above?]
I understand why you need a "base" object like this for individual loops, but if ufuncs start managing the ufunc-level memory buffers internally, then is this still useful? I guess I'm curious to see an example.
PyObject *PyUFunc_GetObject(PyUFuncObject *)
Return the ufunc's current "object".
Oh, are you planning to actually use this to attach some arbitrary metadata, not just attach deallocation callbacks?
Hmm, that's an interesting and tricky point, actually -- I think the way it will work eventually is that signatures will be specified in terms of "dtypetypes" (i.e., subclasses of dtype, rather than ints *or* instances of dtype = PyArray_Descrs). But I guess that's just a challenge we'll have to think about when implementing this stuff -- either it means that the new ufunc API will have to wait a bit for more of the new dtype machinery to be ready, or we'll have to temporarily bridge the gap with an loop registration API that takes new-style loop callbacks but uses int signatures (and then later turn it into a thin wrapper around the final API). -n -- Nathaniel J. Smith -- http://vorpus.org
On Thu, 24 Sep 2015 00:20:23 -0700 Nathaniel Smith <njs@pobox.com> wrote:
Yes... As long as there is a way for us to set the identity value (whatever the exact API) when constructing a ufunc, it should be ok.
Indeed.
"Would be nice" really means "we could make use of it" for letting the user access ufunc metadata. We don't *need* it currently. But generally being able to query the high-level properties of a ufunc, from C, sounds like a good thing, and perhaps other people would be interested.
Well, for example, we dynamically allocate the ufunc's name (and possibly its docstring), so we need to deallocate it when the ufunc is destroyed. Actually, we should probably deallocate more stuff that we currently don't (such as the execution environment)...
No, just deallocation callbacks. I was including the GetObject function for completeness, I'm not sure we would need it (but it sounds trivial to provide and maintain).
Subclasses? I'm not sure what you mean by that, how would one specify e.g. an int64 vs. an int32? Are you referring to Travis' dtypes-as-classes project, or something similar? In that case though, a dtype would still be an instance of a "dtypetype" (metatype), not a subclass :-)
Well, as long as you keep the int typecodes in Numpy (and I guess you'll do for quite some time, for compatibility), bridging should be easy indeed. Regards Antoine.
Hi Nathaniel, On Sun, 20 Sep 2015 21:13:30 -0700 Nathaniel Smith <njs@pobox.com> wrote:
Thanks for a detailed and clear explanation of the proposed changes. As far as Numba is concerned, making changes is ok for us provided Numpy provides APIs to do what we want. Regards Antoine.
Hi Antoine, On Mon, Sep 21, 2015 at 2:44 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Good to hear, thanks! Any interest in designing those new APIs that will do what you want? :-) A no-brainer is that PyUFuncObject should just take responsibility for managing the memory of its own internal arrays instead of assuming that they'll always be statically allocated and forcing elaborate workarounds when they're not, but there is a lot of complicated stuff going on in numba/npyufunc/_internal.c... I am even wondering whether we should go ahead and reify a first-class "ufunc loop" object, so it can have its own refcounting. -n -- Nathaniel J. Smith -- http://vorpus.org
We have the PyArrayObject vs PyArrayObject_fields definition in ndarraytypes.h that is used to enforce access to the members through inline functions rather than directly, which seems to me like the right way to go: don't leave stones unturned, hide everything and provide PyUFunc_NIN, PyUFunc_NOUT and friends to handle those too. On Sun, Sep 20, 2015 at 9:13 PM, Nathaniel Smith <njs@pobox.com> wrote:
-- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.
On Mon, Sep 21, 2015 at 7:29 AM, Jaime Fernández del Río <jaime.frio@gmail.com> wrote:
The PyArrayObject vs PyArrayObject_fields distinction is only enabled if a downstream library explicitly requests it with #define NPY_NO_DEPRECATED_API, though -- the idea is that the changes in this NEP would be enabled unconditionally, even for old code. So the reason nin/nout/nargs are left exposed in this proposal is that there's some existing code out there that would break (until updated) if we hid them, and not much benefit to breaking it. If we're fine with breaking that code then we could just hide them unconditionally too. The only code I found in the wild that would be affected is the "rational" user-defined dtype, which would be trivially fixable since the only thing it does with ufunc->nargs is a quick consistency check: https://github.com/numpy/numpy-dtypes/blob/c0175a6b1c5aa89b4520b29487f06d0e2... Also it's not 100% clear right now whether we even want to keep supporting the old user-defined dtype API that this particular code is based around. But if this code uses ufunc->nargs then perhaps other code does too? I'm open to opinions -- I doubt it matters that much either way. I just want to make sure that we can hide the other stuff :-). When it comes to evolving these APIs in general: one unfortunate thing about the PyArrayObject changes in 1.7 is that because they were implemented using *inline* functions (/macros) they haven't affected the a*B*i exposure at all, even in code that has upgraded to the new calling conventions. While user code no longer *names* the internal fields directly, we still have to implement exactly the same fields and put them in exactly the same place in memory or else break ABI. And the other unfortunate thing is that we don't really have a mechanism for saying "okay, we're dropping support for the old way of doing things in 1.xx" -- in particular the current NPY_NO_DEPRECATED_API mechanism doesn't give us any way to detect and error out if someone tries to use an old version of the APIs, so ABI breaks still mean segfaults. I'm thinking that if/when we figure out how to implement the "sliding window" API/ABI idea that we talked about at SciPy, then that will give us a strategy for cleanly transitioning to a world with a maintainable API+ABI and it becomes worth sitting down and making up a set of setters/getters for the attributes that we want to make public in a maintainable way. But until then our only real options are either hard breaks or nothing, so unless we want to do a hard break there's not much point talking about it. -n
-- Nathaniel J. Smith -- http://vorpus.org
On Mon, Sep 21, 2015 at 10:23 PM, Nathaniel Smith <njs@pobox.com> wrote:
One thing we might consider along the way is separating numpy.multiarray and friends into an actual library plus a module. That way the new numpy api would be exposed in the library rather than by importing an array of pointers from the module. the a*B*i exposure at all, even in code that has upgraded to the new
<snip> Chuck
On Tue, Sep 22, 2015 at 3:43 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
Agreed. This would help the cythonizing process as well (on which I will try to write more about in a separate thread later). David
That sounds like a very good idea. I know that one of the original motivations for the odd import mechanism of NumPy was the AIX platform and it's lack of a shared library. I can't imagine that is still actually a problem. A simpler, library-based mechanism would be a welcome change from my perspective. -Travis On Tue, Sep 22, 2015 at 5:43 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
-- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io
On Tue, Sep 22, 2015 at 3:43 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:
On Mon, Sep 21, 2015 at 10:23 PM, Nathaniel Smith <njs@pobox.com> wrote:
[...]
I'm not sure whether we'll be able to pull this off at the technical level? Partly because anything involving cross-platform linker behavior is a recipe for unpleasantness, but mostly because doing sliding-window API/ABI tracking requires that we have some way to check which of multiple APIs a given third-party package is requesting, and provide a nice error if the one they want isn't available, and I'm not certain how to accomplish that via the regular linker. But sure, something to look into when we reach that point :-) -n -- Nathaniel J. Smith -- http://vorpus.org
On Tue, Sep 22, 2015 at 10:19 PM, Nathaniel Smith <njs@pobox.com> wrote:
I'd recommend the Henry Ford approach, "Any customer can have a car painted any color that he wants so long as it is *black*". Essentially, an ABI break split between a backward compatible layer on top, and a bare metal layer below, with the latter recommended. We would still need to solve the 'hide the structure" problem, but that is probably unavoidable whatever approach we take. In any case, it might be worthwhile making a list of the functions such a library would expose. I'm not sure how big a problem linking would be, likely Windows would continue to be the largest source of problems if we go the shared library route. Chuck
Hi, This e-mail is an attempt at proposing an API to solve Numba's needs. Attribute access ---------------- int PyUFunc_Nin(PyUFuncObject *) Replaces ufunc->nin. int PyUFunc_Nout(PyUFuncObject *) Replaces ufunc->nout. int PyUFunc_Nargs(PyUFuncObject *) Replaces ufunc->nargs. PyObject *PyUFunc_Name(PyUFuncObject *) Replaces ufunc->name, returns a unicode object. (alternative: return a const char *) For introspection, the following would be nice too: int PyUFunc_Identity(PyFuncObject *) Replaces ufunc->identity. const char *PyUFunc_Signature(PyUFuncObject *, int i) Gives a pointer to the types of the i'th signature. (equivalent today to &ufunc->ntypes[i * ufunc->nargs]) Lifetime control ---------------- PyObject *PyUFunc_SetObject(PyUFuncObject *, PyObject *) Sets the ufunc's "object" to the given object. The object has no special semantics except that it is DECREF'ed when the ufunc is deallocated (this is today's ufunc->obj). The DECREF should happen only after the ufunc has accessed any internal resources (since the DECREF could deallocate some of those resources). PyObject *PyUFunc_GetObject(PyUFuncObject *) Return the ufunc's current "object". Loop registration ----------------- int PyUFunc_RegisterLoopForSignature( PyUFuncObject* ufunc, PyUFuncGenericFunction function, int *arg_types, void *data, PyObject *obj) Register a loop implementation for the given arg_types (built-in types, presumably). This either appends the loop to the types and functions array (reallocating it if necessary), or replaces an existing one with the same signature. A copy of arg_types is done, such that the caller does not have to manage its lifetime. The optional "PyObject *obj" is an object which gets DECREF'ed when the loop is relinquished (for example when the ufunc is destroyed, or when the loop gets replaced with another by calling this function again). I cannot say I'm 100% sure this is sufficient, but this seems it should cover our current needs. Note this is a minimal proposal. For example, Numpy could instead decide to pass and return all argument types as PyArray_Descr pointers rather than raw integers, and that would probably work for us too. Regards Antoine.
On Tue, Sep 22, 2015 at 7:57 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Hi,
This e-mail is an attempt at proposing an API to solve Numba's needs.
Thanks!
These all seem trivially supportable going forward.
Hmm, I can imagine cases where we might want to change how this works. (E.g. if np.dot were a ufunc then the existing identity settings wouldn't work very well... and I have some vague memory that there might already some delicate code in a few places because of difficulties in defining "zero" and "one" for arbitrary dtypes.)
I assume the 'i' part isn't actually interesting here (since there's no longer any parallel vector of function pointers accessible), and the high-level semantics that you're looking for are "please give me the set of signatures that have a loop defined"? [Edit: Also, see the discussion below about integer type pointers. The consequences here are that we can certainly provide an operation like this, but if we do then we might be abandoning it in a few releases (e.g. it might start telling you about only a subset of defined signatures). So can you expand a bit on what you mean by "would be nice" above?]
I understand why you need a "base" object like this for individual loops, but if ufuncs start managing the ufunc-level memory buffers internally, then is this still useful? I guess I'm curious to see an example.
PyObject *PyUFunc_GetObject(PyUFuncObject *)
Return the ufunc's current "object".
Oh, are you planning to actually use this to attach some arbitrary metadata, not just attach deallocation callbacks?
Hmm, that's an interesting and tricky point, actually -- I think the way it will work eventually is that signatures will be specified in terms of "dtypetypes" (i.e., subclasses of dtype, rather than ints *or* instances of dtype = PyArray_Descrs). But I guess that's just a challenge we'll have to think about when implementing this stuff -- either it means that the new ufunc API will have to wait a bit for more of the new dtype machinery to be ready, or we'll have to temporarily bridge the gap with an loop registration API that takes new-style loop callbacks but uses int signatures (and then later turn it into a thin wrapper around the final API). -n -- Nathaniel J. Smith -- http://vorpus.org
On Thu, 24 Sep 2015 00:20:23 -0700 Nathaniel Smith <njs@pobox.com> wrote:
Yes... As long as there is a way for us to set the identity value (whatever the exact API) when constructing a ufunc, it should be ok.
Indeed.
"Would be nice" really means "we could make use of it" for letting the user access ufunc metadata. We don't *need* it currently. But generally being able to query the high-level properties of a ufunc, from C, sounds like a good thing, and perhaps other people would be interested.
Well, for example, we dynamically allocate the ufunc's name (and possibly its docstring), so we need to deallocate it when the ufunc is destroyed. Actually, we should probably deallocate more stuff that we currently don't (such as the execution environment)...
No, just deallocation callbacks. I was including the GetObject function for completeness, I'm not sure we would need it (but it sounds trivial to provide and maintain).
Subclasses? I'm not sure what you mean by that, how would one specify e.g. an int64 vs. an int32? Are you referring to Travis' dtypes-as-classes project, or something similar? In that case though, a dtype would still be an instance of a "dtypetype" (metatype), not a subclass :-)
Well, as long as you keep the int typecodes in Numpy (and I guess you'll do for quite some time, for compatibility), bridging should be easy indeed. Regards Antoine.
participants (7)
-
Antoine Pitrou -
Bryan Van de Ven -
Charles R Harris -
David Cournapeau -
Jaime Fernández del Río -
Nathaniel Smith -
Travis Oliphant