Mailman 3 September 2019 - NumPy-Discussion

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

numpy C-API :: use numpy's random number generator in a ufunc
by Daniel Knüttel 18 Oct '19

18 Oct '19

Hi folks, I currently have a project that requires randomness in a ufunc. In order to keep the ufuncs as reproducible as possible I would like to use numpy's random number generator for that; basically because setting the seed will be more intuitive this way. However I cannot find the documentation of the numpy.random C-API (does it have one?). How would one do that? Cheers, -- Daniel Knüttel <daniel.knuettel(a)daknuett.eu>

2 1

NEP 31 — Context-local and global overrides of the NumPy API
by Hameer Abbasi 10 Oct '19

10 Oct '19

Hello all, It was recently brought to my attention that my mails to NumPy-discussion were probably going into the spam folder for many people, so here I am trying from another email. Probably Google trying to force people onto their products as usual. 😉 Me, Ralf Gommers and Peter Bell (both cc’d) have come up with a proposal on how to solve the array creation and duck array problems. The solution is outlined in NEP-31, currently in the form of a PR, [1] Following the high level discussion in NEP-22. [2] It would be nice to get some feedback. Full-text of the NEP: ============================================================ NEP 31 — Context-local and global overrides of the NumPy API ============================================================ :Author: Hameer Abbasi <habbasi(a)quansight.com><mailto:habbasi@quansight.com> :Author: Ralf Gommers <rgommers(a)quansight.com><mailto:rgommers@quansight.com> :Author: Peter Bell <peterbell10(a)live.co.uk><mailto:peterbell10@live.co.uk> :Status: Draft :Type: Standards Track :Created: 2019-08-22 Abstract -------- This NEP proposes to make all of NumPy's public API overridable via an extensible backend mechanism, using a library called ``uarray`` `[1]`_ ``uarray`` provides global and context-local overrides, as well as a dispatch mechanism similar to NEP-18 `[2]`_. First experiences with ``__array_function__`` show that it is necessary to be able to override NumPy functions that *do not take an array-like argument*, and hence aren't overridable via ``__array_function__``. The most pressing need is array creation and coercion functions - see e.g. NEP-30 `[9]`_. This NEP proposes to allow, in an opt-in fashion, overriding any part of the NumPy API. It is intended as a comprehensive resolution to NEP-22 `[3]`_, and obviates the need to add an ever-growing list of new protocols for each new type of function or object that needs to become overridable. Motivation and Scope -------------------- The motivation behind ``uarray`` is manyfold: First, there have been several attempts to allow dispatch of parts of the NumPy API, including (most prominently), the ``__array_ufunc__`` protocol in NEP-13 `[4]`_, and the ``__array_function__`` protocol in NEP-18 `[2]`_, but this has shown the need for further protocols to be developed, including a protocol for coercion (see `[5]`_). The reasons these overrides are needed have been extensively discussed in the references, and this NEP will not attempt to go into the details of why these are needed. Another pain point requiring yet another protocol is the duck-array protocol (see `[9]`_). This NEP takes a more holistic approach: It assumes that there are parts of the API that need to be overridable, and that these will grow over time. It provides a general framework and a mechanism to avoid a design of a new protocol each time this is required. This NEP proposes the following: That ``unumpy`` `[8]`_ becomes the recommended override mechanism for the parts of the NumPy API not yet covered by ``__array_function__`` or ``__array_ufunc__``, and that ``uarray`` is vendored into a new namespace within NumPy to give users and downstream dependencies access to these overrides. This vendoring mechanism is similar to what SciPy decided to do for making ``scipy.fft`` overridable (see `[10]`_). Detailed description -------------------- **Note:** *This section will not attempt to explain the specifics or the mechanism of ``uarray``, that is explained in the ``uarray`` documentation.* `[1]`_ *However, the NumPy community will have input into the design of ``uarray``, and any backward-incompatible changes will be discussed on the mailing list.* The way we propose the overrides will be used by end users is:: import numpy.overridable as np with np.set_backend(backend): x = np.asarray(my_array, dtype=dtype) And a library that implements a NumPy-like API will use it in the following manner (as an example):: import numpy.overridable as np _ua_implementations = {} __ua_domain__ = "numpy" def __ua_function__(func, args, kwargs): fn = _ua_implementations.get(func, None) return fn(*args, **kwargs) if fn is not None else NotImplemented def implements(ua_func): def inner(func): _ua_implementations[ua_func] = func return func return inner @implements(np.asarray) def asarray(a, dtype=None, order=None): # Code here # Either this method or __ua_convert__ must # return NotImplemented for unsupported types, # Or they shouldn't be marked as dispatchable. # Provides a default implementation for ones and zeros. @implements(np.full) def full(shape, fill_value, dtype=None, order='C'): # Code here The only change this NEP proposes at its acceptance, is to make ``unumpy`` the officially recommended way to override NumPy. ``unumpy`` will remain a separate repository/package (which we propose to vendor to avoid a hard dependency, and use the separate ``unumpy`` package only if it is installed) rather than depend on for the time being), and will be developed primarily with the input of duck-array authors and secondarily, custom dtype authors, via the usual GitHub workflow. There are a few reasons for this: * Faster iteration in the case of bugs or issues. * Faster design changes, in the case of needed functionality. * ``unumpy`` will work with older versions of NumPy as well. * The user and library author opt-in to the override process, rather than breakages happening when it is least expected. In simple terms, bugs in ``unumpy`` mean that ``numpy`` remains unaffected. Advantanges of ``unumpy`` over other solutions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``unumpy`` offers a number of advantanges over the approach of defining a new protocol for every problem encountered: Whenever there is something requiring an override, ``unumpy`` will be able to offer a unified API with very minor changes. For example: * ``ufunc`` objects can be overridden via their ``__call__``, ``reduce`` and other methods. * ``dtype`` objects can be overridden via the dispatch/backend mechanism, going as far as to allow ``np.float32`` et. al. to be overridden by overriding ``__get__``. * Other functions can be overridden in a similar fashion. * ``np.asduckarray`` goes away, and becomes ``np.asarray`` with a backend set. * The same holds for array creation functions such as ``np.zeros``, ``np.empty`` and so on. This also holds for the future: Making something overridable would require only minor changes to ``unumpy``. Another promise ``unumpy`` holds is one of default implementations. Default implementations can be provided for any multimethod, in terms of others. This allows one to override a large part of the NumPy API by defining only a small part of it. This is to ease the creation of new duck-arrays, by providing default implementations of many functions that can be easily expressed in terms of others, as well as a repository of utility functions that help in the implementation of duck-arrays that most duck-arrays would require. The last benefit is a clear way to coerce to a given backend, and a protocol for coercing not only arrays, but also ``dtype`` objects and ``ufunc`` objects with similar ones from other libraries. This is due to the existence of actual, third party dtype packages, and their desire to blend into the NumPy ecosystem (see `[6]`_). This is a separate issue compared to the C-level dtype redesign proposed in `[7]`_, it's about allowing third-party dtype implementations to work with NumPy, much like third-party array implementations. Mixing NumPy and ``unumpy`` in the same file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Normally, one would only want to import only one of ``unumpy`` or ``numpy``, you would import it as ``np`` for familiarity. However, there may be situations where one wishes to mix NumPy and the overrides, and there are a few ways to do this, depending on the user's style:: import numpy.overridable as unumpy import numpy as np or:: import numpy as np # Use unumpy via np.overridable Related Work ------------ Previous override mechanisms ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * NEP-18, the ``__array_function__`` protocol. `[2]`_ * NEP-13, the ``__array_ufunc__`` protocol. `[3]`_ Existing NumPy-like array implementations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Dask: https://dask.org/ * CuPy: https://cupy.chainer.org/ * PyData/Sparse: https://sparse.pydata.org/ * Xnd: https://xnd.readthedocs.io/ * Astropy's Quantity: https://docs.astropy.org/en/stable/units/ Existing and potential consumers of alternative arrays ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Dask: https://dask.org/ * scikit-learn: https://scikit-learn.org/ * Xarray: https://xarray.pydata.org/ * TensorLy: http://tensorly.org/ Existing alternate dtype implementations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * ``ndtypes``: https://ndtypes.readthedocs.io/en/latest/ * Datashape: https://datashape.readthedocs.io * Plum: https://plum-py.readthedocs.io/ Implementation -------------- The implementation of this NEP will require the following steps: * Implementation of ``uarray`` multimethods corresponding to the NumPy API, including classes for overriding ``dtype``, ``ufunc`` and ``array`` objects, in the ``unumpy`` repository. * Moving backends from ``unumpy`` into the respective array libraries. Backward compatibility ---------------------- There are no backward incompatible changes proposed in this NEP. Alternatives ------------ The current alternative to this problem is NEP-30 plus adding more protocols (not yet specified) in addition to it. Even then, some parts of the NumPy API will remain non-overridable, so it's a partial alternative. The main alternative to vendoring ``unumpy`` is to simply move it into NumPy completely and not distribute it as a separate package. This would also achieve the proposed goals, however we prefer to keep it a separate package for now, for reasons already stated above. Discussion ---------- * ``uarray`` blogpost: https://labs.quansight.org/blog/2019/07/uarray-update-api-changes-overhead-… * The discussion section of NEP-18: https://numpy.org/neps/nep-0018-array-function-protocol.html#discussion * NEP-22: https://numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html * Dask issue #4462: https://github.com/dask/dask/issues/4462 * PR #13046: https://github.com/numpy/numpy/pull/13046 * Dask issue #4883: https://github.com/dask/dask/issues/4883 * Issue #13831: https://github.com/numpy/numpy/issues/13831 * Discussion PR 1: https://github.com/hameerabbasi/numpy/pull/3 * Discussion PR 2: https://github.com/hameerabbasi/numpy/pull/4 References and Footnotes ------------------------ .. _[1]: [1] uarray, A general dispatch mechanism for Python: https://uarray.readthedocs.io .. _[2]: [2] NEP 18 — A dispatch mechanism for NumPy’s high level array functions: https://numpy.org/neps/nep-0018-array-function-protocol.html .. _[3]: [3] NEP 22 — Duck typing for NumPy arrays – high level overview: https://numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html .. _[4]: [4] NEP 13 — A Mechanism for Overriding Ufuncs: https://numpy.org/neps/nep-0013-ufunc-overrides.html .. _[5]: [5] Reply to Adding to the non-dispatched implementation of NumPy methods: http://numpy-discussion.10968.n7.nabble.com/Adding-to-the-non-dispatched-im… .. _[6]: [6] Custom Dtype/Units discussion: http://numpy-discussion.10968.n7.nabble.com/Custom-Dtype-Units-discussion-t… .. _[7]: [7] The epic dtype cleanup plan: https://github.com/numpy/numpy/issues/2899 .. _[8]: [8] unumpy: NumPy, but implementation-independent: https://unumpy.readthedocs.io .. _[9]: [9] NEP 30 — Duck Typing for NumPy Arrays - Implementation: https://www.numpy.org/neps/nep-0030-duck-array-protocol.html .. _[10]: [10] http://scipy.github.io/devdocs/fft.html#backend-control Copyright --------- This document has been placed in the public domain. Best regards, Hameer Abbasi [1] https://github.com/numpy/numpy/pull/14389 [2] https://numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html

11 46

Error handling in a ufunc inner loop.
by Warren Weckesser 29 Sep '19

29 Sep '19

This is a new thread to address the question of error handling in a ufunc loop that was brought up in the thread on handling core dimensions of length zero. I'm attempting to answer my own question about the idiomatic way to handle an error in an inner loop. The use of the GIL with a ufunc loop is documented at https://numpy.org/devdocs/reference/internals.code-explanations.html#functi… So an inner loop is running without the GIL if the macro NPY_ALLOW_THREADS is defined and the loop is not an object-type loop. If the inner loop is running without the GIL, it must acquire the GIL before calling, say, PyErr_SetString to set an exception. The NumPy macros for acquiring the GIL are documented at https://docs.scipy.org/doc/numpy/reference/c-api.array.html#group-2 These macros are defined in numpy/core/include/numpy/ndarraytypes.h. If NPY_ALLOW_THREADS is defined, these macros wrap calls to PyGILState_Ensure() and PyGILState_Release() ( https://docs.python.org/3/c-api/init.html#non-python-created-threads): ``` #define NPY_ALLOW_C_API_DEF PyGILState_STATE __save__; #define NPY_ALLOW_C_API do {__save__ = PyGILState_Ensure();} while (0); #define NPY_DISABLE_C_API do {PyGILState_Release(__save__);} while (0); ``` If NPY_ALLOW_THREADS is not defined, those macros are defined with empty values. Now suppose I want to change the following inner loop to set an exception instead of returning nan when the input is negative: ``` static void logfactorial_loop(char **args, npy_intp *dimensions, npy_intp* steps, void* data) { char *in = args[0]; char *out = args[1]; npy_intp in_step = steps[0]; npy_intp out_step = steps[1]; for (npy_intp i = 0; i < dimensions[0]; ++i, in += in_step, out += out_step) { int64_t x = *(int64_t *)in; if (x < 0) { *((double *)out) = NAN; } else { *((double *)out) = logfactorial(x); } } } ``` Based on the documentation linked above, the changed inner loop is simply: ``` static void logfactorial_loop(char **args, npy_intp *dimensions, npy_intp* steps, void* data) { char *in = args[0]; char *out = args[1]; npy_intp in_step = steps[0]; npy_intp out_step = steps[1]; for (npy_intp i = 0; i < dimensions[0]; ++i, in += in_step, out += out_step) { int64_t x = *(int64_t *)in; if (x < 0) { NPY_ALLOW_C_API_DEF NPY_ALLOW_C_API PyErr_SetString(PyExc_ValueError, "math domain error in logfactorial: x < 0"); NPY_DISABLE_C_API return; } else { *((double *)out) = logfactorial(x); } } } ``` That worked as expected, but I haven't tried it yet with a NumPy installation where NPY_ALLOW_THREADS is not defined. Is that change correct? Would that be considered the (or an) idiomatic way to handle errors in an inner loop? Are there any potential problems that I'm missing? Warren

1 1

Forcing gufunc to error with size zero input
by Warren Weckesser 29 Sep '19

29 Sep '19

I'm experimenting with gufuncs, and I just created a simple one with signature '(i)->()'. Is there a way to configure the gufunc itself so that an empty array results in an error? Or would I have to create a Python wrapper around the gufunc that does the error checking? Currently, when passed an empty array, the ufunc loop is called with the core dimension associated with i set to 0. It would be nice if the code didn't get that far, and the ufunc machinery "knew" that this gufunc didn't accept a core dimension that is 0. I'd like to automatically get an error, something like the error produced by `np.max([])`. Warren

3 5

NEP 32 is accepted. Now the work begins...
by Warren Weckesser 29 Sep '19

29 Sep '19

NumPy devs, NEP 32 to remove the financial functions (https://numpy.org/neps/nep-0032-remove-financial-functions.html) has been accepted. The next step is to create the numpy-financial package that will replace them. The repository for the new package is https://github.com/numpy/numpy-financial. I have a work-in-progress pull request there to get the initial structure set up. Reviews of the PR would be helpful, as would contributions to set up Sphinx-based documentation, continuous integration, PyPI packaging, and anything else that goes into setting up a "proper" package. Any help would be greatly appreciated! Warren

2 3

error during pip install
by Alan Isaac 28 Sep '19

28 Sep '19

Upgrading numpy with pip on Python 3.8b4 on Win 10 produced: ERROR: Could not install packages due to an EnvironmentError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '"C:' However, the install appears to have been successful. fwiw, Alan Isaac

2 6

UFunc out argument not forcing high precision loop?
by Sebastian Berg 28 Sep '19

28 Sep '19

Hi all, Looking at the ufunc dispatching rules with an `out` argument, I was a bit surprised to realize this little gem is how things work: ``` arr = np.arange(10, dtype=np.uint16) + 2**15 print(arr) # array([ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18], dtype=uint16) out = np.zeros(10) np.add(arr, arr, out=out) print(repr(out)) # array([ 0., 2., 4., 6., 8., 10., 12., 14., 16., 18.]) ``` This is strictly speaking correct/consistent. What the ufunc tries to ensure is that whatever the loop produces fits into `out`. However, I still find it unexpected that it does not pick the full precision loop. There is currently only one way to achieve that, and this by using `dtype=out.dtype` (or similar incarnations) which specify the exact dtype [0]. Of course this is also because I would like to simplify things for a new dispatching system, but I would like to propose to disable the above behaviour. This would mean: ``` # make the call: np.add(arr, arr, out=out) # Equivalent to the current [1]: np.add(arr, arr, out=out, dtype=(None, None, out.dtype)) # Getting the old behaviour requires (assuming inputs have same dtype): np.add(arr, arr, out=out, dtypes=arr.dtype) ``` and thus force the high precision loop. In very rare cases, this could lead to no loop being found. The main incompatibility is if someone actually makes use of the above (integer over/underflow) behaviour, but wants to store it in a higher precision array. I personally currently think we should change it, but am curious if we think that we may be able to get away with an accelerate process and not a year long FutureWarning. Cheers, Sebastian [0] You can also use `casting="no"` but in all relevant cases that should find no loop, since the we typically only have homogeneous loop definitions, and [1] Which is normally the same as the shorter spelling `dtype=out.dtype` of course.

2 3

Low-level API for Random
by Kevin Sheppard 25 Sep '19

25 Sep '19

There are some users of the NumPy C code in randomkit. This was never officially supported. There has been a long open issue to provide this officially. When I wrote randomgen I supplied .pdx files that make it simpler to write Cython code that uses the components. The lower-level API has not had much scrutiny and is in need of a clean-up. I thought this would also encourage users to extend the random machinery themselves as part of their project or code so as to minimize the requests for new (exotic) distributions to be included in Generator. Most of the generator functions follow a pattern random_DISTRIBUTION. Some have a bit more name mangling which can easily be cleaned up (like ranomd_gauss_zig, which should become PREFIX_standard_normal). Ralf Gommers suggested unprefixed names. I tried this in a local branch and it was a bit ugly since some of the distributions have common math names (e.g., gamma) and others are very short (e.g., t or f). I think a prefix is needed, and after looking through the C API docs npy_random_ seemed like a reasonable choice (since these live in numpy.random). Any thoughts on the following questions are welcome (others too): 1. Should there be a prefix on the C functions? 2. If so, what should the prefix be? 3. Should the legacy C functions be part of the API -- these are mostly the ones that produce or depend on polar transform normals (Box-Muller). I have a feeling no, but there may be reasons to prefer BM since they do not depend on rejection sampling. 4. Should low-level API be consumable like any other numpy C API by including the usual header locations and library locations? Right now, the pxd simplifies writing Cython but users have sp specify the location of the headers and source manually An alternative would be to provide a function like np.get_include() -> np.random.get_include() that would specialize in random. Kevin

8 19

NumPy-Discussion September 2019