Mailman 3 January 2018 - NumPy-Discussion

Re: [Numpy-discussion] Moving NumPy's PRNG Forward
by Kevin Sheppard 29 Jan '18

29 Jan '18

I am a firm believer that the current situation is not sustainable. There are a lot of improvements that can practically be incorporated. While many of these are performance related, there are also improvements in accuracy over some ranges of parameters that cannot be incorporated. I also think that perfect stream reproducibility is a bit of a myth across versions since this really would require identical OS, compiler and possibly CPU for some of the generators that produce floats. I believe there is a case for separating the random generator from core NumPy. Some points that favor becoming a subproject: 1. It is a pure consumer of NumPy API. Other parts of the API do no depend on random. 2. A stand alone package could be installed along side many different version of core NumPy which would reduce the pressure on freezing the stream. In terms of what is needed, I think that the underlying PRNG should be swappable. The will provide a simple mechanism to allow certain types of advancement while easily providing backward compat. In the current design this is very hard and requires compiling many nearly identical copies of RandomState. In pseudocode something like standard_normal(prng) where prng is a basic class that retains the PRNG state and has a small set of core random number generators that belong to the underlying PRNG -- probably something like int32, int64, double, and possibly int53. I am not advocating explicitly passing the PRNG as an argument, but having generators which can take any suitable PRNG would add a lot of flexibility in terms of taking advantage of improvements in the underlying PRNGs (see, e.g., xoroshiro128/xorshift1024). The "small" core PRNG would have responsibility over state and streams. The remainder of the module would transform the underlying PRNG into the required distributions. This would also simplify making improvements, since old versions could be saved or improved versions could be added to the API. For example, from numpy.random import standard_normal, prng # Preferred versions standard_normal(prng) # Ziggurat from numpy.random.legacy import standard_normal_bm, mt19937 # legacy generators standard_normal_bm(mt19937) # Box-Muller Kevin

3 3

f2py bug in numpy v1.12.0 and above?
by Solbrig,Jeremy 29 Jan '18

29 Jan '18

I have a suite of fortran code that I compile with f2py and use as a plugin to a python package. I am using Python v2.7 from Anaconda. When compiled using numpy v1.11.3 or lower, everything works fine, but if I upgrade to any more recent version I begin running into a runtime error. Presumably I am just not linking something that needs to be linked, but I'm not sure what library that would be. The error: ImportError: .../libstddev_kernel.so: undefined symbol: _gfortran_stop_numeric_f08 Where libestddev_kernel.so is built using this command: f2py --fcompiler=gnu95 --quiet --opt="-O3" -L. -L/usr/lib -I. -I.../include -I/usr/include -m libstddev_kernel -c stddev_kernel/stddev_kernel.f90 config.o Is there something additional I should be linking to within numpy or my anaconda installation? Thank you, Jeremy Jeremy Solbrig Research Associate Cooperative Institute for Research in the Atmosphere Colorado State University jeremy.solbrig(a)colostate.edu (970) 491-8805

3 4

linspaces
by Vincent Douce Mathoscope 28 Jan '18

28 Jan '18

hi i have 2 questions - i have a parametric plot : T=np.linespace(things) X=f(T)*np.cos(T) Y=f(T)*np.sin(T) i would like to get the first and the last point of this polar curve in the specs of "linspace" (https://docs.scipy.org/doc/numpy/reference/generated/numpy.linspace.html <https://docs.scipy.org/doc/numpy/reference/generated/numpy.linspace.html>) i dont see how to do that automatically is there something simple like "first(X)" and "last(X)" - i have a function : rotation(vect,theta) which takes vect=[a,b] and theta-rotates it by a matrix how to use this function or such a function to rotate a tuple (X,Y) with X,Y as described above in this message ? –––––––––––––––––––––––––– Vincent Douce :=: Mathoscope :=: http://mathoscope.xyz <http://mathoscope.xyz/> 06°13°11°07°26 Bagnères de Bigorre 65200

2 1

Multiple-field indexing: view vs copy in 1.14+
by Allan Haldane 28 Jan '18

28 Jan '18

Hello all, We are making a decision (again) about what to do about the behavior of multiple-field indexing of structured arrays: Should it return a view or a copy, and on what release schedule? As a reminder, this refers to operations like (1.13 behavior): >>> a = np.zeros(3, dtype=[('a', 'i4'), ('b', 'i4'), ('c', 'f4')]) >>> a[['a', 'c']] array([(0, 0.), (0, 0.), (0, 0.)], dtype=[('a', '<i4'), ('c', '<f4')] In numpy 1.14.0 we made this return a view instead of a copy, but downstream test failures suggest we reconsider. In our current implementation for 1.14.1, we have reverted this change, but still plan to go through with it in 1.15. See here for our discussion the problem and solutions: https://github.com/numpy/numpy/pull/10411 The two main options we have discussed are either to try to make the change in 1.15, or never make the change at all and always return a copy. Here are some pros and cons: Pros (change to view in 1.15) ============================= * Views are useful and convenient. Other forms of indexing also often return views so this is more consistent. * This change has been planned since numpy 1.7 in 2009, and there have been visible FutureWarnings about it since then. Anyone whose code will break should have seen the warnings. It has been extensively warned about in recent release notes. * Past discussions have supported the change. See my comment in the PR with many links to them and to other history. * Users have requested the change on the list. * Possibly a majority of the reported code failures were not actually caused by the change, but by another bug (#8100) involving np.load/np.save which this change exposed. If we push it off to 1.15, we will have time to fix this other bug. (There were no FutureWarnings for this breakage, of course). * The code that really will break is of the form a[['a', 'c']].view('i8') because the returned itemsize is different. This has raised FutureWarnings since numpy 1.7, and no users reported failures due to this change. In the PR we still try to mitigate this breakage by introducing a new method `pack_fields`, which converts the result into the 1.13 form, so that np.pack_fields(a[['a', 'c']]).view('i8') will work. Cons (keep returning a copy) ============================ * The extra convenience is not really that much, and fancy indexing also returns a copy instead of a view, so there is a precedent there. * We want to minimize compatibility breaks with old behavior. We've had a fair amount of discussion and complaints about how we break things in general. * We have lived with a "copy" for 8 years now. At some point the behavior gets set in stone for compatibility reasons. * Users have written to the list and github about their code breaking in 1.14.0. As far as I am aware, they all refer to the #8100 problem. * If a new function `pack_fields` is needed to guard against mishaps with the view behavior, that seems like a sign that keeping the copy behavior is the best option from an API perspective. My initial vote is go with the change in 1.15: The "view" code that will ultimately break (not the code related to #8100) has been sending FutureWarnings for many years, and I am not aware of any user complaints involving it: All the complaints so far would be fixed with #8100 in 1.15. Feel free to also discuss the related proposed change, to make np.diag return a view instead of a copy. That change has not been implemented yet, only proposed. Cheers, Allan

4 11

runtime warning in linalg.det
by josef.pktd＠gmail.com 28 Jan '18

28 Jan '18

I'm trying to figure out some warnings in the statsmodels test suite Why do the following raise RuntimeWarnings? np.linalg.det(np.ones((3,3))) C:\...\python-3.4.4.amd64\lib\site-packages\numpy\linalg\linalg.py:1776: RuntimeWarning: invalid value encountered in det r = _umath_linalg.det(a, signature=signature) Out[21]: 0.0 np.linalg.det(np.zeros((3,3))) C:\...\python-3.4.4.amd64\lib\site-packages\numpy\linalg\linalg.py:1776: RuntimeWarning: invalid value encountered in det r = _umath_linalg.det(a, signature=signature) Out[22]: 0.0 np.__version__ Out[23]: '1.11.2' and is there a way to distinguish those from user problems a = np.ones((3,3)) a[1,1] = np.nan np.linalg.det(a) C:\...\WinPython-64bit-3.4.4.5Qt5\python-3.4.4.amd64\lib\site-packages\numpy\linalg\linalg.py:1776: RuntimeWarning: invalid value encountered in det r = _umath_linalg.det(a, signature=signature) Out[26]: nan Josef

1 0

with np.testing.assert_raises still requires nose.
by Andrew Nelson 23 Jan '18

23 Jan '18

Dear np list, from a recent scipy PR (https://github.com/scipy/scipy/pull/8322) it appears that the `np.testing.assert_raises` context manager still requires nose to be installed: https://ci.appveyor.com/project/scipy/scipy/build/1.0.1550/job/s5nq8xqd0n72… The appveyor test is using numpy-1.14.0 This doesn't seem to be an issue on our travis tests, only the windows tests. Is this a bug? -- _____________________________________ Dr. Andrew Nelson _____________________________________

2 1

Moving NumPy's PRNG Forward
by Robert Kern 20 Jan '18

20 Jan '18

tl;dr: I think that our stream-compatibility policy is holding us back, and I think we can come up with a way forward with a new policy that will allow us to innovate without seriously compromising our reliability. To recap, our current policy for numpy.random is that we guarantee that the stream of random numbers from any of the methods of a seeded `RandomState` does not change from version to version (at least on the same hardware, OS, compiler, etc.), except in the case where we are fixing correctness bugs. That is, for code like this: prng = np.random.RandomState(12345) x = prng.normal(10.0, 3.0, size=100) `x` will be the same exact floats regardless of what version of numpy was installed. There seems to be a lot of pent-up motivation to improve on the random number generation, in particular the distributions, that has been blocked by our policy. I think we've lost a few potential first-time contributors that have run up against this wall. We have been pondering ways to allow for adding new core PRNGs and improve the distribution methods while maintaining stream-compatibility for existing code. Kevin Sheppard, in particular, has been working hard to implement new core PRNGs with a common API. https://github.com/bashtage/ng-numpy-randomstate Kevin has also been working to implement the several proposals that have been made to select different versions of distribution implementations. In particular, one idea is to pass something to the RandomState constructor to select a specific version of distributions (or switch out the core PRNG). Note that to satisfy the policy, the simplest method of seeding a RandomState will always give you the oldest version: what we have now. Kevin has recently come to the conclusion that it's not technically feasible to add the version-selection at all if we keep the stream-compatibility policy. https://github.com/numpy/numpy/pull/10124#issuecomment-350876221 I would argue that our current policy isn't providing the value that it claims to. In the first place, there are substantial holes in the reproducibility of streams across platforms. All of the integers (internal and external) are C `long`s, so integer overflows can cause variable streams if you use any of the rejection algorithms involving integers across Windows and Linux. Plain-old floating point arithmetic differences between platforms can cause similar issues (though rarer). Our policy of fixing bugs interferes with strict reproducibility. And our changes to non-random routines can interfere with the ability to reproduce the results of the whole software, independent of the PRNG stream. The multivariate normal implementation is even more vulnerable, as it uses `np.linalg` routines that may be affected by which LAPACK library numpy is built against much less changes that we might make to them in the normal course of development. At the time I established the policy (2008-9), there was significantly less tooling around for pinning versions of software. The PyPI/pip/setuptools ecosystem was in its infancy, VMs were slow cumbersome beasts mostly used to run Windows programs unavailable on Linux, and containerization a la Docker was merely a dream. A lot of resources have been put into reproducible research since then that pins the whole stack from OS libraries on up. The need to have stream-compatibility across numpy versions for the purpose of reproducible research is much diminished. I think that we can relax the strict stream-compatibility policy to allow innovation without giving up much practically-usable stability. Let's compare with Python's policy: https://docs.python.org/3.6/library/random.html#notes-on-reproducibility """ Most of the random module’s algorithms and seeding functions are subject to change across Python versions, but two aspects are guaranteed not to change: * If a new seeding method is added, then a backward compatible seeder will be offered. * The generator’s random() method will continue to produce the same sequence when the compatible seeder is given the same seed. """ I propose that we adopt a similar policy. This would immediately resolve many of the issues blocking innovation in the random distributions. Improvements to the distributions could be made at the same rhythm as normal features. No version-selection API would be required as you select the version by installing the desired version of numpy. By default, everyone gets the latest, best versions of the sampling algorithms. Selecting a different core PRNG could be easily achieved as ng-numpy-randomstate does it, by instantiating different classes. The different incompatible ways to initialize different core PRNGs (with unique features like selectable streams and the like) are transparently handled: different classes have different constructors. There is no need to jam all options for all core PRNGs into a single constructor. I would add a few more of the simpler distribution methods to the list that *is* guaranteed to remain stream-compatible, probably `randint()` and `bytes()` and maybe a couple of others. I would appreciate input on the matter. The current API should remain available and working, but not necessarily with the same algorithms. That is, for code like the following: prng = np.random.RandomState(12345) x = prng.normal(10.0, 3.0, size=100) `x` is still guaranteed to be 100 normal variates with the appropriate mean and standard deviation, but they might be computed by the ziggurat method from PCG-generated bytes (or whatever new default core PRNG we have). As an alternative, we may also want to leave `np.random.RandomState` entirely fixed in place as deprecated legacy code that is never updated. This would allow current unit tests that depend on the stream-compatibility that we previously promised to still pass until they decide to update. Development would move to a different class hierarchy with new names. I am personally not at all interested in preserving any stream compatibility for the `numpy.random.*` aliases or letting the user swap out the core PRNG for the global PRNG that underlies them. `np.random.seed()` should be discouraged (if not outright deprecated) in favor of explicitly passing around instances. In any case, we have a lot of different options to discuss if we decide to relax our stream-compatibility policy. At the moment, I'm not pushing for any particular changes to the code, just the policy in order to enable a more wide-ranging field of options that we have been able to work with so far. Thanks. -- Robert Kern

5 7

Re: [Numpy-discussion] Moving NumPy's PRNG Forward
by Peter Creasey 19 Jan '18

19 Jan '18

> Date: Fri, 19 Jan 2018 23:55:57 +0900 > From: Robert Kern <robert.kern(a)gmail.com> > > tl;dr: I think that our stream-compatibility policy is holding us back, and > I think we can come up with a way forward with a new policy that will allow > us to innovate without seriously compromising our reliability. > > I propose that we adopt a similar policy. This would immediately resolve > many of the issues blocking innovation in the random distributions. > Improvements to the distributions could be made at the same rhythm as > normal features. No version-selection API would be required as you select > the version by installing the desired version of numpy. By default, > everyone gets the latest, best versions of the sampling algorithms. > Selecting a different core PRNG could be easily achieved as > ng-numpy-randomstate does it, by instantiating different classes. The > different incompatible ways to initialize different core PRNGs (with unique > features like selectable streams and the like) are transparently handled: > different classes have different constructors. There is no need to jam all > options for all core PRNGs into a single constructor. +1 I think I have a general comment that random streams are probably over-used in testing, in particular: 1. If you need a small amount (100s) of random values you should just spit them out to code/files for repeatability and, 2. If you need a large amount of random data then maybe you want to rethink your testing strategy. In particular if you need millions of points to hit a few edge cases then your code is significantly more mature (low failure rate) than your test suite (inefficient), and you could probably target your tests a bit more (I'm as guilty of this as others). Having said that I don't think NumPy today is in a position to fight the inertia of 2 - numpy.random has given everyone easy tools to make large and (nearly always) reproducible sequences, and they get used a lot. On the other hand staying on top of performance requires an active code-base, and for a reproducible PRNG almost any change is a breaking-change. Allowing non-backwards-compatible streams concurrently with the old style seems the logical way forward. So +1. Best, Peter

1 0

Behavior of rint?
by Matthew Brett 19 Jan '18

19 Jan '18

Hi, Sorry for my confusion, but I noticed (as a result of the discussion here [1]) that np.rint and the fallback C function [2] seem to round to even. But - my impression was that C rint, by default, rounds down [3]. Is numpy rint not behaving the same way as the GNU C library rint? In [4]: np.rint(np.arange(0.5, 11)) Out[4]: array([ 0., 2., 2., 4., 4., 6., 6., 8., 8., 10., 10.]) In [5]: np.round(np.arange(0.5, 11)) Out[5]: array([ 0., 2., 2., 4., 4., 6., 6., 8., 8., 10., 10.]) Cheers, Matthew [1] https://github.com/nipy/dipy/issues/1402 [2] https://github.com/numpy/numpy/blob/master/numpy/core/src/npymath/npy_math_… [3] https://www.gnu.org/software/libc/manual/html_node/Rounding-Functions.html

4 10

building numpy with python3.7
by Hannes Breytenbach 18 Jan '18

18 Jan '18

Hi devs! This is my first post to the discussion list! Has anyone tried to build numpy with python3.7.0a3? I get the following gcc errors during compile: . . . compiling C sources C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Inumpy/core/include -Ibuild/src.linux-x86_64-3.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/local/include/python3.7m -Ibuild/src.linux-x86_64-3.7/numpy/core/src/private -Ibuild/src.linux-x86_64-3.7/numpy/core/src/npymath -Ibuild/src.linux-x86_64-3.7/numpy/core/src/private -Ibuild/src.linux-x86_64-3.7/numpy/core/src/npymath -Ibuild/src.linux-x86_64-3.7/numpy/core/src/private -Ibuild/src.linux-x86_64-3.7/numpy/core/src/npymath -c' gcc: numpy/random/mtrand/mtrand.c numpy/random/mtrand/mtrand.c: In function ‘__Pyx__ExceptionSave’: numpy/random/mtrand/mtrand.c:40970:19: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’? *type = tstate->exc_type; ^~ . . . error: Command "gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Inumpy/core/include -Ibuild/src.linux-x86_64-3.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/local/include/python3.7m -Ibuild/src.linux-x86_64-3.7/numpy/core/src/private -Ibuild/src.linux-x86_64-3.7/numpy/core/src/npymath -Ibuild/src.linux-x86_64-3.7/numpy/core/src/private -Ibuild/src.linux-x86_64-3.7/numpy/core/src/npymath -Ibuild/src.linux-x86_64-3.7/numpy/core/src/private -Ibuild/src.linux-x86_64-3.7/numpy/core/src/npymath -c numpy/random/mtrand/mtrand.c -o build/temp.linux-x86_64-3.7/numpy/random/mtrand/mtrand.o -MMD -MF build/temp.linux-x86_64-3.7/numpy/random/mtrand/mtrand.o.d" failed with exit status 1 Version info: ------------- gcc --version: gcc (Ubuntu 6.3.0-12ubuntu2) 6.3.0 2017040 numpy version: 1.15.0.dev0+d233e1f The same error comes up when building via pip. I don't know enough about the underlying C code to know how to debug this. Any help would be greatly appreciated! Cheers, -- Hannes Breytenbach PhD Candidate South African Astronomical Observatory +27 82 726 9311

5 10