[Numpy-discussion] overhauling numpy.random and randomgen

[Matti Picus]

Thanks to the work of Kevin Sheppard, Robert Kern and others, the branch 
to merge randomgen https://github.com/bashtage/randomgen into numpy is 
ready for final review.

The branch is here https://github.com/numpy/numpy/pull/13163. It is 
fully backward compatible: numpy.random.mtrand, 
numpy.random.RandomState, and the various stateful distributions from 
RandomState available as numpy.random.* produce the same streams as the 
current versions. The branch is intended to implement NEP 19 
https://www.numpy.org/neps/nep-0019-rng-policy.html


The biggest change is that now there are a variety of random number 
generators available 
https://6722-908607-gh.circle-artifacts.com/0/home/circleci/repo/doc/build/html/reference/random/brng/index.html, 
and a class numpy.random.RandomGenerator that can produce all the 
distributions from RandomState. A RandomGenerator instance is provided 
for convenience as numpy.random.gen


Additional enhancements 
https://6722-908607-gh.circle-artifacts.com/0/home/circleci/repo/doc/build/html/reference/random/new-or-different.html 
allow convenient use of the new constructs in CFFI, Numba, Ctypes, and 
Cython.


There are a few things to address before merging:

- Review the new constructs and other APIS

- Decide which BRNGs to include in the first release

- Check that your packages still work with the new implementations. You 
can do this by creating a new virtualenv and installing numpy via pip 
install git+https://github.com/mattip/numpy.git@randomgen 
<https://github.com/user/repo.git@branch>


We will try to have a final video call about the branch during the 
upcoming meeting May 10-11, more details will follow once we schedule 
the call. The goal is to merge it for the upcoming 1.17 release.


The expectation is that this first merge will be followed by 
implementation and documentation tweaks and improvements, but we hope to 
get the major pieces in place as much as possible now.


Matti


Notes:


Sorry for the long urls, they link to the generated documentation from 
CI. They may not be available a few weeks from now.

There is a tracking issue for further work related to the PR 
numpy.random https://github.com/numpy/numpy/issues/13164