[Numpy-discussion] NEP: Random Number Generator Policy
Ralf Gommers
ralf.gommers at gmail.com
Sun Jun 10 20:46:36 EDT 2018
On Sun, Jun 3, 2018 at 9:23 PM, Warren Weckesser <warren.weckesser at gmail.com
> wrote:
>
>
> On Sun, Jun 3, 2018 at 11:20 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
>
>>
>>
>> On Sun, Jun 3, 2018 at 6:54 PM, <josef.pktd at gmail.com> wrote:
>>
>>>
>>>
>>> On Sun, Jun 3, 2018 at 9:08 PM, Robert Kern <robert.kern at gmail.com>
>>> wrote:
>>>
>>>> On Sun, Jun 3, 2018 at 5:46 PM <josef.pktd at gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Sun, Jun 3, 2018 at 8:21 PM, Robert Kern <robert.kern at gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> The list of ``StableRandom`` methods should be chosen to support unit
>>>>>>> tests:
>>>>>>>
>>>>>>> * ``.randint()``
>>>>>>> * ``.uniform()``
>>>>>>> * ``.normal()``
>>>>>>> * ``.standard_normal()``
>>>>>>> * ``.choice()``
>>>>>>> * ``.shuffle()``
>>>>>>> * ``.permutation()``
>>>>>>>
>>>>>>
>>>>>> https://github.com/numpy/numpy/pull/11229#discussion_r192604311
>>>>>> @bashtage writes:
>>>>>> > standard_gamma and standard_exponential are important enough to be
>>>>>> included here IMO.
>>>>>>
>>>>>> "Importance" was not my criterion, only whether they are used in unit
>>>>>> test suites. This list was just off the top of my head for methods that I
>>>>>> think were actually used in test suites, so I'd be happy to be shown live
>>>>>> tests that use other methods. I'd like to be a *little* conservative about
>>>>>> what methods we stick in here, but we don't have to be *too* conservative,
>>>>>> since we are explicitly never going to be modifying these.
>>>>>>
>>>>>
>>>>> That's one area where I thought the selection is too narrow.
>>>>> We should be able to get a stable stream from the uniform for some
>>>>> distributions.
>>>>>
>>>>> However, according to the Wikipedia description Poisson doesn't look
>>>>> easy. I just wrote a unit test for statsmodels using Poisson random numbers
>>>>> with hard coded numbers for the regression tests.
>>>>>
>>>>
>>>> I'd really rather people do this than use StableRandom; this is best
>>>> practice, as I see it, if your tests involve making precise comparisons to
>>>> expected results.
>>>>
>>>
>>> I hardcoded the results not the random data. So the unit tests rely on a
>>> reproducible stream of Poisson random numbers.
>>> I don't want to save 500 (100 or 1000) observations in a csv file for
>>> every variation of the unit test that I run.
>>>
>>
>> I agree, hardcoding numbers in every place where seeded random numbers
>> are now used is quite unrealistic.
>>
>> It may be worth having a look at test suites for scipy, statsmodels,
>> scikit-learn, etc. and estimate how much work this NEP causes those
>> projects. If the devs of those packages are forced to do large scale
>> migrations from RandomState to StableState, then why not instead keep
>> RandomState and just add a new API next to it?
>>
>>
>
> As a quick and imperfect test, I monkey-patched numpy so that a call to
> numpy.random.seed(m) actually uses m+1000 as the seed. I ran the tests
> using the `runtests.py` script:
>
> *seed+1000, using 'python runtests.py -n' in the source directory:*
>
> 236 failed, 12881 passed, 1248 skipped, 585 deselected, 84 xfailed, 7
> xpassed
>
>
> Most of the failures are in scipy.stats:
>
> *seed+1000, using 'python runtests.py -n -s stats' in the source
> directory:*
>
> 203 failed, 1034 passed, 4 skipped, 370 deselected, 4 xfailed, 1 xpassed
>
>
> Changing the amount added to the seed or running the tests using the
> function `scipy.test("full")` gives different (but similar magnitude)
> results:
>
> *seed+1000, using 'import scipy; scipy.test("full")' in an ipython shell:*
>
> 269 failed, 13359 passed, 1271 skipped, 134 xfailed, 8 xpassed
>
> *seed+1, using 'python runtests.py -n' in the source directory:*
>
> 305 failed, 12812 passed, 1248 skipped, 585 deselected, 84 xfailed, 7
> xpassed
>
>
> I suspect many of the tests will be easy to update, so fixing 300 or so
> tests does not seem like a monumental task.
>
It's all not monumental, but it adds up quickly. In addition to changing
tests, one will also need compatibility code when supporting multiple numpy
versions (e.g. scipy when get a copy of RandomStable in
scipy/_lib/_numpy_compat.py).
A quick count of just np.random.seed occurrences with ``$ grep -roh
--include \*.py np.random.seed . | wc -w`` for some packages:
numpy: 77
scipy: 462
matplotlib: 204
statsmodels: 461
pymc3: 36
scikit-image: 63
scikit-learn: 69
keras: 46
pytorch: 0
tensorflow: 368
astropy: 24
And note, these are *not* incorrect/broken usages, this is code that works
and has done so for years.
Conclusion: the current proposal will cause work for the vast majority of
libraries that depends on numpy. The total amount of that work will
certainly not be counted in person-days/weeks, and more likely in years
than months. So I'm not convinced yet that the current proposal is the best
way forward.
Ralf
I haven't looked into why there are 585 deselected tests; maybe there are
> many more tests lurking there that will have to be updated.
>
> Warren
>
>
>
> Ralf
>>
>>
>>
>>>
>>>
>>>>
>>>> StableRandom is intended as a crutch so that the pain of moving
>>>> existing unit tests away from the deprecated RandomState is less onerous.
>>>> I'd really rather people write better unit tests!
>>>>
>>>> In particular, I do not want to add any of the integer-domain
>>>> distributions (aside from shuffle/permutation/choice) as these are the ones
>>>> that have the platform-dependency issues with respect to 32/64-bit `long`
>>>> integers. They'd be unreliable for unit tests even if we kept them stable
>>>> over time.
>>>>
>>>>
>>>>> I'm not sure which other distributions are common enough and not
>>>>> easily reproducible by transformation. E.g. negative binomial can be
>>>>> reproduces by a gamma-poisson mixture.
>>>>>
>>>>> On the other hand normal can be easily recreated from standard_normal.
>>>>>
>>>>
>>>> I was mostly motivated by making it a bit easier to mechanically
>>>> replace uses of randn(), which is probably even more common than normal()
>>>> and standard_normal() in unit tests.
>>>>
>>>>
>>>>> Would it be difficult to keep this list large, given that it should be
>>>>> frozen, low maintenance code ?
>>>>>
>>>>
>>>> I admit that I had in mind non-statistical unit tests. That is, tests
>>>> that didn't depend on the precise distribution of the inputs.
>>>>
>>>
>>> The problem is that the unit test in `stats` rely on precise inputs (up
>>> to some numerical noise).
>>> For example p-values themselves are uniformly distributed if the
>>> hypothesis test works correctly. That mean if I don't have control over the
>>> inputs, then my p-value could be anything in (0, 1). So either we need a
>>> real dataset, save all the random numbers in a file or have a reproducible
>>> set of random numbers.
>>>
>>> 95% of the unit tests that I write are for statistics. A large fraction
>>> of them don't rely on the exact distribution, but do rely on a random
>>> numbers that are "good enough".
>>> For example, when writing unit test, then I get every once in a while or
>>> sometimes more often a "bad" stream of random numbers, for which
>>> convergence might fail or where the estimated numbers are far away from the
>>> true numbers, so test tolerance would have to be very high.
>>> If I pick one of the seeds that looks good, then I can have tighter unit
>>> test tolerance to insure results are good in a nice case.
>>>
>>> The problem is that we cannot write robust unit tests for regression
>>> tests without stable inputs.
>>> E.g. I verified my results with a Monte Carlo with 5000 replications and
>>> 1000 Poisson observations in each.
>>> Results look close to expected and won't depend much on the exact stream
>>> of random variables.
>>> But the Monte Carlo for each variant of the test took about 40 seconds.
>>> Doing this for all option combination and dataset specification takes too
>>> long to be feasible in a unit test suite.
>>> So I rely on numpy's stable random numbers and hard code the results for
>>> a specific random sample in the regression unit tests.
>>>
>>> Josef
>>>
>>>
>>>
>>>>
>>>> --
>>>> Robert Kern
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180610/2efe793b/attachment-0001.html>
More information about the NumPy-Discussion
mailing list