<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jun 3, 2018 at 9:08 PM, Robert Kern <span dir="ltr"><<a href="mailto:robert.kern@gmail.com" target="_blank">robert.kern@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><span class=""><div dir="ltr">On Sun, Jun 3, 2018 at 5:46 PM <<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>> wrote:<br></div></span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Sun, Jun 3, 2018 at 8:21 PM, Robert Kern <span dir="ltr"><<a href="mailto:robert.kern@gmail.com" target="_blank">robert.kern@gmail.com</a>></span> wrote:<br></span><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_quote"><div><div class="gmail_quote" style="color:rgb(34,34,34);font-family:sans-serif;font-size:13px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><div><br></div></div></div><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>The list of ``StableRandom`` methods should be chosen to support unit tests:</div><div><br></div><div> * ``.randint()``</div><div> * ``.uniform()``</div><div> * ``.normal()``</div><div> * ``.standard_normal()``</div><div> * ``.choice()``</div><div> * ``.shuffle()``</div><div> * ``.permutation()``</div></div></div></blockquote><div><br></div></span><div><a href="https://github.com/numpy/numpy/pull/11229#discussion_r192604311" target="_blank">https://github.com/numpy/<wbr>numpy/pull/11229#discussion_<wbr>r192604311</a><br></div><div>@bashtage writes:<br></div>> standard_gamma and standard_exponential are important enough to be included here IMO.<div><br></div><div>"Importance" was not my criterion, only whether they are used in unit test suites. This list was just off the top of my head for methods that I think were actually used in test suites, so I'd be happy to be shown live tests that use other methods. I'd like to be a *little* conservative about what methods we stick in here, but we don't have to be *too* conservative, since we are explicitly never going to be modifying these.</div></div></div></div></blockquote><div><br></div><div>That's one area where I thought the selection is too narrow.</div><div>We should be able to get a stable stream from the uniform for some distributions.</div><div><br></div><div>However, according to the Wikipedia description Poisson doesn't look easy. I just wrote a unit test for statsmodels using Poisson random numbers with hard coded numbers for the regression tests.</div></span></div></div></div></blockquote><div><br></div><div>I'd really rather people do this than use StableRandom; this is best practice, as I see it, if your tests involve making precise comparisons to expected results.</div></div></div></blockquote><div><br></div><div>I hardcoded the results not the random data. So the unit tests rely on a reproducible stream of Poisson random numbers.</div><div>I don't want to save 500 (100 or 1000) observations in a csv file for every variation of the unit test that I run.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>StableRandom is intended as a crutch so that the pain of moving existing unit tests away from the deprecated RandomState is less onerous. I'd really rather people write better unit tests!</div><div><br></div><div>In particular, I do not want to add any of the integer-domain distributions (aside from shuffle/permutation/choice) as these are the ones that have the platform-dependency issues with respect to 32/64-bit `long` integers. They'd be unreliable for unit tests even if we kept them stable over time.</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>I'm not sure which other distributions are common enough and not easily reproducible by transformation. E.g. negative binomial can be reproduces by a gamma-poisson mixture.</div><div><br></div><div>On the other hand normal can be easily recreated from standard_normal.</div></div></div></div></blockquote><div><br></div></span><div>I was mostly motivated by making it a bit easier to mechanically replace uses of randn(), which is probably even more common than normal() and standard_normal() in unit tests.</div><span class=""><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Would it be difficult to keep this list large, given that it should be frozen, low maintenance code ?</div></div></div></div></blockquote><div><br></div></span><div>I admit that I had in mind non-statistical unit tests. That is, tests that didn't depend on the precise distribution of the inputs.</div></div></div></blockquote><div><br></div><div>The problem is that the unit test in `stats` rely on precise inputs (up to some numerical noise).</div><div>For example p-values themselves are uniformly distributed if the hypothesis test works correctly. That mean if I don't have control over the inputs, then my p-value could be anything in (0, 1). So either we need a real dataset, save all the random numbers in a file or have a reproducible set of random numbers.</div><div><br></div><div>95% of the unit tests that I write are for statistics. A large fraction of them don't rely on the exact distribution, but do rely on a random numbers that are "good enough".</div><div>For example, when writing unit test, then I get every once in a while or sometimes more often a "bad" stream of random numbers, for which convergence might fail or where the estimated numbers are far away from the true numbers, so test tolerance would have to be very high.</div><div>If I pick one of the seeds that looks good, then I can have tighter unit test tolerance to insure results are good in a nice case.</div><div><br></div><div>The problem is that we cannot write robust unit tests for regression tests without stable inputs.</div><div>E.g. I verified my results with a Monte Carlo with 5000 replications and 1000 Poisson observations in each. </div><div>Results look close to expected and won't depend much on the exact stream of random variables.</div><div>But the Monte Carlo for each variant of the test took about 40 seconds. Doing this for all option combination and dataset specification takes too long to be feasible in a unit test suite.<br></div><div>So I rely on numpy's stable random numbers and hard code the results for a specific random sample in the regression unit tests.</div><div><br></div><div>Josef</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><span class="HOEnZb"><font color="#888888"><div> </div></font></span></div><span class="HOEnZb"><font color="#888888">-- <br><div dir="ltr" class="m_-2208704920581523974gmail_signature" data-smartmail="gmail_signature">Robert Kern</div></font></span></div>
<br>______________________________<wbr>_________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org">NumPy-Discussion@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/numpy-<wbr>discussion</a><br>
<br></blockquote></div><br></div></div>