Mailman 3 September 2017 - NumPy-Discussion

ENH: faster histograms (PR #9627)
by Johann Goetz 09 Sep '17

09 Sep '17

I have received and addressed many great suggestions and critiques from @juliantaylor <https://github.com/juliantaylor> and @eric-wieser <https://github.com/eric-wieser> for pull request #9627 <https://github.com/numpy/numpy/pull/9627> which moves the np.histogram() and np.histogramdd() methods into C. Speed ups of 2x to 20x were realized for large sample data depending on the percentage of sample points that lay outside the histogramming range. For details more see my report here <https://gist.github.com/theodoregoetz/10d2351421689bf2660b4f2fca350e6e>. I'd like to know now how to proceed with this pull request. I.e., how can I move the process along. Additionally, I'd like to propose a new feature which I'm sure requires some discussion: The inspiration for this effort came from the fast-histogram <https://pypi.python.org/pypi/fast-histogram> python package which is still faster because it ignores ULP-level correctness. Towards the bottom of my report, I suggest adding a new option to the histogramming methods to ignore ULP corrections which would make the numpy implementation on-par with fast-histogram's. Something like: np.histogram(sample, bins=10, range=(0, 10), fast=True) which would raise an exception or ignore the "fast" parameter perhaps if bins were given as a list of edges: np.histogram(sample, bins=[0,1,2,3], fast=True) # not fast. I think I'd shy away from testing the bin-uniformity since it is very hard to do without a specified tolerance. This can be done by the user with something like this: np.all(np.abs(np.diff(np.diff(edges))) <= \ 2**6 * np.finfo(edges.dtype).eps) Or by comparison with the output of np.linspace(). -- Johann.

1 0

NumPy default citation
by Stefan van der Walt 09 Sep '17

09 Sep '17

Hi, everyone I see that the NumPy homepage does not have a "Citation" section. Furthermore, on scipy.org, the default NumPy citation points to a short summary paper that I wrote with Gael V & Stephen C. While it's a reasonable introduction to three core concepts behind NumPy, attribution should certainly go to Travis & the community. Shall we add a citation to Travis's "Guide to NumPy (2nd ed.)" on both pages? Best regards Stéfan

6 6

ENH: Add complex random normal generator (PR #9561)
by Kevin Sheppard 08 Sep '17

08 Sep '17

I would like to add a complex random normal generator to mtrand/RandomState. A scalar complex normal is a (double) bivariate normal. The main motivation is to simplify the construction of complex normals where are generally parameterized in terms of three values: location, covariance and relation. location is the same as in a standard normal. The covariance and the relation jointly determine the variance of the real and imaginary parts as well as the covariance between the two. #1 The initial implementation in the PR has followed the standard template for scalar RV generators with three paths, scalar->scalar, scalar->array and array->array. It is bulky since the existing array fillers that handle the scalar->array and array->array for double rvs cannot be used. It supports broadcasting and has a similar API to other scalar RV generators (e.g. normal). #2 The PR discussion has moved towards exploiting the relationship with the multivariate normal. Currently the MV normal doesn't broadcast, and so following this path would only allow the scalar->scalar and the scalar->array paths. This could theoretically be extended to allow broadcasting if multivariate_normal was extended to allow broadcasting. #3 If broadcasting is off the table, then it might make more sense to skip a scalar complex normal and just move directly to a multivariate_complex_normal since this is also just a higher dimension (double) multivariate normal. This function could just wrap multivariate_normal and would be relatively straightforward forward. The only down side of this path is that it would not easily support a scalar->scalar path, although this could be added. This probably isn't much of a performance hit for following #2 or #3. I checked how normal and multivariate normal perform for large draws: %timeit np.random.normal(2.0,4.0,size=1000000) 30.8 ms ± 125 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) %timeit np.random.multivariate_normal([2.0],[[4.0]],size=1000000) 32.2 ms ± 308 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) For smaller draws the performance difference is larger: %timeit np.random.normal(2.0,4.0,size=10) 2.95 µs ± 16.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) %timeit np.random.multivariate_normal([2.0],[[4.0]],size=10) 49.4 µs ± 249 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each) And for scalars the scalar path is about 30x faster than the multivariate path. It is also worth noting that multivariate_normal will only return a vector even if the inputs only generate a single scalar. %timeit np.random.normal(2.0,4.0) 1.42 µs ± 3.05 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) %timeit np.random.multivariate_normal([2.0],[[4.0]]) 47.9 µs ± 167 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each) It would be helpful do determine which path, #1 Clone standard scalar generator with 3 paths including broadcasting #2 Scalar generator using multivariate normal, excluding broadcasting #3 Multivariate generator using multivariate normal, excluding broadcasting is the preferred one. Kevin

1 0

np.array, copy=False and memmap
by Nisoli Isaia 06 Sep '17

06 Sep '17

Dear all, I have a question about the behaviour of y = np.array(x, copy=False, dtype='float32') when x is a memmap. If we check the memmap attribute of mmap print "mmap attribute", y._mmap numpy tells us that y is not a memmap. But the following code snippet crashes the python interpreter # opens the memmap with open(filename,'r+b') as f: mm = mmap.mmap(f.fileno(),0) x = np.frombuffer(mm, dtype='float32') # builds an array from the memmap, with the option copy=False y = np.array(x, copy=False, dtype='float32') print "before", y # closes the file mm.close() print "after", y In my code I use memmaps to share read-only objects when doing parallel processing and the behaviour of np.array, even if not consistent, it's desirable. I share scipy sparse matrices over many processes and if np.array would make a copy when dealing with memmaps this would force me to rewrite part of the sparse matrices code. Would it be possible in the future releases of numpy to have np.array check, if copy is false, if y is a memmap and in that case return a full memmap object instead of slicing it? Best wishes Isaia P.S. A longer account of the issue may be found on my university blog http://www.im.ufrj.br/nisoli/blog/?p=131 -- Isaia Nisoli

4 5

New distribution
by Charles R Harris 06 Sep '17

06 Sep '17

Hi All, This is a heads up that there is a pull request <https://github.com/numpy/numpy/pull/9561> adding a univariate complex_normal <https://en.wikipedia.org/wiki/Complex_normal_distribution>distribution. Anyone interested in this should take a look at the PR. I'd also be interested if there was a desire for the multivariate version. Chuck

1 0

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 132, Issue 1
by David Nicholson 05 Sep '17

05 Sep '17

@paul: not quite the same as building a .citation method into every module but there is https://github.com/duecredit/duecredit On Tue, Sep 5, 2017 at 5:21 PM, <numpy-discussion-request(a)python.org> wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)python.org > > You can reach the person managing the list at > numpy-discussion-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. NumPy default citation (Stefan van der Walt) > 2. Re: NumPy default citation (Charles R Harris) > 3. Re: NumPy default citation (Stefan van der Walt) > 4. Re: NumPy default citation (Paul Hobson) > 5. Re: NumPy default citation (Feng Yu) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 05 Sep 2017 11:36:58 -0700 > From: Stefan van der Walt <stefanv(a)berkeley.edu> > To: Discussion of Numerical Python <numpy-discussion(a)python.org> > Subject: [Numpy-discussion] NumPy default citation > Message-ID: > <1504636618.478273.1096130488.4D658E7C(a)webmail.messagingengine.com > > > Content-Type: text/plain; charset="utf-8" > > Hi, everyone > > I see that the NumPy homepage does not have a "Citation" section. > Furthermore, on scipy.org, the default NumPy citation points to a short > summary paper that I wrote with Gael V & Stephen C. While it's a > reasonable introduction to three core concepts behind NumPy, attribution > should certainly go to Travis & the community. > > Shall we add a citation to Travis's "Guide to NumPy (2nd ed.)" on both > pages? > > Best regards > St?fan > > > ------------------------------ > > Message: 2 > Date: Tue, 5 Sep 2017 14:25:20 -0600 > From: Charles R Harris <charlesr.harris(a)gmail.com> > To: Discussion of Numerical Python <numpy-discussion(a)python.org> > Subject: Re: [Numpy-discussion] NumPy default citation > Message-ID: > <CAB6mnx+cKc=8htXjDTnbs4_2=gMwnTf8HT4ziQkcCWM6W5FQTg@mail. > gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Tue, Sep 5, 2017 at 12:36 PM, Stefan van der Walt <stefanv(a)berkeley.edu > > > wrote: > > > Hi, everyone > > > > I see that the NumPy homepage does not have a "Citation" section. > > Furthermore, on scipy.org, the default NumPy citation points to a short > > summary paper that I wrote with Gael V & Stephen C. While it's a > > reasonable introduction to three core concepts behind NumPy, attribution > > should certainly go to Travis & the community. > > > > Shall we add a citation to Travis's "Guide to NumPy (2nd ed.)" on both > > > > What is the citation for? > > Chuck >

1 0