From nico.schloemer at gmail.com Thu Mar 2 05:27:45 2017 From: nico.schloemer at gmail.com (=?UTF-8?Q?Nico_Schl=C3=B6mer?=) Date: Thu, 02 Mar 2017 10:27:45 +0000 Subject: [Numpy-discussion] vectorization vs. numpy.linalg (shape (3, 3, 777) vs shape (777, 3, 3)) Message-ID: Hi everyone, When trying to speed up my code, I noticed that simply by reordering my data I could get more than twice as fast for the simplest operations: ``` import numpy a = numpy.random.rand(50, 50, 50) %timeit a[0] + a[1] 1000000 loops, best of 3: 1.7 ?s per loop %timeit a[:, 0] + a[:, 1] 100000 loops, best of 3: 4.42 ?s per loop %timeit a[..., 0] + a[..., 1] 100000 loops, best of 3: 5.99 ?s per loop ``` This makes sense considering the fact that, by default, NumPy features C-style memory allocation: the last index runs fastest. The blocks that are added with `a[0] + a[1]` are contiguous in memory, so cache is nicely made use of. As opposed to that, the blocks that are added with `a[:, 0] + a[:, 1]` are not contiguous, even more so with `a[..., 0] + a[..., 1]`; hence the slowdown. Would that be the correct explanation? If yes, I'm wondering why most numpy.linalg methods, when vectorized, put the vector index up front. E.g., to mass-compute determinants, one has to do ``` a = numpy.random.rand(777, 3, 3) numpy.linalg.det(a) ``` This way, all 3x3 matrices form a memory block, so if you do `det` block by block, that'll be fine. However, vectorized operations (like `+` above) will be slower than necessary. Any background on this? (I came across this when having to rearrange my data (swapaxes, rollaxis) from shape (3, 3, 777) (which allows for fast vectorized operations in the rest of the code) to (777, 3, 3) for using numpy's svd.) Cheers, Nico -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Sun Mar 5 09:52:55 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sun, 05 Mar 2017 15:52:55 +0100 Subject: [Numpy-discussion] vectorization vs. numpy.linalg (shape (3, 3, 777) vs shape (777, 3, 3)) In-Reply-To: References: Message-ID: <1488725575.6765.1.camel@sipsolutions.net> On Thu, 2017-03-02 at 10:27 +0000, Nico Schl?mer wrote: > Hi everyone, > > When trying to speed up my code, I noticed that simply by reordering > my data I could get more than twice as fast for the simplest > operations: > ``` > import numpy > a = numpy.random.rand(50, 50, 50) > > %timeit a[0] + a[1] > 1000000 loops, best of 3: 1.7 ?s per loop > > %timeit a[:, 0] + a[:, 1] > 100000 loops, best of 3: 4.42 ?s per loop > > %timeit a[..., 0] + a[..., 1] > 100000 loops, best of 3: 5.99 ?s per loop > ``` > This makes sense considering the fact that, by default, NumPy > features C-style memory allocation: the last index runs fastest. The > blocks that are added with `a[0]?+ a[1]` are contiguous in memory, so > cache is nicely made use of. As opposed to that, the blocks that are > added with `a[:, 0] + a[:, 1]` are not contiguous, even more so with > `a[..., 0] + a[..., 1]`; hence the slowdown. Would that be the > correct explanation? > > If yes, I'm wondering why most numpy.linalg methods, when vectorized, > put the vector index up front. E.g., to mass-compute determinants, > one has to do > ``` > a = numpy.random.rand(777, 3, 3) > numpy.linalg.det(a) > ``` > This way, all 3x3 matrices form a memory block, so if you do `det` > block by block, that'll be fine. However, vectorized operations (like > `+` above) will be slower than necessary. > Any background on this?? > I am honestly not sure where you are going at. This order seems the more natural order for most operations. Also numpy does not create copies normally even for transposed data (though some functions may internally, numpy just _defaults_ to C-order). So of course what is faster depends on your use-case, but if you have an operation on many 3x3 arrays the way numpy does it is the more natural way. If you do other things that are faster the other way around, you will have to decide which operation is the more important one overall. - Sebastian > (I came across this when having to rearrange my data (swapaxes, > rollaxis) from shape (3, 3, 777) (which allows for fast vectorized > operations in the rest of the code) to (777, 3, 3) for using numpy's > svd.) > > Cheers, > Nico > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From nico.schloemer at gmail.com Sun Mar 5 10:46:50 2017 From: nico.schloemer at gmail.com (=?UTF-8?Q?Nico_Schl=C3=B6mer?=) Date: Sun, 05 Mar 2017 15:46:50 +0000 Subject: [Numpy-discussion] vectorization vs. numpy.linalg (shape (3, 3, 777) vs shape (777, 3, 3)) In-Reply-To: <1488725575.6765.1.camel@sipsolutions.net> References: <1488725575.6765.1.camel@sipsolutions.net> Message-ID: > I am honestly not sure where you are going at. This order seems the more natural order for most operations. Not sure what you mean by "natural". The most basic operations like `a[0] + a[1]` are several times faster than `a[...,0] + a[..., 1]`. Perhaps one can come up with code examples that don't use those operations at all, but I would guess that'll be a rare case. My point: If your code uses vector operations (as `+`) _and_ numpy.linalg functions, your vector operations will be several times slower than necessary. One way around this would perhaps be to initialize all your data in Fortran-style data layout, where ` a[...,0] + a[..., 1]` is indeed faster than `a[0] + a[1]`. What I'll end up doing is probably to rewrite whatever numpy.linalg function I need for C-style ordering. For dets of 2x2 or 3x3 matrices, this shouldn't be to hard (`a[0][0] +a[1][1] - a[0][1] - a[1][0]`). :) Cheers, Nico On Sun, Mar 5, 2017 at 3:53 PM Sebastian Berg wrote: On Thu, 2017-03-02 at 10:27 +0000, Nico Schl?mer wrote: > Hi everyone, > > When trying to speed up my code, I noticed that simply by reordering > my data I could get more than twice as fast for the simplest > operations: > ``` > import numpy > a = numpy.random.rand(50, 50, 50) > > %timeit a[0] + a[1] > 1000000 loops, best of 3: 1.7 ?s per loop > > %timeit a[:, 0] + a[:, 1] > 100000 loops, best of 3: 4.42 ?s per loop > > %timeit a[..., 0] + a[..., 1] > 100000 loops, best of 3: 5.99 ?s per loop > ``` > This makes sense considering the fact that, by default, NumPy > features C-style memory allocation: the last index runs fastest. The > blocks that are added with `a[0] + a[1]` are contiguous in memory, so > cache is nicely made use of. As opposed to that, the blocks that are > added with `a[:, 0] + a[:, 1]` are not contiguous, even more so with > `a[..., 0] + a[..., 1]`; hence the slowdown. Would that be the > correct explanation? > > If yes, I'm wondering why most numpy.linalg methods, when vectorized, > put the vector index up front. E.g., to mass-compute determinants, > one has to do > ``` > a = numpy.random.rand(777, 3, 3) > numpy.linalg.det(a) > ``` > This way, all 3x3 matrices form a memory block, so if you do `det` > block by block, that'll be fine. However, vectorized operations (like > `+` above) will be slower than necessary. > Any background on this? > I am honestly not sure where you are going at. This order seems the more natural order for most operations. Also numpy does not create copies normally even for transposed data (though some functions may internally, numpy just _defaults_ to C-order). So of course what is faster depends on your use-case, but if you have an operation on many 3x3 arrays the way numpy does it is the more natural way. If you do other things that are faster the other way around, you will have to decide which operation is the more important one overall. - Sebastian > (I came across this when having to rearrange my data (swapaxes, > rollaxis) from shape (3, 3, 777) (which allows for fast vectorized > operations in the rest of the code) to (777, 3, 3) for using numpy's > svd.) > > Cheers, > Nico > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Sun Mar 5 11:31:45 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sun, 05 Mar 2017 17:31:45 +0100 Subject: [Numpy-discussion] vectorization vs. numpy.linalg (shape (3, 3, 777) vs shape (777, 3, 3)) In-Reply-To: References: <1488725575.6765.1.camel@sipsolutions.net> Message-ID: <1488731505.6765.3.camel@sipsolutions.net> On Sun, 2017-03-05 at 15:46 +0000, Nico Schl?mer wrote: > >?I am honestly not sure where you are going at.?This order seems > the?more natural order for most operations.? > > Not sure what you mean by "natural". The most basic operations > like?`a[0] + a[1]` are several times faster than `a[...,0]?+ a[..., > 1]`. Perhaps one can come up with code examples that don't use those > operations at all, but I would guess that'll be a rare case.?My > point: If?your code uses vector operations (as `+`) _and_ > numpy.linalg functions, your vector operations will be several times > slower than necessary. One way around this would perhaps be to > initialize all your data in Fortran-style data layout, where > `a[...,0]?+ a[..., 1]` is indeed faster than `a[0] + a[1]`. > > What I'll end up doing is probably to rewrite whatever numpy.linalg > function I need for C-style ordering. For dets of 2x2 or 3x3 > matrices, this shouldn't be to hard (`a[0][0]?+a[1][1] - a[0][1] - > a[1][0]`). :) Well, Linalg functions are (stacked) low-level calls into lapack. Now for some functions (not simple math), it is possible that if you have chosen a non C-order memory layout, numpy has to jump through hoops to do your operations or iterates in C-order anyway. And yes, this is also the case for most of linalg, since most of them call into highly optimized code and algorithms defined for a single array. In that case manual solutions like you suggested may be faster. Although, I would suggest to be very careful with them, since there are may be subtleties one may miss, and it probably does make the code less easy to maintain and more error prone. In general numpy is smart about memory/iteration order for most cases, if you work on stacked arrays (and not on a single slices of them as in your examples) the memory order often has little to no effect on the operation speed, e.g. while `a[0] + a[1]` it is faster in your examples, whether you do `a.sum(0)` or `a.sum(-1)` makes typically little difference. Of course there are some cases were you can out-smart numpy, and of course in general choosing the right memory order can have a huge impact on your performance. But again, trying to out-smart also comes at a cost and I would be very reluctant to do it, and typically there are probably lower hanging fruits to get first. - Sebastian > > Cheers, > Nico > > > On Sun, Mar 5, 2017 at 3:53 PM Sebastian Berg .net> wrote: > > On Thu, 2017-03-02 at 10:27 +0000, Nico Schl?mer wrote: > > > Hi everyone, > > > > > > When trying to speed up my code, I noticed that simply by > > reordering > > > my data I could get more than twice as fast for the simplest > > > operations: > > > ``` > > > import numpy > > > a = numpy.random.rand(50, 50, 50) > > > > > > %timeit a[0] + a[1] > > > 1000000 loops, best of 3: 1.7 ?s per loop > > > > > > %timeit a[:, 0] + a[:, 1] > > > 100000 loops, best of 3: 4.42 ?s per loop > > > > > > %timeit a[..., 0] + a[..., 1] > > > 100000 loops, best of 3: 5.99 ?s per loop > > > ``` > > > This makes sense considering the fact that, by default, NumPy > > > features C-style memory allocation: the last index runs fastest. > > The > > > blocks that are added with `a[0]?+ a[1]` are contiguous in > > memory, so > > > cache is nicely made use of. As opposed to that, the blocks that > > are > > > added with `a[:, 0] + a[:, 1]` are not contiguous, even more so > > with > > > `a[..., 0] + a[..., 1]`; hence the slowdown. Would that be the > > > correct explanation? > > > > > > If yes, I'm wondering why most numpy.linalg methods, when > > vectorized, > > > put the vector index up front. E.g., to mass-compute > > determinants, > > > one has to do > > > ``` > > > a = numpy.random.rand(777, 3, 3) > > > numpy.linalg.det(a) > > > ``` > > > This way, all 3x3 matrices form a memory block, so if you do > > `det` > > > block by block, that'll be fine. However, vectorized operations > > (like > > > `+` above) will be slower than necessary. > > > Any background on this?? > > > > > > > I am honestly not sure where you are going at. This order seems the > > more natural order for most operations. Also numpy does not create > > copies normally even for transposed data (though some functions may > > internally, numpy just _defaults_ to C-order). So of course what is > > faster depends on your use-case, but if you have an operation on > > many > > 3x3 arrays the way numpy does it is the more natural way. If you do > > other things that are faster the other way around, you will have to > > decide which operation is the more important one overall. > > > > - Sebastian > > > > > (I came across this when having to rearrange my data (swapaxes, > > > rollaxis) from shape (3, 3, 777) (which allows for fast > > vectorized > > > operations in the rest of the code) to (777, 3, 3) for using > > numpy's > > > svd.) > > > > > > Cheers, > > > Nico > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion_________ > > ______________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From njs at pobox.com Sun Mar 5 14:20:19 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 5 Mar 2017 11:20:19 -0800 Subject: [Numpy-discussion] vectorization vs. numpy.linalg (shape (3, 3, 777) vs shape (777, 3, 3)) In-Reply-To: References: <1488725575.6765.1.camel@sipsolutions.net> Message-ID: On Sun, Mar 5, 2017 at 7:46 AM, Nico Schl?mer wrote: >> I am honestly not sure where you are going at. This order seems the more >> natural order for most operations. > > Not sure what you mean by "natural". The most basic operations like `a[0] + > a[1]` are several times faster than `a[...,0] + a[..., 1]`. Perhaps one can > come up with code examples that don't use those operations at all, but I > would guess that'll be a rare case. My point: If your code uses vector > operations (as `+`) _and_ numpy.linalg functions, your vector operations > will be several times slower than necessary. I guess it seems odd that you're thinking of "vector operations" as always meaning "operations on two slices of the same array"? I feel like that's a vanishingly small percentage of vector operations. Surely a + b is a lot more common than any of those? In any case, the reason linalg works that way is to be consistent with the general numpy broadcasting rule where you match indices from the right, which is certainly not possible to change. The reason broadcasting works that way is that in the common case of C memory layout, it makes vectorized operations like a + b faster :-). In theory it should also make the linalg functions faster, because it means that each call to the underlying 'det' routine is working on a contiguous block. If they worked from the left then we'd almost always have to copy the whole matrix into a temporary before we could actually do any linear algebra. (Though since linear algebra routines usually have super-linear complexity this might not matter much in practice.) -n -- Nathaniel J. Smith -- https://vorpus.org From charlesr.harris at gmail.com Mon Mar 6 22:57:25 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 6 Mar 2017 20:57:25 -0700 Subject: [Numpy-discussion] NumPy pre-release 1.12.1rc1 Message-ID: Hi All, I'm pleased to announce the release of NumPy 1.12.1rc1. NumPy 1.12.1rc1 supports Python 2.7 and 3.4 - 3.6 and fixes bugs and regressions found in NumPy 1.12.0. In particular, the regression in f2py constant parsing is fixed. Wheels for Linux, Windows, and OSX can be found on pypi. Archives can be downloaded from github . *Contributors* A total of 10 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Charles Harris * Eric Wieser * Greg Young * Joerg Behrmann + * John Kirkham * Julian Taylor * Marten van Kerkwijk * Matthew Brett * Shota Kawabuchi * Jean Utke + *Fixes Backported* * #8483: BUG: Fix wrong future nat warning and equiv type logic error... * #8489: BUG: Fix wrong masked median for some special cases * #8490: DOC: Place np.average in inline code * #8491: TST: Work around isfinite inconsistency on i386 * #8494: BUG: Guard against replacing constants without `'_'` spec in f2py. * #8524: BUG: Fix mean for float 16 non-array inputs for 1.12 * #8571: BUG: Fix calling python api with error set and minor leaks for... * #8602: BUG: Make iscomplexobj compatible with custom dtypes again * #8618: BUG: Fix undefined behaviour induced by bad `__array_wrap__` * #8648: BUG: Fix `MaskedArray.__setitem__` * #8659: BUG: PPC64el machines are POWER for Fortran in f2py * #8665: BUG: Look up methods on MaskedArray in `_frommethod` * #8674: BUG: Remove extra digit in `binary_repr` at limit * #8704: BUG: Fix deepcopy regression for empty arrays. * #8707: BUG: Fix ma.median for empty ndarrays Cheers, Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Wed Mar 8 21:48:45 2017 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Thu, 9 Mar 2017 13:48:45 +1100 Subject: [Numpy-discussion] Why do mgrid and meshgrid not return broadcast arrays? Message-ID: <214929e3-452a-486d-bcc9-a30a61b7486b@Spark> I was a bit surprised to discover that both meshgrid nor mgrid return fully instantiated arrays, when simple broadcasting (ie with stride=0 for other axes) is functionally identical and happens much, much faster. I wrote my own function to do this: def broadcast_mgrid(arrays): ? ? shape = tuple(map(len, arrays)) ? ? ndim = len(shape) ? ? result = [] ? ? for i, arr in enumerate(arrays, start=1): ? ? ? ? reshaped = np.broadcast_to(arr[[...] + [np.newaxis] * (ndim - i)], ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?shape) ? ? ? ? result.append(reshaped) ? ? return result For even a modest-sized 512 x 512 grid, this version is close to 100x faster: In [154]: %timeit th.broadcast_mgrid((np.arange(512), np.arange(512))) 10000 loops, best of 3: 25.9 ?s per loop In [156]: %timeit np.meshgrid(np.arange(512), np.arange(512)) 100 loops, best of 3: 2.02 ms per loop In [157]: %timeit np.mgrid[:512, :512] 100 loops, best of 3: 4.84 ms per loop Is there a conscious design decision as to why this isn?t what meshgrid/mgrid do already? Or would a PR be welcome to do this? Thanks, Juan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Wed Mar 8 22:05:21 2017 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Wed, 8 Mar 2017 22:05:21 -0500 Subject: [Numpy-discussion] Why do mgrid and meshgrid not return broadcast arrays? In-Reply-To: <214929e3-452a-486d-bcc9-a30a61b7486b@Spark> References: <214929e3-452a-486d-bcc9-a30a61b7486b@Spark> Message-ID: On Wed, Mar 8, 2017 at 9:48 PM, Juan Nunez-Iglesias wrote: > I was a bit surprised to discover that both meshgrid nor mgrid return > fully instantiated arrays, when simple broadcasting (ie with stride=0 for > other axes) is functionally identical and happens much, much faster. > > Take a look at ogrid: https://docs.scipy.org/doc/numpy/reference/generated/numpy.ogrid.html Warren I wrote my own function to do this: > > > *def broadcast_mgrid(arrays):* > * shape = tuple(map(len, arrays))* > * ndim = len(shape)* > * result = []* > * for i, arr in enumerate(arrays, start=1):* > * reshaped = np.broadcast_to(arr[[...] + [np.newaxis] * (ndim - > i)],* > * shape)* > * result.append(reshaped)* > * return result* > > > For even a modest-sized 512 x 512 grid, this version is close to 100x > faster: > > > *In [154]: %timeit th.broadcast_mgrid((np.arange(512), np.arange(512)))* > *10000 loops, best of 3: 25.9 ?s per loop* > > *In [156]: %timeit np.meshgrid(np.arange(512), np.arange(512))* > *100 loops, best of 3: 2.02 ms per loop* > > *In [157]: %timeit np.mgrid[:512, :512]* > *100 loops, best of 3: 4.84 ms per loop* > > > Is there a conscious design decision as to why this isn?t what > meshgrid/mgrid do already? Or would a PR be welcome to do this? > > Thanks, > > Juan. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Wed Mar 8 22:19:37 2017 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Thu, 9 Mar 2017 14:19:37 +1100 Subject: [Numpy-discussion] Why do mgrid and meshgrid not return broadcast arrays? In-Reply-To: References: <214929e3-452a-486d-bcc9-a30a61b7486b@Spark> Message-ID: Hi Warren, ogrid doesn?t solve my problem. Note that my code returns arrays that would evaluate as equal to the mgrid output. It?s just that they are copied in mgrid into a giant array, instead of broadcast: In [176]: a0, b0 = np.mgrid[:5, :5] In [177]: a1, b1 = th.broadcast_mgrid((np.arange(5), np.arange(5))) In [178]: a0 Out[178]: array([[0, 0, 0, 0, 0], ? ? ? ?[1, 1, 1, 1, 1], ? ? ? ?[2, 2, 2, 2, 2], ? ? ? ?[3, 3, 3, 3, 3], ? ? ? ?[4, 4, 4, 4, 4]]) In [179]: a1 Out[179]: array([[0, 0, 0, 0, 0], ? ? ? ?[1, 1, 1, 1, 1], ? ? ? ?[2, 2, 2, 2, 2], ? ? ? ?[3, 3, 3, 3, 3], ? ? ? ?[4, 4, 4, 4, 4]]) In [180]: a0.strides Out[180]: (40, 8) In [181]: a1.strides Out[181]: (8, 0) On 9 Mar 2017, 2:05 PM +1100, Warren Weckesser , wrote: > > > > On Wed, Mar 8, 2017 at 9:48 PM, Juan Nunez-Iglesias wrote: > > > I was a bit surprised to discover that both meshgrid nor mgrid return fully instantiated arrays, when simple broadcasting (ie with stride=0 for other axes) is functionally identical and happens much, much faster. > > > > > > > > > Take a look at ogrid: https://docs.scipy.org/doc/numpy/reference/generated/numpy.ogrid.html > > > > Warren > > > > > > > I wrote my own function to do this: > > > > > > > > > def broadcast_mgrid(arrays): > > > ? ? shape = tuple(map(len, arrays)) > > > ? ? ndim = len(shape) > > > ? ? result = [] > > > ? ? for i, arr in enumerate(arrays, start=1): > > > ? ? ? ? reshaped = np.broadcast_to(arr[[...] + [np.newaxis] * (ndim - i)], > > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?shape) > > > ? ? ? ? result.append(reshaped) > > > ? ? return result > > > > > > > > > For even a modest-sized 512 x 512 grid, this version is close to 100x faster: > > > > > > > > > In [154]: %timeit th.broadcast_mgrid((np.arange(512), np.arange(512))) > > > 10000 loops, best of 3: 25.9 ?s per loop > > > > > > In [156]: %timeit np.meshgrid(np.arange(512), np.arange(512)) > > > 100 loops, best of 3: 2.02 ms per loop > > > > > > In [157]: %timeit np.mgrid[:512, :512] > > > 100 loops, best of 3: 4.84 ms per loop > > > > > > > > > Is there a conscious design decision as to why this isn?t what meshgrid/mgrid do already? Or would a PR be welcome to do this? > > > > > > Thanks, > > > > > > Juan. > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From Per.Brodtkorb at ffi.no Thu Mar 9 02:28:56 2017 From: Per.Brodtkorb at ffi.no (Per.Brodtkorb at ffi.no) Date: Thu, 9 Mar 2017 07:28:56 +0000 Subject: [Numpy-discussion] Why do mgrid and meshgrid not return broadcast arrays? In-Reply-To: References: <214929e3-452a-486d-bcc9-a30a61b7486b@Spark> Message-ID: <8114F0AADAECD745AF1FC2047A5DC7ED201EF37F@HBU-POST2.ffi.no> Hi, Juan. Meshgrid can actually give what you want, but you must use the options: copy=False and indexing=?ij?. In [7]: %timeit np.meshgrid(np.arange(512), np.arange(512)) 1000 loops, best of 3: 1.24 ms per loop In [8]: %timeit np.meshgrid(np.arange(512), np.arange(512), copy=False) 10000 loops, best of 3: 27 ?s per loop In [9]: %timeit np.meshgrid(np.arange(512), np.arange(512), copy=False, indexing='ij') 10000 loops, best of 3: 23 ?s per loop Best regards Per A. Brodtkorb From: NumPy-Discussion [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Juan Nunez-Iglesias Sent: 9. mars 2017 04:20 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Why do mgrid and meshgrid not return broadcast arrays? Hi Warren, ogrid doesn?t solve my problem. Note that my code returns arrays that would evaluate as equal to the mgrid output. It?s just that they are copied in mgrid into a giant array, instead of broadcast: In [176]: a0, b0 = np.mgrid[:5, :5] In [177]: a1, b1 = th.broadcast_mgrid((np.arange(5), np.arange(5))) In [178]: a0 Out[178]: array([[0, 0, 0, 0, 0], [1, 1, 1, 1, 1], [2, 2, 2, 2, 2], [3, 3, 3, 3, 3], [4, 4, 4, 4, 4]]) In [179]: a1 Out[179]: array([[0, 0, 0, 0, 0], [1, 1, 1, 1, 1], [2, 2, 2, 2, 2], [3, 3, 3, 3, 3], [4, 4, 4, 4, 4]]) In [180]: a0.strides Out[180]: (40, 8) In [181]: a1.strides Out[181]: (8, 0) On 9 Mar 2017, 2:05 PM +1100, Warren Weckesser >, wrote: On Wed, Mar 8, 2017 at 9:48 PM, Juan Nunez-Iglesias > wrote: I was a bit surprised to discover that both meshgrid nor mgrid return fully instantiated arrays, when simple broadcasting (ie with stride=0 for other axes) is functionally identical and happens much, much faster. Take a look at ogrid: https://docs.scipy.org/doc/numpy/reference/generated/numpy.ogrid.html Warren I wrote my own function to do this: def broadcast_mgrid(arrays): shape = tuple(map(len, arrays)) ndim = len(shape) result = [] for i, arr in enumerate(arrays, start=1): reshaped = np.broadcast_to(arr[[...] + [np.newaxis] * (ndim - i)], shape) result.append(reshaped) return result For even a modest-sized 512 x 512 grid, this version is close to 100x faster: In [154]: %timeit th.broadcast_mgrid((np.arange(512), np.arange(512))) 10000 loops, best of 3: 25.9 ?s per loop In [156]: %timeit np.meshgrid(np.arange(512), np.arange(512)) 100 loops, best of 3: 2.02 ms per loop In [157]: %timeit np.mgrid[:512, :512] 100 loops, best of 3: 4.84 ms per loop Is there a conscious design decision as to why this isn?t what meshgrid/mgrid do already? Or would a PR be welcome to do this? Thanks, Juan. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Thu Mar 9 02:34:04 2017 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Thu, 9 Mar 2017 18:34:04 +1100 Subject: [Numpy-discussion] Why do mgrid and meshgrid not return broadcast arrays? In-Reply-To: <8114F0AADAECD745AF1FC2047A5DC7ED201EF37F@HBU-POST2.ffi.no> References: <214929e3-452a-486d-bcc9-a30a61b7486b@Spark> <8114F0AADAECD745AF1FC2047A5DC7ED201EF37F@HBU-POST2.ffi.no> Message-ID: Ah, fantastic, thanks Per! I'd still be interested to hear from the core devs as to why this isn't the default, both with meshgrid and mgrid... Juan. On 9 Mar 2017, 6:29 PM +1100, Per.Brodtkorb at ffi.no, wrote: > Hi, Juan. > > Meshgrid can actually give what you want, but you must use the options: copy=False ?and indexing=?ij?. > > In [7]: %timeit np.meshgrid(np.arange(512), np.arange(512)) > 1000 loops, best of 3: 1.24 ms per loop > > In [8]: %timeit np.meshgrid(np.arange(512), np.arange(512), copy=False) > 10000 loops, best of 3: 27 ?s per loop > > In [9]: %timeit np.meshgrid(np.arange(512), np.arange(512), copy=False, indexing='ij') > 10000 loops, best of 3: 23 ?s per loop > > Best regards > Per A. Brodtkorb > > From: NumPy-Discussion [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Juan Nunez-Iglesias > Sent: 9. mars 2017 04:20 > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] Why do mgrid and meshgrid not return broadcast arrays? > > Hi Warren, > > ogrid doesn?t solve my problem. Note that my code returns arrays that would evaluate as equal to the mgrid output. It?s just that they are copied in mgrid into a giant array, instead of broadcast: > > > In [176]: a0, b0 = np.mgrid[:5, :5] > > In [177]: a1, b1 = th.broadcast_mgrid((np.arange(5), np.arange(5))) > > In [178]: a0 > Out[178]: > array([[0, 0, 0, 0, 0], > ? ? ? ?[1, 1, 1, 1, 1], > ? ? ? ?[2, 2, 2, 2, 2], > ? ? ? ?[3, 3, 3, 3, 3], > ? ? ? ?[4, 4, 4, 4, 4]]) > > In [179]: a1 > Out[179]: > array([[0, 0, 0, 0, 0], > ? ? ? ?[1, 1, 1, 1, 1], > ? ? ? ?[2, 2, 2, 2, 2], > ? ? ? ?[3, 3, 3, 3, 3], > ? ? ? ?[4, 4, 4, 4, 4]]) > > In [180]: a0.strides > Out[180]: (40, 8) > > In [181]: a1.strides > Out[181]: (8, 0) > > > > On 9 Mar 2017, 2:05 PM +1100, Warren Weckesser , wrote: > > > > On Wed, Mar 8, 2017 at 9:48 PM, Juan Nunez-Iglesias wrote: > I was a bit surprised to discover that both meshgrid nor mgrid return fully instantiated arrays, when simple broadcasting (ie with stride=0 for other axes) is functionally identical and happens much, much faster. > > > Take a look at ogrid: https://docs.scipy.org/doc/numpy/reference/generated/numpy.ogrid.html > Warren > > > I wrote my own function to do this: > > > > > > def broadcast_mgrid(arrays): > > ? ? shape = tuple(map(len, arrays)) > > ? ? ndim = len(shape) > > ? ? result = [] > > ? ? for i, arr in enumerate(arrays, start=1): > > ? ? ? ? reshaped = np.broadcast_to(arr[[...] + [np.newaxis] * (ndim - i)], > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?shape) > > ? ? ? ? result.append(reshaped) > > ? ? return result > > > > > > For even a modest-sized 512 x 512 grid, this version is close to 100x faster: > > > > > > In [154]: %timeit th.broadcast_mgrid((np.arange(512), np.arange(512))) > > 10000 loops, best of 3: 25.9 ?s per loop > > > > In [156]: %timeit np.meshgrid(np.arange(512), np.arange(512)) > > 100 loops, best of 3: 2.02 ms per loop > > > > In [157]: %timeit np.mgrid[:512, :512] > > 100 loops, best of 3: 4.84 ms per loop > > > > > > Is there a conscious design decision as to why this isn?t what meshgrid/mgrid do already? Or would a PR be welcome to do this? > > > > Thanks, > > > > Juan. > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From Per.Brodtkorb at ffi.no Thu Mar 9 03:01:06 2017 From: Per.Brodtkorb at ffi.no (Per.Brodtkorb at ffi.no) Date: Thu, 9 Mar 2017 08:01:06 +0000 Subject: [Numpy-discussion] Why do mgrid and meshgrid not return broadcast arrays? In-Reply-To: References: <214929e3-452a-486d-bcc9-a30a61b7486b@Spark> <8114F0AADAECD745AF1FC2047A5DC7ED201EF37F@HBU-POST2.ffi.no> Message-ID: <8114F0AADAECD745AF1FC2047A5DC7ED201EF3A3@HBU-POST2.ffi.no> The reason for returning copies from meshgrid as default instead of views into to input arrays, was to not break backwards compatibility. The old meshgrid returned copied arrays, which is safe if you need to write to those arrays. If you use copy=False, a view into the original arrays are returned in order to conserve memory, but will likely return non-contiguous arrays. Furthermore, more than one element of a broadcast array may refer to a single memory location. Per A From: NumPy-Discussion [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Juan Nunez-Iglesias Sent: 9. mars 2017 08:34 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Why do mgrid and meshgrid not return broadcast arrays? Ah, fantastic, thanks Per! I'd still be interested to hear from the core devs as to why this isn't the default, both with meshgrid and mgrid... Juan. On 9 Mar 2017, 6:29 PM +1100, Per.Brodtkorb at ffi.no, wrote: Hi, Juan. Meshgrid can actually give what you want, but you must use the options: copy=False and indexing=?ij?. In [7]: %timeit np.meshgrid(np.arange(512), np.arange(512)) 1000 loops, best of 3: 1.24 ms per loop In [8]: %timeit np.meshgrid(np.arange(512), np.arange(512), copy=False) 10000 loops, best of 3: 27 ?s per loop In [9]: %timeit np.meshgrid(np.arange(512), np.arange(512), copy=False, indexing='ij') 10000 loops, best of 3: 23 ?s per loop Best regards Per A. Brodtkorb From: NumPy-Discussion [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Juan Nunez-Iglesias Sent: 9. mars 2017 04:20 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Why do mgrid and meshgrid not return broadcast arrays? Hi Warren, ogrid doesn?t solve my problem. Note that my code returns arrays that would evaluate as equal to the mgrid output. It?s just that they are copied in mgrid into a giant array, instead of broadcast: In [176]: a0, b0 = np.mgrid[:5, :5] In [177]: a1, b1 = th.broadcast_mgrid((np.arange(5), np.arange(5))) In [178]: a0 Out[178]: array([[0, 0, 0, 0, 0], [1, 1, 1, 1, 1], [2, 2, 2, 2, 2], [3, 3, 3, 3, 3], [4, 4, 4, 4, 4]]) In [179]: a1 Out[179]: array([[0, 0, 0, 0, 0], [1, 1, 1, 1, 1], [2, 2, 2, 2, 2], [3, 3, 3, 3, 3], [4, 4, 4, 4, 4]]) In [180]: a0.strides Out[180]: (40, 8) In [181]: a1.strides Out[181]: (8, 0) On 9 Mar 2017, 2:05 PM +1100, Warren Weckesser >, wrote: On Wed, Mar 8, 2017 at 9:48 PM, Juan Nunez-Iglesias > wrote: I was a bit surprised to discover that both meshgrid nor mgrid return fully instantiated arrays, when simple broadcasting (ie with stride=0 for other axes) is functionally identical and happens much, much faster. Take a look at ogrid: https://docs.scipy.org/doc/numpy/reference/generated/numpy.ogrid.html Warren I wrote my own function to do this: def broadcast_mgrid(arrays): shape = tuple(map(len, arrays)) ndim = len(shape) result = [] for i, arr in enumerate(arrays, start=1): reshaped = np.broadcast_to(arr[[...] + [np.newaxis] * (ndim - i)], shape) result.append(reshaped) return result For even a modest-sized 512 x 512 grid, this version is close to 100x faster: In [154]: %timeit th.broadcast_mgrid((np.arange(512), np.arange(512))) 10000 loops, best of 3: 25.9 ?s per loop In [156]: %timeit np.meshgrid(np.arange(512), np.arange(512)) 100 loops, best of 3: 2.02 ms per loop In [157]: %timeit np.mgrid[:512, :512] 100 loops, best of 3: 4.84 ms per loop Is there a conscious design decision as to why this isn?t what meshgrid/mgrid do already? Or would a PR be welcome to do this? Thanks, Juan. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From nico.schloemer at gmail.com Thu Mar 9 05:26:44 2017 From: nico.schloemer at gmail.com (=?UTF-8?Q?Nico_Schl=C3=B6mer?=) Date: Thu, 09 Mar 2017 10:26:44 +0000 Subject: [Numpy-discussion] float16/32: wrong number of digits? Message-ID: Hi everyone, I wondered how to express a numpy float exactly in terms of format, and found `%r` quite useful: `float(repr(a)) == a` is guaranteed for Python `float`s. When trying the same thing with lower-precision Python floats, I found this identity not quite fulfilled: ``` import numpy b = numpy.array([1.0 / 3.0], dtype=np.float16) float(repr(b[0])) - b[0] Out[12]: -1.9531250000093259e-06 ``` Indeed, ``` b Out[6]: array([ 0.33325195], dtype=float16) ``` ``` repr(b[0]) Out[7]: '0.33325' ``` When counting the bits, a float16 should hold 4.8 decimal digits, so `repr()` seems right. Where does the garbage tail -1.9531250000093259e-06 come from though? Cheers, Nico -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Thu Mar 9 05:57:19 2017 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 09 Mar 2017 10:57:19 +0000 Subject: [Numpy-discussion] float16/32: wrong number of digits? In-Reply-To: References: Message-ID: On Thu, Mar 9, 2017, 11:27 Nico Schl?mer wrote: > Hi everyone, > > I wondered how to express a numpy float exactly in terms of format, and > found `%r` quite useful: `float(repr(a)) == a` is guaranteed for Python > `float`s. When trying the same thing with lower-precision Python floats, I > found this identity not quite fulfilled: > ``` > import numpy > b = numpy.array([1.0 / 3.0], dtype=np.float16) > float(repr(b[0])) - b[0] > Out[12]: -1.9531250000093259e-06 > ``` > Indeed, > ``` > b > Out[6]: array([ 0.33325195], dtype=float16) > ``` > ``` > repr(b[0]) > Out[7]: '0.33325' > ``` > When counting the bits, a float16 should hold 4.8 decimal digits, so > `repr()` seems right. Where does the garbage tail -1.9531250000093259e-06 > come from though? > Even more troubling, the high precision numpy types - np.longdouble and its complex version - lose intimation when used with repr. The basic problem is (roughly) that all floating-point numbers are converted to python floats before printing. I put some effort into cleaning this up, but the code is messy (actually there are several independent code paths for converting numbers to strings) and the algorithms python uses to make repr work out nicely are nontrivial. Anne > Cheers, > Nico > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Thu Mar 9 09:45:47 2017 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Thu, 9 Mar 2017 15:45:47 +0100 Subject: [Numpy-discussion] PyData Barcelona this May Message-ID: There will be a PyData conference in Barcelona this May: http://pydata.org/barcelona2017/ I am planning on attending, and was thinking of maybe proposing to organize a numpy-themed workshop or tutorial. My personal inclination would be to look at some advanced topic that I know well, like writing gufuncs in Cython, but wouldn't mind doing a more run of the mill thing. Anyone has any thoughts or experiences on what has worked well in similar situations? Any specific topic you always wanted to attend a workshop on, but were afraid to ask? Alternatively, or on top of the workshop, I could propose to do a talk: talking last year at PyData Madrid about the new indexing was a lot of fun! Thing is, I have been quite disconnected from the project this past year, and can't really think of any worthwhile topic. Is there any message that we as a project would like to get out to the larger community? And if you are planning on attending, please give me a shout. Thanks, Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Thu Mar 9 10:54:31 2017 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Thu, 9 Mar 2017 18:54:31 +0300 Subject: [Numpy-discussion] ANN: SciPy 0.19.0 Message-ID: On behalf of the Scipy development team I am pleased to announce the availability of Scipy 0.19.0. This release contains several great new features and a large number of bug fixes and various improvements, as detailed in the release notes below. 121 people contributed to this release over the course of seven months. Thanks to everyone who contributed! This release requires Python 2.7 or 3.4-3.6 and NumPy 1.8.2 or greater. Source tarballs and release notes can be found at https://github.com/scipy/scipy/releases/tag/v0.19.0. OS X and Linux wheels are available from PyPI. For security-conscious, the wheels themselves are signed with my GPG key. Additionally, you can checksum the wheels and verify the checksums with those listed below or in the README file at https://github.com/scipy/scipy/releases/tag/v0.19.0. Cheers, Evgeni -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 ========================== SciPy 0.19.0 Release Notes ========================== .. contents:: SciPy 0.19.0 is the culmination of 7 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.19.x branch, and on adding new features on the master branch. This release requires Python 2.7 or 3.4-3.6 and NumPy 1.8.2 or greater. Highlights of this release include: - - A unified foreign function interface layer, `scipy.LowLevelCallable`. - - Cython API for scalar, typed versions of the universal functions from the `scipy.special` module, via `cimport scipy.special.cython_special`. New features ============ Foreign function interface improvements - --------------------------------------- `scipy.LowLevelCallable` provides a new unified interface for wrapping low-level compiled callback functions in the Python space. It supports Cython imported "api" functions, ctypes function pointers, CFFI function pointers, ``PyCapsules``, Numba jitted functions and more. See `gh-6509 `_ for details. `scipy.linalg` improvements - --------------------------- The function `scipy.linalg.solve` obtained two more keywords ``assume_a`` and ``transposed``. The underlying LAPACK routines are replaced with "expert" versions and now can also be used to solve symmetric, hermitian and positive definite coefficient matrices. Moreover, ill-conditioned matrices now cause a warning to be emitted with the estimated condition number information. Old ``sym_pos`` keyword is kept for backwards compatibility reasons however it is identical to using ``assume_a='pos'``. Moreover, the ``debug`` keyword, which had no function but only printing the ``overwrite_`` values, is deprecated. The function `scipy.linalg.matrix_balance` was added to perform the so-called matrix balancing using the LAPACK xGEBAL routine family. This can be used to approximately equate the row and column norms through diagonal similarity transformations. The functions `scipy.linalg.solve_continuous_are` and `scipy.linalg.solve_discrete_are` have numerically more stable algorithms. These functions can also solve generalized algebraic matrix Riccati equations. Moreover, both gained a ``balanced`` keyword to turn balancing on and off. `scipy.spatial` improvements - ---------------------------- `scipy.spatial.SphericalVoronoi.sort_vertices_of_regions` has been re-written in Cython to improve performance. `scipy.spatial.SphericalVoronoi` can handle > 200 k points (at least 10 million) and has improved performance. The function `scipy.spatial.distance.directed_hausdorff` was added to calculate the directed Hausdorff distance. ``count_neighbors`` method of `scipy.spatial.cKDTree` gained an ability to perform weighted pair counting via the new keywords ``weights`` and ``cumulative``. See `gh-5647 `_ for details. `scipy.spatial.distance.pdist` and `scipy.spatial.distance.cdist` now support non-double custom metrics. `scipy.ndimage` improvements - ---------------------------- The callback function C API supports PyCapsules in Python 2.7 Multidimensional filters now allow having different extrapolation modes for different axes. `scipy.optimize` improvements - ----------------------------- The `scipy.optimize.basinhopping` global minimizer obtained a new keyword, `seed`, which can be used to seed the random number generator and obtain repeatable minimizations. The keyword `sigma` in `scipy.optimize.curve_fit` was overloaded to also accept the covariance matrix of errors in the data. `scipy.signal` improvements - --------------------------- The function `scipy.signal.correlate` and `scipy.signal.convolve` have a new optional parameter `method`. The default value of `auto` estimates the fastest of two computation methods, the direct approach and the Fourier transform approach. A new function has been added to choose the convolution/correlation method, `scipy.signal.choose_conv_method` which may be appropriate if convolutions or correlations are performed on many arrays of the same size. New functions have been added to calculate complex short time fourier transforms of an input signal, and to invert the transform to recover the original signal: `scipy.signal.stft` and `scipy.signal.istft`. This implementation also fixes the previously incorrect ouput of `scipy.signal.spectrogram` when complex output data were requested. The function `scipy.signal.sosfreqz` was added to compute the frequency response from second-order sections. The function `scipy.signal.unit_impulse` was added to conveniently generate an impulse function. The function `scipy.signal.iirnotch` was added to design second-order IIR notch filters that can be used to remove a frequency component from a signal. The dual function `scipy.signal.iirpeak` was added to compute the coefficients of a second-order IIR peak (resonant) filter. The function `scipy.signal.minimum_phase` was added to convert linear-phase FIR filters to minimum phase. The functions `scipy.signal.upfirdn` and `scipy.signal.resample_poly` are now substantially faster when operating on some n-dimensional arrays when n > 1. The largest reduction in computation time is realized in cases where the size of the array is small (<1k samples or so) along the axis to be filtered. `scipy.fftpack` improvements - ---------------------------- Fast Fourier transform routines now accept `np.float16` inputs and upcast them to `np.float32`. Previously, they would raise an error. `scipy.cluster` improvements - ---------------------------- Methods ``"centroid"`` and ``"median"`` of `scipy.cluster.hierarchy.linkage` have been significantly sped up. Long-standing issues with using ``linkage`` on large input data (over 16 GB) have been resolved. `scipy.sparse` improvements - --------------------------- The functions `scipy.sparse.save_npz` and `scipy.sparse.load_npz` were added, providing simple serialization for some sparse formats. The `prune` method of classes `bsr_matrix`, `csc_matrix`, and `csr_matrix` was updated to reallocate backing arrays under certain conditions, reducing memory usage. The methods `argmin` and `argmax` were added to classes `coo_matrix`, `csc_matrix`, `csr_matrix`, and `bsr_matrix`. New function `scipy.sparse.csgraph.structural_rank` computes the structural rank of a graph with a given sparsity pattern. New function `scipy.sparse.linalg.spsolve_triangular` solves a sparse linear system with a triangular left hand side matrix. `scipy.special` improvements - ---------------------------- Scalar, typed versions of universal functions from `scipy.special` are available in the Cython space via ``cimport`` from the new module `scipy.special.cython_special`. These scalar functions can be expected to be significantly faster then the universal functions for scalar arguments. See the `scipy.special` tutorial for details. Better control over special-function errors is offered by the functions `scipy.special.geterr` and `scipy.special.seterr` and the context manager `scipy.special.errstate`. The names of orthogonal polynomial root functions have been changed to be consistent with other functions relating to orthogonal polynomials. For example, `scipy.special.j_roots` has been renamed `scipy.special.roots_jacobi` for consistency with the related functions `scipy.special.jacobi` and `scipy.special.eval_jacobi`. To preserve back-compatibility the old names have been left as aliases. Wright Omega function is implemented as `scipy.special.wrightomega`. `scipy.stats` improvements - -------------------------- The function `scipy.stats.weightedtau` was added. It provides a weighted version of Kendall's tau. New class `scipy.stats.multinomial` implements the multinomial distribution. New class `scipy.stats.rv_histogram` constructs a continuous univariate distribution with a piecewise linear CDF from a binned data sample. New class `scipy.stats.argus` implements the Argus distribution. `scipy.interpolate` improvements - -------------------------------- New class `scipy.interpolate.BSpline` represents splines. ``BSpline`` objects contain knots and coefficients and can evaluate the spline. The format is consistent with FITPACK, so that one can do, for example:: >>> t, c, k = splrep(x, y, s=0) >>> spl = BSpline(t, c, k) >>> np.allclose(spl(x), y) ``spl*`` functions, `scipy.interpolate.splev`, `scipy.interpolate.splint`, `scipy.interpolate.splder` and `scipy.interpolate.splantider`, accept both ``BSpline`` objects and ``(t, c, k)`` tuples for backwards compatibility. For multidimensional splines, ``c.ndim > 1``, ``BSpline`` objects are consistent with piecewise polynomials, `scipy.interpolate.PPoly`. This means that ``BSpline`` objects are not immediately consistent with `scipy.interpolate.splprep`, and one *cannot* do ``>>> BSpline(*splprep([x, y])[0])``. Consult the `scipy.interpolate` test suite for examples of the precise equivalence. In new code, prefer using ``scipy.interpolate.BSpline`` objects instead of manipulating ``(t, c, k)`` tuples directly. New function `scipy.interpolate.make_interp_spline` constructs an interpolating spline given data points and boundary conditions. New function `scipy.interpolate.make_lsq_spline` constructs a least-squares spline approximation given data points. `scipy.integrate` improvements - ------------------------------ Now `scipy.integrate.fixed_quad` supports vector-valued functions. Deprecated features =================== `scipy.interpolate.splmake`, `scipy.interpolate.spleval` and `scipy.interpolate.spline` are deprecated. The format used by `splmake/spleval` was inconsistent with `splrep/splev` which was confusing to users. `scipy.special.errprint` is deprecated. Improved functionality is available in `scipy.special.seterr`. calling `scipy.spatial.distance.pdist` or `scipy.spatial.distance.cdist` with arguments not needed by the chosen metric is deprecated. Also, metrics `"old_cosine"` and `"old_cos"` are deprecated. Backwards incompatible changes ============================== The deprecated ``scipy.weave`` submodule was removed. `scipy.spatial.distance.squareform` now returns arrays of the same dtype as the input, instead of always float64. `scipy.special.errprint` now returns a boolean. The function `scipy.signal.find_peaks_cwt` now returns an array instead of a list. `scipy.stats.kendalltau` now computes the correct p-value in case the input contains ties. The p-value is also identical to that computed by `scipy.stats.mstats.kendalltau` and by R. If the input does not contain ties there is no change w.r.t. the previous implementation. The function `scipy.linalg.block_diag` will not ignore zero-sized matrices anymore. Instead it will insert rows or columns of zeros of the appropriate size. See gh-4908 for more details. Other changes ============= SciPy wheels will now report their dependency on ``numpy`` on all platforms. This change was made because Numpy wheels are available, and because the pip upgrade behavior is finally changing for the better (use ``--upgrade-strategy=only-if-needed`` for ``pip >= 8.2``; that behavior will become the default in the next major version of ``pip``). Numerical values returned by `scipy.interpolate.interp1d` with ``kind="cubic"`` and ``"quadratic"`` may change relative to previous scipy versions. If your code depended on specific numeric values (i.e., on implementation details of the interpolators), you may want to double-check your results. Authors ======= * @endolith * Max Argus + * Herv? Audren * Alessandro Pietro Bardelli + * Michael Benfield + * Felix Berkenkamp * Matthew Brett * Per Brodtkorb * Evgeni Burovski * Pierre de Buyl * CJ Carey * Brandon Carter + * Tim Cera * Klesk Chonkin * Christian H?ggstr?m + * Luca Citi * Peadar Coyle + * Daniel da Silva + * Greg Dooper + * John Draper + * drlvk + * David Ellis + * Yu Feng * Baptiste Fontaine + * Jed Frey + * Siddhartha Gandhi + * Wim Glenn + * Akash Goel + * Christoph Gohlke * Ralf Gommers * Alexander Goncearenco + * Richard Gowers + * Alex Griffing * Radoslaw Guzinski + * Charles Harris * Callum Jacob Hays + * Ian Henriksen * Randy Heydon + * Lindsey Hiltner + * Gerrit Holl + * Hiroki IKEDA + * jfinkels + * Mher Kazandjian + * Thomas Keck + * keuj6 + * Kornel Kielczewski + * Sergey B Kirpichev + * Vasily Kokorev + * Eric Larson * Denis Laxalde * Gregory R. Lee * Josh Lefler + * Julien Lhermitte + * Evan Limanto + * Jin-Guo Liu + * Nikolay Mayorov * Geordie McBain + * Josue Melka + * Matthieu Melot * michaelvmartin15 + * Surhud More + * Brett M. Morris + * Chris Mutel + * Paul Nation * Andrew Nelson * David Nicholson + * Aaron Nielsen + * Joel Nothman * nrnrk + * Juan Nunez-Iglesias * Mikhail Pak + * Gavin Parnaby + * Thomas Pingel + * Ilhan Polat + * Aman Pratik + * Sebastian Pucilowski * Ted Pudlik * puenka + * Eric Quintero * Tyler Reddy * Joscha Reimer * Antonio Horta Ribeiro + * Edward Richards + * Roman Ring + * Rafael Rossi + * Colm Ryan + * Sami Salonen + * Alvaro Sanchez-Gonzalez + * Johannes Schmitz * Kari Schoonbee * Yurii Shevchuk + * Jonathan Siebert + * Jonathan Tammo Siebert + * Scott Sievert + * Sourav Singh * Byron Smith + * Srikiran + * Samuel St-Jean + * Yoni Teitelbaum + * Bhavika Tekwani * Martin Thoma * timbalam + * Svend Vanderveken + * Sebastiano Vigna + * Aditya Vijaykumar + * Santi Villalba + * Ze Vinicius * Pauli Virtanen * Matteo Visconti * Yusuke Watanabe + * Warren Weckesser * Phillip Weinberg + * Nils Werner * Jakub Wilk * Josh Wilson * wirew0rm + * David Wolever + * Nathan Woods * ybeltukov + * G Young * Evgeny Zhurko + A total of 121 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 0.19.0 - ------------------------ - - `#1767 `__: Function definitions in __fitpack.h should be moved. (Trac #1240) - - `#1774 `__: _kmeans chokes on large thresholds (Trac #1247) - - `#2089 `__: Integer overflows cause segfault in linkage function with large... - - `#2190 `__: Are odd-length window functions supposed to be always symmetrical?... - - `#2251 `__: solve_discrete_are in scipy.linalg does (sometimes) not solve... - - `#2580 `__: scipy.interpolate.UnivariateSpline (or a new superclass of it)... - - `#2592 `__: scipy.stats.anderson assumes gumbel_l - - `#3054 `__: scipy.linalg.eig does not handle infinite eigenvalues - - `#3160 `__: multinomial pmf / logpmf - - `#3904 `__: scipy.special.ellipj dn wrong values at quarter period - - `#4044 `__: Inconsistent code book initialization in kmeans - - `#4234 `__: scipy.signal.flattop documentation doesn't list a source for... - - `#4831 `__: Bugs in C code in __quadpack.h - - `#4908 `__: bug: unnessesary validity check for block dimension in scipy.sparse.block_diag - - `#4917 `__: BUG: indexing error for sparse matrix with ix_ - - `#4938 `__: Docs on extending ndimage need to be updated. - - `#5056 `__: sparse matrix element-wise multiplying dense matrix returns dense... - - `#5337 `__: Formula in documentation for correlate is wrong - - `#5537 `__: use OrderedDict in io.netcdf - - `#5750 `__: [doc] missing data index value in KDTree, cKDTree - - `#5755 `__: p-value computation in scipy.stats.kendalltau() in broken in... - - `#5757 `__: BUG: Incorrect complex output of signal.spectrogram - - `#5964 `__: ENH: expose scalar versions of scipy.special functions to cython - - `#6107 `__: scipy.cluster.hierarchy.single segmentation fault with 2**16... - - `#6278 `__: optimize.basinhopping should take a RandomState object - - `#6296 `__: InterpolatedUnivariateSpline: check_finite fails when w is unspecified - - `#6306 `__: Anderson-Darling bad results - - `#6314 `__: scipy.stats.kendaltau() p value not in agreement with R, SPSS... - - `#6340 `__: Curve_fit bounds and maxfev - - `#6377 `__: expm_multiply, complex matrices not working using start,stop,ect... - - `#6382 `__: optimize.differential_evolution stopping criterion has unintuitive... - - `#6391 `__: Global Benchmarking times out at 600s. - - `#6397 `__: mmwrite errors with large (but still 64-bit) integers - - `#6413 `__: scipy.stats.dirichlet computes multivariate gaussian differential... - - `#6428 `__: scipy.stats.mstats.mode modifies input - - `#6440 `__: Figure out ABI break policy for scipy.special Cython API - - `#6441 `__: Using Qhull for halfspace intersection : segfault - - `#6442 `__: scipy.spatial : In incremental mode volume is not recomputed - - `#6451 `__: Documentation for scipy.cluster.hierarchy.to_tree is confusing... - - `#6490 `__: interp1d (kind=zero) returns wrong value for rightmost interpolation... - - `#6521 `__: scipy.stats.entropy does *not* calculate the KL divergence - - `#6530 `__: scipy.stats.spearmanr unexpected NaN handling - - `#6541 `__: Test runner does not run scipy._lib/tests? - - `#6552 `__: BUG: misc.bytescale returns unexpected results when using cmin/cmax... - - `#6556 `__: RectSphereBivariateSpline(u, v, r) fails if min(v) >= pi - - `#6559 `__: Differential_evolution maxiter causing memory overflow - - `#6565 `__: Coverage of spectral functions could be improved - - `#6628 `__: Incorrect parameter name in binomial documentation - - `#6634 `__: Expose LAPACK's xGESVX family for linalg.solve ill-conditioned... - - `#6657 `__: Confusing documentation for `scipy.special.sph_harm` - - `#6676 `__: optimize: Incorrect size of Jacobian returned by `minimize(...,... - - `#6681 `__: add a new context manager to wrap `scipy.special.seterr` - - `#6700 `__: BUG: scipy.io.wavfile.read stays in infinite loop, warns on wav... - - `#6721 `__: scipy.special.chebyt(N) throw a 'TypeError' when N > 64 - - `#6727 `__: Documentation for scipy.stats.norm.fit is incorrect - - `#6764 `__: Documentation for scipy.spatial.Delaunay is partially incorrect - - `#6811 `__: scipy.spatial.SphericalVoronoi fails for large number of points - - `#6841 `__: spearmanr fails when nan_policy='omit' is set - - `#6869 `__: Currently in gaussian_kde, the logpdf function is calculated... - - `#6875 `__: SLSQP inconsistent handling of invalid bounds - - `#6876 `__: Python stopped working (Segfault?) with minimum/maximum filter... - - `#6889 `__: dblquad gives different results under scipy 0.17.1 and 0.18.1 - - `#6898 `__: BUG: dblquad ignores error tolerances - - `#6901 `__: Solving sparse linear systems in CSR format with complex values - - `#6903 `__: issue in spatial.distance.pdist docstring - - `#6917 `__: Problem in passing drop_rule to scipy.sparse.linalg.spilu - - `#6926 `__: signature mismatches for LowLevelCallable - - `#6961 `__: Scipy contains shebang pointing to /usr/bin/python and /bin/bash... - - `#6972 `__: BUG: special: `generate_ufuncs.py` is broken - - `#6984 `__: Assert raises test failure for test_ill_condition_warning - - `#6990 `__: BUG: sparse: Bad documentation of the `k` argument in `sparse.linalg.eigs` - - `#6991 `__: Division by zero in linregress() - - `#7011 `__: possible speed improvment in rv_continuous.fit() - - `#7015 `__: Test failure with Python 3.5 and numpy master - - `#7055 `__: SciPy 0.19.0rc1 test errors and failures on Windows - - `#7096 `__: macOS test failues for test_solve_continuous_are - - `#7100 `__: test_distance.test_Xdist_deprecated_args test error in 0.19.0rc2 Pull requests for 0.19.0 - ------------------------ - - `#2908 `__: Scipy 1.0 Roadmap - - `#3174 `__: add b-splines - - `#4606 `__: ENH: Add a unit impulse waveform function - - `#5608 `__: Adds keyword argument to choose faster convolution method - - `#5647 `__: ENH: Faster count_neighour in cKDTree / + weighted input data - - `#6021 `__: Netcdf append - - `#6058 `__: ENH: scipy.signal - Add stft and istft - - `#6059 `__: ENH: More accurate signal.freqresp for zpk systems - - `#6195 `__: ENH: Cython interface for special - - `#6234 `__: DOC: Fixed a typo in ward() help - - `#6261 `__: ENH: add docstring and clean up code for signal.normalize - - `#6270 `__: MAINT: special: add tests for cdflib - - `#6271 `__: Fix for scipy.cluster.hierarchy.is_isomorphic - - `#6273 `__: optimize: rewrite while loops as for loops - - `#6279 `__: MAINT: Bessel tweaks - - `#6291 `__: Fixes gh-6219: remove runtime warning from genextreme distribution - - `#6294 `__: STY: Some PEP8 and cleaning up imports in stats/_continuous_distns.py - - `#6297 `__: Clarify docs in misc/__init__.py - - `#6300 `__: ENH: sparse: Loosen input validation for `diags` with empty inputs - - `#6301 `__: BUG: standardizes check_finite behavior re optional weights,... - - `#6303 `__: Fixing example in _lazyselect docstring. - - `#6307 `__: MAINT: more improvements to gammainc/gammaincc - - `#6308 `__: Clarified documentation of hypergeometric distribution. - - `#6309 `__: BUG: stats: Improve calculation of the Anderson-Darling statistic. - - `#6315 `__: ENH: Descending order of x in PPoly - - `#6317 `__: ENH: stats: Add support for nan_policy to stats.median_test - - `#6321 `__: TST: fix a typo in test name - - `#6328 `__: ENH: sosfreqz - - `#6335 `__: Define LinregressResult outside of linregress - - `#6337 `__: In anderson test, added support for right skewed gumbel distribution. - - `#6341 `__: Accept several spellings for the curve_fit max number of function... - - `#6342 `__: DOC: cluster: clarify hierarchy.linkage usage - - `#6352 `__: DOC: removed brentq from its own 'see also' - - `#6362 `__: ENH: stats: Use explicit formulas for sf, logsf, etc in weibull... - - `#6369 `__: MAINT: special: add a comment to hyp0f1_complex - - `#6375 `__: Added the multinomial distribution. - - `#6387 `__: MAINT: special: improve accuracy of ellipj's `dn` at quarter... - - `#6388 `__: BenchmarkGlobal - getting it to work in Python3 - - `#6394 `__: ENH: scipy.sparse: add save and load functions for sparse matrices - - `#6400 `__: MAINT: moves global benchmark run from setup_cache to track_all - - `#6403 `__: ENH: seed kwd for basinhopping. Closes #6278 - - `#6404 `__: ENH: signal: added irrnotch and iirpeak functions. - - `#6406 `__: ENH: special: extend `sici`/`shichi` to complex arguments - - `#6407 `__: ENH: Window functions should not accept non-integer or negative... - - `#6408 `__: MAINT: _differentialevolution now uses _lib._util.check_random_state - - `#6427 `__: MAINT: Fix gmpy build & test that mpmath uses gmpy - - `#6439 `__: MAINT: ndimage: update callback function c api - - `#6443 `__: BUG: Fix volume computation in incremental mode - - `#6447 `__: Fixes issue #6413 - Minor documentation fix in the entropy function... - - `#6448 `__: ENH: Add halfspace mode to Qhull - - `#6449 `__: ENH: rtol and atol for differential_evolution termination fixes... - - `#6453 `__: DOC: Add some See Also links between similar functions - - `#6454 `__: DOC: linalg: clarify callable signature in `ordqz` - - `#6457 `__: ENH: spatial: enable non-double dtypes in squareform - - `#6459 `__: BUG: Complex matrices not handled correctly by expm_multiply... - - `#6465 `__: TST DOC Window docs, tests, etc. - - `#6469 `__: ENH: linalg: better handling of infinite eigenvalues in `eig`/`eigvals` - - `#6475 `__: DOC: calling interp1d/interp2d with NaNs is undefined - - `#6477 `__: Document magic numbers in optimize.py - - `#6481 `__: TST: Supress some warnings from test_windows - - `#6485 `__: DOC: spatial: correct typo in procrustes - - `#6487 `__: Fix Bray-Curtis formula in pdist docstring - - `#6493 `__: ENH: Add covariance functionality to scipy.optimize.curve_fit - - `#6494 `__: ENH: stats: Use log1p() to improve some calculations. - - `#6495 `__: BUG: Use MST algorithm instead of SLINK for single linkage clustering - - `#6497 `__: MRG: Add minimum_phase filter function - - `#6505 `__: reset scipy.signal.resample window shape to 1-D - - `#6507 `__: BUG: linkage: Raise exception if y contains non-finite elements - - `#6509 `__: ENH: _lib: add common machinery for low-level callback functions - - `#6520 `__: scipy.sparse.base.__mul__ non-numpy/scipy objects with 'shape'... - - `#6522 `__: Replace kl_div by rel_entr in entropy - - `#6524 `__: DOC: add next_fast_len to list of functions - - `#6527 `__: DOC: Release notes to reflect the new covariance feature in optimize.curve_fit - - `#6532 `__: ENH: Simplify _cos_win, document it, add symmetric/periodic arg - - `#6535 `__: MAINT: sparse.csgraph: updating old cython loops - - `#6540 `__: DOC: add to documentation of orthogonal polynomials - - `#6544 `__: TST: Ensure tests for scipy._lib are run by scipy.test() - - `#6546 `__: updated docstring of stats.linregress - - `#6553 `__: commited changes that I originally submitted for scipy.signal.cspline? - - `#6561 `__: BUG: modify signal.find_peaks_cwt() to return array and accept... - - `#6562 `__: DOC: Negative binomial distribution clarification - - `#6563 `__: MAINT: be more liberal in requiring numpy - - `#6567 `__: MAINT: use xrange for iteration in differential_evolution fixes... - - `#6572 `__: BUG: "sp.linalg.solve_discrete_are" fails for random data - - `#6578 `__: BUG: misc: allow both cmin/cmax and low/high params in bytescale - - `#6581 `__: Fix some unfortunate typos - - `#6582 `__: MAINT: linalg: make handling of infinite eigenvalues in `ordqz`... - - `#6585 `__: DOC: interpolate: correct seealso links to ndimage - - `#6588 `__: Update docstring of scipy.spatial.distance_matrix - - `#6592 `__: DOC: Replace 'first' by 'smallest' in mode - - `#6593 `__: MAINT: remove scipy.weave submodule - - `#6594 `__: DOC: distance.squareform: fix html docs, add note about dtype... - - `#6598 `__: [DOC] Fix incorrect error message in medfilt2d - - `#6599 `__: MAINT: linalg: turn a `solve_discrete_are` test back on - - `#6600 `__: DOC: Add SOS goals to roadmap - - `#6601 `__: DEP: Raise minimum numpy version to 1.8.2 - - `#6605 `__: MAINT: 'new' module is deprecated, don't use it - - `#6607 `__: DOC: add note on change in wheel dependency on numpy and pip... - - `#6609 `__: Fixes #6602 - Typo in docs - - `#6616 `__: ENH: generalization of continuous and discrete Riccati solvers... - - `#6621 `__: DOC: improve cluster.hierarchy docstrings. - - `#6623 `__: CS matrix prune method should copy data from large unpruned arrays - - `#6625 `__: DOC: special: complete documentation of `eval_*` functions - - `#6626 `__: TST: special: silence some deprecation warnings - - `#6631 `__: fix parameter name doc for discrete distributions - - `#6632 `__: MAINT: stats: change some instances of `special` to `sc` - - `#6633 `__: MAINT: refguide: py2k long integers are equal to py3k integers - - `#6638 `__: MAINT: change type declaration in cluster.linkage, prevent overflow - - `#6640 `__: BUG: fix issue with duplicate values used in cluster.vq.kmeans - - `#6641 `__: BUG: fix corner case in cluster.vq.kmeans for large thresholds - - `#6643 `__: MAINT: clean up truncation modes of dendrogram - - `#6645 `__: MAINT: special: rename `*_roots` functions - - `#6646 `__: MAINT: clean up mpmath imports - - `#6647 `__: DOC: add sqrt to Mahalanobis description for pdist - - `#6648 `__: DOC: special: add a section on `cython_special` to the tutorial - - `#6649 `__: ENH: Added scipy.spatial.distance.directed_hausdorff - - `#6650 `__: DOC: add Sphinx roles for DOI and arXiv links - - `#6651 `__: BUG: mstats: make sure mode(..., None) does not modify its input - - `#6652 `__: DOC: special: add section to tutorial on functions not in special - - `#6653 `__: ENH: special: add the Wright Omega function - - `#6656 `__: ENH: don't coerce input to double with custom metric in cdist... - - `#6658 `__: Faster/shorter code for computation of discordances - - `#6659 `__: DOC: special: make __init__ summaries and html summaries match - - `#6661 `__: general.rst: Fix a typo - - `#6664 `__: TST: Spectral functions' window correction factor - - `#6665 `__: [DOC] Conditions on v in RectSphereBivariateSpline - - `#6668 `__: DOC: Mention negative masses for center of mass - - `#6675 `__: MAINT: special: remove outdated README - - `#6677 `__: BUG: Fixes computation of p-values. - - `#6679 `__: BUG: optimize: return correct Jacobian for method 'SLSQP' in... - - `#6680 `__: ENH: Add structural rank to sparse.csgraph - - `#6686 `__: TST: Added Airspeed Velocity benchmarks for SphericalVoronoi - - `#6687 `__: DOC: add section "deciding on new features" to developer guide. - - `#6691 `__: ENH: Clearer error when fmin_slsqp obj doesn't return scalar - - `#6702 `__: TST: Added airspeed velocity benchmarks for scipy.spatial.distance.cdist - - `#6707 `__: TST: interpolate: test fitpack wrappers, not _impl - - `#6709 `__: TST: fix a number of test failures on 32-bit systems - - `#6711 `__: MAINT: move function definitions from __fitpack.h to _fitpackmodule.c - - `#6712 `__: MAINT: clean up wishlist in stats.morestats, and copyright statement. - - `#6715 `__: DOC: update the release notes with BSpline et al. - - `#6716 `__: MAINT: scipy.io.wavfile: No infinite loop when trying to read... - - `#6717 `__: some style cleanup - - `#6723 `__: BUG: special: cast to float before in-place multiplication in... - - `#6726 `__: address performance regressions in interp1d - - `#6728 `__: DOC: made code examples in `integrate` tutorial copy-pasteable - - `#6731 `__: DOC: scipy.optimize: Added an example for wrapping complex-valued... - - `#6732 `__: MAINT: cython_special: remove `errprint` - - `#6733 `__: MAINT: special: fix some pyflakes warnings - - `#6734 `__: DOC: sparse.linalg: fixed matrix description in `bicgstab` doc - - `#6737 `__: BLD: update `cythonize.py` to detect changes in pxi files - - `#6740 `__: DOC: special: some small fixes to docstrings - - `#6741 `__: MAINT: remove dead code in interpolate.py - - `#6742 `__: BUG: fix ``linalg.block_diag`` to support zero-sized matrices. - - `#6744 `__: ENH: interpolate: make PPoly.from_spline accept BSpline objects - - `#6746 `__: DOC: special: clarify use of Condon-Shortley phase in `sph_harm`/`lpmv` - - `#6750 `__: ENH: sparse: avoid densification on broadcasted elem-wise mult - - `#6751 `__: sinm doc explained cosm - - `#6753 `__: ENH: special: allow for more fine-tuned error handling - - `#6759 `__: Move logsumexp and pade from scipy.misc to scipy.special and... - - `#6761 `__: ENH: argmax and argmin methods for sparse matrices - - `#6762 `__: DOC: Improve docstrings of sparse matrices - - `#6763 `__: ENH: Weighted tau - - `#6768 `__: ENH: cythonized spherical Voronoi region polygon vertex sorting - - `#6770 `__: Correction of Delaunay class' documentation - - `#6775 `__: ENH: Integrating LAPACK "expert" routines with conditioning warnings... - - `#6776 `__: MAINT: Removing the trivial f2py warnings - - `#6777 `__: DOC: Update rv_continuous.fit doc. - - `#6778 `__: MAINT: cluster.hierarchy: Improved wording of error msgs - - `#6786 `__: BLD: increase minimum Cython version to 0.23.4 - - `#6787 `__: DOC: expand on ``linalg.block_diag`` changes in 0.19.0 release... - - `#6789 `__: ENH: Add further documentation for norm.fit - - `#6790 `__: MAINT: Fix a potential problem in nn_chain linkage algorithm - - `#6791 `__: DOC: Add examples to scipy.ndimage.fourier - - `#6792 `__: DOC: fix some numpydoc / Sphinx issues. - - `#6793 `__: MAINT: fix circular import after moving functions out of misc - - `#6796 `__: TST: test importing each submodule. Regression test for gh-6793. - - `#6799 `__: ENH: stats: Argus distribution - - `#6801 `__: ENH: stats: Histogram distribution - - `#6803 `__: TST: make sure tests for ``_build_utils`` are run. - - `#6804 `__: MAINT: more fixes in `loggamma` - - `#6806 `__: ENH: Faster linkage for 'centroid' and 'median' methods - - `#6810 `__: ENH: speed up upfirdn and resample_poly for n-dimensional arrays - - `#6812 `__: TST: Added ConvexHull asv benchmark code - - `#6814 `__: ENH: Different extrapolation modes for different dimensions in... - - `#6826 `__: Signal spectral window default fix - - `#6828 `__: BUG: SphericalVoronoi Space Complexity (Fixes #6811) - - `#6830 `__: RealData docstring correction - - `#6834 `__: DOC: Added reference for skewtest function. See #6829 - - `#6836 `__: DOC: Added mode='mirror' in the docstring for the functions accepting... - - `#6838 `__: MAINT: sparse: start removing old BSR methods - - `#6844 `__: handle incompatible dimensions when input is not an ndarray in... - - `#6847 `__: Added maxiter to golden search. - - `#6850 `__: BUG: added check for optional param scipy.stats.spearmanr - - `#6858 `__: MAINT: Removing redundant tests - - `#6861 `__: DEP: Fix escape sequences deprecated in Python 3.6. - - `#6862 `__: DOC: dx should be float, not int - - `#6863 `__: updated documentation curve_fit - - `#6866 `__: DOC : added some documentation to j1 referring to spherical_jn - - `#6867 `__: DOC: cdist move long examples list into Notes section - - `#6868 `__: BUG: Make stats.mode return a ModeResult namedtuple on empty... - - `#6871 `__: Corrected documentation. - - `#6874 `__: ENH: gaussian_kde.logpdf based on logsumexp - - `#6877 `__: BUG: ndimage: guard against footprints of all zeros - - `#6881 `__: python 3.6 - - `#6885 `__: Vectorized integrate.fixed_quad - - `#6886 `__: fixed typo - - `#6891 `__: TST: fix failures for linalg.dare/care due to tightened test... - - `#6892 `__: DOC: fix a bunch of Sphinx errors. - - `#6894 `__: TST: Added asv benchmarks for scipy.spatial.Voronoi - - `#6908 `__: BUG: Fix return dtype for complex input in spsolve - - `#6909 `__: ENH: fftpack: use float32 routines for float16 inputs. - - `#6911 `__: added min/max support to binned_statistic - - `#6913 `__: Fix 6875: SLSQP raise ValueError for all invalid bounds. - - `#6914 `__: DOCS: GH6903 updating docs of Spatial.distance.pdist - - `#6916 `__: MAINT: fix some issues for 32-bit Python - - `#6924 `__: BLD: update Bento build for scipy.LowLevelCallable - - `#6932 `__: ENH: Use OrderedDict in io.netcdf. Closes gh-5537 - - `#6933 `__: BUG: fix LowLevelCallable issue on 32-bit Python. - - `#6936 `__: BUG: sparse: handle size-1 2D indexes correctly - - `#6938 `__: TST: fix test failures in special on 32-bit Python. - - `#6939 `__: Added attributes list to cKDTree docstring - - `#6940 `__: improve efficiency of dok_matrix.tocoo - - `#6942 `__: DOC: add link to liac-arff package in the io.arff docstring. - - `#6943 `__: MAINT: Docstring fixes and an additional test for linalg.solve - - `#6944 `__: DOC: Add example of odeint with a banded Jacobian to the integrate... - - `#6946 `__: ENH: hypergeom.logpmf in terms of betaln - - `#6947 `__: TST: speedup distance tests - - `#6948 `__: DEP: Deprecate the keyword "debug" from linalg.solve - - `#6950 `__: BUG: Correctly treat large integers in MMIO (fixes #6397) - - `#6952 `__: ENH: Minor user-friendliness cleanup in LowLevelCallable - - `#6956 `__: DOC: improve description of 'output' keyword for convolve - - `#6957 `__: ENH more informative error in sparse.bmat - - `#6962 `__: Shebang fixes - - `#6964 `__: DOC: note argmin/argmax addition - - `#6965 `__: BUG: Fix issues passing error tolerances in dblquad and tplquad. - - `#6971 `__: fix the docstring of signaltools.correlate - - `#6973 `__: Silence expected numpy warnings in scipy.ndimage.interpolation.zoom() - - `#6975 `__: BUG: special: fix regex in `generate_ufuncs.py` - - `#6976 `__: Update docstring for griddata - - `#6978 `__: Avoid division by zero in zoom factor calculation - - `#6979 `__: BUG: ARE solvers did not check the generalized case carefully - - `#6985 `__: ENH: sparse: add scipy.sparse.linalg.spsolve_triangular - - `#6994 `__: MAINT: spatial: updates to plotting utils - - `#6995 `__: DOC: Bad documentation of k in sparse.linalg.eigs See #6990 - - `#6997 `__: TST: Changed the test with a less singular example - - `#7000 `__: DOC: clarify interp1d 'zero' argument - - `#7007 `__: BUG: Fix division by zero in linregress() for 2 data points - - `#7009 `__: BUG: Fix problem in passing drop_rule to scipy.sparse.linalg.spilu - - `#7012 `__: speed improvment in _distn_infrastructure.py - - `#7014 `__: Fix Typo: add a single quotation mark to fix a slight typo - - `#7021 `__: MAINT: stats: use machine constants from np.finfo, not machar - - `#7026 `__: MAINT: update .mailmap - - `#7032 `__: Fix layout of rv_histogram docs - - `#7035 `__: DOC: update 0.19.0 release notes - - `#7036 `__: ENH: Add more boundary options to signal.stft - - `#7040 `__: TST: stats: skip too slow tests - - `#7042 `__: MAINT: sparse: speed up setdiag tests - - `#7043 `__: MAINT: refactory and code cleaning Xdist - - `#7053 `__: Fix msvc 9 and 10 compile errors - - `#7060 `__: DOC: updated release notes with #7043 and #6656 - - `#7062 `__: MAINT: Change defaut STFT boundary kwarg to "zeros" - - `#7064 `__: Fix ValueError: path is on mount 'X:', start on mount 'D:' on... - - `#7067 `__: TST: Fix PermissionError: [Errno 13] Permission denied on Windows - - `#7068 `__: TST: Fix UnboundLocalError: local variable 'data' referenced... - - `#7069 `__: Fix OverflowError: Python int too large to convert to C long... - - `#7071 `__: TST: silence RuntimeWarning for nan test of stats.spearmanr - - `#7072 `__: Fix OverflowError: Python int too large to convert to C long... - - `#7084 `__: TST: linalg: bump tolerance in test_falker - - `#7095 `__: TST: linalg: bump more tolerances in test_falker - - `#7101 `__: TST: Relax solve_continuous_are test case 2 and 12 - - `#7106 `__: BUG: stop cdist "correlation" modifying input - - `#7116 `__: Backports to 0.19.0rc2 Checksums ========= MD5 ~~~ dde4d5d44a0274a5abb01be4a3cd486a scipy-0.19.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 08809612b46e660e567e3272ec11c808 scipy-0.19.0-cp27-cp27m-manylinux1_i686.whl 0e49f7fc8d31c1c79f0a4d63b29e8a1f scipy-0.19.0-cp27-cp27m-manylinux1_x86_64.whl a2669158cf847856d292b8a60cdaa170 scipy-0.19.0-cp27-cp27mu-manylinux1_i686.whl adfa1f5127a789165dfe9ff140ec0d6e scipy-0.19.0-cp27-cp27mu-manylinux1_x86_64.whl d568c9f60683c33b81ebc1c39eea198a scipy-0.19.0-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl a90148fec477c1950578b40a1197509f scipy-0.19.0-cp34-cp34m-manylinux1_i686.whl ed27be5380e0aaf0229adf747e760f8c scipy-0.19.0-cp34-cp34m-manylinux1_x86_64.whl 4cda63dc7b73bd03bdf9e8ebc6027526 scipy-0.19.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl ff8e652b5e918b276793f1ce542a5959 scipy-0.19.0-cp35-cp35m-manylinux1_i686.whl 60741a900a145eb924ec861ec2743582 scipy-0.19.0-cp35-cp35m-manylinux1_x86_64.whl 81685a961d6118459b7787e8465c8d36 scipy-0.19.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 83f0750862c80a659686797d4ec9bca0 scipy-0.19.0-cp36-cp36m-manylinux1_i686.whl 3cbb30615496fbbf9b52c9a643c6fe5e scipy-0.19.0-cp36-cp36m-manylinux1_x86_64.whl 735cdb6fbfcb9917535749816202d0af scipy-0.19.0.tar.gz b21466e87a642940fb9ba35be74940a3 scipy-0.19.0.tar.xz 91b8396231eec780222a57703d3ec550 scipy-0.19.0.zip SHA256 ~~~~~~ 517a85600d6574fef1a67e6d2001b847c27c8bfd136f7a12879c3f91e7bb291f scipy-0.19.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 3c34987ee52fd98b34f2e4a6d277b452d49056f1383550acc54c5bab408a194c scipy-0.19.0-cp27-cp27m-manylinux1_i686.whl eabdfde8276007e9aec9a400f9528645001a30f7d78b04a0ab215183d9523e2a scipy-0.19.0-cp27-cp27m-manylinux1_x86_64.whl fa67bbb0a3225fcd8610d693e7b2ca08fda107359e48229f7b83593bbb70cc97 scipy-0.19.0-cp27-cp27mu-manylinux1_i686.whl 4147b97709e75822e73d312e4d262410baafa961a7b11649a7b4b7c2d41fb4fe scipy-0.19.0-cp27-cp27mu-manylinux1_x86_64.whl 663e78bfa197376547424aff9fb5009e7b2f26855ee5aaf1a2ddbb2f4dc6af3b scipy-0.19.0-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 2e70ded029d51f6a48d4b2a154f583b85aa2e3290dfd71e0b6bbcfe9454cffdd scipy-0.19.0-cp34-cp34m-manylinux1_i686.whl 4b2731a191dfa48a05b2f5bc18881595a1418092092ecdd8d3feab80f72adc96 scipy-0.19.0-cp34-cp34m-manylinux1_x86_64.whl 57f7be33f1009ad6199132e8a7e5d4c9727224680d8cbc4596a2a8935a86f96b scipy-0.19.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 0496f2b204a63cde3797e5452bf671ee25afc11bd9489ae69cd4dccee13083a1 scipy-0.19.0-cp35-cp35m-manylinux1_i686.whl 1bcf71f2e534a1aabf9f075700701bf3af434120b1b114dfa4723d02e076ed1f scipy-0.19.0-cp35-cp35m-manylinux1_x86_64.whl e1c45905f550b5f14e1f47697c92bab5c1e6ba77da5a441bd2affa4621c41b26 scipy-0.19.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 47f537214293a5d74b05d217afa49b582b7fd9428ec9ea64be69210cfc56611a scipy-0.19.0-cp36-cp36m-manylinux1_i686.whl f5a3a7dcbeb345c227770029870aeb547a3c207a6cbc0106d6157139fd0c23e9 scipy-0.19.0-cp36-cp36m-manylinux1_x86_64.whl eba12b5f757a8c839b26a06f4936ecb80b65cb3674981ee25449b2a55663abe8 scipy-0.19.0.tar.gz ed52232afb2b321a4978e39040d94bf81af90176ba64f58c4499dc305a024437 scipy-0.19.0.tar.xz 4190d34bf9a09626cd42100bbb12e3d96b2daf1a8a3244e991263eb693732122 scipy-0.19.0.zip -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQEcBAEBCAAGBQJYwWscAAoJEIp0pQ0zQcu+JG8H+gOy04ecVl+IBisy/Fz5wfuy xrsCT5yhRPQpxKph8g/6Us5Oh7s8ixrVt9gTVccAspmWXQJhMy5kcppC3s5WsvU+ jFOwUnW7c9QvtIf2ZD9Ay/56WojlXg1ui17MqoCbmkEn2QE8KTKu93hIZpVD5wmV 1fhd7u/ieeQ7sfj6gMZzt0AGpVjnGedEzHRY4zI0PkiCY+Ex8sc2W8G2h5Qbnx9r KoqECIuesLQzVbNgPhWaWaiE1TNX0EJYdWQll0T8scI4opUdg6vEaR05aPhxeR1r KaGEnvfASTZ369COkuVB4JINlKQj0dwLBFIr9NGVzX4vU74GMh5TuDfJlA/mvGU= =bOnA -----END PGP SIGNATURE----- From faltet at gmail.com Thu Mar 9 14:15:41 2017 From: faltet at gmail.com (Francesc Alted) Date: Thu, 9 Mar 2017 20:15:41 +0100 Subject: [Numpy-discussion] PyData Barcelona this May In-Reply-To: References: Message-ID: Hola Jaime! 2017-03-09 15:45 GMT+01:00 Jaime Fern?ndez del R?o : > There will be a PyData conference in Barcelona this May: > > http://pydata.org/barcelona2017/ > > I am planning on attending, and was thinking of maybe proposing to > organize a numpy-themed workshop or tutorial. > > My personal inclination would be to look at some advanced topic that I > know well, like writing gufuncs in Cython, but wouldn't mind doing a more > run of the mill thing. Anyone has any thoughts or experiences on what has > worked well in similar situations? Any specific topic you always wanted to > attend a workshop on, but were afraid to ask? > ?Writing gufuncs in Cython seems a quite advanced? topic for a workshop, but an interesting one indeed. Numba also supports creating gufuncs ( http://numba.pydata.org/numba-doc/dev/reference/numpysupported.html), so this perhaps may work as a first approach before going deeper into Cython. > > Alternatively, or on top of the workshop, I could propose to do a talk: > talking last year at PyData Madrid about the new indexing was a lot of fun! > Thing is, I have been quite disconnected from the project this past year, > and can't really think of any worthwhile topic. Is there any message that > we as a project would like to get out to the larger community? > ?Not a message in particular, but perhaps it would be nice talking about the temporaries removal ?in expressions that Julian implemented recently ( https://github.com/numpy/numpy/pull/7997) and that is to be released in 1.13. It is a really cool (and somewhat scary) patch ;) > > And if you are planning on attending, please give me a shout. > ?It would be nice to attend and see you again, but unfortunately I am quite swamped. Will see.? ?Have fun in Barcelona!? -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Thu Mar 9 14:29:09 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 09 Mar 2017 20:29:09 +0100 Subject: [Numpy-discussion] PyData Barcelona this May In-Reply-To: References: Message-ID: <1489087749.4278.8.camel@sipsolutions.net> On Thu, 2017-03-09 at 15:45 +0100, Jaime Fern?ndez del R?o wrote: > There will be a PyData conference in Barcelona this May: > > http://pydata.org/barcelona2017/ > > I am planning on attending, and was thinking of maybe proposing to > organize a numpy-themed workshop or tutorial. > > My personal inclination would be to look at some advanced topic that > I know well, like writing gufuncs in Cython, but wouldn't mind doing > a more run of the mill thing. Anyone has any thoughts or experiences > on what has worked well in similar situations? Any specific topic you > always wanted to attend a workshop on, but were afraid to ask? > > Alternatively, or on top of the workshop, I could propose to do a > talk: talking last year at PyData Madrid about the new indexing was a > lot of fun! Thing is, I have been quite disconnected from the project > this past year, and can't really think of any worthwhile topic. Is > there any message that we as a project would like to get out to the > larger community? > Francesc already pointed out the temporary optimization. From what I remember, my personal highlight would probably be Pauli's work on the memory overlap detection. Though both are rather passive improvements I guess (you don't really have to learn them to use them), its very cool! And if its about highlighting new stuff, these can probably easily fill a talk. > And if you are planning on attending, please give me a shout. > Barcelona :). Maybe I should think about it, but probably not. > Thanks, > > Jaime > > --? > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus > planes de dominaci?n mundial. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From matthew.brett at gmail.com Fri Mar 10 17:25:27 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 10 Mar 2017 14:25:27 -0800 Subject: [Numpy-discussion] Building with Clang on Windows? Message-ID: Hi, Has anyone succeeded in building numpy with Clang on Windows? Or, building any other Python extension with Clang on Windows? I'd love to hear about how you did it... Cheers, Matthew From jtaylor.debian at googlemail.com Mon Mar 13 07:21:24 2017 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 13 Mar 2017 12:21:24 +0100 Subject: [Numpy-discussion] caching large allocations on gnu/linux Message-ID: Hi, As numpy often allocates large arrays and one factor in its performance is faulting memory from the kernel to the process. This has some cost that is relatively significant. For example in this operation on large arrays it accounts for 10-15% of the runtime: import numpy as np a = np.ones(10000000) b = np.ones(10000000) %timeit (a * b)**2 + 3 54.45% ipython umath.so [.] sse2_binary_multiply_DOUBLE 20.43% ipython umath.so [.] DOUBLE_add 16.66% ipython [kernel.kallsyms] [k] clear_page The reason for this is that the glibc memory allocator uses memory mapping for large allocations instead of reusing already faulted memory. The reason for this is to return memory back to the system immediately when it is free to keep the whole system more robust. This makes a lot of sense in general but not so much for many numerical applications that often are the only thing running. But despite if have been shown in an old paper that caching memory in numpy speeds up many applications, numpys usage is diverse so we couldn't really diverge from the glibc behaviour. Until Linux 4.5 added support for madvise(MADV_FREE). This flag of the madvise syscall tells the kernel that a piece of memory can be reused by other processes if there is memory pressure. Should another process claim the memory and the original process want to use it again the kernel will fault new memory into its place so it behaves exactly as if it was just freed regularly. But when no other process claims the memory and the original process wants to reuse it, the memory do not need to be faulted again. So effectively this flag allows us to cache memory inside numpy that can be reused by the rest of the system if required. Doing gives the expected speedup in the above example. An issue is that the memory usage of numpy applications will seem to increase. The memory that is actually free will still show up in the usual places you look at memory usage. Namely the resident memory usage of the process in top, /proc etc. The usage will only go down when the memory is actually needed by other processes. This probably would break some of the memory profiling tools so probably we need a switch to disable the caching for the profiling tools to use. Another concern is that using this functionality is actually the job of the system memory allocator but I had a look at glibcs allocator and it does not look like an easy job to make good use of MADV_FREE retroactively, so I don't expect this to happen anytime soon. Should it be agreed that caching is worthwhile I would propose a very simple implementation. We only really need to cache a small handful of array data pointers for the fast allocate deallocate cycle that appear in common numpy usage. For example a small list of maybe 4 pointers storing the 4 largest recent deallocations. New allocations just pick the first memory block of sufficient size. The cache would only be active on systems that support MADV_FREE (which is linux 4.5 and probably BSD too). So what do you think of this idea? cheers, Julian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 845 bytes Desc: OpenPGP digital signature URL: From wieser.eric+numpy at gmail.com Mon Mar 13 07:57:48 2017 From: wieser.eric+numpy at gmail.com (Eric Wieser) Date: Mon, 13 Mar 2017 04:57:48 -0700 (MST) Subject: [Numpy-discussion] float16/32: wrong number of digits? In-Reply-To: References: Message-ID: <1489406268634-44046.post@n7.nabble.com> > `float(repr(a)) == a` is guaranteed for Python `float` And `np.float16(repr(a)) == a` is guaranteed for `np.float16`(and the same is true up to `float128`, which can be platform-dependent). Your code doesn't work because you're deserializing to a higher precision format than you serialized to. -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/float16-32-wrong-number-of-digits-tp44037p44046.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From peridot.faceted at gmail.com Mon Mar 13 10:18:40 2017 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 13 Mar 2017 14:18:40 +0000 Subject: [Numpy-discussion] float16/32: wrong number of digits? In-Reply-To: <1489406268634-44046.post@n7.nabble.com> References: <1489406268634-44046.post@n7.nabble.com> Message-ID: On Mon, Mar 13, 2017 at 12:57 PM Eric Wieser wrote: > > `float(repr(a)) == a` is guaranteed for Python `float` > > And `np.float16(repr(a)) == a` is guaranteed for `np.float16`(and the same > is true up to `float128`, which can be platform-dependent). Your code > doesn't work because you're deserializing to a higher precision format than > you serialized to. > I would hesitate to make this guarantee - certainly for old versions of numpy, np.float128(repr(x))!=x in many cases. I submitted a patch, now accepted, that probably accomplishes this on most systems (in fact this is now in the test suite) but if you are using a version of numpy that is a couple of years old, there is no way to convert long doubles to human-readable or back that doesn't lose precision. To repeat: only in recent versions of numpy can long doubles be converted to human-readable and back without passing through doubles. It is still not possible to use % or format() on them without discarding all precision beyond doubles. If you actually need long doubles (and if you don't, why use them?) make sure your application includes a test for this ability. I recommend checking repr(1+np.finfo(np.longdouble).eps). Anne P.S. You can write (I have) a short piece of cython code that will reliably repr and back long doubles, but on old versions of numpy it's just not possible from within python. -A -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Mon Mar 13 11:21:57 2017 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 13 Mar 2017 15:21:57 +0000 Subject: [Numpy-discussion] caching large allocations on gnu/linux In-Reply-To: References: Message-ID: On Mon, Mar 13, 2017 at 12:21 PM Julian Taylor < jtaylor.debian at googlemail.com> wrote: Should it be agreed that caching is worthwhile I would propose a very > simple implementation. We only really need to cache a small handful of > array data pointers for the fast allocate deallocate cycle that appear > in common numpy usage. > For example a small list of maybe 4 pointers storing the 4 largest > recent deallocations. New allocations just pick the first memory block > of sufficient size. > The cache would only be active on systems that support MADV_FREE (which > is linux 4.5 and probably BSD too). > > So what do you think of this idea? > This is an interesting thought, and potentially a nontrivial speedup with zero user effort. But coming up with an appropriate caching policy is going to be tricky. The thing is, for each array, numpy grabs a block "the right size", and that size can easily vary by orders of magnitude, even within the temporaries of a single expression as a result of broadcasting. So simply giving each new array the smallest cached block that will fit could easily result in small arrays in giant allocated blocks, wasting non-reclaimable memory. So really you want to recycle blocks of the same size, or nearly, which argues for a fairly large cache, with smart indexing of some kind. How much difference is this likely to make? Note that numpy is now in some cases able to eliminate allocation of temporary arrays. I think the only way to answer these questions is to set up a trial implementation, with user-switchable behaviour (which should include the ability for users to switch it on even when MADV_FREE is not available) and sensible statistics reporting. Then volunteers can run various numpy workloads past it. Anne -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Mon Mar 13 13:11:40 2017 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 13 Mar 2017 18:11:40 +0100 Subject: [Numpy-discussion] caching large allocations on gnu/linux In-Reply-To: References: Message-ID: <398b801d-fdb9-5d4f-5594-15c8d7592d92@googlemail.com> On 13.03.2017 16:21, Anne Archibald wrote: > > > On Mon, Mar 13, 2017 at 12:21 PM Julian Taylor > > > wrote: > > Should it be agreed that caching is worthwhile I would propose a very > simple implementation. We only really need to cache a small handful of > array data pointers for the fast allocate deallocate cycle that appear > in common numpy usage. > For example a small list of maybe 4 pointers storing the 4 largest > recent deallocations. New allocations just pick the first memory block > of sufficient size. > The cache would only be active on systems that support MADV_FREE (which > is linux 4.5 and probably BSD too). > > So what do you think of this idea? > > > This is an interesting thought, and potentially a nontrivial speedup > with zero user effort. But coming up with an appropriate caching policy > is going to be tricky. The thing is, for each array, numpy grabs a block > "the right size", and that size can easily vary by orders of magnitude, > even within the temporaries of a single expression as a result of > broadcasting. So simply giving each new array the smallest cached block > that will fit could easily result in small arrays in giant allocated > blocks, wasting non-reclaimable memory. So really you want to recycle > blocks of the same size, or nearly, which argues for a fairly large > cache, with smart indexing of some kind. > The nice thing about MADV_FREE is that we don't need any clever cache. The same process that marked the pages free can reclaim them in another allocation, at least that is what my testing indicates it allows. So a small allocation getting a huge memory block does not waste memory as the top unused part will get reclaimed when needed, either by numpy itself doing another allocation or a different program on the system. An issue that does arise though is that this memory is not available for the page cache used for caching on disk data. A too large cache might then be detrimental for IO heavy workloads that rely on the page cache. So we might want to cap it to some max size, provide an explicit on/off switch and/or have numpy IO functions clear the cache. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 845 bytes Desc: OpenPGP digital signature URL: From faltet at gmail.com Mon Mar 13 14:54:20 2017 From: faltet at gmail.com (Francesc Alted) Date: Mon, 13 Mar 2017 19:54:20 +0100 Subject: [Numpy-discussion] caching large allocations on gnu/linux In-Reply-To: <398b801d-fdb9-5d4f-5594-15c8d7592d92@googlemail.com> References: <398b801d-fdb9-5d4f-5594-15c8d7592d92@googlemail.com> Message-ID: 2017-03-13 18:11 GMT+01:00 Julian Taylor : > On 13.03.2017 16:21, Anne Archibald wrote: > > > > > > On Mon, Mar 13, 2017 at 12:21 PM Julian Taylor > > > > > wrote: > > > > Should it be agreed that caching is worthwhile I would propose a very > > simple implementation. We only really need to cache a small handful > of > > array data pointers for the fast allocate deallocate cycle that > appear > > in common numpy usage. > > For example a small list of maybe 4 pointers storing the 4 largest > > recent deallocations. New allocations just pick the first memory > block > > of sufficient size. > > The cache would only be active on systems that support MADV_FREE > (which > > is linux 4.5 and probably BSD too). > > > > So what do you think of this idea? > > > > > > This is an interesting thought, and potentially a nontrivial speedup > > with zero user effort. But coming up with an appropriate caching policy > > is going to be tricky. The thing is, for each array, numpy grabs a block > > "the right size", and that size can easily vary by orders of magnitude, > > even within the temporaries of a single expression as a result of > > broadcasting. So simply giving each new array the smallest cached block > > that will fit could easily result in small arrays in giant allocated > > blocks, wasting non-reclaimable memory. So really you want to recycle > > blocks of the same size, or nearly, which argues for a fairly large > > cache, with smart indexing of some kind. > > > > The nice thing about MADV_FREE is that we don't need any clever cache. > The same process that marked the pages free can reclaim them in another > allocation, at least that is what my testing indicates it allows. > So a small allocation getting a huge memory block does not waste memory > as the top unused part will get reclaimed when needed, either by numpy > itself doing another allocation or a different program on the system. > ?Well, what you say makes a lot of sense to me, so if you have tested that then I'd say that this is worth a PR and see how it works on different workloads?. > > An issue that does arise though is that this memory is not available for > the page cache used for caching on disk data. A too large cache might > then be detrimental for IO heavy workloads that rely on the page cache. > ?Yeah. Also, memory mapped arrays use the page cache intensively, so we should test this use case? and see how the caching affects memory map performance. > So we might want to cap it to some max size, provide an explicit on/off > switch and/or have numpy IO functions clear the cache. > ?Definitely? dynamically allowing the disabling ?this feature would be desirable. That would provide an easy path for testing how it affects performance. Would that be feasible? Francesc -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Mon Mar 13 15:16:05 2017 From: rmay31 at gmail.com (Ryan May) Date: Mon, 13 Mar 2017 13:16:05 -0600 Subject: [Numpy-discussion] docs.scipy.org down Message-ID: Is https://docs.scipy.org/ being down known issue? Ryan -- Ryan May -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Mar 13 15:20:02 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 14 Mar 2017 08:20:02 +1300 Subject: [Numpy-discussion] docs.scipy.org down In-Reply-To: References: Message-ID: On Tue, Mar 14, 2017 at 8:16 AM, Ryan May wrote: > Is https://docs.scipy.org/ being down known issue? > It is. Is being worked on; tracking issue is https://github.com/numpy/numpy/issues/8779 Thanks for reporting, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Tue Mar 14 11:21:33 2017 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Tue, 14 Mar 2017 16:21:33 +0100 Subject: [Numpy-discussion] caching large allocations on gnu/linux In-Reply-To: References: <398b801d-fdb9-5d4f-5594-15c8d7592d92@googlemail.com> Message-ID: On 13.03.2017 19:54, Francesc Alted wrote: > 2017-03-13 18:11 GMT+01:00 Julian Taylor >: > > On 13.03.2017 16:21, Anne Archibald wrote: > > > > > > On Mon, Mar 13, 2017 at 12:21 PM Julian Taylor > > > >> > > wrote: > > > > Should it be agreed that caching is worthwhile I would propose a very > > simple implementation. We only really need to cache a small handful of > > array data pointers for the fast allocate deallocate cycle that appear > > in common numpy usage. > > For example a small list of maybe 4 pointers storing the 4 largest > > recent deallocations. New allocations just pick the first memory block > > of sufficient size. > > The cache would only be active on systems that support MADV_FREE (which > > is linux 4.5 and probably BSD too). > > > > So what do you think of this idea? > > > > > > This is an interesting thought, and potentially a nontrivial speedup > > with zero user effort. But coming up with an appropriate caching policy > > is going to be tricky. The thing is, for each array, numpy grabs a block > > "the right size", and that size can easily vary by orders of magnitude, > > even within the temporaries of a single expression as a result of > > broadcasting. So simply giving each new array the smallest cached block > > that will fit could easily result in small arrays in giant allocated > > blocks, wasting non-reclaimable memory. So really you want to recycle > > blocks of the same size, or nearly, which argues for a fairly large > > cache, with smart indexing of some kind. > > > > The nice thing about MADV_FREE is that we don't need any clever cache. > The same process that marked the pages free can reclaim them in another > allocation, at least that is what my testing indicates it allows. > So a small allocation getting a huge memory block does not waste memory > as the top unused part will get reclaimed when needed, either by numpy > itself doing another allocation or a different program on the system. > > > ?Well, what you say makes a lot of sense to me, so if you have tested > that then I'd say that this is worth a PR and see how it works on > different workloads?. > > > > An issue that does arise though is that this memory is not available for > the page cache used for caching on disk data. A too large cache might > then be detrimental for IO heavy workloads that rely on the page cache. > > > ?Yeah. Also, memory mapped arrays use the page cache intensively, so we > should test this use case? and see how the caching affects memory map > performance. > > > So we might want to cap it to some max size, provide an explicit on/off > switch and/or have numpy IO functions clear the cache. > > > ?Definitely? dynamically > allowing the disabling > ?this feature would be desirable. That would provide an easy path for > testing how it affects performance. Would that be feasible? > > I have created a PR with such a simple cache implemented: https://github.com/numpy/numpy/pull/8783 This sets the max amount of memory pointers to save and returns the old value: np.core.multiarray.set_memory_cache_size(4) On system where it works it return a value greater 0 (4 currently). The size of the cache in bytes is currently unbounded. Setting the value to 0 clears and disables the cache. You should probably not expect too large performance improvements. It will only have an effect in applications that have measurable page faulting overhead which only happens if you have lots of relatively short operations that create copies of large arrays. So mostly operations with temporaries and maybe some indexing operations. From matthew.brett at gmail.com Tue Mar 14 22:21:12 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 14 Mar 2017 19:21:12 -0700 Subject: [Numpy-discussion] Move scipy.org docs to Github? Message-ID: Hi, The scipy.org site is down at the moment, and has been for more than 36 hours: https://github.com/numpy/numpy/issues/8779#issue-213781439 This has happened before: https://github.com/scipy/scipy.org/issues/187#issue-186426408 I think it was down for about 24 hours that time. >From the number of people opening issues or commenting on the scipy.org website this time, it seems to be causing quite a bit of disruption. It seems to me that we would have a much better chances of avoiding significant down-time, if we switched to hosting the docs on github pages. What do y'all think? Cheers, Matthew From ralf.gommers at gmail.com Wed Mar 15 05:47:36 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 15 Mar 2017 22:47:36 +1300 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: Message-ID: On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett wrote: > Hi, > > The scipy.org site is down at the moment, and has been for more than 36 > hours: > > https://github.com/numpy/numpy/issues/8779#issue-213781439 > > This has happened before: > > https://github.com/scipy/scipy.org/issues/187#issue-186426408 > > I think it was down for about 24 hours that time. > > From the number of people opening issues or commenting on the > scipy.org website this time, it seems to be causing quite a bit of > disruption. > > It seems to me that we would have a much better chances of avoiding > significant down-time, if we switched to hosting the docs on github > pages. > > What do y'all think? > Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1]. Rough bandwidth estimate (using page size from http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB. Ralf [1] https://help.github.com/articles/what-is-github-pages/#usage-limits -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Wed Mar 15 06:55:47 2017 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Wed, 15 Mar 2017 11:55:47 +0100 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: Message-ID: In the meantime maybe it's a good idea to keep one of the issues open so that people can see that this is an open issue? As we close them they disappear from the issues tab on Github On Wed, Mar 15, 2017 at 10:47 AM, Ralf Gommers wrote: > > > On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett > wrote: > >> Hi, >> >> The scipy.org site is down at the moment, and has been for more than 36 >> hours: >> >> https://github.com/numpy/numpy/issues/8779#issue-213781439 >> >> This has happened before: >> >> https://github.com/scipy/scipy.org/issues/187#issue-186426408 >> >> I think it was down for about 24 hours that time. >> >> From the number of people opening issues or commenting on the >> scipy.org website this time, it seems to be causing quite a bit of >> disruption. >> >> It seems to me that we would have a much better chances of avoiding >> significant down-time, if we switched to hosting the docs on github >> pages. >> >> What do y'all think? >> > > Once the site is back up we should look at migrating to a better (hosted) > infrastructure. I suspect that Github Pages won't work, we'll exceed or be > close to exceeding both the 1 GB site size limit and the 100 GB/month > bandwidth limit [1]. > > Rough bandwidth estimate (using page size from http://scipy.github.io/ > devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per > visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 > MB in size: if only a small fraction of visitors download those, we'll be > at >100 GB. > > Ralf > > [1] https://help.github.com/articles/what-is-github-pages/#usage-limits > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Wed Mar 15 06:59:53 2017 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Wed, 15 Mar 2017 11:59:53 +0100 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: Message-ID: By the way, this is not bad at all in the absence of the actual documentation http://devdocs.io/numpy~1.12/ On Wed, Mar 15, 2017 at 11:55 AM, Ilhan Polat wrote: > In the meantime maybe it's a good idea to keep one of the issues open so > that people can see that this is an open issue? As we close them they > disappear from the issues tab on Github > > On Wed, Mar 15, 2017 at 10:47 AM, Ralf Gommers > wrote: > >> >> >> On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett >> wrote: >> >>> Hi, >>> >>> The scipy.org site is down at the moment, and has been for more than 36 >>> hours: >>> >>> https://github.com/numpy/numpy/issues/8779#issue-213781439 >>> >>> This has happened before: >>> >>> https://github.com/scipy/scipy.org/issues/187#issue-186426408 >>> >>> I think it was down for about 24 hours that time. >>> >>> From the number of people opening issues or commenting on the >>> scipy.org website this time, it seems to be causing quite a bit of >>> disruption. >>> >>> It seems to me that we would have a much better chances of avoiding >>> significant down-time, if we switched to hosting the docs on github >>> pages. >>> >>> What do y'all think? >>> >> >> Once the site is back up we should look at migrating to a better (hosted) >> infrastructure. I suspect that Github Pages won't work, we'll exceed or be >> close to exceeding both the 1 GB site size limit and the 100 GB/month >> bandwidth limit [1]. >> >> Rough bandwidth estimate (using page size from >> http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per >> month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to >> that pdf docs, which are ~20 MB in size: if only a small fraction of >> visitors download those, we'll be at >100 GB. >> >> Ralf >> >> [1] https://help.github.com/articles/what-is-github-pages/#usage-limits >> >> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Mar 15 08:24:41 2017 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 15 Mar 2017 05:24:41 -0700 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: Message-ID: On Mar 15, 2017 02:47, "Ralf Gommers" wrote: On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett wrote: > Hi, > > The scipy.org site is down at the moment, and has been for more than 36 > hours: > > https://github.com/numpy/numpy/issues/8779#issue-213781439 > > This has happened before: > > https://github.com/scipy/scipy.org/issues/187#issue-186426408 > > I think it was down for about 24 hours that time. > > From the number of people opening issues or commenting on the > scipy.org website this time, it seems to be causing quite a bit of > disruption. > > It seems to me that we would have a much better chances of avoiding > significant down-time, if we switched to hosting the docs on github > pages. > > What do y'all think? > Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1]. Rough bandwidth estimate (using page size from http://scipy.github.io/ devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB. No matter where we go, we can likely reduce the endpoint bandwidth requirements substantially by putting something like cloudflare's free tier in front. That doesn't help for the actual disk size though of course... -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryanv at continuum.io Wed Mar 15 09:10:30 2017 From: bryanv at continuum.io (Bryan Van de ven) Date: Wed, 15 Mar 2017 08:10:30 -0500 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: Message-ID: Doesn't help all the CI builds that are failing because they utilize intersphinx to link to the official docs, unfortunately. Bryan > On Mar 15, 2017, at 05:59, Ilhan Polat wrote: > > By the way, this is not bad at all in the absence of the actual documentation http://devdocs.io/numpy~1.12/ > > On Wed, Mar 15, 2017 at 11:55 AM, Ilhan Polat wrote: > In the meantime maybe it's a good idea to keep one of the issues open so that people can see that this is an open issue? As we close them they disappear from the issues tab on Github > > On Wed, Mar 15, 2017 at 10:47 AM, Ralf Gommers wrote: > > > On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett wrote: > Hi, > > The scipy.org site is down at the moment, and has been for more than 36 hours: > > https://github.com/numpy/numpy/issues/8779#issue-213781439 > > This has happened before: > > https://github.com/scipy/scipy.org/issues/187#issue-186426408 > > I think it was down for about 24 hours that time. > > From the number of people opening issues or commenting on the > scipy.org website this time, it seems to be causing quite a bit of > disruption. > > It seems to me that we would have a much better chances of avoiding > significant down-time, if we switched to hosting the docs on github > pages. > > What do y'all think? > > Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1]. > > Rough bandwidth estimate (using page size from http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB. > > Ralf > > [1] https://help.github.com/articles/what-is-github-pages/#usage-limits > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion From bryanv at continuum.io Wed Mar 15 09:16:52 2017 From: bryanv at continuum.io (Bryan Van de ven) Date: Wed, 15 Mar 2017 08:16:52 -0500 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: Message-ID: <22FB7ACD-7566-4B6E-8118-6E2C6D8E8CCA@continuum.io> NumPy is a NumFocus fiscally sponsored project, perhaps they can help with the costs of different/better hosting. Bryan > On Mar 15, 2017, at 07:24, Nathaniel Smith wrote: > > On Mar 15, 2017 02:47, "Ralf Gommers" wrote: > > > On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett wrote: > Hi, > > The scipy.org site is down at the moment, and has been for more than 36 hours: > > https://github.com/numpy/numpy/issues/8779#issue-213781439 > > This has happened before: > > https://github.com/scipy/scipy.org/issues/187#issue-186426408 > > I think it was down for about 24 hours that time. > > From the number of people opening issues or commenting on the > scipy.org website this time, it seems to be causing quite a bit of > disruption. > > It seems to me that we would have a much better chances of avoiding > significant down-time, if we switched to hosting the docs on github > pages. > > What do y'all think? > > Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1]. > > Rough bandwidth estimate (using page size from http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB. > > No matter where we go, we can likely reduce the endpoint bandwidth requirements substantially by putting something like cloudflare's free tier in front. That doesn't help for the actual disk size though of course... > > -n > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion From ben.v.root at gmail.com Wed Mar 15 11:07:02 2017 From: ben.v.root at gmail.com (Benjamin Root) Date: Wed, 15 Mar 2017 11:07:02 -0400 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: <22FB7ACD-7566-4B6E-8118-6E2C6D8E8CCA@continuum.io> References: <22FB7ACD-7566-4B6E-8118-6E2C6D8E8CCA@continuum.io> Message-ID: How is it that scipy.org's bandwidth usage is that much greater than matplotlib.org's? We are quite image-heavy, but we haven't hit any bandwidth limits that I am aware of. Ben Root On Wed, Mar 15, 2017 at 9:16 AM, Bryan Van de ven wrote: > NumPy is a NumFocus fiscally sponsored project, perhaps they can help with > the costs of different/better hosting. > > Bryan > > > > On Mar 15, 2017, at 07:24, Nathaniel Smith wrote: > > > > On Mar 15, 2017 02:47, "Ralf Gommers" wrote: > > > > > > On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett > wrote: > > Hi, > > > > The scipy.org site is down at the moment, and has been for more than 36 > hours: > > > > https://github.com/numpy/numpy/issues/8779#issue-213781439 > > > > This has happened before: > > > > https://github.com/scipy/scipy.org/issues/187#issue-186426408 > > > > I think it was down for about 24 hours that time. > > > > From the number of people opening issues or commenting on the > > scipy.org website this time, it seems to be causing quite a bit of > > disruption. > > > > It seems to me that we would have a much better chances of avoiding > > significant down-time, if we switched to hosting the docs on github > > pages. > > > > What do y'all think? > > > > Once the site is back up we should look at migrating to a better > (hosted) infrastructure. I suspect that Github Pages won't work, we'll > exceed or be close to exceeding both the 1 GB site size limit and the 100 > GB/month bandwidth limit [1]. > > > > Rough bandwidth estimate (using page size from http://scipy.github.io/ > devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per > visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 > MB in size: if only a small fraction of visitors download those, we'll be > at >100 GB. > > > > No matter where we go, we can likely reduce the endpoint bandwidth > requirements substantially by putting something like cloudflare's free tier > in front. That doesn't help for the actual disk size though of course... > > > > -n > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Mar 15 11:33:54 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 15 Mar 2017 08:33:54 -0700 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: Message-ID: Hi, On Wed, Mar 15, 2017 at 2:47 AM, Ralf Gommers wrote: > > > On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett > wrote: >> >> Hi, >> >> The scipy.org site is down at the moment, and has been for more than 36 >> hours: >> >> https://github.com/numpy/numpy/issues/8779#issue-213781439 >> >> This has happened before: >> >> https://github.com/scipy/scipy.org/issues/187#issue-186426408 >> >> I think it was down for about 24 hours that time. >> >> From the number of people opening issues or commenting on the >> scipy.org website this time, it seems to be causing quite a bit of >> disruption. >> >> It seems to me that we would have a much better chances of avoiding >> significant down-time, if we switched to hosting the docs on github >> pages. >> >> What do y'all think? > > > Once the site is back up we should look at migrating to a better (hosted) > infrastructure. I suspect that Github Pages won't work, we'll exceed or be > close to exceeding both the 1 GB site size limit and the 100 GB/month > bandwidth limit [1]. > > Rough bandwidth estimate (using page size from > http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per > month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that > pdf docs, which are ~20 MB in size: if only a small fraction of visitors > download those, we'll be at >100 GB. Maybe we could host the PDF docs somewhere else? I wonder if Github would consider allowing us to go a bit over if necessary? Cheers, Matthew From davidmenhur at gmail.com Wed Mar 15 11:36:56 2017 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Wed, 15 Mar 2017 16:36:56 +0100 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: Message-ID: What about readthedocs? I haven't seen any explicit limit in traffic. On 15 March 2017 at 16:33, Matthew Brett wrote: > Hi, > > On Wed, Mar 15, 2017 at 2:47 AM, Ralf Gommers wrote: >> >> >> On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett >> wrote: >>> >>> Hi, >>> >>> The scipy.org site is down at the moment, and has been for more than 36 >>> hours: >>> >>> https://github.com/numpy/numpy/issues/8779#issue-213781439 >>> >>> This has happened before: >>> >>> https://github.com/scipy/scipy.org/issues/187#issue-186426408 >>> >>> I think it was down for about 24 hours that time. >>> >>> From the number of people opening issues or commenting on the >>> scipy.org website this time, it seems to be causing quite a bit of >>> disruption. >>> >>> It seems to me that we would have a much better chances of avoiding >>> significant down-time, if we switched to hosting the docs on github >>> pages. >>> >>> What do y'all think? >> >> >> Once the site is back up we should look at migrating to a better (hosted) >> infrastructure. I suspect that Github Pages won't work, we'll exceed or be >> close to exceeding both the 1 GB site size limit and the 100 GB/month >> bandwidth limit [1]. >> >> Rough bandwidth estimate (using page size from >> http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per >> month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that >> pdf docs, which are ~20 MB in size: if only a small fraction of visitors >> download those, we'll be at >100 GB. > > Maybe we could host the PDF docs somewhere else? I wonder if Github > would consider allowing us to go a bit over if necessary? > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion From m.h.vankerkwijk at gmail.com Wed Mar 15 12:11:09 2017 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Wed, 15 Mar 2017 12:11:09 -0400 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: Message-ID: Astropy uses readthedocs quite happily (auto-updates on merges to master too). -- Marten From pav at iki.fi Wed Mar 15 15:28:18 2017 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Mar 2017 19:28:18 +0000 (UTC) Subject: [Numpy-discussion] Move scipy.org docs to Github? References: Message-ID: Wed, 15 Mar 2017 12:11:09 -0400, Marten van Kerkwijk kirjoitti: > Astropy uses readthedocs quite happily (auto-updates on merges to master > too). AFAIK, scipy cannot be built on readthedocs. From njs at pobox.com Wed Mar 15 16:02:57 2017 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 15 Mar 2017 13:02:57 -0700 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: Message-ID: On Mar 15, 2017 12:28 PM, "Pauli Virtanen" wrote: Wed, 15 Mar 2017 12:11:09 -0400, Marten van Kerkwijk kirjoitti: > Astropy uses readthedocs quite happily (auto-updates on merges to master > too). AFAIK, scipy cannot be built on readthedocs. Another issue is that switching to rtd would (I think?) force us into their URL structure and break all incoming links, which would be really bad. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Mar 15 17:56:52 2017 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 15 Mar 2017 14:56:52 -0700 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: <22FB7ACD-7566-4B6E-8118-6E2C6D8E8CCA@continuum.io> References: <22FB7ACD-7566-4B6E-8118-6E2C6D8E8CCA@continuum.io> Message-ID: On Wed, Mar 15, 2017 at 6:16 AM, Bryan Van de ven wrote: > NumPy is a NumFocus fiscally sponsored project, perhaps they can help with the costs of different/better hosting. Enthought already provides hosting and operations support (thanks!) ? the problem is that it doesn't make sense to have a full-time ops person just for numpy, but if we're taking a tiny slice of someone's time then occasional outages are going to happen. The advantage of something like github pages is that it's big enough that it *does* have dedicated ops support. As long as we can fit under the 1 gig size limit then GH pages seems like the best option so far... it's reliable, widely understood, and all of the limits besides the 1 gig size are soft limits where they say they'll work with us to figure things out. -n -- Nathaniel J. Smith -- https://vorpus.org From toddrjen at gmail.com Wed Mar 15 18:07:20 2017 From: toddrjen at gmail.com (Todd) Date: Wed, 15 Mar 2017 18:07:20 -0400 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: Message-ID: On Mar 15, 2017 05:47, "Ralf Gommers" wrote: On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett wrote: > Hi, > > The scipy.org site is down at the moment, and has been for more than 36 > hours: > > https://github.com/numpy/numpy/issues/8779#issue-213781439 > > This has happened before: > > https://github.com/scipy/scipy.org/issues/187#issue-186426408 > > I think it was down for about 24 hours that time. > > From the number of people opening issues or commenting on the > scipy.org website this time, it seems to be causing quite a bit of > disruption. > > It seems to me that we would have a much better chances of avoiding > significant down-time, if we switched to hosting the docs on github > pages. > > What do y'all think? > Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1]. Rough bandwidth estimate (using page size from http://scipy.github.io/ devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB. Ralf [1] https://help.github.com/articles/what-is-github-pages/#usage-limits Would it be possible to include the PDF and zipped HTML as sources in GitHub releases and (if desired) link to those on the documentation page? That way only the individual HTML pages would need to be stored in GitHub pages, reducing both the size and bandwidth. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Mar 15 19:20:03 2017 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Mar 2017 23:20:03 +0000 (UTC) Subject: [Numpy-discussion] Move scipy.org docs to Github? References: <22FB7ACD-7566-4B6E-8118-6E2C6D8E8CCA@continuum.io> Message-ID: Wed, 15 Mar 2017 14:56:52 -0700, Nathaniel Smith kirjoitti: [clip] > As long as we can fit under the 1 gig size limit then GH pages seems > like the best option so far... it's reliable, widely understood, and all > of the limits besides the 1 gig size are soft limits where they say > they'll work with us to figure things out. The Scipy html docs weigh 60M apiece and numpy is 35M, so it can be done if a limited number of releases are kept, and the rest and the auxiliary files are put as release downloads. Otherwise there's probably no problem as you can stick a CDN in front if it's too heavy otherwise. Sounds sensible? Certainly it's the lowest-effort approach, and would simplify management of S3/etc origin site access permissions. Pauli From matthew.brett at gmail.com Thu Mar 16 00:14:24 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 15 Mar 2017 21:14:24 -0700 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: <22FB7ACD-7566-4B6E-8118-6E2C6D8E8CCA@continuum.io> Message-ID: Hi, On Wed, Mar 15, 2017 at 4:20 PM, Pauli Virtanen wrote: > Wed, 15 Mar 2017 14:56:52 -0700, Nathaniel Smith kirjoitti: > [clip] >> As long as we can fit under the 1 gig size limit then GH pages seems >> like the best option so far... it's reliable, widely understood, and all >> of the limits besides the 1 gig size are soft limits where they say >> they'll work with us to figure things out. > > The Scipy html docs weigh 60M apiece and numpy is 35M, so it can be done > if a limited number of releases are kept, and the rest and the auxiliary > files are put as release downloads. > > Otherwise there's probably no problem as you can stick a CDN in front if > it's too heavy otherwise. > > Sounds sensible? Certainly it's the lowest-effort approach, and would > simplify management of S3/etc origin site access permissions. Sounds very sensible to me, Cheers, Matthew From dpinte at enthought.com Thu Mar 16 03:15:08 2017 From: dpinte at enthought.com (Didrik Pinte) Date: Thu, 16 Mar 2017 08:15:08 +0100 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: <22FB7ACD-7566-4B6E-8118-6E2C6D8E8CCA@continuum.io> Message-ID: On 15 March 2017 at 22:56, Nathaniel Smith wrote: > On Wed, Mar 15, 2017 at 6:16 AM, Bryan Van de ven > wrote: > > NumPy is a NumFocus fiscally sponsored project, perhaps they can help > with the costs of different/better hosting. > > Enthought already provides hosting and operations support (thanks!) ? > the problem is that it doesn't make sense to have a full-time ops > person just for numpy, but if we're taking a tiny slice of someone's > time then occasional outages are going to happen. > The key issue is this specific case is that the entire system was totally undocumented. If it had been, the outage would have lasted much less than an hour! The advantage of something like github pages is that it's big enough > that it *does* have dedicated ops support. > Agreed. One issue is that we are working with a lot of legacy. Github will more than likely be a great solution to host static web pages but the evaluation for the shift needs to get into all the funky legacy redirects/rewrites we have in place, etc. This is probably not a real issue for docs.scipy.org but would be for other services. > As long as we can fit under the 1 gig size limit then GH pages seems > like the best option so far... it's reliable, widely understood, and > all of the limits besides the 1 gig size are soft limits where they > say they'll work with us to figure things out. > Another option would be to just host the content under S3 with Cloudfront. It will also be pretty simple as a setup, scale nicely and won't have much restrictions on sizing. -- Didrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Mar 16 17:08:31 2017 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 16 Mar 2017 21:08:31 +0000 (UTC) Subject: [Numpy-discussion] Move scipy.org docs to Github? References: <22FB7ACD-7566-4B6E-8118-6E2C6D8E8CCA@continuum.io> Message-ID: Thu, 16 Mar 2017 08:15:08 +0100, Didrik Pinte kirjoitti: >> The advantage of something like github pages is that it's big enough >> that it *does* have dedicated ops support. > > Agreed. One issue is that we are working with a lot of legacy. Github > will more than likely be a great solution to host static web pages but > the evaluation for the shift needs to get into all the funky legacy > redirects/rewrites we have in place, etc. This is probably not a real > issue for docs.scipy.org but would be for other services. IIRC, there's not that many of them, so in principle it could be possible to cobble them with redirects. >> As long as we can fit under the 1 gig size limit then GH pages seems >> like the best option so far... it's reliable, widely understood, and >> all of the limits besides the 1 gig size are soft limits where they say >> they'll work with us to figure things out. > > Another option would be to just host the content under S3 with > Cloudfront. > It will also be pretty simple as a setup, scale nicely and won't have > much restrictions on sizing. Some minor-ish disadvantages of this are that it brings a new set of credentials to manage, it will be somewhat less transparent, and the tooling will be less familiar to people (eg release managers) who have to deal with it. From rmcgibbo at gmail.com Thu Mar 16 18:18:47 2017 From: rmcgibbo at gmail.com (Robert T. McGibbon) Date: Thu, 16 Mar 2017 18:18:47 -0400 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: <22FB7ACD-7566-4B6E-8118-6E2C6D8E8CCA@continuum.io> Message-ID: I have always put my docs on Amazon S3 (examples: http://mdtraj.org/1.8.0/ , .http://msmbuilder.org/3.7.0/) For static webpages, you can't beat the cost, and there's a lot of tooling in the wild for uploading pages to S3. It might be an option to consider. -Robert On Thu, Mar 16, 2017 at 5:08 PM, Pauli Virtanen wrote: > Thu, 16 Mar 2017 08:15:08 +0100, Didrik Pinte kirjoitti: > >> The advantage of something like github pages is that it's big enough > >> that it *does* have dedicated ops support. > > > > Agreed. One issue is that we are working with a lot of legacy. Github > > will more than likely be a great solution to host static web pages but > > the evaluation for the shift needs to get into all the funky legacy > > redirects/rewrites we have in place, etc. This is probably not a real > > issue for docs.scipy.org but would be for other services. > > IIRC, there's not that many of them, so in principle it could be possible > to cobble them with redirects. > > >> As long as we can fit under the 1 gig size limit then GH pages seems > >> like the best option so far... it's reliable, widely understood, and > >> all of the limits besides the 1 gig size are soft limits where they say > >> they'll work with us to figure things out. > > > > Another option would be to just host the content under S3 with > > Cloudfront. > > It will also be pretty simple as a setup, scale nicely and won't have > > much restrictions on sizing. > > Some minor-ish disadvantages of this are that it brings a new set of > credentials to manage, it will be somewhat less transparent, and the > tooling will be less familiar to people (eg release managers) who have to > deal with it. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- -Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From dpinte at enthought.com Fri Mar 17 06:49:57 2017 From: dpinte at enthought.com (Didrik Pinte) Date: Fri, 17 Mar 2017 11:49:57 +0100 Subject: [Numpy-discussion] Move scipy.org docs to Github? In-Reply-To: References: <22FB7ACD-7566-4B6E-8118-6E2C6D8E8CCA@continuum.io> Message-ID: Quick update: - the current static content for docs.scipy.org is about 2.7Gb. Some clean can happen but probably not going below 1Gb. - www.scipy.org is really small. -- Didrik On 16 March 2017 at 23:18, Robert T. McGibbon wrote: > I have always put my docs on Amazon S3 (examples: http://mdtraj.org/1.8.0/ > , .http://msmbuilder.org/3.7.0/) For static webpages, you can't beat the > cost, and there's a lot of tooling in the wild for uploading pages to S3. > > It might be an option to consider. > > -Robert > > On Thu, Mar 16, 2017 at 5:08 PM, Pauli Virtanen wrote: > >> Thu, 16 Mar 2017 08:15:08 +0100, Didrik Pinte kirjoitti: >> >> The advantage of something like github pages is that it's big enough >> >> that it *does* have dedicated ops support. >> > >> > Agreed. One issue is that we are working with a lot of legacy. Github >> > will more than likely be a great solution to host static web pages but >> > the evaluation for the shift needs to get into all the funky legacy >> > redirects/rewrites we have in place, etc. This is probably not a real >> > issue for docs.scipy.org but would be for other services. >> >> IIRC, there's not that many of them, so in principle it could be possible >> to cobble them with redirects. >> >> >> As long as we can fit under the 1 gig size limit then GH pages seems >> >> like the best option so far... it's reliable, widely understood, and >> >> all of the limits besides the 1 gig size are soft limits where they say >> >> they'll work with us to figure things out. >> > >> > Another option would be to just host the content under S3 with >> > Cloudfront. >> > It will also be pretty simple as a setup, scale nicely and won't have >> > much restrictions on sizing. >> >> Some minor-ish disadvantages of this are that it brings a new set of >> credentials to manage, it will be somewhat less transparent, and the >> tooling will be less familiar to people (eg release managers) who have to >> deal with it. >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > > -- > -Robert > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Didrik Pinte +32 475 665 668 CTO +44 1223 969515 Enthought Inc. dpinte at enthought.com Scientific Computing Solutions http://www.enthought.com The information contained in this message is Enthought confidential & not to be disseminated to outside parties without explicit prior approval from sender. This message is intended solely for the addressee(s), If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Fri Mar 17 07:37:30 2017 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Fri, 17 Mar 2017 12:37:30 +0100 Subject: [Numpy-discussion] PyData Barcelona this May In-Reply-To: <1489087749.4278.8.camel@sipsolutions.net> References: <1489087749.4278.8.camel@sipsolutions.net> Message-ID: Last night I gave a short talk to the PyData Z?rich meetup on Julian's temporary elision PR, and Pauli's overlapping memory one. My learnings from that experiment are: - there is no way to talk about both things in a 30 minute talk: I barely scraped the surface and ended up needing 25 minutes. - many people that use numpy in their daily work don't know what strides are, this was a BIG surprise for me. Based on that experience, I was thinking that maybe a good topic for a workshop would be NumPy's memory model: views, reshaping, strides, some hints of buffering in the iterator... And Julian's temporary work lends itself to a very nice talk, more on Python internals than on NumPy, but it's a very cool subject nonetheless. So my thinking is that I am going to propose those two, as a workshop and a talk. Thoughts? Jaime On Thu, Mar 9, 2017 at 8:29 PM, Sebastian Berg wrote: > On Thu, 2017-03-09 at 15:45 +0100, Jaime Fern?ndez del R?o wrote: > > There will be a PyData conference in Barcelona this May: > > > > http://pydata.org/barcelona2017/ > > > > I am planning on attending, and was thinking of maybe proposing to > > organize a numpy-themed workshop or tutorial. > > > > My personal inclination would be to look at some advanced topic that > > I know well, like writing gufuncs in Cython, but wouldn't mind doing > > a more run of the mill thing. Anyone has any thoughts or experiences > > on what has worked well in similar situations? Any specific topic you > > always wanted to attend a workshop on, but were afraid to ask? > > > > Alternatively, or on top of the workshop, I could propose to do a > > talk: talking last year at PyData Madrid about the new indexing was a > > lot of fun! Thing is, I have been quite disconnected from the project > > this past year, and can't really think of any worthwhile topic. Is > > there any message that we as a project would like to get out to the > > larger community? > > > > Francesc already pointed out the temporary optimization. From what I > remember, my personal highlight would probably be Pauli's work on the > memory overlap detection. Though both are rather passive improvements I > guess (you don't really have to learn them to use them), its very cool! > And if its about highlighting new stuff, these can probably easily fill > a talk. > > > And if you are planning on attending, please give me a shout. > > > > Barcelona :). Maybe I should think about it, but probably not. > > > > Thanks, > > > > Jaime > > > > -- > > (\__/) > > ( O.o) > > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus > > planes de dominaci?n mundial. > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at gmail.com Fri Mar 17 07:44:06 2017 From: faltet at gmail.com (Francesc Alted) Date: Fri, 17 Mar 2017 12:44:06 +0100 Subject: [Numpy-discussion] PyData Barcelona this May In-Reply-To: References: <1489087749.4278.8.camel@sipsolutions.net> Message-ID: 2017-03-17 12:37 GMT+01:00 Jaime Fern?ndez del R?o : > Last night I gave a short talk to the PyData Z?rich meetup on Julian's > temporary elision PR, and Pauli's overlapping memory one. My learnings from > that experiment are: > > - there is no way to talk about both things in a 30 minute talk: I > barely scraped the surface and ended up needing 25 minutes. > - many people that use numpy in their daily work don't know what > strides are, this was a BIG surprise for me. > > Based on that experience, I was thinking that maybe a good topic for a > workshop would be NumPy's memory model: views, reshaping, strides, some > hints of buffering in the iterator... > ?Yeah, I think that workshop would represent a very valuable insight to many people using NumPy?. > > And Julian's temporary work lends itself to a very nice talk, more on > Python internals than on NumPy, but it's a very cool subject nonetheless. > > So my thinking is that I am going to propose those two, as a workshop and > a talk. Thoughts? > ?+1? > > Jaime > > On Thu, Mar 9, 2017 at 8:29 PM, Sebastian Berg > wrote: > >> On Thu, 2017-03-09 at 15:45 +0100, Jaime Fern?ndez del R?o wrote: >> > There will be a PyData conference in Barcelona this May: >> > >> > http://pydata.org/barcelona2017/ >> > >> > I am planning on attending, and was thinking of maybe proposing to >> > organize a numpy-themed workshop or tutorial. >> > >> > My personal inclination would be to look at some advanced topic that >> > I know well, like writing gufuncs in Cython, but wouldn't mind doing >> > a more run of the mill thing. Anyone has any thoughts or experiences >> > on what has worked well in similar situations? Any specific topic you >> > always wanted to attend a workshop on, but were afraid to ask? >> > >> > Alternatively, or on top of the workshop, I could propose to do a >> > talk: talking last year at PyData Madrid about the new indexing was a >> > lot of fun! Thing is, I have been quite disconnected from the project >> > this past year, and can't really think of any worthwhile topic. Is >> > there any message that we as a project would like to get out to the >> > larger community? >> > >> >> Francesc already pointed out the temporary optimization. From what I >> remember, my personal highlight would probably be Pauli's work on the >> memory overlap detection. Though both are rather passive improvements I >> guess (you don't really have to learn them to use them), its very cool! >> And if its about highlighting new stuff, these can probably easily fill >> a talk. >> >> > And if you are planning on attending, please give me a shout. >> > >> >> Barcelona :). Maybe I should think about it, but probably not. >> >> >> > Thanks, >> > >> > Jaime >> > >> > -- >> > (\__/) >> > ( O.o) >> > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus >> > planes de dominaci?n mundial. >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes > de dominaci?n mundial. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.corlay at gmail.com Fri Mar 17 09:18:30 2017 From: sylvain.corlay at gmail.com (Sylvain Corlay) Date: Fri, 17 Mar 2017 14:18:30 +0100 Subject: [Numpy-discussion] ANN: xtensor 0.7.1 numpy-style syntax in C++ with bindings to numpy arrays Message-ID: Hi All, On behalf of the xtensor development team, I am pleased to announce the releases of - xtensor 0.7.1 https://github.com/QuantStack/xtensor/ - xtensor-python 0.6.0 https://github.com/QuantStack/xtensor-python/ *What is xtensor?* xtensor is a C++ library meant for numerical analysis with multi-dimensional array expressions. xtensor provides - an extensible expression system enabling* lazy broadcasting.* - an API following the idioms of the *C++ standard library*. - an increasing support of numpy features which you can see in the *NumPy to xtensor cheat sheet*. - tools to manipulate array expressions and build upon xtensor. - *numpy bindings* enabling the inplace use of numpy arrays as xtensor expressions in C++ extensions. *What is new in this release?* In this release, we have added the reducers functionality and the real and imaginary views for complex arrays. We have increased the performance of xtensor universal functions. *Where can I learn more about xtensor?* Check out the extensive documentation: http://xtensor.readthedocs.io/en/latest/ Or the numpy to xtensor cheat sheet: http://xtensor.readthedocs.io/en/latest/numpy.html Or join us in the chat room: https://gitter.im/QuantStack/Lobby Thanks! Sylvain >From NumPy to xtensor Containers Two container types are provided. xarray (dynamic number of dimensions) and xtensor (static number of dimensions). Python 3 - numpyC++ 14 - xtensor np.array([[3, 4], [5, 6]]) xt::xarray({{3, 4}, {5, 6}}) xt::xtensor({{3, 4}, {5, 6}}) arr.reshape([3, 4]) arr.reshape{{3, 4}) Initializers Lazy helper functions return tensor expressions. Return types don?t hold any value and are evaluated upon access or assignment. They can be assigned to a container or directly used in expressions. Python 3 - numpyC++ 14 - xtensor np.linspace(1.0, 10.0, 100) xt::linspace(1.0, 10.0, 100) np.logspace(2.0, 3.0, 4) xt::logspace(2.0, 3.0, 4) np.arange(3, 7) xt::arange(3, 7) np.eye(4) xt::eye(4) np.zeros([3, 4]) xt::zeros({3, 4}) np.ones([3, 4]) xt::ones({3, 4}) np.meshgrid(x0, x1, x2, indexing='ij') xt::meshgrid(x0, x1, x2) xtensor?s meshgrid implementation corresponds to numpy?s 'ij' indexing order. Broadcasting xtensor offers lazy numpy-style broadcasting, and universal functions. Unlike numpy, no copy or temporary variables are created. Python 3 - numpyC++ 14 - xtensor a[:, np.newaxis] a[:5, 1:] a[5:1:-1, :] xt::view(a, xt::all(), xt::newaxis()) xt::view(a, xt::range(_, 5), xt::range(1, _)) xt::view(a, xt::range(5, 1, -1), xt::all()) np.broadcast(a, [4, 5, 7]) xt::broadcast(a, {4, 5, 7}) np.vectorize(f) xt::vectorize(f) a[a > 5] xt::filter(a, a > 5) a[[0, 1], [0, 0]] xt::index_view(a, {{0, 0}, {1, 0}}) Random The random module provides simple ways to create random tensor expressions, lazily. Python 3 - numpyC++ 14 - xtensor np.random.randn(10, 10) xt::random::randn({10, 10}) np.random.randint(10, 10) xt::random::randint({10, 10}) np.random.rand(3, 4) xt::random::rand({3, 4}) Concatenation Concatenating expressions does not allocate memory, it returns a tensor expression holding closures on the specified arguments. Python 3 - numpyC++ 14 - xtensor np.stack([a, b, c], axis=1) xt::stack(xtuple(a, b, c), 1) np.concatenate([a, b, c], axis=1) xt::concatenate(xtuple(a, b, c), 1) Diagonal, triangular and flip In the same spirit as concatenation, the following operations do not allocate any memory and do not modify the underlying xexpression. Python 3 - numpyC++ 14 - xtensor np.diag(a) xt::diag(a) np.diagonal(a) xt::diagonal(a) np.triu(a) xt::triu(a) np.tril(a, k=1) xt::tril(a, 1) np.flip(a, axis=3) xt::flip(a, 3) np.flipud(a) xt::flip(a, 0) np.fliplr(a) xt::flip(a, 1) Iteration xtensor follows the idioms of the C++ STL providing iterator pairs to iterate on arrays in different fashions. Python 3 - numpyC++ 14 - xtensor for x in np.nditer(a): for(auto it=a.xbegin(); it!=a.xend(); ++it) Iterating with a prescribed broadcasting shape for(auto it=a.xbegin({3, 4}); it!=a.xend({3, 4}); ++it) Logical Logical universal functions are truly lazy. xt::where(condition, a, b) does not evaluate a where condition is falsy, and it does not evaluate b where condition is truthy. Python 3 - numpyC++ 14 - xtensor np.where(a > 5, a, b) xt::where(a > 5, a, b) np.where(a > 5) xt::where(a > 5) np.any(a) xt::any(a) np.all(a) xt::all(a) np.logical_and(a, b) a && b np.logical_or(a, b) a || b Comparisons Python 3 - numpyC++ 14 - xtensor np.equal(a, b) xt::equal(a, b) np.not_equal(a) xt::not_equal(a) np.nonzero(a) xt::nonzero(a) Complex numbers Functions xt::real and xt::imag respectively return views on the real and imaginary part of a complex expression. The returned value is an expression holding a closure on the passed argument. Python 3 - numpyC++ 14 - xtensor np.real(a) xt::real(a) np.imag(a) xt::imag(a) - The constness and value category (rvalue / lvalue) of real(a) is the same as that of a. Hence, if a is a non-const lvalue, real(a) is an non-const lvalue reference, to which one can assign a real expression. - If a has complex values, the same holds for imag(a). The constness and value category ofimag(a) is the same as that of a. - If a has real values, imag(a) returns zeros(a.shape()). Reducers Reducers accumulate values of tensor expressions along specified axes. When no axis is specified, values are accumulated along all axes. Reducers are lazy, meaning that returned expressons don?t hold any values and are computed upon access or assigmnent. Python 3 - numpyC++ 14 - xtensor np.sum(a, axis=[0, 1]) xt::sum(a, {0, 1}) np.sum(a) xt::sum(a) np.prod(a, axis=1) xt::prod(a, {1}) np.prod(a) xt::prod(a) np.mean(a, axis=1) xt::mean(a, {1}) np.mean(a) xt::mean(a) More generally, one can use the xt::reduce(function, input, axes) which allows the specification of an arbitrary binary function for the reduction. The binary function must be cummutative and associative up to rounding errors. Mathematical functions xtensor universal functions are provided for a large set number of mathematical functions. *Basic functions:* Python 3 - numpyC++ 14 - xtensor np.isnan(a) xt::isnan(a) np.absolute(a) xt::abs(a) np.sign(a) xt::sign(a) np.remainder(a, b) xt::remainder(a, b) np.clip(a, min, max) xt::clip(a, min, max) xt::fma(a, b, c) *Exponential functions:* Python 3 - numpyC++ 14 - xtensor np.exp(a) xt::exp(a) np.expm1(a) xt::expm1(a) np.log(a) xt::log(a) np.log1p(a) xt::log1p(a) *Power functions:* Python 3 - numpyC++ 14 - xtensor np.power(a, p) xt::pow(a, b) np.sqrt(a) xt::sqrt(a) np.cbrt(a) xt::cbrt(a) *Trigonometric functions:* Python 3 - numpyC++ 14 - xtensor np.sin(a) xt::sin(a) np.cos(a) xt::cos(a) np.tan(a) xt::tan(a) *Hyperbolic functions:* Python 3 - numpyC++ 14 - xtensor np.sinh(a) xt::sinh(a) np.cosh(a) xt::cosh(a) np.tang(a) xt::tanh(a) *Error and gamma functions:* Python 3 - numpyC++ 14 - xtensor scipy.special.erf(a) xt::erf(a) scipy.special.gamma(a) xt::tgamma(a) scipy.special.gammaln(a) xt::lgamma(a) -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Mar 17 10:23:44 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 17 Mar 2017 08:23:44 -0600 Subject: [Numpy-discussion] ANN: xtensor 0.7.1 numpy-style syntax in C++ with bindings to numpy arrays In-Reply-To: References: Message-ID: On Fri, Mar 17, 2017 at 7:18 AM, Sylvain Corlay wrote: > Hi All, > > On behalf of the xtensor development team, I am pleased to announce the > releases of > > - xtensor 0.7.1 https://github.com/QuantStack/xtensor/ > - xtensor-python 0.6.0 https://github.com/QuantStack/xtensor-python/ > > > That's cool stuff! Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Mar 17 15:41:42 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 17 Mar 2017 12:41:42 -0700 Subject: [Numpy-discussion] PyData Barcelona this May In-Reply-To: References: <1489087749.4278.8.camel@sipsolutions.net> Message-ID: On Fri, Mar 17, 2017 at 4:37 AM, Jaime Fern?ndez del R?o < jaime.frio at gmail.com> wrote: > > - many people that use numpy in their daily work don't know what > strides are, this was a BIG surprise for me. > > Based on that experience, I was thinking that maybe a good topic for a > workshop would be NumPy's memory model: views, reshaping, strides, some > hints of buffering in the iterator... > I think this is a great idea. In fact, when I do an intro to numpy, I spend a bit of time on those issues, 'cause I think it's key to "Getting" numpy, and not something that people end up learning on their own from tutorials, etc. However, in my case, I try to jam it into a low-level intro, and I think that fails :-( So doing it on it's own with the assumption that participant already know the basics of the high level python interface is a great idea. Maybe a "advanced" numpy tutorial for SciPy 2017 in Austin also??? Here is my last talk -- maybe it'll be helpful. http://uwpce-pythoncert.github.io/SystemDevelopment/scipy.html#scipy the strides stuff is covered in a notebook here: https://github.com/UWPCE-PythonCert/SystemDevelopment/blob/master/Examples/numpy/stride_tricks.ipynb other notebooks here: https://github.com/UWPCE-PythonCert/SystemDevelopment/tree/master/Examples/numpy and the source for the whole thing is here: https://github.com/UWPCE-PythonCert/SystemDevelopment/blob/master/slides_sources/source/scipy.rst All licensed under: Creative Commons Attribution-ShareAlike -- so please use anything you find useful. -CHB And Julian's temporary work lends itself to a very nice talk, more on > Python internals than on NumPy, but it's a very cool subject nonetheless. > > So my thinking is that I am going to propose those two, as a workshop and > a talk. Thoughts? > > Jaime > > On Thu, Mar 9, 2017 at 8:29 PM, Sebastian Berg > wrote: > >> On Thu, 2017-03-09 at 15:45 +0100, Jaime Fern?ndez del R?o wrote: >> > There will be a PyData conference in Barcelona this May: >> > >> > http://pydata.org/barcelona2017/ >> > >> > I am planning on attending, and was thinking of maybe proposing to >> > organize a numpy-themed workshop or tutorial. >> > >> > My personal inclination would be to look at some advanced topic that >> > I know well, like writing gufuncs in Cython, but wouldn't mind doing >> > a more run of the mill thing. Anyone has any thoughts or experiences >> > on what has worked well in similar situations? Any specific topic you >> > always wanted to attend a workshop on, but were afraid to ask? >> > >> > Alternatively, or on top of the workshop, I could propose to do a >> > talk: talking last year at PyData Madrid about the new indexing was a >> > lot of fun! Thing is, I have been quite disconnected from the project >> > this past year, and can't really think of any worthwhile topic. Is >> > there any message that we as a project would like to get out to the >> > larger community? >> > >> >> Francesc already pointed out the temporary optimization. From what I >> remember, my personal highlight would probably be Pauli's work on the >> memory overlap detection. Though both are rather passive improvements I >> guess (you don't really have to learn them to use them), its very cool! >> And if its about highlighting new stuff, these can probably easily fill >> a talk. >> >> > And if you are planning on attending, please give me a shout. >> > >> >> Barcelona :). Maybe I should think about it, but probably not. >> >> >> > Thanks, >> > >> > Jaime >> > >> > -- >> > (\__/) >> > ( O.o) >> > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus >> > planes de dominaci?n mundial. >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes > de dominaci?n mundial. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Mar 17 17:59:42 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 18 Mar 2017 10:59:42 +1300 Subject: [Numpy-discussion] PyData Barcelona this May In-Reply-To: References: <1489087749.4278.8.camel@sipsolutions.net> Message-ID: On Sat, Mar 18, 2017 at 8:41 AM, Chris Barker wrote: > On Fri, Mar 17, 2017 at 4:37 AM, Jaime Fern?ndez del R?o < > jaime.frio at gmail.com> wrote: > >> >> - many people that use numpy in their daily work don't know what >> strides are, this was a BIG surprise for me. >> >> I'm not surprised at all. To start with, the majority of users are self-taught programmers that never used something lower level than Python or Matlab. Even talking to them about memory layout presents challenges. > >> - >> >> Based on that experience, I was thinking that maybe a good topic for a >> workshop would be NumPy's memory model: views, reshaping, strides, some >> hints of buffering in the iterator... >> > This material has been used multiple times in EuroScipy tutorials and may be of use: http://www.scipy-lectures.org/advanced/advanced_numpy/index.html Ralf > I think this is a great idea. In fact, when I do an intro to numpy, I > spend a bit of time on those issues, 'cause I think it's key to "Getting" > numpy, and not something that people end up learning on their own from > tutorials, etc. However, in my case, I try to jam it into a low-level > intro, and I think that fails :-( > > So doing it on it's own with the assumption that participant already know > the basics of the high level python interface is a great idea. > > Maybe a "advanced" numpy tutorial for SciPy 2017 in Austin also??? > > Here is my last talk -- maybe it'll be helpful. > > http://uwpce-pythoncert.github.io/SystemDevelopment/scipy.html#scipy > > the strides stuff is covered in a notebook here: > > https://github.com/UWPCE-PythonCert/SystemDevelopment/ > blob/master/Examples/numpy/stride_tricks.ipynb > > other notebooks here: > > https://github.com/UWPCE-PythonCert/SystemDevelopment/ > tree/master/Examples/numpy > > and the source for the whole thing is here: > > https://github.com/UWPCE-PythonCert/SystemDevelopment/ > blob/master/slides_sources/source/scipy.rst > > > All licensed under: Creative Commons Attribution-ShareAlike -- so please > use anything you find useful. > > -CHB > > > > And Julian's temporary work lends itself to a very nice talk, more on >> Python internals than on NumPy, but it's a very cool subject nonetheless. >> >> So my thinking is that I am going to propose those two, as a workshop and >> a talk. Thoughts? >> >> Jaime >> >> On Thu, Mar 9, 2017 at 8:29 PM, Sebastian Berg < >> sebastian at sipsolutions.net> wrote: >> >>> On Thu, 2017-03-09 at 15:45 +0100, Jaime Fern?ndez del R?o wrote: >>> > There will be a PyData conference in Barcelona this May: >>> > >>> > http://pydata.org/barcelona2017/ >>> > >>> > I am planning on attending, and was thinking of maybe proposing to >>> > organize a numpy-themed workshop or tutorial. >>> > >>> > My personal inclination would be to look at some advanced topic that >>> > I know well, like writing gufuncs in Cython, but wouldn't mind doing >>> > a more run of the mill thing. Anyone has any thoughts or experiences >>> > on what has worked well in similar situations? Any specific topic you >>> > always wanted to attend a workshop on, but were afraid to ask? >>> > >>> > Alternatively, or on top of the workshop, I could propose to do a >>> > talk: talking last year at PyData Madrid about the new indexing was a >>> > lot of fun! Thing is, I have been quite disconnected from the project >>> > this past year, and can't really think of any worthwhile topic. Is >>> > there any message that we as a project would like to get out to the >>> > larger community? >>> > >>> >>> Francesc already pointed out the temporary optimization. From what I >>> remember, my personal highlight would probably be Pauli's work on the >>> memory overlap detection. Though both are rather passive improvements I >>> guess (you don't really have to learn them to use them), its very cool! >>> And if its about highlighting new stuff, these can probably easily fill >>> a talk. >>> >>> > And if you are planning on attending, please give me a shout. >>> > >>> >>> Barcelona :). Maybe I should think about it, but probably not. >>> >>> >>> > Thanks, >>> > >>> > Jaime >>> > >>> > -- >>> > (\__/) >>> > ( O.o) >>> > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus >>> > planes de dominaci?n mundial. >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussion at scipy.org >>> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> >> -- >> (\__/) >> ( O.o) >> ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes >> de dominaci?n mundial. >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Mar 18 13:57:28 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 18 Mar 2017 11:57:28 -0600 Subject: [Numpy-discussion] NumPy 1.12.1 released Message-ID: Hi All, I'm pleased to announce the release of NumPy 1.12.1. NumPy 1.12.1 supports Python 2.7 and 3.4 - 3.6 and fixes bugs and regressions found in NumPy 1.12.0. In particular, the regression in f2py constant parsing is fixed. Wheels for Linux, Windows, and OSX can be found on pypi. Archives can be downloaded from github . *Contributors* A total of 10 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Charles Harris * Eric Wieser * Greg Young * Joerg Behrmann + * John Kirkham * Julian Taylor * Marten van Kerkwijk * Matthew Brett * Shota Kawabuchi * Jean Utke + *Fixes Backported* * #8483: BUG: Fix wrong future nat warning and equiv type logic error... * #8489: BUG: Fix wrong masked median for some special cases * #8490: DOC: Place np.average in inline code * #8491: TST: Work around isfinite inconsistency on i386 * #8494: BUG: Guard against replacing constants without `'_'` spec in f2py. * #8524: BUG: Fix mean for float 16 non-array inputs for 1.12 * #8571: BUG: Fix calling python api with error set and minor leaks for... * #8602: BUG: Make iscomplexobj compatible with custom dtypes again * #8618: BUG: Fix undefined behaviour induced by bad `__array_wrap__` * #8648: BUG: Fix `MaskedArray.__setitem__` * #8659: BUG: PPC64el machines are POWER for Fortran in f2py * #8665: BUG: Look up methods on MaskedArray in `_frommethod` * #8674: BUG: Remove extra digit in `binary_repr` at limit * #8704: BUG: Fix deepcopy regression for empty arrays. * #8707: BUG: Fix ma.median for empty ndarrays Cheers, Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Mon Mar 20 14:58:30 2017 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Mon, 20 Mar 2017 19:58:30 +0100 Subject: [Numpy-discussion] PyData Barcelona this May In-Reply-To: References: <1489087749.4278.8.camel@sipsolutions.net> Message-ID: Thanks for the links, Chris and Ralf, will be very helpful eventually. I have just submitted a workshop proposal with the following short description: Taking NumPy In Stride This workshop is aimed at users already familiar with NumPy. We will dissect the NumPy memory model with the help of a very powerful abstraction: strides. Participants will learn how to create different views out of the same data, including multidimensional ones, get a new angle on how and why broadcasting works, and explore related techniques to write faster, more efficient code. Let's see what the organizers think of it... Jaime On Fri, Mar 17, 2017 at 10:59 PM, Ralf Gommers wrote: > > > On Sat, Mar 18, 2017 at 8:41 AM, Chris Barker > wrote: > >> On Fri, Mar 17, 2017 at 4:37 AM, Jaime Fern?ndez del R?o < >> jaime.frio at gmail.com> wrote: >> >>> >>> - many people that use numpy in their daily work don't know what >>> strides are, this was a BIG surprise for me. >>> >>> I'm not surprised at all. To start with, the majority of users are > self-taught programmers that never used something lower level than Python > or Matlab. Even talking to them about memory layout presents challenges. > > >> >>> - >>> >>> Based on that experience, I was thinking that maybe a good topic for a >>> workshop would be NumPy's memory model: views, reshaping, strides, some >>> hints of buffering in the iterator... >>> >> > This material has been used multiple times in EuroScipy tutorials and may > be of use: http://www.scipy-lectures.org/advanced/advanced_numpy/index. > html > > Ralf > > > >> I think this is a great idea. In fact, when I do an intro to numpy, I >> spend a bit of time on those issues, 'cause I think it's key to "Getting" >> numpy, and not something that people end up learning on their own from >> tutorials, etc. However, in my case, I try to jam it into a low-level >> intro, and I think that fails :-( >> >> So doing it on it's own with the assumption that participant already know >> the basics of the high level python interface is a great idea. >> >> Maybe a "advanced" numpy tutorial for SciPy 2017 in Austin also??? >> >> Here is my last talk -- maybe it'll be helpful. >> >> http://uwpce-pythoncert.github.io/SystemDevelopment/scipy.html#scipy >> >> the strides stuff is covered in a notebook here: >> >> https://github.com/UWPCE-PythonCert/SystemDevelopment/blob/ >> master/Examples/numpy/stride_tricks.ipynb >> >> other notebooks here: >> >> https://github.com/UWPCE-PythonCert/SystemDevelopment/tree/ >> master/Examples/numpy >> >> and the source for the whole thing is here: >> >> https://github.com/UWPCE-PythonCert/SystemDevelopment/blob/ >> master/slides_sources/source/scipy.rst >> >> >> All licensed under: Creative Commons Attribution-ShareAlike -- so please >> use anything you find useful. >> >> -CHB >> >> >> >> And Julian's temporary work lends itself to a very nice talk, more on >>> Python internals than on NumPy, but it's a very cool subject nonetheless. >>> >>> So my thinking is that I am going to propose those two, as a workshop >>> and a talk. Thoughts? >>> >>> Jaime >>> >>> On Thu, Mar 9, 2017 at 8:29 PM, Sebastian Berg < >>> sebastian at sipsolutions.net> wrote: >>> >>>> On Thu, 2017-03-09 at 15:45 +0100, Jaime Fern?ndez del R?o wrote: >>>> > There will be a PyData conference in Barcelona this May: >>>> > >>>> > http://pydata.org/barcelona2017/ >>>> > >>>> > I am planning on attending, and was thinking of maybe proposing to >>>> > organize a numpy-themed workshop or tutorial. >>>> > >>>> > My personal inclination would be to look at some advanced topic that >>>> > I know well, like writing gufuncs in Cython, but wouldn't mind doing >>>> > a more run of the mill thing. Anyone has any thoughts or experiences >>>> > on what has worked well in similar situations? Any specific topic you >>>> > always wanted to attend a workshop on, but were afraid to ask? >>>> > >>>> > Alternatively, or on top of the workshop, I could propose to do a >>>> > talk: talking last year at PyData Madrid about the new indexing was a >>>> > lot of fun! Thing is, I have been quite disconnected from the project >>>> > this past year, and can't really think of any worthwhile topic. Is >>>> > there any message that we as a project would like to get out to the >>>> > larger community? >>>> > >>>> >>>> Francesc already pointed out the temporary optimization. From what I >>>> remember, my personal highlight would probably be Pauli's work on the >>>> memory overlap detection. Though both are rather passive improvements I >>>> guess (you don't really have to learn them to use them), its very cool! >>>> And if its about highlighting new stuff, these can probably easily fill >>>> a talk. >>>> >>>> > And if you are planning on attending, please give me a shout. >>>> > >>>> >>>> Barcelona :). Maybe I should think about it, but probably not. >>>> >>>> >>>> > Thanks, >>>> > >>>> > Jaime >>>> > >>>> > -- >>>> > (\__/) >>>> > ( O.o) >>>> > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus >>>> > planes de dominaci?n mundial. >>>> > _______________________________________________ >>>> > NumPy-Discussion mailing list >>>> > NumPy-Discussion at scipy.org >>>> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> >>> -- >>> (\__/) >>> ( O.o) >>> ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus >>> planes de dominaci?n mundial. >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Mar 20 17:13:52 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 20 Mar 2017 14:13:52 -0700 Subject: [Numpy-discussion] PyData Barcelona this May In-Reply-To: References: <1489087749.4278.8.camel@sipsolutions.net> Message-ID: On Mon, Mar 20, 2017 at 11:58 AM, Jaime Fern?ndez del R?o < jaime.frio at gmail.com> wrote: > I have just submitted a workshop proposal with the following short > description: > > Taking NumPy In Stride > This workshop is aimed at users already familiar with NumPy. We will > dissect > the NumPy memory model with the help of a very powerful abstraction: > strides. > Participants will learn how to create different views out of the same data, > including multidimensional ones, get a new angle on how and why > broadcasting > works, and explore related techniques to write faster, more efficient code. > I'd go! And nice title :-) Any thoughts on a similar one for SciPy in Austin? -CHB > Let's see what the organizers think of it... > > Jaime > > > On Fri, Mar 17, 2017 at 10:59 PM, Ralf Gommers > wrote: > >> >> >> On Sat, Mar 18, 2017 at 8:41 AM, Chris Barker >> wrote: >> >>> On Fri, Mar 17, 2017 at 4:37 AM, Jaime Fern?ndez del R?o < >>> jaime.frio at gmail.com> wrote: >>> >>>> >>>> - many people that use numpy in their daily work don't know what >>>> strides are, this was a BIG surprise for me. >>>> >>>> I'm not surprised at all. To start with, the majority of users are >> self-taught programmers that never used something lower level than Python >> or Matlab. Even talking to them about memory layout presents challenges. >> >> >>> >>>> - >>>> >>>> Based on that experience, I was thinking that maybe a good topic for a >>>> workshop would be NumPy's memory model: views, reshaping, strides, some >>>> hints of buffering in the iterator... >>>> >>> >> This material has been used multiple times in EuroScipy tutorials and may >> be of use: http://www.scipy-lectures.org/advanced/advanced_numpy/index. >> html >> >> Ralf >> >> >> >>> I think this is a great idea. In fact, when I do an intro to numpy, I >>> spend a bit of time on those issues, 'cause I think it's key to "Getting" >>> numpy, and not something that people end up learning on their own from >>> tutorials, etc. However, in my case, I try to jam it into a low-level >>> intro, and I think that fails :-( >>> >>> So doing it on it's own with the assumption that participant already >>> know the basics of the high level python interface is a great idea. >>> >>> Maybe a "advanced" numpy tutorial for SciPy 2017 in Austin also??? >>> >>> Here is my last talk -- maybe it'll be helpful. >>> >>> http://uwpce-pythoncert.github.io/SystemDevelopment/scipy.html#scipy >>> >>> the strides stuff is covered in a notebook here: >>> >>> https://github.com/UWPCE-PythonCert/SystemDevelopment/blob/m >>> aster/Examples/numpy/stride_tricks.ipynb >>> >>> other notebooks here: >>> >>> https://github.com/UWPCE-PythonCert/SystemDevelopment/tree/m >>> aster/Examples/numpy >>> >>> and the source for the whole thing is here: >>> >>> https://github.com/UWPCE-PythonCert/SystemDevelopment/blob/m >>> aster/slides_sources/source/scipy.rst >>> >>> >>> All licensed under: Creative Commons Attribution-ShareAlike -- so please >>> use anything you find useful. >>> >>> -CHB >>> >>> >>> >>> And Julian's temporary work lends itself to a very nice talk, more on >>>> Python internals than on NumPy, but it's a very cool subject nonetheless. >>>> >>>> So my thinking is that I am going to propose those two, as a workshop >>>> and a talk. Thoughts? >>>> >>>> Jaime >>>> >>>> On Thu, Mar 9, 2017 at 8:29 PM, Sebastian Berg < >>>> sebastian at sipsolutions.net> wrote: >>>> >>>>> On Thu, 2017-03-09 at 15:45 +0100, Jaime Fern?ndez del R?o wrote: >>>>> > There will be a PyData conference in Barcelona this May: >>>>> > >>>>> > http://pydata.org/barcelona2017/ >>>>> > >>>>> > I am planning on attending, and was thinking of maybe proposing to >>>>> > organize a numpy-themed workshop or tutorial. >>>>> > >>>>> > My personal inclination would be to look at some advanced topic that >>>>> > I know well, like writing gufuncs in Cython, but wouldn't mind doing >>>>> > a more run of the mill thing. Anyone has any thoughts or experiences >>>>> > on what has worked well in similar situations? Any specific topic you >>>>> > always wanted to attend a workshop on, but were afraid to ask? >>>>> > >>>>> > Alternatively, or on top of the workshop, I could propose to do a >>>>> > talk: talking last year at PyData Madrid about the new indexing was a >>>>> > lot of fun! Thing is, I have been quite disconnected from the project >>>>> > this past year, and can't really think of any worthwhile topic. Is >>>>> > there any message that we as a project would like to get out to the >>>>> > larger community? >>>>> > >>>>> >>>>> Francesc already pointed out the temporary optimization. From what I >>>>> remember, my personal highlight would probably be Pauli's work on the >>>>> memory overlap detection. Though both are rather passive improvements I >>>>> guess (you don't really have to learn them to use them), its very cool! >>>>> And if its about highlighting new stuff, these can probably easily fill >>>>> a talk. >>>>> >>>>> > And if you are planning on attending, please give me a shout. >>>>> > >>>>> >>>>> Barcelona :). Maybe I should think about it, but probably not. >>>>> >>>>> >>>>> > Thanks, >>>>> > >>>>> > Jaime >>>>> > >>>>> > -- >>>>> > (\__/) >>>>> > ( O.o) >>>>> > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus >>>>> > planes de dominaci?n mundial. >>>>> > _______________________________________________ >>>>> > NumPy-Discussion mailing list >>>>> > NumPy-Discussion at scipy.org >>>>> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> >>>> -- >>>> (\__/) >>>> ( O.o) >>>> ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus >>>> planes de dominaci?n mundial. >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> >>> -- >>> >>> Christopher Barker, Ph.D. >>> Oceanographer >>> >>> Emergency Response Division >>> NOAA/NOS/OR&R (206) 526-6959 voice >>> 7600 Sand Point Way NE (206) 526-6329 fax >>> Seattle, WA 98115 (206) 526-6317 main reception >>> >>> Chris.Barker at noaa.gov >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes > de dominaci?n mundial. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Tue Mar 21 05:02:01 2017 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Tue, 21 Mar 2017 10:02:01 +0100 Subject: [Numpy-discussion] PyData Barcelona this May In-Reply-To: References: <1489087749.4278.8.camel@sipsolutions.net> Message-ID: On Mon, Mar 20, 2017 at 10:13 PM, Chris Barker wrote: > On Mon, Mar 20, 2017 at 11:58 AM, Jaime Fern?ndez del R?o < > jaime.frio at gmail.com> wrote: > >> I have just submitted a workshop proposal with the following short >> description: >> >> Taking NumPy In Stride >> This workshop is aimed at users already familiar with NumPy. We will >> dissect >> the NumPy memory model with the help of a very powerful abstraction: >> strides. >> Participants will learn how to create different views out of the same >> data, >> including multidimensional ones, get a new angle on how and why >> broadcasting >> works, and explore related techniques to write faster, more efficient >> code. >> > > I'd go! > > And nice title :-) > > Any thoughts on a similar one for SciPy in Austin? > I'll be more than happy to share presentations, notebooks and whatnot with someone wanting to run the tutorial over there. But Austin is a looong way from Z?rich, and the dates conflict with my son's birthday, so I don't think I will be going... Jaime > > -CHB > > > > > >> Let's see what the organizers think of it... >> >> Jaime >> >> >> On Fri, Mar 17, 2017 at 10:59 PM, Ralf Gommers >> wrote: >> >>> >>> >>> On Sat, Mar 18, 2017 at 8:41 AM, Chris Barker >>> wrote: >>> >>>> On Fri, Mar 17, 2017 at 4:37 AM, Jaime Fern?ndez del R?o < >>>> jaime.frio at gmail.com> wrote: >>>> >>>>> >>>>> - many people that use numpy in their daily work don't know what >>>>> strides are, this was a BIG surprise for me. >>>>> >>>>> I'm not surprised at all. To start with, the majority of users are >>> self-taught programmers that never used something lower level than Python >>> or Matlab. Even talking to them about memory layout presents challenges. >>> >>> >>>> >>>>> - >>>>> >>>>> Based on that experience, I was thinking that maybe a good topic for a >>>>> workshop would be NumPy's memory model: views, reshaping, strides, some >>>>> hints of buffering in the iterator... >>>>> >>>> >>> This material has been used multiple times in EuroScipy tutorials and >>> may be of use: http://www.scipy-lectures.org/ >>> advanced/advanced_numpy/index.html >>> >>> Ralf >>> >>> >>> >>>> I think this is a great idea. In fact, when I do an intro to numpy, I >>>> spend a bit of time on those issues, 'cause I think it's key to "Getting" >>>> numpy, and not something that people end up learning on their own from >>>> tutorials, etc. However, in my case, I try to jam it into a low-level >>>> intro, and I think that fails :-( >>>> >>>> So doing it on it's own with the assumption that participant already >>>> know the basics of the high level python interface is a great idea. >>>> >>>> Maybe a "advanced" numpy tutorial for SciPy 2017 in Austin also??? >>>> >>>> Here is my last talk -- maybe it'll be helpful. >>>> >>>> http://uwpce-pythoncert.github.io/SystemDevelopment/scipy.html#scipy >>>> >>>> the strides stuff is covered in a notebook here: >>>> >>>> https://github.com/UWPCE-PythonCert/SystemDevelopment/blob/m >>>> aster/Examples/numpy/stride_tricks.ipynb >>>> >>>> other notebooks here: >>>> >>>> https://github.com/UWPCE-PythonCert/SystemDevelopment/tree/m >>>> aster/Examples/numpy >>>> >>>> and the source for the whole thing is here: >>>> >>>> https://github.com/UWPCE-PythonCert/SystemDevelopment/blob/m >>>> aster/slides_sources/source/scipy.rst >>>> >>>> >>>> All licensed under: Creative Commons Attribution-ShareAlike -- so >>>> please use anything you find useful. >>>> >>>> -CHB >>>> >>>> >>>> >>>> And Julian's temporary work lends itself to a very nice talk, more on >>>>> Python internals than on NumPy, but it's a very cool subject nonetheless. >>>>> >>>>> So my thinking is that I am going to propose those two, as a workshop >>>>> and a talk. Thoughts? >>>>> >>>>> Jaime >>>>> >>>>> On Thu, Mar 9, 2017 at 8:29 PM, Sebastian Berg < >>>>> sebastian at sipsolutions.net> wrote: >>>>> >>>>>> On Thu, 2017-03-09 at 15:45 +0100, Jaime Fern?ndez del R?o wrote: >>>>>> > There will be a PyData conference in Barcelona this May: >>>>>> > >>>>>> > http://pydata.org/barcelona2017/ >>>>>> > >>>>>> > I am planning on attending, and was thinking of maybe proposing to >>>>>> > organize a numpy-themed workshop or tutorial. >>>>>> > >>>>>> > My personal inclination would be to look at some advanced topic that >>>>>> > I know well, like writing gufuncs in Cython, but wouldn't mind doing >>>>>> > a more run of the mill thing. Anyone has any thoughts or experiences >>>>>> > on what has worked well in similar situations? Any specific topic >>>>>> you >>>>>> > always wanted to attend a workshop on, but were afraid to ask? >>>>>> > >>>>>> > Alternatively, or on top of the workshop, I could propose to do a >>>>>> > talk: talking last year at PyData Madrid about the new indexing was >>>>>> a >>>>>> > lot of fun! Thing is, I have been quite disconnected from the >>>>>> project >>>>>> > this past year, and can't really think of any worthwhile topic. Is >>>>>> > there any message that we as a project would like to get out to the >>>>>> > larger community? >>>>>> > >>>>>> >>>>>> Francesc already pointed out the temporary optimization. From what I >>>>>> remember, my personal highlight would probably be Pauli's work on the >>>>>> memory overlap detection. Though both are rather passive improvements >>>>>> I >>>>>> guess (you don't really have to learn them to use them), its very >>>>>> cool! >>>>>> And if its about highlighting new stuff, these can probably easily >>>>>> fill >>>>>> a talk. >>>>>> >>>>>> > And if you are planning on attending, please give me a shout. >>>>>> > >>>>>> >>>>>> Barcelona :). Maybe I should think about it, but probably not. >>>>>> >>>>>> >>>>>> > Thanks, >>>>>> > >>>>>> > Jaime >>>>>> > >>>>>> > -- >>>>>> > (\__/) >>>>>> > ( O.o) >>>>>> > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus >>>>>> > planes de dominaci?n mundial. >>>>>> > _______________________________________________ >>>>>> > NumPy-Discussion mailing list >>>>>> > NumPy-Discussion at scipy.org >>>>>> > https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> (\__/) >>>>> ( O.o) >>>>> ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus >>>>> planes de dominaci?n mundial. >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Christopher Barker, Ph.D. >>>> Oceanographer >>>> >>>> Emergency Response Division >>>> NOAA/NOS/OR&R (206) 526-6959 voice >>>> 7600 Sand Point Way NE (206) 526-6329 fax >>>> Seattle, WA 98115 (206) 526-6317 main reception >>>> >>>> Chris.Barker at noaa.gov >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> >> -- >> (\__/) >> ( O.o) >> ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes >> de dominaci?n mundial. >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.h.vankerkwijk at gmail.com Tue Mar 21 09:50:46 2017 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Tue, 21 Mar 2017 09:50:46 -0400 Subject: [Numpy-discussion] PyData Barcelona this May In-Reply-To: References: <1489087749.4278.8.camel@sipsolutions.net> Message-ID: "Taking numpy in stride, and the essential role of 0" ;-) -- Marten From davidmenhur at gmail.com Tue Mar 21 10:45:57 2017 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Tue, 21 Mar 2017 15:45:57 +0100 Subject: [Numpy-discussion] PyData Barcelona this May In-Reply-To: References: <1489087749.4278.8.camel@sipsolutions.net> Message-ID: On 20 March 2017 at 19:58, Jaime Fern?ndez del R?o wrote: > > Taking NumPy In Stride > This workshop is aimed at users already familiar with NumPy. We will dissect > the NumPy memory model with the help of a very powerful abstraction: > strides. > Participants will learn how to create different views out of the same data, > including multidimensional ones, get a new angle on how and why broadcasting > works, and explore related techniques to write faster, more efficient code. I think I only understand this abstract because I know what views are. Maybe you could add a line explaining what they are? (I cannot think of one myself). From stevenbocco at gmail.com Tue Mar 21 10:52:23 2017 From: stevenbocco at gmail.com (Steven Bocco) Date: Tue, 21 Mar 2017 10:52:23 -0400 Subject: [Numpy-discussion] Announcing Theano 0.9.0 Message-ID: Announcing Theano 0.9.0 This is a release for a major version, with lots of new features, bug fixes, and some interface changes (deprecated or potentially misleading features were removed). This release is the last major version that features the old GPU back-end ( theano.sandbox.cuda, accessible through device=gpu*). All GPU users are encouraged to transition to the new GPU back-end, based on libgpuarray ( theano.gpuarray, accessible through device=cuda*). For more information, see https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29 . Upgrading to Theano 0.9.0 is recommended for everyone, but you should first make sure that your code does not raise deprecation warnings with Theano 0.8*. Otherwise either results can change, or warnings may have been turned into errors. For those using the bleeding edge version in the git repository, we encourage you to update to the rel-0.9.0 tag. What's New Highlights (since 0.8.0): - Better Python 3.5 support - Better numpy 1.12 support - Conda packages for Mac, Linux and Windows - Support newer Mac and Windows versions - More Windows integration: - Theano scripts (theano-cache and theano-nose) now works on Windows - Better support for Windows end-lines into C codes - Support for space in paths on Windows - Scan improvements: - More scan optimizations, with faster compilation and gradient computation - Support for checkpoint in scan (trade off between speed and memory usage, useful for long sequences) - Fixed broadcast checking in scan - Graphs improvements: - More numerical stability by default for some graphs - Better handling of corner cases for theano functions and graph optimizations - More graph optimizations with faster compilation and execution - smaller and more readable graph - New GPU back-end: - Removed warp-synchronous programming to get good results with newer CUDA drivers - More pooling support on GPU when cuDNN isn't available - Full support of ignore_border option for pooling - Inplace storage for shared variables - float16 storage - Using PCI bus ID of graphic cards for a better mapping between theano device number and nvidia-smi number - Fixed offset error in GpuIncSubtensor - Less C code compilation - Added support for bool dtype - Updated and more complete documentation - Bug fixes related to merge optimizer and shape inference - Lot of other bug fixes, crashes fixes and warning improvements Interface changes: - Merged CumsumOp/CumprodOp into CumOp - In MRG module: - Replaced method multinomial_wo_replacement() with new method choice() - Random generator now tries to infer the broadcast pattern of its output - New pooling interface - Pooling parameters can change at run time - Moved softsign out of sandbox to theano.tensor.nnet.softsign - Using floatX dtype when converting empty list/tuple - Roll make the shift be modulo the size of the axis we roll on - round() default to the same as NumPy: half_to_even Convolution updates: - Support of full and half modes for 2D and 3D convolutions including in conv3d2d - Allowed pooling of empty batch - Implement conv2d_transpose convenience function - Multi-cores convolution and pooling on CPU - New abstract 3d convolution interface similar to the 2d convolution interface - Dilated convolution GPU: - cuDNN: support versoin 5.1 and wrap batch normalization (2d and 3d) and RNN functions - Multiple-GPU, synchrone update (via platoon, use NCCL) - Gemv(matrix-vector product) speed up for special shape - cublas gemv workaround when we reduce on an axis with a dimensions size of 0 - Warn user that some cuDNN algorithms may produce unexpected results in certain environments for convolution backward filter operations - GPUMultinomialFromUniform op now supports multiple dtypes - Support for MaxAndArgMax for some axis combination - Support for solve (using cusolver), erfinv and erfcinv - Implemented GpuAdvancedSubtensor New features: - OpFromGraph now allows gradient overriding for every input - Added Abstract Ops for batch normalization that use cuDNN when available and pure Theano CPU/GPU alternatives otherwise - Added gradient of solve, tensorinv (CPU), tensorsolve (CPU), searchsorted (CPU), DownsampleFactorMaxGradGrad (CPU) - Added Multinomial Without Replacement - Allowed partial evaluation of compiled function - More Rop support - Indexing support ellipsis: a[..., 3]`, a[1,...,3] - Added theano.tensor.{tensor5,dtensor5, ...} - compiledir_format support device - Added New Theano flag conv.assert_shape to check user-provided shapes at runtime (for debugging) - Added new Theano flag cmodule.age_thresh_use - Added new Theano flag cuda.enabled - Added new Theano flag nvcc.cudafe to enable faster compilation and import with old CUDA back-end - Added new Theano flag print_global_stats to print some global statistics (time spent) at the end - Added new Theano flag profiling.ignore_first_call, useful to profile the new gpu back-end - remove ProfileMode (use Theano flag profile=True instead) Others: - Split op now has C code for CPU and GPU - theano-cache list now includes compilation times - Speed up argmax only on GPU (without also needing the max) - More stack trace in error messages - Speed up cholesky grad - log(sum(exp(...))) now get stability optimized Other more detailed changes: - Added Jenkins (gpu tests run on pull requests in addition to daily buildbot) - Removed old benchmark directory and other old files not used anymore - Use of 64-bit indexing in sparse ops to allow matrix with more then 231-1 elements - Allowed more then one output to be an destructive inplace - More support of negative axis - Added the keepdims parameter to the norm function - Make scan gradient more deterministic Download and Install You can download Theano from http://pypi.python.org/pypi/Theano Installation instructions are available at http://deeplearning.net/software/theano/install.html Description Theano is a Python library that allows you to define, optimize, and efficiently evaluate mathematical expressions involving multi-dimensional arrays. It is built on top of NumPy. Theano features: - tight integration with NumPy: a similar interface to NumPy's. numpy.ndarrays are also used internally in Theano-compiled functions. - transparent use of a GPU: perform data-intensive computations much faster than on a CPU. - efficient symbolic differentiation: Theano can compute derivatives for functions of one or many inputs. - speed and stability optimizations: avoid nasty bugs when computing expressions such as log(1+ exp(x)) for large values of x. - dynamic C code generation: evaluate expressions faster. - extensive unit-testing and self-verification: includes tools for detecting and diagnosing bugs and/or potential problems. Theano has been powering large-scale computationally intensive scientific research since 2007, but it is also approachable enough to be used in the classroom (IFT6266 at the University of Montreal). Resources About Theano: http://deeplearning.net/software/theano/ Theano-related projects: http://github.com/Theano/Theano/wiki/Related-projects About NumPy: http://numpy.scipy.org/ About SciPy: http://www.scipy.org/ Machine Learning Tutorial with Theano on Deep Architectures: http://deeplearning.net/tutorial/ Acknowledgments I would like to thank all contributors of Theano. For this particular release, many people have helped, notably (in alphabetical order): - affanv14 - Alexander Matyasko - Alexandre de Brebisson - Amjad Almahairi - Andr?s Gottlieb - Anton Chechetka - Arnaud Bergeron - Benjamin Scellier - Ben Poole - Bhavishya Pohani - Bryn Keller - Caglar - Carl Thom? - Cesar Laurent - Chiheb Trabelsi - Chinnadhurai Sankar - Christos Tsirigotis - Ciyong Chen - David Bau - Dimitar Dimitrov - Evelyn Mitchell - F?bio Perez - Faruk Ahmed - Fei Wang - Fei Zhan - Florian Bordes - Francesco Visin - Frederic Bastien - Fuchai - Gennadiy Tupitsin - Gijs van Tulder - Gilles Louppe - Gokula Krishnan - Greg Ciccarelli - gw0 [http://gw.tnode.com/] - happygds - Harm de Vries - He - hexahedria - hsintone - Huan Zhang - Ilya Kulikov - Iulian Vlad Serban - jakirkham - Jakub Sygnowski - Jan Schl?ter - Jesse Livezey - Jonas Degrave - joncrall - Kaixhin - Karthik Karanth - Kelvin Xu - Kevin Keraudren - khaotik - Kirill Bobyrev - Kumar Krishna Agrawal - Kv Manohar - Liwei Cai - Lucas Beyer - Maltimore - Marc-Alexandre Cote - Marco - Marius F. Killinger - Martin Drawitsch - Mathieu Germain - Matt Graham - Maxim Kochurov - Micah Bojrab - Michael Harradon - Mikhail Korobov - mockingjamie - Mohammad Pezeshki - Morgan Stuart - Nan Rosemary Ke - Neil - Nicolas Ballas - Nizar Assaf - Olivier Mastropietro - Ozan ?a?layan - p - Pascal Lamblin - Pierre Luc Carrier - RadhikaG - Ramana Subramanyam - Ray Donnelly - Rebecca N. Palmer - Reyhane Askari - Rithesh Kumar - Rizky Luthfianto - Robin Millette - Roman Ring - root - Ruslana Makovetsky - Saizheng Zhang - Samira Ebrahimi Kahou - Samira Shabanian - Sander Dieleman - Sebastin Santy - Shawn Tan - Simon Lefrancois - Sina Honari - Steven Bocco - superantichrist - Taesup (TS) Kim - texot - Thomas George - tillahoffmann - Tim Cooijmans - Tim Gasper - valtron - Vincent Dumoulin - Vincent Michalski - Vitaliy Kurlin - Wazeer Zulfikar - wazeerzulfikar - Wojciech G?ogowski - Xavier Bouthillier - Yang Zhang - Yann N. Dauphin - Yaroslav Ganin - Ying Zhang - you-n-g - Zhouhan LIN Also, thank you to all NumPy and Scipy developers as Theano builds on their strengths. All questions/comments are always welcome on the Theano mailing-lists ( http://deeplearning.net/software/theano/#community ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Mar 21 20:39:08 2017 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 21 Mar 2017 17:39:08 -0700 Subject: [Numpy-discussion] Advanced numpy tutorial for SciPy? Message-ID: <-4678996778209545455@unknownmsgid> In another thread, there is a discussion of a workshop on "Taking NumPy In Stride" for PyData Barcelona. I think it would be great to have something like that at SciPy in Austin this year. Jaime can't make it, and I don't think strides are going to fill a four hour tutorial, so it would be good as part of an advanced numpy tutorial. I don't have the bandwidth to put together an entire tutorial, but maybe someone would like to join forces? Or if someone is already planning an advanced numpy tutorial, perhaps I could contribute. Not much time left to get a proposal in! -Chris From ralf.gommers at gmail.com Wed Mar 22 05:23:15 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 22 Mar 2017 22:23:15 +1300 Subject: [Numpy-discussion] migration of all scipy.org mailing lists Message-ID: Hi all, The server for the scipy.org mailing list is in very bad shape, so we (led by Didrik Pinte) are planning to complete the migration of active mailing lists to the python.org infrastructure and to decommission the lists than seem dormant/obsolete. The scipy-user mailing list was already moved to python.org a month or two ago, and that migration went smoothly. These are the lists we plan to migrate: astropy ipython-dev ipython-user numpy-discussion numpy-svn scipy-dev scipy-organizers scipy-svn And these we plan to retire: Announce APUG Ipython-tickets Mailman numpy-refactor numpy-refactor-git numpy-tickets Pyxg scipy-tickets NiPy-devel There will be a short period (<24 hours) where messages to the list may be refused, with an informative message as to why. The mailing list address will change from listname at scipy.org to listname at python.org This will happen asap, likely within a day or two. So two requests: 1. If you see any issue with this plan, please reply and keep Didrik and myself on Cc (we are not subscribed to all lists). 2. If you see this message on a numpy/scipy list, but not on another list (could be due to a moderation queue) then please forward this message again to that other list. Thanks, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Mar 22 05:24:37 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 22 Mar 2017 22:24:37 +1300 Subject: [Numpy-discussion] migration of all scipy.org mailing lists In-Reply-To: References: Message-ID: (and now with Didrik on Cc - apologies) On Wed, Mar 22, 2017 at 10:23 PM, Ralf Gommers wrote: > Hi all, > > The server for the scipy.org mailing list is in very bad shape, so we > (led by Didrik Pinte) are planning to complete the migration of active > mailing lists to the python.org infrastructure and to decommission the > lists than seem dormant/obsolete. > > The scipy-user mailing list was already moved to python.org a month or > two ago, and that migration went smoothly. > > These are the lists we plan to migrate: > > astropy > ipython-dev > ipython-user > numpy-discussion > numpy-svn > scipy-dev > scipy-organizers > scipy-svn > > And these we plan to retire: > > Announce > APUG > Ipython-tickets > Mailman > numpy-refactor > numpy-refactor-git > numpy-tickets > Pyxg > scipy-tickets > NiPy-devel > > There will be a short period (<24 hours) where messages to the list may be > refused, with an informative message as to why. The mailing list address > will change from listname at scipy.org to listname at python.org > > This will happen asap, likely within a day or two. So two requests: > 1. If you see any issue with this plan, please reply and keep Didrik and > myself on Cc (we are not subscribed to all lists). > 2. If you see this message on a numpy/scipy list, but not on another list > (could be due to a moderation queue) then please forward this message again > to that other list. > > Thanks, > Ralf > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Wed Mar 22 07:18:04 2017 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 22 Mar 2017 07:18:04 -0400 Subject: [Numpy-discussion] migration of all scipy.org mailing lists References: Message-ID: Has anyone taken care of notifying gmane about this? Ralf Gommers wrote: > Hi all, > > The server for the scipy.org mailing list is in very bad shape, so we (led > by Didrik Pinte) are planning to complete the migration of active mailing > lists to the python.org infrastructure and to decommission the lists than > seem dormant/obsolete. > > The scipy-user mailing list was already moved to python.org a month or two > ago, and that migration went smoothly. > > These are the lists we plan to migrate: > > astropy > ipython-dev > ipython-user > numpy-discussion > numpy-svn > scipy-dev > scipy-organizers > scipy-svn > > And these we plan to retire: > > Announce > APUG > Ipython-tickets > Mailman > numpy-refactor > numpy-refactor-git > numpy-tickets > Pyxg > scipy-tickets > NiPy-devel > > There will be a short period (<24 hours) where messages to the list may be > refused, with an informative message as to why. The mailing list address > will change from listname at scipy.org to listname at python.org > > This will happen asap, likely within a day or two. So two requests: > 1. If you see any issue with this plan, please reply and keep Didrik and > myself on Cc (we are not subscribed to all lists). > 2. If you see this message on a numpy/scipy list, but not on another list > (could be due to a moderation queue) then please forward this message > again to that other list. > > Thanks, > Ralf From ralf.gommers at gmail.com Wed Mar 22 08:52:07 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 23 Mar 2017 01:52:07 +1300 Subject: [Numpy-discussion] migration of all scipy.org mailing lists In-Reply-To: References: Message-ID: On Thu, Mar 23, 2017 at 12:18 AM, Neal Becker wrote: > Has anyone taken care of notifying gmane about this? > We will have to update this info in quite a few places after the move is done. Including Gmane, although that site hasn't been working for half a year so is pretty low on the priority list. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan12343 at gmail.com Wed Mar 22 10:11:56 2017 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Wed, 22 Mar 2017 09:11:56 -0500 Subject: [Numpy-discussion] migration of all scipy.org mailing lists In-Reply-To: References: Message-ID: Are you sure about astropy? They recently moved to google groups. On Wednesday, March 22, 2017, Ralf Gommers wrote: > Hi all, > > The server for the scipy.org mailing list is in very bad shape, so we > (led by Didrik Pinte) are planning to complete the migration of active > mailing lists to the python.org infrastructure and to decommission the > lists than seem dormant/obsolete. > > The scipy-user mailing list was already moved to python.org a month or > two ago, and that migration went smoothly. > > These are the lists we plan to migrate: > > astropy > ipython-dev > ipython-user > numpy-discussion > numpy-svn > scipy-dev > scipy-organizers > scipy-svn > > And these we plan to retire: > > Announce > APUG > Ipython-tickets > Mailman > numpy-refactor > numpy-refactor-git > numpy-tickets > Pyxg > scipy-tickets > NiPy-devel > > There will be a short period (<24 hours) where messages to the list may be > refused, with an informative message as to why. The mailing list address > will change from listname at scipy.org > to > listname at python.org > > This will happen asap, likely within a day or two. So two requests: > 1. If you see any issue with this plan, please reply and keep Didrik and > myself on Cc (we are not subscribed to all lists). > 2. If you see this message on a numpy/scipy list, but not on another list > (could be due to a moderation queue) then please forward this message again > to that other list. > > Thanks, > Ralf > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.v.root at gmail.com Wed Mar 22 10:44:09 2017 From: ben.v.root at gmail.com (Benjamin Root) Date: Wed, 22 Mar 2017 10:44:09 -0400 Subject: [Numpy-discussion] migration of all scipy.org mailing lists In-Reply-To: References: Message-ID: nabble is the other site that probably should be notified of the mailing list move. On Wed, Mar 22, 2017 at 10:11 AM, Nathan Goldbaum wrote: > Are you sure about astropy? They recently moved to google groups. > > > On Wednesday, March 22, 2017, Ralf Gommers wrote: > >> Hi all, >> >> The server for the scipy.org mailing list is in very bad shape, so we >> (led by Didrik Pinte) are planning to complete the migration of active >> mailing lists to the python.org infrastructure and to decommission the >> lists than seem dormant/obsolete. >> >> The scipy-user mailing list was already moved to python.org a month or >> two ago, and that migration went smoothly. >> >> These are the lists we plan to migrate: >> >> astropy >> ipython-dev >> ipython-user >> numpy-discussion >> numpy-svn >> scipy-dev >> scipy-organizers >> scipy-svn >> >> And these we plan to retire: >> >> Announce >> APUG >> Ipython-tickets >> Mailman >> numpy-refactor >> numpy-refactor-git >> numpy-tickets >> Pyxg >> scipy-tickets >> NiPy-devel >> >> There will be a short period (<24 hours) where messages to the list may >> be refused, with an informative message as to why. The mailing list address >> will change from listname at scipy.org to listname at python.org >> >> This will happen asap, likely within a day or two. So two requests: >> 1. If you see any issue with this plan, please reply and keep Didrik and >> myself on Cc (we are not subscribed to all lists). >> 2. If you see this message on a numpy/scipy list, but not on another list >> (could be due to a moderation queue) then please forward this message again >> to that other list. >> >> Thanks, >> Ralf >> >> >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thoger.emil at gmail.com Wed Mar 22 22:06:02 2017 From: thoger.emil at gmail.com (=?UTF-8?Q?Th=c3=b8ger_Emil_Rivera-Thorsen?=) Date: Thu, 23 Mar 2017 03:06:02 +0100 Subject: [Numpy-discussion] Possible bug in Numpy.convolve Message-ID: <0ff408fb-cbb0-91c0-79cd-9090fa99f2e1@gmail.com> Dear list; I am honestly not certain whether this, or the SciPy list, is the appropriate place to post this; please let me know if I got it wrong. I am convolving a 1D data set containing a relatively narrow peak, with a relatively narrow Gaussian kernel, in order to emulate the effect of atmospheric seeing on astrophysical observations. I have a 1D data array 45 pixels long, and a Gaussian kernel, and run np.convolve(data, kernel, mode='same') on the two arrays, the resulting array's peak is shifted relative to the origin. I have attached a plot to illustrate. The original data is shown in blue. When I convolve it with a symmetric kernel (black), I get an offset resulting peak (magenta). If I flip the kernel -- even though it is perfectly symmetric -- the resulting curve is offset in the opposite direction (yellow). However, if I offset the kernel so it is centered exactly one pixel below the central value, the output array gets centered correct (red), even if I flip the (now no longer symmetric) kernel. This is using Numpy 1.11.3, python 2.7.13, on Anaconda 4.3.0 64-bit on Ubuntu 16.10 Using astropy.convolution, reproduces the correct red curve, so I can use that for now, but it seems to me this is either a bug or, if it is indeed the intended behavior, a word of caution would be merited in the docstring. Cheers, Emil Rivera-Thorsen -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cjlfglpfehnobnll.png Type: image/png Size: 32973 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Selection_043.png Type: image/png Size: 32973 bytes Desc: not available URL: From josef.pktd at gmail.com Wed Mar 22 22:38:47 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Mar 2017 22:38:47 -0400 Subject: [Numpy-discussion] Possible bug in Numpy.convolve In-Reply-To: <0ff408fb-cbb0-91c0-79cd-9090fa99f2e1@gmail.com> References: <0ff408fb-cbb0-91c0-79cd-9090fa99f2e1@gmail.com> Message-ID: On Wed, Mar 22, 2017 at 10:06 PM, Th?ger Emil Rivera-Thorsen < thoger.emil at gmail.com> wrote: > Dear list; > > I am honestly not certain whether this, or the SciPy list, is the > appropriate place to post this; please let me know if I got it wrong. > > I am convolving a 1D data set containing a relatively narrow peak, with a > relatively narrow Gaussian kernel, in order to emulate the effect of > atmospheric seeing on astrophysical observations. > > I have a 1D data array 45 pixels long, and a Gaussian kernel, and run > np.convolve(data, kernel, mode='same') on the two arrays, the resulting > array's peak is shifted relative to the origin. I have attached a plot to > illustrate. > > The original data is shown in blue. When I convolve it with a symmetric > kernel (black), I get an offset resulting peak (magenta). If I flip the > kernel -- even though it is perfectly symmetric -- the resulting curve is > offset in the opposite direction (yellow). However, if I offset the kernel > so it is centered exactly one pixel below the central value, the output > array gets centered correct (red), even if I flip the (now no longer > symmetric) kernel. > > This is using Numpy 1.11.3, python 2.7.13, on Anaconda 4.3.0 64-bit on > Ubuntu 16.10 > > Using astropy.convolution, reproduces the correct red curve, so I can use > that for now, but it seems to me this is either a bug or, if it is indeed > the intended behavior, a word of caution would be merited in the docstring. > > Cheers, > > Emil Rivera-Thorsen > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > Can you provide an example to replicate? I haven't seen this behavior, it looks centered to me, at least for odd window length.. AFAIR, I had to try a bit in the past for how to set the window and location with even window length. >>> np.__version__ '1.11.2' >>> x = np.linspace(-1, 1, 21) >>> w = stats.norm.pdf(np.linspace(-3, 3, 5)) >>> np.column_stack((x, np.convolve(x, w, mode='same')))[8:13] array([[ -2.00000000e-01, -1.33368234e-01], [ -1.00000000e-01, -6.66841169e-02], [ 0.00000000e+00, 1.51788304e-17], [ 1.00000000e-01, 6.66841169e-02], [ 2.00000000e-01, 1.33368234e-01]]) >>> x = np.abs(np.linspace(-1, 1, 21)) >>> w = stats.norm.pdf(np.linspace(-3, 3, 4)) >>> np.column_stack((x, np.convolve(x, w, mode='same')))[8:13] array([[ 0.2 , 0.12320129], [ 0.1 , 0.07392077], [ 0. , 0.02552663], [ 0.1 , 0.02552663], [ 0.2 , 0.07392077]]) >>> w = stats.norm.pdf(np.linspace(-3, 3, 5)) >>> np.column_stack((x, np.convolve(x, w, mode='same')))[8:13] array([[ 0.2 , 0.13336823], [ 0.1 , 0.06757049], [ 0. , 0.02767626], [ 0.1 , 0.06757049], [ 0.2 , 0.13336823]]) Josef -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cjlfglpfehnobnll.png Type: image/png Size: 32973 bytes Desc: not available URL: From thoger.emil at gmail.com Wed Mar 22 22:44:04 2017 From: thoger.emil at gmail.com (=?UTF-8?Q?Th=c3=b8ger_Emil_Rivera-Thorsen?=) Date: Thu, 23 Mar 2017 03:44:04 +0100 Subject: [Numpy-discussion] Possible bug in Numpy.convolve In-Reply-To: References: <0ff408fb-cbb0-91c0-79cd-9090fa99f2e1@gmail.com> Message-ID: OK, this is embarrassing. I had in fact made the kernel only almost symmetric, it was slightly offset to one side. This caused it. Only a wetware bug. Sorry for having wasted your time! On 03/23/2017 03:38 AM, josef.pktd at gmail.com wrote: > > > On Wed, Mar 22, 2017 at 10:06 PM, Th?ger Emil Rivera-Thorsen > > wrote: > > Dear list; > > I am honestly not certain whether this, or the SciPy list, is the > appropriate place to post this; please let me know if I got it wrong. > > I am convolving a 1D data set containing a relatively narrow peak, > with a relatively narrow Gaussian kernel, in order to emulate the > effect of atmospheric seeing on astrophysical observations. > > I have a 1D data array 45 pixels long, and a Gaussian kernel, and > run np.convolve(data, kernel, mode='same') on the two arrays, the > resulting array's peak is shifted relative to the origin. I have > attached a plot to illustrate. > > The original data is shown in blue. When I convolve it with a > symmetric kernel (black), I get an offset resulting peak > (magenta). If I flip the kernel -- even though it is perfectly > symmetric -- the resulting curve is offset in the opposite > direction (yellow). However, if I offset the kernel so it is > centered exactly one pixel below the central value, the output > array gets centered correct (red), even if I flip the (now no > longer symmetric) kernel. > > This is using Numpy 1.11.3, python 2.7.13, on Anaconda 4.3.0 > 64-bit on Ubuntu 16.10 > > Using astropy.convolution, reproduces the correct red curve, so I > can use that for now, but it seems to me this is either a bug or, > if it is indeed the intended behavior, a word of caution would be > merited in the docstring. > > Cheers, > > Emil Rivera-Thorsen > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > Can you provide an example to replicate? > > I haven't seen this behavior, it looks centered to me, at least for > odd window length.. > AFAIR, I had to try a bit in the past for how to set the window and > location with even window length. > > >>> np.__version__ > '1.11.2' > > >>> x = np.linspace(-1, 1, 21) > >>> w = stats.norm.pdf(np.linspace(-3, 3, 5)) > >>> np.column_stack((x, np.convolve(x, w, mode='same')))[8:13] > array([[ -2.00000000e-01, -1.33368234e-01], > [ -1.00000000e-01, -6.66841169e-02], > [ 0.00000000e+00, 1.51788304e-17], > [ 1.00000000e-01, 6.66841169e-02], > [ 2.00000000e-01, 1.33368234e-01]]) > > > >>> x = np.abs(np.linspace(-1, 1, 21)) > >>> w = stats.norm.pdf(np.linspace(-3, 3, 4)) > >>> np.column_stack((x, np.convolve(x, w, mode='same')))[8:13] > array([[ 0.2 , 0.12320129], > [ 0.1 , 0.07392077], > [ 0. , 0.02552663], > [ 0.1 , 0.02552663], > [ 0.2 , 0.07392077]]) > > > >>> w = stats.norm.pdf(np.linspace(-3, 3, 5)) > >>> np.column_stack((x, np.convolve(x, w, mode='same')))[8:13] > array([[ 0.2 , 0.13336823], > [ 0.1 , 0.06757049], > [ 0. , 0.02767626], > [ 0.1 , 0.06757049], > [ 0.2 , 0.13336823]]) > > Josef > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 32973 bytes Desc: not available URL: From ralf.gommers at gmail.com Thu Mar 23 06:08:44 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 23 Mar 2017 23:08:44 +1300 Subject: [Numpy-discussion] migration of all scipy.org mailing lists In-Reply-To: References: Message-ID: On Thu, Mar 23, 2017 at 3:11 AM, Nathan Goldbaum wrote: > Are you sure about astropy? They recently moved to google groups. https://mail.scipy.org/pipermail/astropy/ shows messages as recent as today. So I'm sure I think (but happy to be corrected if moving to python.org is not desired for whatever reason). Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Thu Mar 23 07:27:36 2017 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 23 Mar 2017 07:27:36 -0400 Subject: [Numpy-discussion] migration of all scipy.org mailing lists References: Message-ID: Ralf Gommers wrote: > On Thu, Mar 23, 2017 at 12:18 AM, Neal Becker wrote: > >> Has anyone taken care of notifying gmane about this? >> > > We will have to update this info in quite a few places after the move is > done. Including Gmane, although that site hasn't been working for half a > year so is pretty low on the priority list. > > Ralf I'm reading/writing to you via gmane, so I think it is working :) From nathan12343 at gmail.com Thu Mar 23 08:44:33 2017 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Thu, 23 Mar 2017 12:44:33 +0000 Subject: [Numpy-discussion] migration of all scipy.org mailing lists In-Reply-To: References: Message-ID: The switchover happened on almost the same day, see: https://mail.scipy.org/pipermail/astropy/2017-March/004498.html On Thu, Mar 23, 2017 at 5:09 AM Ralf Gommers wrote: > On Thu, Mar 23, 2017 at 3:11 AM, Nathan Goldbaum > wrote: > > Are you sure about astropy? They recently moved to google groups. > > > https://mail.scipy.org/pipermail/astropy/ shows messages as recent as > today. So I'm sure I think (but happy to be corrected if moving to > python.org is not desired for whatever reason). > > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Thu Mar 23 10:56:22 2017 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Thu, 23 Mar 2017 15:56:22 +0100 Subject: [Numpy-discussion] PyData Barcelona this May In-Reply-To: References: <1489087749.4278.8.camel@sipsolutions.net> Message-ID: Thanks for the suggestion! That abstract is limited to 400 characters, and it's already at 350+, so there isn't much room left. If it gets accepted I will eventually need to fill out an extended abstract, where I will make sure to explain that better. Jaime On Tue, Mar 21, 2017 at 3:45 PM, Da?id wrote: > On 20 March 2017 at 19:58, Jaime Fern?ndez del R?o > wrote: > > > > Taking NumPy In Stride > > This workshop is aimed at users already familiar with NumPy. We will > dissect > > the NumPy memory model with the help of a very powerful abstraction: > > strides. > > Participants will learn how to create different views out of the same > data, > > including multidimensional ones, get a new angle on how and why > broadcasting > > works, and explore related techniques to write faster, more efficient > code. > > I think I only understand this abstract because I know what views are. > Maybe you could add a line explaining what they are? (I cannot think > of one myself). > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Thu Mar 23 14:10:25 2017 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 23 Mar 2017 20:10:25 +0200 Subject: [Numpy-discussion] Release tags Message-ID: I am searching for the exact git hash corresponding to numpy 1.12.1 Looking at the result of `git tag -l` yielded tags for v1.11.0, but nothing higher. Also the documentation site https://docs.scipy.org/doc/numpy/release.html does not seem to list 1.12.1, but I guess that is a separate issue? Matti From kirillbalunov at gmail.com Thu Mar 23 14:16:28 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Thu, 23 Mar 2017 21:16:28 +0300 Subject: [Numpy-discussion] Structured array creation with list of lists and others Message-ID: It was the first time I tried to create a structured array in numpy. Usually I use pandas for heterogeneous arrays, but it is one more dependency to my project. It took me some time (really, much more than some), to understand the problem with structured array creation. As example: I had list of list of this kind: b=[[ 1, 10.3, 12.1, 2.12 ],...] And tried: np.array(b, dtype='i4,f4,f4,f4') Which raises some weird exception: TypeError: a bytes-like object is required, not 'int' Two hours later I found that I need list of tuples. I didn't find any help in documentation and could not realize that the problem with the inner lists... Why there is such restriction - 'list of tuples' to create structured array? What is the idea behind that, why not list of lists, or tuple of lists or ...? Also the exception does not help at all... p.s.: It looks like that dtype also accepts only list of tuples. But I can not catch the idea for this restrictions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Thu Mar 23 14:27:28 2017 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Thu, 23 Mar 2017 19:27:28 +0100 Subject: [Numpy-discussion] Release tags In-Reply-To: References: Message-ID: <09d6e9aa-eed6-7131-65f8-6247d5268816@googlemail.com> probably the git quirk of not pulling tags by default, you need to run: git pull --tags On 23.03.2017 19:10, Matti Picus wrote: > I am searching for the exact git hash corresponding to numpy 1.12.1 > > Looking at the result of `git tag -l` yielded tags for v1.11.0, but > nothing higher. > > Also the documentation site > https://docs.scipy.org/doc/numpy/release.html does not seem to list > 1.12.1, but I guess that is a separate issue? > > Matti > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion From ralf.gommers at gmail.com Fri Mar 24 05:09:50 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 24 Mar 2017 22:09:50 +1300 Subject: [Numpy-discussion] mailing list moved Message-ID: Hi all, This mailing list moved, it's now numpy-discussion at python.org. Please update the stored address in the contacts list of your email client - messages sent to @scipy.org will not arrive anymore. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Fri Mar 24 07:30:57 2017 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 24 Mar 2017 11:30:57 +0000 Subject: [Numpy-discussion] Set #threads from within python code Message-ID: I don't want my python code to run multi-thread. So I can do: MKL_NUM_THREAD=1 NUMEXPR_NUM_THREADS=1 OMP_NUM_THREADS=1 my_program... But I don't seem to be able to achieve this effect without setting env variables on the command line; within my_program. Using os.environ doesn't work. I don't understand why. I'd like to not have to put this on the command line because I sometimes forget. -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Fri Mar 24 07:36:38 2017 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Fri, 24 Mar 2017 12:36:38 +0100 Subject: [Numpy-discussion] Set #threads from within python code In-Reply-To: References: Message-ID: On 24 March 2017 at 12:30, Neal Becker wrote: > Using os.environ doesn't > work. I don't understand why. It should, I do that for other variables. Are you setting the variables before importing other libraries? They may only get read at import time. From ndbecker2 at gmail.com Fri Mar 24 07:38:41 2017 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 24 Mar 2017 11:38:41 +0000 Subject: [Numpy-discussion] Set #threads from within python code In-Reply-To: References: Message-ID: Ah, that probably explains it! On Fri, Mar 24, 2017 at 7:37 AM Da?id wrote: > On 24 March 2017 at 12:30, Neal Becker wrote: > > Using os.environ doesn't > > work. I don't understand why. > > It should, I do that for other variables. Are you setting the > variables before importing other libraries? They may only get read at > import time. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jslavin at cfa.harvard.edu Fri Mar 24 09:59:23 2017 From: jslavin at cfa.harvard.edu (Slavin, Jonathan) Date: Fri, 24 Mar 2017 09:59:23 -0400 Subject: [Numpy-discussion] Structured array creation with list of lists and others Message-ID: Hi Kirill, ?T? he idea is that each tuple assigns a name to the field and a data type. There are a variety of ways to create structured arrays but they all involve giving both a name and data type to each field (I think). See https://docs.scipy.org/doc/numpy/user/basics.rec.html ?Jon? On Fri, Mar 24, 2017 at 5:09 AM, wrote: > From: Kirill Balunov > To: numpy-discussion at scipy.org > Cc: > Bcc: > Date: Thu, 23 Mar 2017 21:16:28 +0300 > Subject: [Numpy-discussion] Structured array creation with list of lists and others > It was the first time I tried to create a structured array in numpy. Usually I use pandas for heterogeneous arrays, but it is one more dependency to my project. > > It took me some time (really, much more than some), to understand the problem with structured array creation. As example: > > I had list of list of this kind: > b=[[ 1, 10.3, 12.1, 2.12 ],...] > > And tried: > np.array(b, dtype='i4,f4,f4,f4') > > Which raises some weird exception: > TypeError: a bytes-like object is required, not 'int' > > Two hours later I found that I need list of tuples. I didn't find any help in documentation and could not realize that the problem with the inner lists... > > Why there is such restriction - 'list of tuples' to create structured array? What is the idea behind that, why not list of lists, or tuple of lists or ...? > > Also the exception does not help at all... > p.s.: It looks like that dtype also accepts only list of tuples. But I can not catch the idea for this restrictions. -- ________________________________________________________ Jonathan D. Slavin Harvard-Smithsonian CfA jslavin at cfa.harvard.edu 60 Garden Street, MS 83 phone: (617) 496-7981 Cambridge, MA 02138-1516 cell: (781) 363-0035 USA ________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From allanhaldane at gmail.com Fri Mar 24 12:48:32 2017 From: allanhaldane at gmail.com (Allan Haldane) Date: Fri, 24 Mar 2017 12:48:32 -0400 Subject: [Numpy-discussion] Structured array creation with list of lists and others In-Reply-To: References: Message-ID: On 03/23/2017 02:16 PM, Kirill Balunov wrote: > It was the first time I tried to create a structured array in numpy. > Usually I use pandas for heterogeneous arrays, but it is one more > dependency to my project. > > It took me some time (really, much more than some), to understand the > problem with structured array creation. As example: > > I had list of list of this kind: > b=[[ 1, 10.3, 12.1, 2.12 ],...] > > And tried: > np.array(b, dtype='i4,f4,f4,f4') > > Which raises some weird exception: > TypeError: a bytes-like object is required, not 'int' > > Two hours later I found that I need list of tuples. I didn't find any help > in documentation and could not realize that the problem with the inner > lists... > > Why there is such restriction - 'list of tuples' to create structured > array? What is the idea behind that, why not list of lists, or tuple of > lists or ...? > > Also the exception does not help at all... > p.s.: It looks like that dtype also accepts only list of tuples. But I can > not catch the idea for this restrictions. > The problem is that numpy needs to distinguish between multidimensional arrays and structured elements. A "list of lists" will often trigger numpy's broadcasting rules, which is not what you want here. For instance, should numpy interpret your input list as a 2d array of dimension Lx4 containing integer elements, or a 1d array of length L of structs with 4 fields? In this particular case maybe numpy could, in principle, figure it out from what you gave it by calculating that the innermost dimension is the same length as the number of fields. But there are other cases (such as assignment) where similar ambiguities arise that are harder to resolve. So to preserve our sanity we want to require that structures be formatted as tuples all the time. I have a draft of potential updated structured array docs you can read here: https://gist.github.com/ahaldane/7d1873d33d4d0f80ba7a54ccf1052eee See the section "Assignment from Python Native Types (Tuples)", which hopefully better warns that tuples are needed. Let me know if you think something is missing from the draft. (WARNING: the section about multi-field assignment in the doc draft is incorrect for current numpy - that's what I'm proposing for the next release. The rest of the docs are accurate for current numpy) Agreed that the error message could be changed. Allan From oleksandr.pavlyk at intel.com Fri Mar 24 15:42:54 2017 From: oleksandr.pavlyk at intel.com (Pavlyk, Oleksandr) Date: Fri, 24 Mar 2017 19:42:54 +0000 Subject: [Numpy-discussion] Set #threads from within python code In-Reply-To: References: Message-ID: <4C9EDA7282E297428F3986994EB0FBD38DA6C4@ORSMSX110.amr.corp.intel.com> Rather than keeping the number of threads MKL uses to 1, it is better to use MKL_THREADING_LAYER=SEQUENTIAL https://software.intel.com/en-us/node/528380 --Sasha From: NumPy-Discussion [mailto:numpy-discussion-bounces+oleksandr.pavlyk=intel.com at python.org] On Behalf Of Neal Becker Sent: Friday, March 24, 2017 6:31 AM To: numpy-discussion at python.org Subject: [Numpy-discussion] Set #threads from within python code I don't want my python code to run multi-thread. So I can do: MKL_NUM_THREAD=1 NUMEXPR_NUM_THREADS=1 OMP_NUM_THREADS=1 my_program... But I don't seem to be able to achieve this effect without setting env variables on the command line; within my_program. Using os.environ doesn't work. I don't understand why. I'd like to not have to put this on the command line because I sometimes forget. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at xgm.de Sat Mar 25 13:46:52 2017 From: mailinglists at xgm.de (Florian Lindner) Date: Sat, 25 Mar 2017 18:46:52 +0100 Subject: [Numpy-discussion] Optimize evaluation of function on matrix Message-ID: <0227278d-48e9-4ce5-bd34-2a95f9765574@xgm.de> Hello, I have this function: def eval_BF(self, meshA, meshB): """ Evaluates single BF or list of BFs on the meshes. """ if type(self.basisfunction) is list: A = np.empty((len(meshA), len(meshB))) for i, row in enumerate(meshA): for j, col in enumerate(meshB): A[i, j] = self.basisfunction[j](row - col) else: mgrid = np.meshgrid(meshB, meshA) A = self.basisfunction( np.abs(mgrid[0] - mgrid[1]) ) return A meshA and meshB are 1-dimensional numpy arrays. self.basisfunction is e.g. def Gaussian(radius, shape): """ Gaussian Basis Function """ return np.exp( -np.power(shape*abs(radius), 2)) or a list of partial instantations of such functions (from functools.partial). How can I optimize eval_BF? Esp. in the case of basisfunction being a list. Thanks! Florian From sebastian at sipsolutions.net Sat Mar 25 17:31:12 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sat, 25 Mar 2017 22:31:12 +0100 Subject: [Numpy-discussion] Optimize evaluation of function on matrix In-Reply-To: <0227278d-48e9-4ce5-bd34-2a95f9765574@xgm.de> References: <0227278d-48e9-4ce5-bd34-2a95f9765574@xgm.de> Message-ID: <1490477472.21453.1.camel@sipsolutions.net> On Sat, 2017-03-25 at 18:46 +0100, Florian Lindner wrote: > Hello, > > I have this function: > > def eval_BF(self, meshA, meshB): > ????????""" Evaluates single BF or list of BFs on the meshes. """ > ????????if type(self.basisfunction) is list: > ????????????A = np.empty((len(meshA), len(meshB))) > ????????????for i, row in enumerate(meshA): > ????????????????for j, col in enumerate(meshB): > ????????????????????A[i, j] = self.basisfunction[j](row - col) > ????????else: > ????????????mgrid = np.meshgrid(meshB, meshA) > ????????????A = self.basisfunction( np.abs(mgrid[0] - mgrid[1]) ) > ????????return A > > > meshA and meshB are 1-dimensional numpy arrays. self.basisfunction is > e.g. > > def Gaussian(radius, shape): > ????""" Gaussian Basis Function """ > ????return np.exp( -np.power(shape*abs(radius), 2)) > > > or a list of partial instantations of such functions (from > functools.partial). > > How can I optimize eval_BF? Esp. in the case of basisfunction being a > list. > Are you sure you need to optimize it? If they have a couple of hundred elements or so for each row, the math is probably the problem and most of that might be the `exp`. You can get rid of the `row` loop though in case row if an individual row is a pretty small array. To be honest, I am a bit surprised that its a problem, since "basis function" sounds a bit like you have to do this once and then use the result many times. - Sebastian > Thanks! > Florian > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From ralf.gommers at gmail.com Sun Mar 26 04:05:47 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 26 Mar 2017 21:05:47 +1300 Subject: [Numpy-discussion] migration of all scipy.org mailing lists In-Reply-To: References: Message-ID: On Fri, Mar 24, 2017 at 9:57 PM, Ralf Gommers wrote: > On Fri, Mar 24, 2017 at 12:27 AM, Neal Becker wrote: > >> Ralf Gommers wrote: >> >> > On Thu, Mar 23, 2017 at 12:18 AM, Neal Becker >> wrote: >> > >> >> Has anyone taken care of notifying gmane about this? >> >> >> > >> > We will have to update this info in quite a few places after the move is >> > done. Including Gmane, although that site hasn't been working for half a >> > year so is pretty low on the priority list. >> > >> > Ralf >> >> I'm reading/writing to you via gmane, so I think it is working :) >> > > No it's not, archives are the key feature of Gmane (and what we link to > from http://scipy.org/scipylib/mailing-lists.html) and those haven't been > working since last September. See https://lars.ingebrigtsen.no/ > 2016/07/28/the-end-of-gmane/ for why. > > > > Your mail forwarding still happens to work, but that's not nearly as > interesting a feature. Since Gmane is more or less unmaintained and at the > moment http://gmane.org/ gives me a blank page, I don't think I'll bother > to contact them (unless archives come back). > The move is complete, and I have updated the mailing list addresses at: - http://scipy.org/scipylib/mailing-lists.html (done) - numpy codebase (PR https://github.com/numpy/numpy/pull/8840) - scipy codebase (PR https://github.com/scipy/scipy/pull/7226) - Nabble (opened issue http://support.nabble.com/All-scipy-org-mailing-list-moved-to-python-org-td7597902.html), also for ipython-dev If anyone knows of other places, please let me know (or fix it). Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Sun Mar 26 09:53:51 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sun, 26 Mar 2017 15:53:51 +0200 Subject: [Numpy-discussion] heads up: gufuncs on empty arrays and NpyIter removal of empty axis Message-ID: <1490536431.4266.4.camel@sipsolutions.net> Hi all, just a small heads up for gufunc hackers and low level iterator users. We will probably very soon put in a commit into master that will allow the removal of empty axis from NpyIter/nditer, effectively removing the error: "ValueError: cannot remove a zero-sized axis from an iterator" and allowing: ``` arr = np.zeros((100, 0)) it = np.nditer((arr,), ???????????????flags=["zerosize_ok", "multi_index"]) it.remove_axis(1) ``` As a follow up step, we also allow that gufuncs may be called with empty inner loop sizes. In some cases that may mean that your gufuncs may need special handling for lets say: ``` arr = np.zeros((100, 0)) # note the 0 dimension. my_gufunc(arr) ``` If this creates problems for you, please tell, so that we can slow down or undo the change. As an example, we have a matrix_multiply gufunc for testing purpose, which did not zero out the output for the case of `matrix_multiply(np.ones((10, 0)), np.ones((0, 10)))`. So this could turn code that errored out for weird reasons into wrong results in rare cases. - Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From kirillbalunov at gmail.com Sun Mar 26 12:44:11 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Sun, 26 Mar 2017 19:44:11 +0300 Subject: [Numpy-discussion] Structured array creation with list of lists and others In-Reply-To: References: Message-ID: Allan thank you for your draft! I agree with you that (not in mine ) in general case it would be hard to resolve all corner cases. Also I think if someone read numpy reference linearly, he/she will have some insight that list of tuples are necessary (but it was not my case). For me one problem is that in some cases numpy allows a lot freedom, but in other it is unnecessarily strict. Another one is exception messages (but this is certainly subjective). 2017-03-24 19:48 GMT+03:00 Allan Haldane : > On 03/23/2017 02:16 PM, Kirill Balunov wrote: > > It was the first time I tried to create a structured array in numpy. > > Usually I use pandas for heterogeneous arrays, but it is one more > > dependency to my project. > > > > It took me some time (really, much more than some), to understand the > > problem with structured array creation. As example: > > > > I had list of list of this kind: > > b=[[ 1, 10.3, 12.1, 2.12 ],...] > > > > And tried: > > np.array(b, dtype='i4,f4,f4,f4') > > > > Which raises some weird exception: > > TypeError: a bytes-like object is required, not 'int' > > > > Two hours later I found that I need list of tuples. I didn't find any > help > > in documentation and could not realize that the problem with the inner > > lists... > > > > Why there is such restriction - 'list of tuples' to create structured > > array? What is the idea behind that, why not list of lists, or tuple of > > lists or ...? > > > > Also the exception does not help at all... > > p.s.: It looks like that dtype also accepts only list of tuples. But I > can > > not catch the idea for this restrictions. > > > > The problem is that numpy needs to distinguish between multidimensional > arrays and structured elements. A "list of lists" will often trigger > numpy's broadcasting rules, which is not what you want here. > > For instance, should numpy interpret your input list as a 2d array of > dimension Lx4 containing integer elements, or a 1d array of length L of > structs with 4 fields? > > In this particular case maybe numpy could, in principle, figure it out > from what you gave it by calculating that the innermost dimension is > the same length as the number of fields. But there are other cases (such > as assignment) where similar ambiguities arise that are harder to > resolve. So to preserve our sanity we want to require that structures be > formatted as tuples all the time. > > I have a draft of potential updated structured array docs you can read > here: > https://gist.github.com/ahaldane/7d1873d33d4d0f80ba7a54ccf1052eee > > See the section "Assignment from Python Native Types (Tuples)", which > hopefully better warns that tuples are needed. Let me know if you think > something is missing from the draft. > > (WARNING: the section about multi-field assignment in the doc draft is > incorrect for current numpy - that's what I'm proposing for the next > release. The rest of the docs are accurate for current numpy) > > Agreed that the error message could be changed. > > Allan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Mar 27 04:41:35 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 27 Mar 2017 21:41:35 +1300 Subject: [Numpy-discussion] Fwd: [numfocus] Grants up to $3k available to NumFOCUS projects (sponsored & affiliated) In-Reply-To: References: <1489688042-5554705.54580375.fv2GIDbTc031721@rs159.luxsci.com> Message-ID: Hi all, For those who did not see the call on the NumFOCUS list, the below may be of interest. I also have a proposal idea: a redesign of numpy.org. Our website is very poor, both in terms of content and design. If a good designer spends 40 hours on it, that should be enough to change it into something nice (or at least not embarassing). What do you think? Other ideas? Cheers, Ralf ---------- Forwarded message ---------- From: Gina Helfrich Date: Fri, Mar 17, 2017 at 10:51 AM Subject: Re: [numfocus] Grants up to $3k available to NumFOCUS projects (sponsored & affiliated) To: numfocus at googlegroups.com, devs at numfocus.org There is no specific template, but proposals should be kept under 2 pages. Maximum 1 submission per project. Required elements of the proposal are: - title - project description - benefit to project/community - project team - and budget Submit proposals to info at numfocus.org Best, Gina On Thu, Mar 16, 2017 at 1:13 PM, Bob Carpenter wrote: > Is there a format you're expecting for proposals? > > Are applications limited to one per project? > > Hard to imagine a project won't have something to do with $3K, > so I imagine you'll get applications from all of the projects if > the proposals take less than an hour or two to put together. > > - Bob > > > On Mar 16, 2017, at 12:14 PM, Gina Helfrich wrote: > > > > Call for Proposals - Small Development Grants > > > > NumFOCUS is asking for proposals from its sponsored and affiliated > projects for targeted small development projects with a clear benefit to > those projects. This call is motivated by the success of our 2016 > end-of-year fundraising drive; we want to direct the donated funds to our > projects in a way that has impact and visibility to donors and the wider > community. > > > > There are no restrictions on what the funding can be used for. Whether > it?s code development, documentation work, an educational, sustainability > or diversity initiative, or yet something else, we trust the projects > themselves to understand what they need and explain that need in the > proposal. > > > > Available Funding: > > ? Up to $3,000 per proposal > > ? Allocated funding is $9,000; depending on the number and quality > of proposals this may be adjusted up or down. > > > > Eligibility: > > ? Proposals must be approved by the leadership of a NumFOCUS > sponsored or affiliated project. > > ? Proposed work must have a clear outcome, achievable within 2017. > > ? The call is open to applicants from any nationality and can be > performed at any university, institute or business worldwide (US export > laws permitting). > > > > Timeline: > > ? Mid-March 2017: Call for Proposals released > > ? 3 April 2017: deadline for proposal submissions > > ? 17 April: successful proposals announced > > > > -- > > Dr. Gina Helfrich > > Communications Director, NumFOCUS > > gina at numfocus.org > > 512-222-5449 > > > > > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "NumFOCUS" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to numfocus+unsubscribe at googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "NumFOCUS" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to numfocus+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Dr. Gina Helfrich Communications Director, NumFOCUS gina at numfocus.org 512-222-5449 <(512)%20222-5449> -- You received this message because you are subscribed to the Google Groups "NumFOCUS" group. To unsubscribe from this group and stop receiving emails from it, send an email to numfocus+unsubscribe at googlegroups.com. For more options, visit https://groups.google.com/d/optout. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Mon Mar 27 06:33:01 2017 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 27 Mar 2017 12:33:01 +0200 Subject: [Numpy-discussion] Fwd: [numfocus] Grants up to $3k available to NumFOCUS projects (sponsored & affiliated) In-Reply-To: References: <1489688042-5554705.54580375.fv2GIDbTc031721@rs159.luxsci.com> Message-ID: <78cad834-ff24-3a21-ed14-912309d8089d@googlemail.com> I have two ideas under one big important topic: make numpy python3 compatible. The first fits pretty well with the grant size and nobody wants to do it for free: - fix our text IO functions under python3 and support multiple encodings, not only latin1. Reasonably simple to do, slap encoding arguments on the functions, generate test cases and somehow keep backward compatibility. Some prelimary unfinished work is in https://github.com/numpy/numpy/pull/4208 - add ascii/latin1 dtype to support a compact python3 string array, deprecate 's' dtype which has different meaning in python2 and 3 This one is probably too big for 3k though. On 27.03.2017 10:41, Ralf Gommers wrote: > Hi all, > > For those who did not see the call on the NumFOCUS list, the below may > be of interest. > > I also have a proposal idea: a redesign of numpy.org . > Our website is very poor, both in terms of content and design. If a good > designer spends 40 hours on it, that should be enough to change it into > something nice (or at least not embarassing). > > What do you think? Other ideas? > > Cheers, > Ralf > > > > > ---------- Forwarded message ---------- > From: *Gina Helfrich* > > Date: Fri, Mar 17, 2017 at 10:51 AM > Subject: Re: [numfocus] Grants up to $3k available to NumFOCUS projects > (sponsored & affiliated) > To: numfocus at googlegroups.com , > devs at numfocus.org > > > There is no specific template, but proposals should be kept under 2 > pages. Maximum 1 submission per project. > > Required elements of the proposal are: > > * title > * project description > * benefit to project/community > * project team > * and budget > > Submit proposals to info at numfocus.org > > Best, > Gina > > On Thu, Mar 16, 2017 at 1:13 PM, Bob Carpenter > wrote: > > Is there a format you're expecting for proposals? > > Are applications limited to one per project? > > Hard to imagine a project won't have something to do with $3K, > so I imagine you'll get applications from all of the projects if > the proposals take less than an hour or two to put together. > > - Bob > > > On Mar 16, 2017, at 12:14 PM, Gina Helfrich > wrote: > > > > Call for Proposals - Small Development Grants > > > > NumFOCUS is asking for proposals from its sponsored and affiliated > projects for targeted small development projects with a clear > benefit to those projects. This call is motivated by the success of > our 2016 end-of-year fundraising drive; we want to direct the > donated funds to our projects in a way that has impact and > visibility to donors and the wider community. > > > > There are no restrictions on what the funding can be used for. > Whether it?s code development, documentation work, an educational, > sustainability or diversity initiative, or yet something else, we > trust the projects themselves to understand what they need and > explain that need in the proposal. > > > > Available Funding: > > ? Up to $3,000 per proposal > > ? Allocated funding is $9,000; depending on the number and > quality of proposals this may be adjusted up or down. > > > > Eligibility: > > ? Proposals must be approved by the leadership of a NumFOCUS > sponsored or affiliated project. > > ? Proposed work must have a clear outcome, achievable within > 2017. > > ? The call is open to applicants from any nationality and > can be performed at any university, institute or business worldwide > (US export laws permitting). > > > > Timeline: > > ? Mid-March 2017: Call for Proposals released > > ? 3 April 2017: deadline for proposal submissions > > ? 17 April: successful proposals announced > > > > -- > > Dr. Gina Helfrich > > Communications Director, NumFOCUS > > gina at numfocus.org > > 512-222-5449 > > > > > > > > > > -- > > You received this message because you are subscribed to the Google Groups "NumFOCUS" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to numfocus+unsubscribe at googlegroups.com > . > > For more options, visit https://groups.google.com/d/optout . > > -- > You received this message because you are subscribed to the Google > Groups "NumFOCUS" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to numfocus+unsubscribe at googlegroups.com > . > For more options, visit https://groups.google.com/d/optout > . > > > > > -- > Dr. Gina Helfrich > Communications Director, NumFOCUS > gina at numfocus.org > 512-222-5449 > > > > > -- > You received this message because you are subscribed to the Google > Groups "NumFOCUS" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to numfocus+unsubscribe at googlegroups.com > . > For more options, visit https://groups.google.com/d/optout > . > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > From ralf.gommers at gmail.com Mon Mar 27 06:42:59 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 27 Mar 2017 23:42:59 +1300 Subject: [Numpy-discussion] Fwd: [numfocus] Grants up to $3k available to NumFOCUS projects (sponsored & affiliated) In-Reply-To: <78cad834-ff24-3a21-ed14-912309d8089d@googlemail.com> References: <1489688042-5554705.54580375.fv2GIDbTc031721@rs159.luxsci.com> <78cad834-ff24-3a21-ed14-912309d8089d@googlemail.com> Message-ID: On Mon, Mar 27, 2017 at 11:33 PM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > I have two ideas under one big important topic: make numpy python3 > compatible. > > The first fits pretty well with the grant size and nobody wants to do it > for free: > - fix our text IO functions under python3 and support multiple > encodings, not only latin1. > Reasonably simple to do, slap encoding arguments on the functions, > generate test cases and somehow keep backward compatibility. Some > prelimary unfinished work is in https://github.com/numpy/numpy/pull/4208 I like that idea, it's a recurring pain point. Are you interested to work on it, or are you thinking to advertise the idea here to see if anyone steps up? - add ascii/latin1 dtype to support a compact python3 string array, > deprecate 's' dtype which has different meaning in python2 and 3 > This one is probably too big for 3k though. > Agreed, that won't fit in such a small grant. Ralf > > On 27.03.2017 10:41, Ralf Gommers wrote: > > Hi all, > > > > For those who did not see the call on the NumFOCUS list, the below may > > be of interest. > > > > I also have a proposal idea: a redesign of numpy.org . > > Our website is very poor, both in terms of content and design. If a good > > designer spends 40 hours on it, that should be enough to change it into > > something nice (or at least not embarassing). > > > > What do you think? Other ideas? > > > > Cheers, > > Ralf > > > > > > > > > > ---------- Forwarded message ---------- > > From: *Gina Helfrich* > > > Date: Fri, Mar 17, 2017 at 10:51 AM > > Subject: Re: [numfocus] Grants up to $3k available to NumFOCUS projects > > (sponsored & affiliated) > > To: numfocus at googlegroups.com , > > devs at numfocus.org > > > > > > There is no specific template, but proposals should be kept under 2 > > pages. Maximum 1 submission per project. > > > > Required elements of the proposal are: > > > > * title > > * project description > > * benefit to project/community > > * project team > > * and budget > > > > Submit proposals to info at numfocus.org > > > > Best, > > Gina > > > > On Thu, Mar 16, 2017 at 1:13 PM, Bob Carpenter > > wrote: > > > > Is there a format you're expecting for proposals? > > > > Are applications limited to one per project? > > > > Hard to imagine a project won't have something to do with $3K, > > so I imagine you'll get applications from all of the projects if > > the proposals take less than an hour or two to put together. > > > > - Bob > > > > > On Mar 16, 2017, at 12:14 PM, Gina Helfrich > > wrote: > > > > > > Call for Proposals - Small Development Grants > > > > > > NumFOCUS is asking for proposals from its sponsored and affiliated > > projects for targeted small development projects with a clear > > benefit to those projects. This call is motivated by the success of > > our 2016 end-of-year fundraising drive; we want to direct the > > donated funds to our projects in a way that has impact and > > visibility to donors and the wider community. > > > > > > There are no restrictions on what the funding can be used for. > > Whether it?s code development, documentation work, an educational, > > sustainability or diversity initiative, or yet something else, we > > trust the projects themselves to understand what they need and > > explain that need in the proposal. > > > > > > Available Funding: > > > ? Up to $3,000 per proposal > > > ? Allocated funding is $9,000; depending on the number and > > quality of proposals this may be adjusted up or down. > > > > > > Eligibility: > > > ? Proposals must be approved by the leadership of a NumFOCUS > > sponsored or affiliated project. > > > ? Proposed work must have a clear outcome, achievable within > > 2017. > > > ? The call is open to applicants from any nationality and > > can be performed at any university, institute or business worldwide > > (US export laws permitting). > > > > > > Timeline: > > > ? Mid-March 2017: Call for Proposals released > > > ? 3 April 2017: deadline for proposal submissions > > > ? 17 April: successful proposals announced > > > > > > -- > > > Dr. Gina Helfrich > > > Communications Director, NumFOCUS > > > gina at numfocus.org > > > 512-222-5449 > > > > > > > > > > > > > > > -- > > > You received this message because you are subscribed to the Google > Groups "NumFOCUS" group. > > > To unsubscribe from this group and stop receiving emails from it, > send an email to numfocus+unsubscribe at googlegroups.com > > . > > > For more options, visit https://groups.google.com/d/optout < > https://groups.google.com/d/optout>. > > > > -- > > You received this message because you are subscribed to the Google > > Groups "NumFOCUS" group. > > To unsubscribe from this group and stop receiving emails from it, > > send an email to numfocus+unsubscribe at googlegroups.com > > . > > For more options, visit https://groups.google.com/d/optout > > . > > > > > > > > > > -- > > Dr. Gina Helfrich > > Communications Director, NumFOCUS > > gina at numfocus.org > > 512-222-5449 > > > > > > > > > > -- > > You received this message because you are subscribed to the Google > > Groups "NumFOCUS" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to numfocus+unsubscribe at googlegroups.com > > . > > For more options, visit https://groups.google.com/d/optout > > . > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at xgm.de Mon Mar 27 07:06:21 2017 From: mailinglists at xgm.de (Florian Lindner) Date: Mon, 27 Mar 2017 13:06:21 +0200 Subject: [Numpy-discussion] Optimize evaluation of function on matrix In-Reply-To: <1490477472.21453.1.camel@sipsolutions.net> References: <0227278d-48e9-4ce5-bd34-2a95f9765574@xgm.de> <1490477472.21453.1.camel@sipsolutions.net> Message-ID: Hey, I've timed the two versions, one basisfunction being a function: 1 loop, best of 3: 17.3 s per loop the other one, basisfunction being a list of functions: 1 loop, best of 3: 33.5 s per loop > To be honest, I am a bit surprised that its a problem, since "basis > function" sounds a bit like you have to do this once and then use the > result many times. It's part of a radial basis function interpolation algorithm. Yes, in practice the matrix is filled only once and reused a couple of times, but in my case, which is exploration of parameters for the algorithm, I call eval_BF many times. > You can get rid of the `row` loop though in case row if an individual > row is a pretty small array. Would you elaborate on that? Do you mean that the inner col loop produces an array which is then assigned to the row. But I think it stell need to row loop there. Best, Florian Am 25.03.2017 um 22:31 schrieb Sebastian Berg: > On Sat, 2017-03-25 at 18:46 +0100, Florian Lindner wrote: >> Hello, >> >> I have this function: >> >> def eval_BF(self, meshA, meshB): >> """ Evaluates single BF or list of BFs on the meshes. """ >> if type(self.basisfunction) is list: >> A = np.empty((len(meshA), len(meshB))) >> for i, row in enumerate(meshA): >> for j, col in enumerate(meshB): >> A[i, j] = self.basisfunction[j](row - col) >> else: >> mgrid = np.meshgrid(meshB, meshA) >> A = self.basisfunction( np.abs(mgrid[0] - mgrid[1]) ) >> return A >> >> >> meshA and meshB are 1-dimensional numpy arrays. self.basisfunction is >> e.g. >> >> def Gaussian(radius, shape): >> """ Gaussian Basis Function """ >> return np.exp( -np.power(shape*abs(radius), 2)) >> >> >> or a list of partial instantations of such functions (from >> functools.partial). >> >> How can I optimize eval_BF? Esp. in the case of basisfunction being a >> list. >> > > Are you sure you need to optimize it? If they have a couple of hundred > elements or so for each row, the math is probably the problem and most > of that might be the `exp`. > You can get rid of the `row` loop though in case row if an individual > row is a pretty small array. > > To be honest, I am a bit surprised that its a problem, since "basis > function" sounds a bit like you have to do this once and then use the > result many times. > > - Sebastian > > >> Thanks! >> Florian >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From sebastian at sipsolutions.net Mon Mar 27 10:09:40 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 27 Mar 2017 16:09:40 +0200 Subject: [Numpy-discussion] Optimize evaluation of function on matrix In-Reply-To: References: <0227278d-48e9-4ce5-bd34-2a95f9765574@xgm.de> <1490477472.21453.1.camel@sipsolutions.net> Message-ID: <1490623780.18200.4.camel@sipsolutions.net> On Mon, 2017-03-27 at 13:06 +0200, Florian Lindner wrote: > Hey, > > I've timed the two versions, one basisfunction being a function: > > 1 loop, best of 3: 17.3 s per loop > > the other one, basisfunction being a list of functions: > > 1 loop, best of 3: 33.5 s per loop > > > To be honest, I am a bit surprised that its a problem, since "basis > > function" sounds a bit like you have to do this once and then use > > the > > result many times. > > It's part of a radial basis function interpolation algorithm. Yes, in > practice the matrix is filled only once and reused > a couple of times, but in my case, which is exploration of parameters > for the algorithm, I call eval_BF many times. > > > You can get rid of the `row` loop though in case row if an > > individual > > row is a pretty small array. > > Would you elaborate on that? Do you mean that the inner col loop > produces an array which is then assigned to the row. > But I think it stell need to row loop there. Well, I like to not serve the result, but if you exchange the loops: A = np.empty((len(meshA), len(meshB))) for j, col in enumerate(meshB): for i, row in enumerate(meshA): A[i, j] = self.basisfunction[j](row - col) Then you can see that there is broadcasting magic similar (do not want to use too many brain cells now) to: A = np.empty((len(meshA), len(meshB))) for j, col in enumerate(meshB): # possibly insert np.newaxis/None or a reshape in [??] ????A[:, j] = self.basisfunction[j](meshA[??] - col) - Sebastian > > Best, > Florian > > Am 25.03.2017 um 22:31 schrieb Sebastian Berg: > > On Sat, 2017-03-25 at 18:46 +0100, Florian Lindner wrote: > > > Hello, > > > > > > I have this function: > > > > > > def eval_BF(self, meshA, meshB): > > > ????????""" Evaluates single BF or list of BFs on the meshes. """ > > > ????????if type(self.basisfunction) is list: > > > ????????????A = np.empty((len(meshA), len(meshB))) > > > ????????????for i, row in enumerate(meshA): > > > ????????????????for j, col in enumerate(meshB): > > > ????????????????????A[i, j] = self.basisfunction[j](row - col) > > > ????????else: > > > ????????????mgrid = np.meshgrid(meshB, meshA) > > > ????????????A = self.basisfunction( np.abs(mgrid[0] - mgrid[1]) ) > > > ????????return A > > > > > > > > > meshA and meshB are 1-dimensional numpy arrays. > > > self.basisfunction is > > > e.g. > > > > > > def Gaussian(radius, shape): > > > ????""" Gaussian Basis Function """ > > > ????return np.exp( -np.power(shape*abs(radius), 2)) > > > > > > > > > or a list of partial instantations of such functions (from > > > functools.partial). > > > > > > How can I optimize eval_BF? Esp. in the case of basisfunction > > > being a > > > list. > > > > > > > Are you sure you need to optimize it? If they have a couple of > > hundred > > elements or so for each row, the math is probably the problem and > > most > > of that might be the `exp`. > > You can get rid of the `row` loop though in case row if an > > individual > > row is a pretty small array. > > > > To be honest, I am a bit surprised that its a problem, since "basis > > function" sounds a bit like you have to do this once and then use > > the > > result many times. > > > > - Sebastian > > > > > > > Thanks! > > > Florian > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From chris.barker at noaa.gov Mon Mar 27 11:21:37 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 27 Mar 2017 08:21:37 -0700 Subject: [Numpy-discussion] Fwd: [numfocus] Grants up to $3k available to NumFOCUS projects (sponsored & affiliated) In-Reply-To: <78cad834-ff24-3a21-ed14-912309d8089d@googlemail.com> References: <1489688042-5554705.54580375.fv2GIDbTc031721@rs159.luxsci.com> <78cad834-ff24-3a21-ed14-912309d8089d@googlemail.com> Message-ID: On Mon, Mar 27, 2017 at 3:33 AM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > - add ascii/latin1 dtype to support a compact python3 string array, > deprecate 's' dtype which has different meaning in python2 and 3 > This one is probably too big for 3k though. > probably -- but not THAT big -- it seems pretty straightforward to me. The bigger challenge is deciding what to do -- the bikeshedding -- and the backward incompatibility issues. IIRC, when this came up on the list, there was nothing like consensus on exactly what to do and how to do it. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Mar 27 15:14:59 2017 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 27 Mar 2017 21:14:59 +0200 Subject: [Numpy-discussion] Fwd: [numfocus] Grants up to $3k available to NumFOCUS projects, (sponsored & affiliated) Message-ID: <15778743-2fdd-5fe0-1855-7b616b54dbd8@iki.fi> Mon, 27 Mar 2017 08:21:37 -0700, Chris Barker kirjoitti: > On Mon, Mar 27, 2017 at 3:33 AM, Julian Taylor > wrote: > >> - add ascii/latin1 dtype to support a compact python3 string array, >> deprecate 's' dtype which has different meaning in python2 and 3 This >> one is probably too big for 3k though. > > probably -- but not THAT big -- it seems pretty straightforward to me. > > The bigger challenge is deciding what to do -- the bikeshedding -- and > the backward incompatibility issues. IIRC, when this came up on the > list, there was nothing like consensus on exactly what to do and how > to do it. TBH, I don't see why 's' should be deprecated --- the operation is well-specified (byte strings + null stripping) and has the same meaning in python2 and 3. Of course, a true 1-byte unicode subset string may be more useful type for some applications, so it could indeed be added. -- Pauli Virtanen From pav at iki.fi Mon Mar 27 15:22:29 2017 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 27 Mar 2017 21:22:29 +0200 Subject: [Numpy-discussion] migration of all scipy.org mailing lists Message-ID: <8b1e7706-1d79-1197-f3b0-a4b0a4c9662d@iki.fi> Sun, 26 Mar 2017 21:05:47 +1300, Ralf Gommers kirjoitti: [clip] > The move is complete, and I have updated the mailing list addresses at: > - http://scipy.org/scipylib/mailing-lists.html (done) > - numpy codebase (PR https://github.com/numpy/numpy/pull/8840) > - scipy codebase (PR https://github.com/scipy/scipy/pull/7226) > - Nabble (opened issue > http://support.nabble.com/All-scipy-org-mailing-list-moved-to-python- org-td7597902.html), > also for ipython-dev > > If anyone knows of other places, please let me know (or fix it). Should the Read/Search links be removed from this page: https://scipy.org/scipylib/mailing-lists.html They point to Gmane, which currently appears to just serve a blank page. So maybe Gmane really is going away. The Gmane NNTP server still works, but apparently you cannot post via it because the To-address still points to @scipy.org. Not sure if there is someone who can fix this any more. -- Pauli Virtanen From chris.barker at noaa.gov Mon Mar 27 15:22:55 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 27 Mar 2017 12:22:55 -0700 Subject: [Numpy-discussion] Fwd: [numfocus] Grants up to $3k available to NumFOCUS projects, (sponsored & affiliated) In-Reply-To: <15778743-2fdd-5fe0-1855-7b616b54dbd8@iki.fi> References: <15778743-2fdd-5fe0-1855-7b616b54dbd8@iki.fi> Message-ID: On Mon, Mar 27, 2017 at 12:14 PM, Pauli Virtanen wrote: > > The bigger challenge is deciding what to do -- the bikeshedding -- and > > the backward incompatibility issues. IIRC, when this came up on the > > list, there was nothing like consensus on exactly what to do and how > > to do it. > > TBH, I don't see why 's' should be deprecated --- the operation is > well-specified (byte strings + null stripping) and has the same meaning in > python2 and 3. > exactly -- I don't think there was a consensus on this. > Of course, a true 1-byte unicode subset string may be more useful type for > some applications, so it could indeed be added. > That's the idea -- scientist tend to use a lot of ascii text (or at least one-byte per char text), numy requires each element to be the same number of bytes, so the unicode dtype is 4 btes per char -- seemingly very wasteful. but if you use 's' on py3, you get bytestrings back -- not "text" from a py3 perspective. and aside from backwards compatibility, I see no reason for a 's' dtype that returns a bytes object on py3 -- if it's really binary data, you can use the 'b' dtype. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Mar 27 15:50:39 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 28 Mar 2017 08:50:39 +1300 Subject: [Numpy-discussion] migration of all scipy.org mailing lists In-Reply-To: <8b1e7706-1d79-1197-f3b0-a4b0a4c9662d@iki.fi> References: <8b1e7706-1d79-1197-f3b0-a4b0a4c9662d@iki.fi> Message-ID: On Tue, Mar 28, 2017 at 8:22 AM, Pauli Virtanen wrote: > Sun, 26 Mar 2017 21:05:47 +1300, Ralf Gommers kirjoitti: > [clip] > > The move is complete, and I have updated the mailing list addresses at: > > - http://scipy.org/scipylib/mailing-lists.html (done) > > - numpy codebase (PR https://github.com/numpy/numpy/pull/8840) > > - scipy codebase (PR https://github.com/scipy/scipy/pull/7226) > > - Nabble (opened issue > > http://support.nabble.com/All-scipy-org-mailing-list-moved-to-python- > org-td7597902.html), > > also for ipython-dev > > > > If anyone knows of other places, please let me know (or fix it). > > Should the Read/Search links be removed from this page: > > https://scipy.org/scipylib/mailing-lists.html > > They point to Gmane, which currently appears to just serve a blank page. > So maybe Gmane really is going away. > Yes, I think we should replace them with Nabble (unless someone has a better alternative?). I was planning to rework the mailing-lists page for that and a few other things like explaining bottom posting. Ralf > > The Gmane NNTP server still works, but apparently you cannot post via it > because the To-address still points to @scipy.org. Not sure if there is > someone who can fix this any more. > > -- > Pauli Virtanen > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From harrigan.matthew at gmail.com Mon Mar 27 20:19:24 2017 From: harrigan.matthew at gmail.com (Matthew Harrigan) Date: Mon, 27 Mar 2017 20:19:24 -0400 Subject: [Numpy-discussion] Optimize evaluation of function on matrix In-Reply-To: <0227278d-48e9-4ce5-bd34-2a95f9765574@xgm.de> References: <0227278d-48e9-4ce5-bd34-2a95f9765574@xgm.de> Message-ID: The best way to get good optimized code is to find it. Does this do what you want? https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.Rbf.html For some advice here, first avoid loops in Python and instead stick to ufuncs and broadcasting. It looks like the matrix is symmetric with constant diagonals, so much of the work can possibly be avoided. Finally consider threading over blocks or different basis functions. To dial it up to 11 write low level c code to do exactly what you want, writing a custom ufunc or call it cython. On Mar 25, 2017 2:03 PM, "Florian Lindner" wrote: > Hello, > > I have this function: > > def eval_BF(self, meshA, meshB): > """ Evaluates single BF or list of BFs on the meshes. """ > if type(self.basisfunction) is list: > A = np.empty((len(meshA), len(meshB))) > for i, row in enumerate(meshA): > for j, col in enumerate(meshB): > A[i, j] = self.basisfunction[j](row - col) > else: > mgrid = np.meshgrid(meshB, meshA) > A = self.basisfunction( np.abs(mgrid[0] - mgrid[1]) ) > return A > > > meshA and meshB are 1-dimensional numpy arrays. self.basisfunction is e.g. > > def Gaussian(radius, shape): > """ Gaussian Basis Function """ > return np.exp( -np.power(shape*abs(radius), 2)) > > > or a list of partial instantations of such functions (from > functools.partial). > > How can I optimize eval_BF? Esp. in the case of basisfunction being a list. > > Thanks! > Florian > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Mar 28 05:53:38 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 28 Mar 2017 22:53:38 +1300 Subject: [Numpy-discussion] migration of all scipy.org mailing lists In-Reply-To: References: <8b1e7706-1d79-1197-f3b0-a4b0a4c9662d@iki.fi> Message-ID: On Tue, Mar 28, 2017 at 8:50 AM, Ralf Gommers wrote: > > > On Tue, Mar 28, 2017 at 8:22 AM, Pauli Virtanen wrote: > >> Sun, 26 Mar 2017 21:05:47 +1300, Ralf Gommers kirjoitti: >> [clip] >> > The move is complete, and I have updated the mailing list addresses at: >> > - http://scipy.org/scipylib/mailing-lists.html (done) >> > - numpy codebase (PR https://github.com/numpy/numpy/pull/8840) >> > - scipy codebase (PR https://github.com/scipy/scipy/pull/7226) >> > - Nabble (opened issue >> > http://support.nabble.com/All-scipy-org-mailing-list-moved-to-python- >> org-td7597902.html), >> > also for ipython-dev >> > >> > If anyone knows of other places, please let me know (or fix it). >> >> Should the Read/Search links be removed from this page: >> >> https://scipy.org/scipylib/mailing-lists.html >> >> They point to Gmane, which currently appears to just serve a blank page. >> So maybe Gmane really is going away. >> > > Yes, I think we should replace them with Nabble (unless someone has a > better alternative?). > > I was planning to rework the mailing-lists page for that and a few other > things like explaining bottom posting. > Done in https://github.com/scipy/scipy.org/pull/203. There is no scipy-dev Nabble forum. If anyone is familiar with Nabble (or wants to lbecome familiar with it) and can create that forum, that would be much appreciated. Right now there's no good way to search scipy-dev. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Mar 28 19:16:54 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 28 Mar 2017 16:16:54 -0700 Subject: [Numpy-discussion] Fwd: SciPy2017 Sprints FinAid for sprint leaders/core devs In-Reply-To: References: Message-ID: In case anyone is interested in helping run a NumPy sprint at SciPy this year: ---------- Forwarded message ---------- From: Jonathan Rocher Date: Mon, Mar 27, 2017 at 7:06 AM Subject: SciPy2017 Sprints FinAid for sprint leaders/core devs To: Nathaniel Smith , charlesr.harris at gmail.com, jtaylor.debian at googlemail.com, matthew.brett at gmail.com Dear Nathaniel, Chuck, Julian and Matt, cc: SciPy2017 FinAid and Sprints co-chairs This year, SciPy2017 Sprint and Financial Aid comittees would like like to encourage more Scientific Python leaders to lead sprints at the conference and grow our foundations and the contributor base! As such, we're launching, this year, a new "Sprint Leader FinAid", which will cover two nights of lodging for sprint leaders who can offer their knowledge of a package central to our community and pedagogical skills to help make sprints more accessible and grow the contributor pool. Would you be interested in applying? If not, would you be kind enough to help us encourage qualified individuals to lead a sprint for NumPy? Please feel free to forward them our emails and this note! Cheers, Scott Collis, Eric Ma & Jonathan Rocher -- Jonathan Rocher Austin TX, USA twitter:@jonrocher, linkedin:jonathanrocher ------------------------------------------------------------- -- Nathaniel J. Smith -- https://vorpus.org From charlesr.harris at gmail.com Wed Mar 29 16:01:19 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 29 Mar 2017 14:01:19 -0600 Subject: [Numpy-discussion] Fwd: SciPy2017 Sprints FinAid for sprint leaders/core devs In-Reply-To: References: Message-ID: On Tue, Mar 28, 2017 at 5:16 PM, Nathaniel Smith wrote: > In case anyone is interested in helping run a NumPy sprint at SciPy this > year: > I haven't found numpy sprints to be very productive. However, I think it would be useful if we could have a developers meeting sometime this year. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Wed Mar 29 17:55:50 2017 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Wed, 29 Mar 2017 14:55:50 -0700 Subject: [Numpy-discussion] Fwd: SciPy2017 Sprints FinAid for sprint leaders/core devs In-Reply-To: References: Message-ID: <1490824550.2742751.927932680.77CF511C@webmail.messagingengine.com> On Wed, Mar 29, 2017, at 13:01, Charles R Harris wrote: > On Tue, Mar 28, 2017 at 5:16 PM, Nathaniel Smith > wrote: >> In case anyone is interested in helping run a NumPy sprint at SciPy >> this year: > > I haven't found numpy sprints to be very productive. However, I think > it would be useful if we could have a developers meeting sometime > this year. Yes, I think it's helpful to think of the sprints as an onboarding opportunity, rather than as a focused working session. Many of the scikit- image core contributors joined the team this way, at least. St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Mar 30 05:46:59 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 30 Mar 2017 22:46:59 +1300 Subject: [Numpy-discussion] Fwd: SciPy2017 Sprints FinAid for sprint leaders/core devs In-Reply-To: <1490824550.2742751.927932680.77CF511C@webmail.messagingengine.com> References: <1490824550.2742751.927932680.77CF511C@webmail.messagingengine.com> Message-ID: On Thu, Mar 30, 2017 at 10:55 AM, Stefan van der Walt wrote: > On Wed, Mar 29, 2017, at 13:01, Charles R Harris wrote: > > On Tue, Mar 28, 2017 at 5:16 PM, Nathaniel Smith wrote: > > In case anyone is interested in helping run a NumPy sprint at SciPy this > year: > > > I haven't found numpy sprints to be very productive. However, I think it > would be useful if we could have a developers meeting sometime this year. > > > Yes, I think it's helpful to think of the sprints as an onboarding > opportunity, rather than as a focused working session. Many of the > scikit-image core contributors joined the team this way, at least. > > Agreed, and I would call that productive. Getting even one new maintainer involved is worth organizing multiple sprints for. That said, also +1 to a developer meeting this year. It'd be good if we could combine it with the NumFOCUS summit or a relevant conference in the second half of the year. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.haessig at crans.org Thu Mar 30 07:31:19 2017 From: pierre.haessig at crans.org (Pierre Haessig) Date: Thu, 30 Mar 2017 13:31:19 +0200 Subject: [Numpy-discussion] speed of random number generator compared to Julia Message-ID: <0db9ba2d-cb97-821e-62e5-b1b922c785a8@crans.org> Hello, I have a simulation code which uses intensively random number generation (e.g. normal rng). I wanted to compare the performance of this simulation across some Numpy implementations and some Julia implementation. In the end, in both languages, the respective best performing implementations are dominated by the rng, but Julia wins by a factor of 4-5, because the rng is 4-5x faster. Here are some timing results: I*n IPython (Python 3.5.2 from Anaconda)* import numpy as np N = 10**6 %timeit np.random.normal(size=N) 10 loops, best of 3: 37.1 ms per loop %timeit np.random.uniform(size=N) 100 loops, best of 3: 10.2 ms per loop *In Julia (0.4.7 x86_64 from Debian testing)* N = 10^6 @time randn(N); 0.007802 seconds (6 allocations: 7.630 MB) @time rand(N); 0.002059 seconds (8 allocations: 7.630 MB) (with some variations between trials) Results are consistent in the sense that generating Gaussian numbers is 3-4 times slower than uniform in Python and in Julia. But how come Julia is 4-5x faster since Numpy uses C implementation for the entire process ? (Mersenne Twister -> uniform double -> Box-Muller transform to get a Gaussian https://github.com/numpy/numpy/blob/master/numpy/random/mtrand/randomkit.c). Also I noticed that Julia uses a different algorithm (Ziggurat Method from Marsaglia and Tsang , https://github.com/JuliaLang/julia/blob/master/base/random.jl#L700) but this doesn't explain the difference for uniform rng. best, Pierre -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Thu Mar 30 18:59:24 2017 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 30 Mar 2017 18:59:24 -0400 Subject: [Numpy-discussion] Fwd: SciPy2017 Sprints FinAid for sprint leaders/core devs In-Reply-To: References: <1490824550.2742751.927932680.77CF511C@webmail.messagingengine.com> Message-ID: <1490914764.2758.10.camel@sipsolutions.net> On Thu, 2017-03-30 at 22:46 +1300, Ralf Gommers wrote: > > > Agreed, and I would call that productive. Getting even one new > maintainer involved is worth organizing multiple sprints for. > > That said, also +1 to a developer meeting this year. It'd be good if > we could combine it with the NumFOCUS summit or a relevant conference > in the second half of the year. Would be good, even if there is nothing big going on. Can we gather possible dates and possible (personal) preferences? Here is a start: * SciPy (Austin, TX): July 10-16 * EuroScipy (Germany): August 23-27 *?NumFocus Summit? * PyData Events?? Personally, I probably can't make longer trips until some time in July. time around then). We won't find a perfect time anyway probably, so personal preferences or not, whoever is willing to organize a bit can decide on the time and place as far as I am concerned :). - Sebastian > > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: This is a digitally signed message part URL: From charlesr.harris at gmail.com Thu Mar 30 22:40:06 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 30 Mar 2017 20:40:06 -0600 Subject: [Numpy-discussion] __array_ufunc__ counting down to launch, T-24 hrs. Message-ID: Hi All, Just a note that the __array_ufunc__ PR is ready to merge. If you are interested, you can review here . Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Mar 31 01:08:30 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 30 Mar 2017 22:08:30 -0700 Subject: [Numpy-discussion] __array_ufunc__ counting down to launch, T-24 hrs. In-Reply-To: References: Message-ID: On Thu, Mar 30, 2017 at 7:40 PM, Charles R Harris wrote: > Hi All, > > Just a note that the __array_ufunc__ PR is ready to merge. If you are > interested, you can review here. I want to get this in too, but 24 hours seems like a very short deadline for getting feedback on such a large and complex change? I'm pretty sure the ndarray.__array_ufunc__ code that was just added a few hours ago is wrong (see comments on the diff)... My main comment, also relevant to the kind of high-level discussion we tend to use the mailing list for: https://github.com/numpy/numpy/pull/8247#issuecomment-290616432 -n -- Nathaniel J. Smith -- https://vorpus.org From ralf.gommers at gmail.com Fri Mar 31 02:39:57 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 31 Mar 2017 19:39:57 +1300 Subject: [Numpy-discussion] Fwd: SciPy2017 Sprints FinAid for sprint leaders/core devs In-Reply-To: <1490914764.2758.10.camel@sipsolutions.net> References: <1490824550.2742751.927932680.77CF511C@webmail.messagingengine.com> <1490914764.2758.10.camel@sipsolutions.net> Message-ID: On Fri, Mar 31, 2017 at 11:59 AM, Sebastian Berg wrote: > On Thu, 2017-03-30 at 22:46 +1300, Ralf Gommers wrote: > > > > > > > Agreed, and I would call that productive. Getting even one new > > maintainer involved is worth organizing multiple sprints for. > > > > That said, also +1 to a developer meeting this year. It'd be good if > > we could combine it with the NumFOCUS summit or a relevant conference > > in the second half of the year. > > Would be good, even if there is nothing big going on. > > Can we gather possible dates and possible (personal) preferences? Here > is a start: > > * SciPy (Austin, TX): July 10-16 > * EuroScipy (Germany): August 23-27 > * NumFocus Summit? > Austin, October (exact date TBD). I intend to plan a longer trip to the US around this summit, and I think at least one other core dev should go there. So this one has my preference. > * PyData Events?? > Sticking to the ones in the second half of the year and in the US or Western Europe: PyData Berlin, June 30 - July 2 PyData Seattle July 5 -7 Other options: JupyterCon, New York, August 22-25 Strata Data Conference, September 25-28 > > Personally, I probably can't make longer trips until some time in July. > time around then). Same here. Ralf > We won't find a perfect time anyway probably, so > personal preferences or not, whoever is willing to organize a bit can > decide on the time and place as far as I am concerned :). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Mar 31 04:14:55 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 31 Mar 2017 21:14:55 +1300 Subject: [Numpy-discussion] Fwd: [numfocus] Grants up to $3k available to NumFOCUS projects (sponsored & affiliated) In-Reply-To: References: <1489688042-5554705.54580375.fv2GIDbTc031721@rs159.luxsci.com> <78cad834-ff24-3a21-ed14-912309d8089d@googlemail.com> Message-ID: On Mon, Mar 27, 2017 at 11:42 PM, Ralf Gommers wrote: > > > On Mon, Mar 27, 2017 at 11:33 PM, Julian Taylor < > jtaylor.debian at googlemail.com> wrote: > >> I have two ideas under one big important topic: make numpy python3 >> compatible. >> >> The first fits pretty well with the grant size and nobody wants to do it >> for free: >> - fix our text IO functions under python3 and support multiple >> encodings, not only latin1. >> Reasonably simple to do, slap encoding arguments on the functions, >> generate test cases and somehow keep backward compatibility. Some >> prelimary unfinished work is in https://github.com/numpy/numpy/pull/4208 > > > I like that idea, it's a recurring pain point. Are you interested to work > on it, or are you thinking to advertise the idea here to see if anyone > steps up? > More thoughts on this anyone? Or preferences for this idea or the numpy.org one? Submission deadline is April 3rd and we can only put in one proposal this time, so we need to (a) make a choice between these ideas, and (b) write up a proposal. If there's not enough replies to this so the choice is clear cut, I will send out a poll to the core devs. Ralf > > > - add ascii/latin1 dtype to support a compact python3 string array, >> deprecate 's' dtype which has different meaning in python2 and 3 >> This one is probably too big for 3k though. >> > > Agreed, that won't fit in such a small grant. > > Ralf > > > >> >> On 27.03.2017 10:41, Ralf Gommers wrote: >> > Hi all, >> > >> > For those who did not see the call on the NumFOCUS list, the below may >> > be of interest. >> > >> > I also have a proposal idea: a redesign of numpy.org > >. >> > Our website is very poor, both in terms of content and design. If a good >> > designer spends 40 hours on it, that should be enough to change it into >> > something nice (or at least not embarassing). >> > >> > What do you think? Other ideas? >> > >> > Cheers, >> > Ralf >> > >> > >> > >> > >> > ---------- Forwarded message ---------- >> > From: *Gina Helfrich* > >> > Date: Fri, Mar 17, 2017 at 10:51 AM >> > Subject: Re: [numfocus] Grants up to $3k available to NumFOCUS projects >> > (sponsored & affiliated) >> > To: numfocus at googlegroups.com , >> > devs at numfocus.org >> > >> > >> > There is no specific template, but proposals should be kept under 2 >> > pages. Maximum 1 submission per project. >> > >> > Required elements of the proposal are: >> > >> > * title >> > * project description >> > * benefit to project/community >> > * project team >> > * and budget >> > >> > Submit proposals to info at numfocus.org >> > >> > Best, >> > Gina >> > >> > On Thu, Mar 16, 2017 at 1:13 PM, Bob Carpenter > > > wrote: >> > >> > Is there a format you're expecting for proposals? >> > >> > Are applications limited to one per project? >> > >> > Hard to imagine a project won't have something to do with $3K, >> > so I imagine you'll get applications from all of the projects if >> > the proposals take less than an hour or two to put together. >> > >> > - Bob >> > >> > > On Mar 16, 2017, at 12:14 PM, Gina Helfrich > > > wrote: >> > > >> > > Call for Proposals - Small Development Grants >> > > >> > > NumFOCUS is asking for proposals from its sponsored and affiliated >> > projects for targeted small development projects with a clear >> > benefit to those projects. This call is motivated by the success of >> > our 2016 end-of-year fundraising drive; we want to direct the >> > donated funds to our projects in a way that has impact and >> > visibility to donors and the wider community. >> > > >> > > There are no restrictions on what the funding can be used for. >> > Whether it?s code development, documentation work, an educational, >> > sustainability or diversity initiative, or yet something else, we >> > trust the projects themselves to understand what they need and >> > explain that need in the proposal. >> > > >> > > Available Funding: >> > > ? Up to $3,000 per proposal >> > > ? Allocated funding is $9,000; depending on the number and >> > quality of proposals this may be adjusted up or down. >> > > >> > > Eligibility: >> > > ? Proposals must be approved by the leadership of a NumFOCUS >> > sponsored or affiliated project. >> > > ? Proposed work must have a clear outcome, achievable within >> > 2017. >> > > ? The call is open to applicants from any nationality and >> > can be performed at any university, institute or business worldwide >> > (US export laws permitting). >> > > >> > > Timeline: >> > > ? Mid-March 2017: Call for Proposals released >> > > ? 3 April 2017: deadline for proposal submissions >> > > ? 17 April: successful proposals announced >> > > >> > > -- >> > > Dr. Gina Helfrich >> > > Communications Director, NumFOCUS >> > > gina at numfocus.org >> > > 512-222-5449 >> > > >> > > >> > > >> > > >> > > -- >> > > You received this message because you are subscribed to the >> Google Groups "NumFOCUS" group. >> > > To unsubscribe from this group and stop receiving emails from it, >> send an email to numfocus+unsubscribe at googlegroups.com >> > . >> > > For more options, visit https://groups.google.com/d/optout < >> https://groups.google.com/d/optout>. >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "NumFOCUS" group. >> > To unsubscribe from this group and stop receiving emails from it, >> > send an email to numfocus+unsubscribe at googlegroups.com >> > . >> > For more options, visit https://groups.google.com/d/optout >> > . >> > >> > >> > >> > >> > -- >> > Dr. Gina Helfrich >> > Communications Director, NumFOCUS >> > gina at numfocus.org >> > 512-222-5449 >> > >> > >> > >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "NumFOCUS" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an email to numfocus+unsubscribe at googlegroups.com >> > . >> > For more options, visit https://groups.google.com/d/optout >> > . >> > >> > >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at python.org >> > https://mail.python.org/mailman/listinfo/numpy-discussion >> > >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at xgm.de Fri Mar 31 08:26:15 2017 From: mailinglists at xgm.de (Florian Lindner) Date: Fri, 31 Mar 2017 14:26:15 +0200 Subject: [Numpy-discussion] Optimize evaluation of function on matrix In-Reply-To: <1490623780.18200.4.camel@sipsolutions.net> References: <0227278d-48e9-4ce5-bd34-2a95f9765574@xgm.de> <1490477472.21453.1.camel@sipsolutions.net> <1490623780.18200.4.camel@sipsolutions.net> Message-ID: <04b18c37-9719-5d9e-8909-3d0903a87955@xgm.de> Hey, Am 27.03.2017 um 16:09 schrieb Sebastian Berg: > On Mon, 2017-03-27 at 13:06 +0200, Florian Lindner wrote: >> Hey, >> >> I've timed the two versions, one basisfunction being a function: >> >> 1 loop, best of 3: 17.3 s per loop >> >> the other one, basisfunction being a list of functions: >> >> 1 loop, best of 3: 33.5 s per loop >> >>> To be honest, I am a bit surprised that its a problem, since "basis >>> function" sounds a bit like you have to do this once and then use >>> the >>> result many times. >> >> It's part of a radial basis function interpolation algorithm. Yes, in >> practice the matrix is filled only once and reused >> a couple of times, but in my case, which is exploration of parameters >> for the algorithm, I call eval_BF many times. >> >>> You can get rid of the `row` loop though in case row if an >>> individual >>> row is a pretty small array. >> >> Would you elaborate on that? Do you mean that the inner col loop >> produces an array which is then assigned to the row. >> But I think it stell need to row loop there. > > Well, I like to not serve the result, but if you exchange the loops: > > A = np.empty((len(meshA), len(meshB))) > for j, col in enumerate(meshB): > for i, row in enumerate(meshA): > A[i, j] = self.basisfunction[j](row - col) > > Then you can see that there is broadcasting magic similar (do not want > to use too many brain cells now) to: > > A = np.empty((len(meshA), len(meshB))) > for j, col in enumerate(meshB): > # possibly insert np.newaxis/None or a reshape in [??] > A[:, j] = self.basisfunction[j](meshA[??] - col) I have it like that now: A = np.empty((len(meshA), len(meshB))) for j, col in enumerate(meshB): A[:,j] = self.basisfunction[j](meshA - col) which has improved my speeds by a factor of 36. Thanks! Florian > > - Sebastian > >> >> Best, >> Florian >> >> Am 25.03.2017 um 22:31 schrieb Sebastian Berg: >>> On Sat, 2017-03-25 at 18:46 +0100, Florian Lindner wrote: >>>> Hello, >>>> >>>> I have this function: >>>> >>>> def eval_BF(self, meshA, meshB): >>>> """ Evaluates single BF or list of BFs on the meshes. """ >>>> if type(self.basisfunction) is list: >>>> A = np.empty((len(meshA), len(meshB))) >>>> for i, row in enumerate(meshA): >>>> for j, col in enumerate(meshB): >>>> A[i, j] = self.basisfunction[j](row - col) >>>> else: >>>> mgrid = np.meshgrid(meshB, meshA) >>>> A = self.basisfunction( np.abs(mgrid[0] - mgrid[1]) ) >>>> return A >>>> >>>> >>>> meshA and meshB are 1-dimensional numpy arrays. >>>> self.basisfunction is >>>> e.g. >>>> >>>> def Gaussian(radius, shape): >>>> """ Gaussian Basis Function """ >>>> return np.exp( -np.power(shape*abs(radius), 2)) >>>> >>>> >>>> or a list of partial instantations of such functions (from >>>> functools.partial). >>>> >>>> How can I optimize eval_BF? Esp. in the case of basisfunction >>>> being a >>>> list. >>>> >>> >>> Are you sure you need to optimize it? If they have a couple of >>> hundred >>> elements or so for each row, the math is probably the problem and >>> most >>> of that might be the `exp`. >>> You can get rid of the `row` loop though in case row if an >>> individual >>> row is a pretty small array. >>> >>> To be honest, I am a bit surprised that its a problem, since "basis >>> function" sounds a bit like you have to do this once and then use >>> the >>> result many times. >>> >>> - Sebastian >>> >>> >>>> Thanks! >>>> Florian >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at python.org >>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at python.org >>>> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From njs at pobox.com Fri Mar 31 09:51:14 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 31 Mar 2017 06:51:14 -0700 Subject: [Numpy-discussion] Fwd: [numfocus] Grants up to $3k available to NumFOCUS projects (sponsored & affiliated) In-Reply-To: References: <1489688042-5554705.54580375.fv2GIDbTc031721@rs159.luxsci.com> <78cad834-ff24-3a21-ed14-912309d8089d@googlemail.com> Message-ID: On Mar 31, 2017 1:15 AM, "Ralf Gommers" wrote: On Mon, Mar 27, 2017 at 11:42 PM, Ralf Gommers wrote: > > > On Mon, Mar 27, 2017 at 11:33 PM, Julian Taylor < > jtaylor.debian at googlemail.com> wrote: > >> I have two ideas under one big important topic: make numpy python3 >> compatible. >> >> The first fits pretty well with the grant size and nobody wants to do it >> for free: >> - fix our text IO functions under python3 and support multiple >> encodings, not only latin1. >> Reasonably simple to do, slap encoding arguments on the functions, >> generate test cases and somehow keep backward compatibility. Some >> prelimary unfinished work is in https://github.com/numpy/numpy/pull/4208 > > > I like that idea, it's a recurring pain point. Are you interested to work > on it, or are you thinking to advertise the idea here to see if anyone > steps up? > More thoughts on this anyone? Or preferences for this idea or the numpy.org one? Submission deadline is April 3rd and we can only put in one proposal this time, so we need to (a) make a choice between these ideas, and (b) write up a proposal. If there's not enough replies to this so the choice is clear cut, I will send out a poll to the core devs. Do we have anyone interested in doing the work in either case? That seems like the most important consideration to me... -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Fri Mar 31 10:07:20 2017 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Fri, 31 Mar 2017 16:07:20 +0200 Subject: [Numpy-discussion] Fwd: [numfocus] Grants up to $3k available to NumFOCUS projects (sponsored & affiliated) In-Reply-To: References: <1489688042-5554705.54580375.fv2GIDbTc031721@rs159.luxsci.com> <78cad834-ff24-3a21-ed14-912309d8089d@googlemail.com> Message-ID: <9079116f-b13c-a695-e1b8-e9777467c1d9@googlemail.com> On 31.03.2017 15:51, Nathaniel Smith wrote: > On Mar 31, 2017 1:15 AM, "Ralf Gommers" > wrote: > > > > On Mon, Mar 27, 2017 at 11:42 PM, Ralf Gommers > > wrote: > > > > On Mon, Mar 27, 2017 at 11:33 PM, Julian Taylor > > wrote: > > I have two ideas under one big important topic: make numpy > python3 > compatible. > > The first fits pretty well with the grant size and nobody > wants to do it > for free: > - fix our text IO functions under python3 and support multiple > encodings, not only latin1. > Reasonably simple to do, slap encoding arguments on the > functions, > generate test cases and somehow keep backward compatibility. > Some > prelimary unfinished work is in > https://github.com/numpy/numpy/pull/4208 > > > > I like that idea, it's a recurring pain point. Are you > interested to work on it, or are you thinking to advertise the > idea here to see if anyone steps up? > > > More thoughts on this anyone? Or preferences for this idea or the > numpy.org one? Submission deadline is April 3rd > and we can only put in one proposal this time, so we need to (a) > make a choice between these ideas, and (b) write up a proposal. > > If there's not enough replies to this so the choice is clear cut, I > will send out a poll to the core devs. > > > Do we have anyone interested in doing the work in either case? That > seems like the most important consideration to me... > > -n > I could do the textio thing if no one shows up for numpy.org. I can probably check again what is required in the next few days and write a proposal. The change will need reviewing in the end too, should that be compensated too? It feels weird if not. From m.h.vankerkwijk at gmail.com Fri Mar 31 12:46:53 2017 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Fri, 31 Mar 2017 12:46:53 -0400 Subject: [Numpy-discussion] __array_ufunc__ counting down to launch, T-24 hrs. In-Reply-To: References: Message-ID: Hi All, Following Nathaniel's request, I have made a PR that changes the original NEP to describe the current implementation. * PR at https://github.com/charris/numpy/pull/9 * Rendered relevant page at http://www.astro.utoronto.ca/~mhvk/numpy-doc/neps/ufunc-overrides.html It may still be somewhat short on detail, but should now give the rationale for what we want to implement. All the best, Marten From shoyer at gmail.com Fri Mar 31 12:47:26 2017 From: shoyer at gmail.com (Stephan Hoyer) Date: Fri, 31 Mar 2017 09:47:26 -0700 Subject: [Numpy-discussion] __array_ufunc__ counting down to launch, T-24 hrs. In-Reply-To: References: Message-ID: I agree with Nathaniel -- let's finish the design doc first. On Thu, Mar 30, 2017 at 10:08 PM, Nathaniel Smith wrote: > On Thu, Mar 30, 2017 at 7:40 PM, Charles R Harris > wrote: > > Hi All, > > > > Just a note that the __array_ufunc__ PR is ready to merge. If you are > > interested, you can review here. > > I want to get this in too, but 24 hours seems like a very short > deadline for getting feedback on such a large and complex change? I'm > pretty sure the ndarray.__array_ufunc__ code that was just added a few > hours ago is wrong (see comments on the diff)... > > My main comment, also relevant to the kind of high-level discussion we > tend to use the mailing list for: > https://github.com/numpy/numpy/pull/8247#issuecomment-290616432 > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan12343 at gmail.com Fri Mar 31 13:16:49 2017 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Fri, 31 Mar 2017 17:16:49 +0000 Subject: [Numpy-discussion] __array_ufunc__ counting down to launch, T-24 hrs. In-Reply-To: References: Message-ID: Thanks for linking to the updated NEP, I've been looking for a good overview of this discussion. Up until now I haven't wanted to wade through the extensive discussion on this topic. I'm curious, if I want to simultaneously support older Numpy versions as well as newer versions, will I be able to leave implementations of __array_wrap__ and __array_prepare__ defined alongside __array_ufunc__? Optimally in such a way that older numpy versions use __array_wrap__ and newer versions only use __array_ufunc__. There isn't discussion about this in the NEP, but does this also have impacts on non-ufunc numpy operations like concatenate, dot, norm, hstack, and others? We currently make use of wrappers around those functions in yt but unfortunately they have poor discoverability for users, it would be nice if NumPy could do the right thing with nearest subclasses. On Fri, Mar 31, 2017 at 12:04 PM Marten van Kerkwijk < m.h.vankerkwijk at gmail.com> wrote: > Hi All, > > Following Nathaniel's request, I have made a PR that changes the > original NEP to describe the current implementation. > * PR at https://github.com/charris/numpy/pull/9 > * Rendered relevant page at > http://www.astro.utoronto.ca/~mhvk/numpy-doc/neps/ufunc-overrides.html > It may still be somewhat short on detail, but should now give the > rationale for what we want to implement. > > All the best, > > Marten > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.h.vankerkwijk at gmail.com Fri Mar 31 13:41:12 2017 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Fri, 31 Mar 2017 13:41:12 -0400 Subject: [Numpy-discussion] __array_ufunc__ counting down to launch, T-24 hrs. In-Reply-To: References: Message-ID: Hi Nathan, That is a good point: Yes, one can leave __array_prepare__ and __array_wrap__ in place: only for ufuncs will they be ignored if __array_ufunc__ is present; __array_wrap__ in particular will still be used by quite a lot of other numpy functions (other use of __array_prepare__ is usually a mistake, but sadly does happen...). Anyway, with older versions of numpy, code should continue to work as it did. All the best, Marten p.s. It is also important to leave __array_prepare__ and __array_wrap__ in place for possible other classes that assume your class is wrapped that way... p.s.2 The same holds for __array_priority__ From charlesr.harris at gmail.com Fri Mar 31 13:54:18 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 31 Mar 2017 11:54:18 -0600 Subject: [Numpy-discussion] Fwd: [numfocus] Grants up to $3k available to NumFOCUS projects (sponsored & affiliated) In-Reply-To: <9079116f-b13c-a695-e1b8-e9777467c1d9@googlemail.com> References: <1489688042-5554705.54580375.fv2GIDbTc031721@rs159.luxsci.com> <78cad834-ff24-3a21-ed14-912309d8089d@googlemail.com> <9079116f-b13c-a695-e1b8-e9777467c1d9@googlemail.com> Message-ID: On Fri, Mar 31, 2017 at 8:07 AM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > On 31.03.2017 15:51, Nathaniel Smith wrote: > > On Mar 31, 2017 1:15 AM, "Ralf Gommers" > > wrote: > > > > > > > > On Mon, Mar 27, 2017 at 11:42 PM, Ralf Gommers > > > wrote: > > > > > > > > On Mon, Mar 27, 2017 at 11:33 PM, Julian Taylor > > > > wrote: > > > > I have two ideas under one big important topic: make numpy > > python3 > > compatible. > > > > The first fits pretty well with the grant size and nobody > > wants to do it > > for free: > > - fix our text IO functions under python3 and support > multiple > > encodings, not only latin1. > > Reasonably simple to do, slap encoding arguments on the > > functions, > > generate test cases and somehow keep backward compatibility. > > Some > > prelimary unfinished work is in > > https://github.com/numpy/numpy/pull/4208 > > > > > > > > I like that idea, it's a recurring pain point. Are you > > interested to work on it, or are you thinking to advertise the > > idea here to see if anyone steps up? > > > > > > More thoughts on this anyone? Or preferences for this idea or the > > numpy.org one? Submission deadline is April 3rd > > and we can only put in one proposal this time, so we need to (a) > > make a choice between these ideas, and (b) write up a proposal. > > > > If there's not enough replies to this so the choice is clear cut, I > > will send out a poll to the core devs. > > > > > > Do we have anyone interested in doing the work in either case? That > > seems like the most important consideration to me... > > > > -n > > > > I could do the textio thing if no one shows up for numpy.org. I can > probably check again what is required in the next few days and write a > proposal. > The change will need reviewing in the end too, should that be > compensated too? It feels weird if not. > Note that we would also like to bring over the text input implementation from pandas. There was general agreement back when we met in (2015?) that that would be desirable, but it has never come to pass. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Mar 31 14:33:54 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 31 Mar 2017 12:33:54 -0600 Subject: [Numpy-discussion] Fwd: SciPy2017 Sprints FinAid for sprint leaders/core devs In-Reply-To: References: <1490824550.2742751.927932680.77CF511C@webmail.messagingengine.com> <1490914764.2758.10.camel@sipsolutions.net> Message-ID: On Fri, Mar 31, 2017 at 12:39 AM, Ralf Gommers wrote: > > > On Fri, Mar 31, 2017 at 11:59 AM, Sebastian Berg < > sebastian at sipsolutions.net> wrote: > >> On Thu, 2017-03-30 at 22:46 +1300, Ralf Gommers wrote: >> > >> >> > >> > Agreed, and I would call that productive. Getting even one new >> > maintainer involved is worth organizing multiple sprints for. >> > >> > That said, also +1 to a developer meeting this year. It'd be good if >> > we could combine it with the NumFOCUS summit or a relevant conference >> > in the second half of the year. >> >> Would be good, even if there is nothing big going on. >> >> Can we gather possible dates and possible (personal) preferences? Here >> is a start: >> >> * SciPy (Austin, TX): July 10-16 >> * EuroScipy (Germany): August 23-27 >> * NumFocus Summit? >> > > Austin, October (exact date TBD). I intend to plan a longer trip to the US > around this summit, and I think at least one other core dev should go > there. So this one has my preference. > > >> * PyData Events?? >> > > Sticking to the ones in the second half of the year and in the US or > Western Europe: > > PyData Berlin, June 30 - July 2 > PyData Seattle July 5 -7 > > Other options: > > JupyterCon, New York, August 22-25 > Strata Data Conference, September 25-28 > > >> >> Personally, I probably can't make longer trips until some time in July. >> time around then). > > > Same here. > I'm flexible, anytime that works for others will probably work for me. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: