Mailman 3 April 2015 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

C-API: multidimensional array indexing?
by Johann Bauer 31 Mar '16

31 Mar '16

Dear experts, is there a C-API function for numpy which implements Python's multidimensional indexing? Say, I have a 2d-array PyArrayObject * M; and an index int i; how do I extract the i-th row or column M[i,:] respectively M[:,i]? I am looking for a function which gives again a PyArrayObject * and which is a view to M (no copied data; the result should be another PyArrayObject whose data and strides points to the correct memory portion of M). I searched the API documentation, Google and mailing lists for quite a long time but didn't find anything. Can you help me? Thanks, Johann

3 3

Generalize hstack/vstack --> stack; Block matrices like in matlab
by Stefan Otte 09 Jan '16

09 Jan '16

Hey, quite often I work with block matrices. Matlab offers the convenient notation [ a b; c d ] to stack matrices. The numpy equivalent is kinda clumsy: vstack([hstack([a,b]), hstack([c,d])]) I wrote the little function `stack` that does exactly that: stack([[a, b], [c, d]]) In my case `stack` replaced `hstack` and `vstack` almost completely. If you're interested in including it in numpy I created a pull request [1]. I'm looking forward to getting some feedback! Best, Stefan [1] https://github.com/numpy/numpy/pull/5057

12 24

F2PY and multi-dimension arrays - unexpected array size error
by Backing Olof 01 Jul '15

01 Jul '15

Hi I am helping out with a Python and Fortran project. Let me give you some background: * Fortran source: C Bergstrom FCC C User subroutine VUMAT subroutine VUMAT( C Read only - * nblock, ndir, nshr, nstatev, nprops, * stepTime, dt, * props, * density, strainInc, * tempOld, * stressOld, stateOld, enerInternOld, enerInelasOld, C Write only - * stressNew, stateNew, enerInternNew, enerInelasNew ) C include 'vaba_param.inc' C dimension props(nprops), 1 density(nblock), strainInc(nblock,ndir+nshr), 2 tempOld(nblock), 5 stressOld(nblock,ndir+nshr), stateOld(nblock,nstatev), 6 enerInternOld(nblock), enerInelasOld(nblock), 2 stressNew(nblock,ndir+nshr), stateNew(nblock,nstatev), 3 enerInternNew(nblock), enerInelasNew(nblock) * Corresponding .pyf integer :: nblock integer :: ndir integer :: nshr integer :: nstatev integer :: nprops real :: steptime real :: dt real dimension(nprops) :: props real dimension(nblock) :: density real dimension(nblock,ndir+nshr) :: straininc real dimension(nblock) :: tempold real dimension(nblock,ndir+nshr) :: stressold real dimension(nblock,nstatev) :: stateold real dimension(nblock) :: enerinternold real dimension(nblock) :: enerinelasold real dimension(nblock,ndir+nshr),intent(out) :: stressnew real dimension(nblock,nstatev),intent(out) :: statenew real dimension(nblock),intent(out) :: enerinternnew real dimension(nblock),intent(out) :: enerinelasnew * Python source with call of Fortran routine: nblock = 1 ndir = 3 nshr = 3 nstatev = 3 nprops = 11 stepTime = 1 dt = 1 props = np.array([10, 0.5, 1e10, 5, 1e12, 3e-6, 8e-6, 27, 2], float) density = np.array([[7.8e3]], float) strainInc = np.array([[1,-0.5,-0.5,0,0,0]], float) tempOld = np.array([[1]], float) stressOld = np.array([[1,1,1,1,1,1]], float) stateOld = np.array([[1,1,1]], float) enerInternOld = np.array([1], float) enerInelasOld = np.array([1], float) stressNew = np.array([[]], float) stateNew = np.array([[]], float) enerInternNew = np.array([[]], float) enerInelasNew = np.array([[]], float) stressNew, stateNew, enerInternNew, enerInelasNew = vumat(nblock, ndir, nshr, nstatev, nprops, stepTime, dt, props, density, strainInc, tempOld, stressOld, stateOld, enerInternOld, enerInelasOld) When trying to run with Python 2.7 I get: olof@ubuntu:~$ ./demo.py unexpected array size: new_size=4, got array with arr_size=1 Traceback (most recent call last): File "./demo.py", line 33, in <module> main() File "./demo.py", line 30, in main stressNew, stateNew, enerInternNew, enerInelasNew = vumat(nblock, ndir, nshr, nstatev, nprops, stepTime, dt, props, density, strainInc, tempOld, stressOld, stateOld, enerInternOld, enerInelasOld) VUMAT_Bergstrom_FCC.error: failed in converting 9th argument `stressold' of VUMAT_Bergstrom_FCC.vumat to C/Fortran array Other stuff: * python 2.7.6 * numpy/f2py 1.8.2 * gcc/gfortran 4.8.2 * ubuntu 14.04 LTS 32-bit I have tried to google, read the f2py manual, fortran tutorials etc, but to no avail. I must also admit that my knowledge in python is so-so and fortran even less(!). What is the missing statement/syntax that I can’t get correct? Your humble programmer, Olof

1 1

PR added: frozen dimensions in gufunc signatures
by Jaime Fernández del Río 23 Jun '15

23 Jun '15

Hi, I have just sent a PR (https://github.com/numpy/numpy/pull/5015), adding the possibility of having frozen dimensions in gufunc signatures. As a proof of concept, I have added a `cross1d` gufunc to `numpy.core.umath_tests`: In [1]: import numpy as np In [2]: from numpy.core.umath_tests import cross1d In [3]: cross1d.signature Out[3]: '(3),(3)->(3)' In [4]: a = np.random.rand(1000, 3) In [5]: b = np.random.rand(1000, 3) In [6]: np.allclose(np.cross(a, b), cross1d(a, b)) Out[6]: True In [7]: %timeit np.cross(a, b) 10000 loops, best of 3: 76.1 us per loop In [8]: %timeit cross1d(a, b) 100000 loops, best of 3: 13.1 us per loop In [9]: c = np.random.rand(1000, 2) In [10]: d = np.random.rand(1000, 2) In [11]: cross1d(c, d) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-11-72c66212e40c> in <module>() ----> 1 cross1d(c, d) ValueError: cross1d: Operand 0 has a mismatch in its core dimension 0, with gufunc signature (3),(3)->(3) (size 2 is different from 3) The speed up over `np.cross` is nice, and while `np.cross` is not the best of examples, as it needs to handle more sizes, in many cases this will allow producing gufuncs that work without a Python wrapper redoing checks that are best left to the iterator, such as dimension sizes. It still needs tests, but before embarking on fully developing those, I wanted to make sure that there is an interest on this. I would also like to further enhance gufuncs providing computed dimensions, e.g. making it possible to e.g. define `pairwise_cross` with signature '(n, 3)->($m, 3)', where the $ indicates that m is a computed dimension, that would have to be calculated by a function passed to the gufunc constructor and stored in the gufunc object, based on the other core dimensions. In this case it would make $m be n*(n-1), so that all pairwise cross products between 3D vectors could be computed. The syntax with '$' is kind of crappy, so any suggestions on how to better express this in the signature are more than welcome, as well as any feedback on the merits (or lack of them) of implementing this. Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

4 4

numpy vendor repo
by Ralf Gommers 01 May '15

01 May '15

Hi, Today I wanted to add something to https://github.com/numpy/vendor and realised that this repo is in pretty bad shape. A couple of years ago Ondrej took a copy of the ATLAS binaries in that repo and started a new repo (not a fork) at https://github.com/certik/numpy-vendor. The latest improvements were made by Julian and live at https://github.com/juliantaylor/numpy-vendor. I'd like to start from numpy/vendor, then add all commits from Julian's numpy-vendor on top of it, then move things around so we have the binaries/sources/tools layout back and finally update the README so it's clear how to build both the ATLAS binaries and Numpy releases. Any objections or better ideas? Ralf

4 7

code snippet: assert all close or large
by josef.pktd＠gmail.com 30 Apr '15

30 Apr '15

1 1

ANN: SciPy 2015 Tutorial Schedule Posted - Register Today - Already 30% Sold Out
by Courtenay Godshall (Enthought) 30 Apr '15

30 Apr '15

**The <https://plus.google.com/s/%23SciPy2015> #SciPy2015 Conference (Scientific Computing with <https://plus.google.com/s/%23Python> #Python) Tutorial Schedule is up! It is 1st come, 1st served and already 30% sold out. Register today!** http://www.scipy2015.scipy.org/ehome/115969/289057/? <http://www.scipy2015.scipy.org/ehome/115969/289057/?&> &.This year you can choose from 16 different SciPy tutorials OR select the 2 day Software Carpentry course on scientific Python that assumes some programming experience but no Python knowledge. Please share! Tutorials include: *Introduction to NumPy (Beginner) *Machine Learning with Scikit-Learn (Intermediate) *Cython: Blend of the Best of Python and C/++ (Intermediate) *Image Analysis in Python with SciPy and Scikit-Image (Intermediate) *Analyzing and Manipulating Data with Pandas (Beginner) *Machine Learning with Scikit-Learn (Advanced) *Building Python Data Applications with Blaze and Bokeh (Intermediate) *Multibody Dynamics and Control with Python (Intermediate) *Anatomy of Matplotlib (Beginner) *Computational Statistics I (Intermediate) *Efficient Python for High-Performance Parallel Computing (Intermediate) *Geospatial Data with Open Source Tools in Python (Intermediate) *Decorating Drones: Using Drones to Delve Deeper into Intermediate Python (Intermediate) *Computational Statistics II (Intermediate) *Modern Optimization Methods in Python (Advanced) *Jupyter Advanced Topics Tutorial (Advanced)

1 0

performance of numpy.array()
by simona bellavista 30 Apr '15

30 Apr '15

I work on two distinct scientific clusters. I have run the same python code on the two clusters and I have noticed that one is faster by an order of magnitude than the other (1min vs 10min, this is important because I run this function many times). I have investigated with a profiler and I have found that the cause of this is that (same code and same data) is the function numpy.array that is being called 10^5 times. On cluster A it takes 2 s in total, whereas on cluster B it takes ~6 min. For what regards the other functions, they are generally faster on cluster A. I understand that the clusters are quite different, both as hardware and installed libraries. It strikes me that on this particular function the performance is so different. I would have though that this is due to a difference in the available memory, but actually by looking with `top` the memory seems to be used only at 0.1% on cluster B. In theory numpy is compiled with atlas on cluster B, and on cluster A it is not clear, because numpy.__config__.show() returns NOT AVAILABLE for anything. Does anybody has any insight on that, and if I can improve the performance on cluster B?

6 9

Weighted covariance.
by Charles R Harris 30 Apr '15

30 Apr '15

The weighted covariance function in PR #4960 <https://github.com/numpy/numpy/pull/4960> is evolving to the following, where frequency weights are `f` and reliability weights are `a`. Assume that the observations are in the columns of the observation matrix. the steps to compute the weighted covariance are as follows:: >>> w = f * a >>> v1 = np.sum(w) >>> v2 = np.sum(a * w) >>> m -= np.sum(m * w, axis=1, keepdims=True) / v1 >>> cov = np.dot(m * w, m.T) * v1 / (v1**2 - ddof * v2) Note that when ``a == 1``, the normalization factor ``v1 / (v1**2 - ddof * v2)`` goes over to ``1 / (np.sum(f) - ddof)`` as it should. This is probably a good time for comments from all the kibitzers out there. Chuck

1 0

ANN: numexpr 2.4.3 released
by Francesc Alted 29 Apr '15

29 Apr '15

Announcing Numexpr 2.4.3 ========================= Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new ========== This is a maintenance release to cope with an old bug affecting comparisons with empty strings. Fixes #121 and PyTables #184. In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/wiki/Release-Notes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted

5 13

scipy.fftpack.diff operation question
by R Schumacher 29 Apr '15

29 Apr '15

We are looking to plot to time series accelerometer data as velocity and displacement. To this end we tried scipy.fftpack.diff, but in looking at the test code direct_diff() we get odd results, and, why is the doc using "sqrt(-1)*j" in its explanation? So, I tried a few different integration methods. We were first looking at the doc at http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.fftpack.di… and code at https://github.com/scipy/scipy/blob/v0.15.1/scipy/fftpack/pseudo_diffs.py#L… as well as the tests at https://github.com/scipy/scipy/blob/master/benchmarks/benchmarks/fftpack_ps… The direct_diff() test function in bench_pseudo_diffs.py seems odd, since I have to pass period=-1 to get a matching sign plot for the first integration. And why the scaling differences required, even for diff() and direct_diff()? Is my understanding fundamentally flawed? Emacs! Ray

1 0

how to set a fixed sized dtype suitable for bitwise operations
by Benjamin Root 28 Apr '15

28 Apr '15

I have a need to have a numpy array of 17 byte (more specifically, at least 147 bits) values that I would be doing some bit twiddling on. I have found that doing a dtype of "i17" yields a dtype of int32, which is completely not what I intended. Doing 'u17' gets an "data type not understood". I have tried 'a17', but then bitwise_or() and left_shift() do not work (returns "NotImplemented"). How should I be going about this? Cheers! Ben Root

2 2

GSoC'15 accepted students for Scipy/Numpy
by Ralf Gommers 28 Apr '15

28 Apr '15

Hi all, Google has just announced which students got accepted for this year's GSoC. For Scipy these are: - Nikolay Mayorov, "Improve nonlinear least squares minimization functionality in SciPy" mentors: Chuck & Evgeni - Abraham Escalante, "SciPy: scipy.stats improvements" mentor: Ralf (Evgeni is backup mentor) Furthermore, this proposal was accepted for Scikit-image: - Aman Singh, "Scikit-Image: rewriting scipy.ndimage to cython" mentors: Jaime, Ralf & the scikit-image devs Congratulations to all of you! We had a lot of interest this year, which is great to see. GSoC applications are competitive, and unfortunately there are students who didn't make it. To those students I would say: please stay involved, and you're very welcome to apply again next year! Today is also the start of the "community bonding period", where the students aren't yet expected to start working on their project but do get time to further figure out how things work, interact with the community and ensure that they can hit the ground running on day 1 of the coding period: http://googlesummerofcode.blogspot.nl/2007/04/so-what-is-this-community-bon…. It looks like it'll be an interesting and productive summer! Cheers, Ralf P.S. all proposals are linked on https://github.com/scipy/scipy/wiki/GSoC-project-ideas#student-applications… for who's interested in the details. P.P.S. some students have asked to get some feedback about why they were/weren't accepted, in order to learn from it for a next time. Until today we weren't allowed to say much, but now that Google has announced the results I'd be happy to give some feedback - please contact me in private if you want.

1 0

EuroScipy 2015: Submission deadline in 3 days !!!
by Nicolas P. Rougier 27 Apr '15

27 Apr '15

--------------------------------- Submission deadline in 3 days !!! --------------------------------- EuroScipy 2015, the annual conference on Python in science will take place in Cambridge, UK on 26-30 August 2015. The conference features two days of tutorials followed by two days of scientific talks & posters and an extra day dedicated to developer sprints. It is the major event in Europe in the field of technical/scientific computing within the Python ecosystem. Data scientists, analysts, quants, PhD's, scientists and students from more than 20 countries attended the conference last year. The topics presented at EuroSciPy are very diverse, with a focus on advanced software engineering and original uses of Python and its scientific libraries, either in theoretical or experimental research, from both academia and the industry. Submissions for posters, talks & tutorials (beginner and advanced) are welcome on our website at http://www.euroscipy.org/2015/ Sprint proposals should be addressed directly to the organisation at euroscipy-org(a)python.org Important dates =============== Mar 24, 2015 Call for talks, posters & tutorials Apr 30, 2015 Talk and tutorials submission deadline May 1, 2015 Registration opens May 30, 2015 Final program announced Jun 15, 2015 Early-bird registration ends Aug 26-27, 2015 Tutorials Aug 28-29, 2015 Main conference Aug 30, 2015 Sprints We look forward to an exciting conference and hope to see you in Cambridge The EuroSciPy 2015 Team - http://www.euroscipy.org/2015/

1 0

ANN: PyTables 3.2.0 release candidate 1 is out
by Francesc Alted 21 Apr '15

21 Apr '15

=========================== Announcing PyTables 3.2.0rc1 =========================== We are happy to announce PyTables 3.2.0rc1. ******************************* IMPORTANT NOTICE: If you are a user of PyTables, it needs your help to keep going. Please read the next thread as it contains important information about the future of the project: https://groups.google.com/forum/#!topic/pytables-users/yY2aUa4H7W4 Thanks! ******************************* What's new ========== This is a major release of PyTables and it is the result of more than a year of accumulated patches, but most specially it fixes a nasty problem with indexed queries not returning the correct results in some scenarios. There are many usablity and performance improvements too. In case you want to know more in detail what has changed in this version, please refer to: http://pytables.github.io/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/3.2.0rc1 For an online version of the manual, visit: http://pytables.github.io/usersguide/index.html What it is? =========== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than a tenth of a second. Resources ========= About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Developers

1 0

Automatic number of bins for numpy histograms
by Varun 18 Apr '15

18 Apr '15

http://nbviewer.ipython.org/github/nayyarv/matplotlib/blob/master/examples/… tistics/A utomating%20Binwidth%20Choice%20for%20Histogram.ipynb Long story short, histogram visualisations that depend on numpy (such as matplotlib, or nearly all of them) have poor default behaviour as I have to constantly play around with the number of bins to get a good idea of what I'm looking at. The bins=10 works ok for up to 1000 points or very normal data, but has poor performance for anything else, and doesn't account for variability either. I don't have a method easily available to scale the number of bins given the data. R doesn't suffer from these problems and provides methods for use with it's hist method. I would like to provide similar functionality for matplotlib, to at least provide some kind of good starting point, as histograms are very useful for initial data discovery. The notebook above provides an explanation of the problem as well as some proposed alternatives. Use different datasets (type and size) to see the performance of the suggestions. All of the methods proposed exist in R and literature. I've put together an implementation to add this new functionality, but am hesitant to make a pull request as I would like some feedback from a maintainer before doing so. https://github.com/numpy/numpy/compare/master...nayyarv:master I've provided them as functions for easy refactoring, as it can be argued that it should be in it's own function/file/class, or alternatively can be turned into simple if, elif statements. I believe this belongs in numpy as it is where the functionality exists for histogram methods that most libraries build on, and it would useful for them to not require scipy for example. I will update the documentation accordingly before making a pull request, and add in more tests to show it's functionality. I can adapt my ipython notebook into a quick tutorial/help file if need be. I've already attempted to add this into matplotlib before being redirected here https://github.com/matplotlib/matplotlib/issues/4316

11 22

Consider improving numpy.outer's behavior with zero-dimensional vectors
by Neil Girdhar 18 Apr '15

18 Apr '15

Numpy's outer product works fine with vectors. However, I seem to always want len(outer(a, b).shape) to be equal to len(a.shape) + len(b.shape). Wolfram-alpha seems to agree https://reference.wolfram.com/language/ref/Outer.html with respect to matrix outer products. My suggestion is to define outer as defined below. I've contrasted it with numpy's current outer product. In [36]: def a(n): return np.ones(n) In [37]: b = a(()) In [38]: c = a(4) In [39]: d = a(5) In [40]: np.outer(b, d).shape Out[40]: (1, 5) In [41]: np.outer(c, d).shape Out[41]: (4, 5) In [42]: np.outer(c, b).shape Out[42]: (4, 1) In [43]: def outer(a, b): return a[(...,) + len(b.shape) * (np.newaxis,)] * b ....: In [44]: outer(b, d).shape Out[44]: (5,) In [45]: outer(c, d).shape Out[45]: (4, 5) In [46]: outer(c, b).shape Out[46]: (4,) Best, Neil

5 46

EuroScipy 2015 : Call for talks, posters and tutorials [Reminder]
by Nicolas P. Rougier 17 Apr '15

17 Apr '15

[Apology for cross-posting] Dear all, EuroScipy 2015, the annual conference on Python in science will take place in Cambridge, UK on 26-30 August 2015. The conference features two days of tutorials followed by two days of scientific talks & posters and an extra day dedicated to developer sprints. It is the major event in Europe in the field of technical/scientific computing within the Python ecosystem. Data scientists, analysts, quants, PhD's, scientists and students from more than 20 countries attended the conference last year. The topics presented at EuroSciPy are very diverse, with a focus on advanced software engineering and original uses of Python and its scientific libraries, either in theoretical or experimental research, from both academia and the industry. Submissions for posters, talks & tutorials (beginner and advanced) are welcome on our website at http://www.euroscipy.org/2015/ Sprint proposals should be addressed directly to the organisation at euroscipy-org(a)python.org Important dates Mar 24, 2015 Call for talks, posters & tutorials Apr 30, 2015 Talk and tutorials submission deadline May 1, 2015 Registration opens May 30, 2015 Final program announced Jun 15, 2015 Early-bird registration ends Aug 26-27, 2015 Tutorials Aug 28-29, 2015 Main conference Aug 30, 2015 Sprints We look forward to an exciting conference and hope to see you in Cambridge The EuroSciPy 2015 Team - http://www.euroscipy.org/2015/

1 0

Re: [Numpy-discussion] IDE's for numpy development?
by Suzen, Mehmet 16 Apr '15

16 Apr '15

> Spyder supports C. Thanks for correcting this. I wasn't aware of it. How was your experience with it? Best, -m

3 2

[ANN] python-blosc v1.2.5
by Valentin Haenel 16 Apr '15

16 Apr '15

============================= Announcing python-blosc 1.2.5 ============================= What is new? ============ This release contains support for Blosc v1.5.4 including changes to how the GIL is kept. This was required because Blosc was refactored in the v1.5.x line to remove global variables and to use context objects instead. As such, it became necessary to keep the GIL while calling Blosc from Python code that uses the multiprocessing module. In addition, is now possible to change the blocksize used by Blosc using ``set_blocksize``. When using this however, bear in mind that the blocksize has been finely tuned to be a good default value and that randomly messing with this value may have unforeseen and unpredictable consequences on the performance of Blosc. Additionally, we can now compile on Posix architectures, thanks again to Andreas Schwab for that one. For more info, you can have a look at the release notes in: https://github.com/Blosc/python-blosc/wiki/Release-notes More docs and examples are available in the documentation site: http://python-blosc.blosc.org What is it? =========== Blosc (http://www.blosc.org) is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Blosc is the first compressor that is meant not only to reduce the size of large datasets on-disk or in-memory, but also to accelerate object manipulations that are memory-bound (http://www.blosc.org/docs/StarvingCPUs.pdf). See http://www.blosc.org/synthetic-benchmarks.html for some benchmarks on how much speed it can achieve in some datasets. Blosc works well for compressing numerical arrays that contains data with relatively low entropy, like sparse data, time series, grids with regular-spaced values, etc. python-blosc (http://python-blosc.blosc.org/) is the Python wrapper for the Blosc compression library. There is also a handy tool built on Blosc called Bloscpack (https://github.com/Blosc/bloscpack). It features a commmand line interface that allows you to compress large binary datafiles on-disk. It also comes with a Python API that has built-in support for serializing and deserializing Numpy arrays both on-disk and in-memory at speeds that are competitive with regular Pickle/cPickle machinery. Installing ========== python-blosc is in PyPI repository, so installing it is easy: $ pip install -U blosc # yes, you should omit the python- prefix Download sources ================ The sources are managed through github services at: http://github.com/Blosc/python-blosc Documentation ============= There is Sphinx-based documentation site at: http://python-blosc.blosc.org/ Mailing list ============ There is an official mailing list for Blosc at: blosc(a)googlegroups.com http://groups.google.es/group/blosc Licenses ======== Both Blosc and its Python wrapper are distributed using the MIT license. See: https://github.com/Blosc/python-blosc/blob/master/LICENSES for more details. ---- **Enjoy data!**

1 0

ANN: numexpr 2.4.1 released
by Francesc Alted 14 Apr '15

14 Apr '15

========================= Announcing Numexpr 2.4.1 ========================= Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new ========== In this version there is improved support for newer MKL library as well as other minor improvements. This version is meant for production. In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/wiki/Release-Notes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted

1 0

Numpy compilation error
by Peter Kerpedjiev 12 Apr '15

12 Apr '15

Dear all, Upon trying to install numpy using 'pip install numpy' in a virtualenv, I get the following error messages: creating build/temp.linux-x86_64-2.7/numpy/random/mtrand compile options: '-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Inumpy/core/include -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.7 -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -c' gcc: numpy/random/mtrand/distributions.c numpy/random/mtrand/distributions.c: In function ‘loggam’: numpy/random/mtrand/distributions.c:892:1: internal compiler error: Illegal instruction } ^ Please submit a full bug report, with preprocessed source if appropriate. See <http://bugzilla.redhat.com/bugzilla> for instructions. Preprocessed source stored into /tmp/ccjkBSd2.out file, please attach this to your bugreport. This leads to the compilation process failing with this error: Cleaning up... Command /home/mescalin/pkerp/.virtualenvs/notebooks/bin/python -c "import setuptools;__file__='/home/mescalin/pkerp/.virtualenvs/notebooks/build/numpy/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file __, 'exec'))" install --record /tmp/pip-c_Cd7B-record/install-record.txt --single-version-externally-managed --install-headers /home/mescalin/pkerp/.virtualenvs/notebooks/include/site/python2.7 failed with error code 1 in /ho me/mescalin/pkerp/.virtualenvs/notebooks/build/numpy Traceback (most recent call last): File "/home/mescalin/pkerp/.virtualenvs/notebooks/bin/pip", line 9, in <module> load_entry_point('pip==1.4.1', 'console_scripts', 'pip')() File "/home/mescalin/pkerp/.virtualenvs/notebooks/lib/python2.7/site-packages/pip/__init__.py", line 148, in main return command.main(args[1:], options) File "/home/mescalin/pkerp/.virtualenvs/notebooks/lib/python2.7/site-packages/pip/basecommand.py", line 169, in main text = '\n'.join(complete_log) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 72: ordinal not in range(128) Have any of you encountered a similar problem before? Thanks in advance, -Peter ================================================ The gcc version is: [pkerp@fluidspace ~]$ gcc --version gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-7) Copyright (C) 2013 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. PS:

2 1

Non-meta indexing improvements discussion
by Sebastian Berg 10 Apr '15

10 Apr '15

Hi all, Let me take a shot at summing up some suggestions to make the indexing less surprising, and maybe we can gather some more in a more concentrated way now. 1. Implement something like `arr.fancy_index[...]` and `arr.ortho_index[...]` (i.e. Jaimes PR is the start for trying this) 2. Add warnings for non-consecutive advanced indexing (i.e. the original example `arr[0, :, index_array]`). 3. I do not know if it possible or useful, but I could imagine a module wide switch (similar to __future__ imports) to change the default indexing behaviour. One more thing, implementing this (especially the "new" indexing) is non-trivial, so as always help beyond just a discussion is appreciated and in my opinion the best way to push an actual change to happen sooner rather then in some far off future. I do not have time for concentrating much on an implementation for a while myself for a while at least. - Sebastian

3 3

Trouble subclassing ndarray
by Elliot 10 Apr '15

10 Apr '15

Hi all, Sorry if this is the wrong forum for a question like this. I'm trying to create an object with multiple inheritance, one of which is from numpy.ndarray. The other gives it cacheable properties and defines a __getattr__ to deliver the cached properties. The initial instantiation is successful, but ufuncs and slices cause an infinite recursion where the __getattr__ function is used but the _cacheable attribute is not set (ie: from __init__ ) I am using docs.scipy.org/doc/numpy/user/basics.subclassing.html as a reference. Here is code that shows my problem (python 2.7, numpy 1.8.2) ===================== from __future__ import print_function import numpy as np class Cacheable(object): def __init__(self,*args,**kwargs): self._cacheable = {} def __getattr__(self,key): print("getting %s"%key) if key in self._cacheable: print(" found it") self._cacheable[key]() return self.__dict__[key] #if chache function does't update # data you're going to have a bad time else: raise AttributeError def _clear(self): '''clears derived properties''' for key in self._cacheable: if key in self.__dict__: del self.__dict__[key] class BaseGeometry(np.ndarray,Cacheable): '''Numpy array with extra attributes that are cacheable arrays''' def __new__(cls,input_array,dtype=np.float64,*args,**kwargs): # Input array is an already formed ndarray instance # We first cast to be our class type obj = np.asarray(input_array,dtype=dtype).view(cls) # Finally, we must return the newly created object: return obj def __init__(self,dims=None,dtype=np.float64,readonly=True): #TODO: sort through args and kwargs to make better self.readonly=readonly if readonly: self.flags.writeable=False self.dims=dims self._dtype = dtype Cacheable.__init__(self) def writeable_copy(self): ret = np.copy(self) ret.flags.writeable = True return ret def __array_finalize_(self,obj): #New object, will be created in __new__ print("array_finalize") if obj is None: return # created from slice or template print("finalizing slice") self._cacheable = getattr(obj, '_cacheable', None) if __name__ == "__main__": n = 5 test = BaseGeometry(np.random.randint(-25,25,(n,2))) print("this works:",test._cacheable) broken = test[1:4] #interestingly, no problem here print(broken) #infinite recursion =================== array_finalize is never called. output is: this works: {} getting _cacheable getting _cacheable getting _cacheable [and on and on] getting _cacheable getting _cacheable <repr(<__main__.BaseGeometry at 0x7fa42e2921b8>) failed: RuntimeError: maximum recursion depth exceeded while calling a Python object> -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Trouble-subclassing-ndarray-tp4… Sent from the Numpy-discussion mailing list archive at Nabble.com.

4 4

On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
by Nathaniel Smith 10 Apr '15

10 Apr '15

On Apr 5, 2015 7:04 AM, "Robert Kern" <robert.kern(a)gmail.com> wrote: > > On Sat, Apr 4, 2015 at 10:38 PM, Nathaniel Smith <njs(a)pobox.com> wrote: > > > > On Apr 4, 2015 4:12 AM, "Todd" <toddrjen(a)gmail.com> wrote: > > > > > > There was no break as large as this. In fact I would say this is even a larger change than any individual change we saw in the python 2 to 3 switch. The basic mechanics of indexing are just too fundamental and touch on too many things to make this sort of change feasible. > > > > I'm afraid I'm not clever enough to know how large or feasible a change is without even seeing the proposed change. > > It doesn't take any cleverness. The change in question was to make the default indexing semantics to orthogonal indexing. No matter the details of the ultimate proposal to achieve that end, it has known minimum consequences, at least in the broad outline. Current documentation and books become obsolete for a fundamental operation. Current code must be modified by some step to continue working. These are consequences inherent in the end, not just the means to the end; we don't need a concrete proposal in front of us to know what they are. There are ways to mitigate these consequences, but there are no silver bullets that eliminate them. And we can compare those consequences to approaches like Jaime's that achieve a majority of the benefits of such a change without any of the negative consequences. That comparison does not bode well for any proposal. Ok, let me try to make my point another way. I don't actually care at this stage in the discussion whether the change is ultimately viable. And I don't think you should either. (For values of "you" that includes everyone in the discussion, not picking on Robert in particular :-).) My point is that rational, effective discussion requires giving ideas room to breath. Sometimes ideas turn out to be not as bad as they looked. Sometimes it turns out that they are, but there's some clever tweak that gives you 95% of the benefits for 5% of the cost. Sometimes you generate a better understanding of the tradeoffs that subsequently informs later design decisions. Sometimes working through the details makes both sides realize that there's a third option that solves both their problems. Sometimes you merely get a very specific understanding of why the whole approach is unreasonable that you can then, say, take to the pandas and netcdf developers as evidence of that you made a good faith effort and ask them to meet you half way. And all these things require understanding the specifics of what *exactly* works or doesn't work about about idea. IMHO, it's extremely misleading at this stage to make any assertion about whether Jaime's approach gives the "majority of benefits of such a change" is extremely misleading at this stage: not because it's wrong, but because it totally short-circuits the discussion about what benefits we care about. Jaime's patch certainly has merits, but does it help us make numpy and pandas/netcdf's more compatible? Does it make it easier for Eric to teach? Those are some specific merits that we might care about a lot, and for which Jaime's patch may or may not help much. But that kind of nuance gets lost when we jump straight to debating thumbs-up versus thumbs-down. I cross-my-heart promise that under the current regime, no PR breaking fancy indexing would ever get anywhere *near* numpy master without *extensive* discussion and warnings on the list. The core devs just spent weeks quibbling about whether a PR that adds a new field to the end of the dtype struct would break ABI backcompat (we're now pretty sure it doesn't), and the current standard we enforce is that every PR that touches public API needs a list discussion, even minor extensions with no compatibility issues at all. No one is going to sneak anything by anyone. Plus, I dunno, our current approach to discussions just seems to make things hostile and shouty and unpleasant. If a grad student or junior colleague comes to you with an idea where you see some potentially critical flaw, do you yell THAT WILL NEVER WORK and kick them out of your office? Or, do you maybe ask a few leading questions and see where they go? I think things will work better if the next time something like this comes up, *one* person just says "hmm, interesting idea, but the backcompat issues seem pretty severe; do you have any ideas about how to mitigate that?", and then we let that point be taken as having been made and see where the discussion goes. Maybe we can all give it a try? -n

12 27

ANN: HDF5 for Python 2.5.0
by Andrew Collette 10 Apr '15

10 Apr '15

Announcing HDF5 for Python (h5py) 2.5.0 ======================================== The h5py team is happy to announce the availability of h5py 2.5.0. This release introduces experimental support for the highly-anticipated "Single Writer Multiple Reader" (SWMR) feature in the upcoming HDF5 1.10 release. SWMR allows sharing of a single HDF5 file between multiple processes without the complexity of MPI or multiprocessing-based solutions. This is an experimental feature that should NOT be used in production code. We are interested in getting feedback from the broader community with respect to performance and the API design. For more details, check out the h5py user guide: http://docs.h5py.org/en/latest/swmr.html SWMR support was contributed by Ulrik Pedersen. What's h5py? ------------ The h5py package is a Pythonic interface to the HDF5 binary data format. It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want. Documentation is at: http://docs.h5py.org Changes ------- * Experimental SWMR support * Group and AttributeManager classes now inherit from the appropriate ABCs * Fixed an issue with 64-bit float VLENS * Cython warning cleanups related to "const" * Entire code base ported to "six"; 2to3 removed from setup.py Acknowledgements --------------- This release incorporates changes from, among others: * Ulrik Pedersen * James Tocknell * Will Parkin * Antony Lee * Peter H. Li * Peter Colberg * Ghislain Antony Vaillant Where to get it --------------- Downloads, documentation, and more are available at the h5py website: http://www.h5py.org

4 5

what files to include with compiled fortran extension on Mac
by Mark Bakker 09 Apr '15

09 Apr '15

Hello list, I want to send somebody my compiled fortran extension on a Mac (compiled with f2py and gfortran). Problem is that it doesn't work on other Macs unless they also instal xcode (2 GB, yikes!) and gfortran. So apparently there are some additional files missing when I just send the compiled extension. Does anybody know what other files to include or (better) how to compile a fortran extension without needing to send any additional files? Thanks, Mark

1 0

SIAM meeting in Snowbird anyone?
by Sebastian Berg 09 Apr '15

09 Apr '15

Hey, since I am not in the US often and the SIAM conference is pretty large, I was wondering if some more of our community will be at the SIAM conference in Snowbird around May 17th-21st and would like to meet up then. - Sebastian

1 0

FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
by Andreas Hilboll 09 Apr '15

09 Apr '15

Hi all, I'm commonly using function signatures like def myfunc(a, b, c=None): if c is None: # do something ... ... where c is an optional array argument. For some time now, I'm getting a FutureWarning: comparison to `None` will result in an elementwise object comparison in the future from the "c is None" comparison. I'm wondering what would be the best way to do this check in a future-proof way? Best, -- Andreas.

2 2

Behavior of np.random.multivariate_normal with bad covariance matrices
by Blake Griffith 07 Apr '15

07 Apr '15

I have an open PR which lets users control the checks on the input covariance matrix. The matrix is required to be symmetric and positve semi-definite (PSD). The current behavior is that NumPy raises a warning if the matrix is not PSD, and does not even check for symmetry. I added a symmetry check, which raises a warning when the input is not symmetric. And added two keyword args which users can use to turn off the checks/warnings when the matrix is ill formed. So this would only cause another new warning to be raised in existing code. This is needed because sometimes the covariance matrix is only *almost* symmetric or PSD due to roundoff error. Thoughts? PR: https://github.com/numpy/numpy/pull/5726

2 2

Research position in the Brazilian Research Institute for Science and Neurotechnology - BRAINN
by Thiago Franco Moraes 07 Apr '15

07 Apr '15

Research position in the Brazilian Research Institute for Science and Neurotechnology – BRAINN Postdoc researcher to work with software development for medical imaging The Brazilian Research Institute for Neuroscience and Neurotechnology (BRAINN) (www.brainn.org.br) focuses on the investigation of basic mechanisms leading to epilepsy and stroke, and the injury mechanisms that follow disease onset and progression. This research has important applications related to prevention, diagnosis, treatment and rehabilitation and will serve as a model for better understanding normal and abnormal brain function. The BRAINN Institute is composed of 10 institutions from Brazil and abroad and hosted by State University of Campinas (UNICAMP). Among the associated institutions is Renato Archer Information Technology Center (CTI) that has a specialized team in open-source software development for medical imaging (www.cti.gov.br/invesalius) and 3D printing applications for healthcare. CTI is located close the UNICAMP in the city of Campinas, State of São Paulo in a very technological region of Brazil and is looking for a postdoc researcher to work with software development for medical imaging related to the imaging analysis, diagnosis and treatment of brain diseases. The postdoc position is for two years with the possibility of being renovated for more two years. Education - PhD in computer science, computer engineering, mathematics, physics or related. Requirements - Digital image processing (Medical imaging) - Computer graphics (basic) Benefits 6.143,40 Reais per month free of taxes (about US$ 2.000,00); 15% technical reserve for conferences participation and specific materials acquisition; Interested Send curriculum to: jorge.silva(a)cti.gov.br with subject “Postdoc position” Applications reviews will begin April 30, 2015 and continue until the position is filled.

1 0

Multidimensional Indexing
by Nicholas Devenish 07 Apr '15

07 Apr '15

With the indexing example from the documentation: y = np.arange(35).reshape(5,7) Why does selecting an item from explicitly every row work as I’d expect: >>> y[np.array([0,1,2,3,4]),np.array([0,0,0,0,0])] array([ 0, 7, 14, 21, 28]) But doing so from a full slice (which, I would naively expect to mean “Every Row”) has some…other… behaviour: >>> y[:,np.array([0,0,0,0,0])] array([[ 0, 0, 0, 0, 0], [ 7, 7, 7, 7, 7], [14, 14, 14, 14, 14], [21, 21, 21, 21, 21], [28, 28, 28, 28, 28]]) What is going on in this example, and how do I get what I expect? By explicitly passing in an extra array with value===index? What is the rationale for this difference in behaviour? Thanks, Nick

4 3

OS X wheels: speed versus multiprocessing
by Nathaniel Smith 07 Apr '15

07 Apr '15

Hi all, Starting with 1.9.1, the official numpy OS X wheels (the ones you get by doing "pip install numpy") have been built to use Apple's Accelerate library for linear algebra. This is fast, but it breaks multiprocessing in obscure ways (e.g. see this user report: https://github.com/numpy/numpy/issues/5752). Unfortunately, there is no obvious best solution to what linear algebra package to use, so we have to make a decision as to which set of compromises we prefer. Options: Accelerate: fast, but breaks multiprocessing as above. OpenBLAS: fast, but Julian raised concerns about its trustworthiness last year ( http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069659.html). Possibly things have improved since then (I get the impression that they've gotten some additional developer attention from the Julia community), but I don't know. Atlas: slower (faster than reference blas but definitely slower than fancy options like the above), but solid. My feeling is that for wheels in particular it's more important that everything "just work" than that we get the absolute fastest speeds. And this is especially true for the multiprocessing issue, given that it's a widely used part of the stdlib, the failures are really obscure/confusing, and there is no workaround for python 2 which is still where a majority of our users still are. So I'd vote for using either atlas or OpenBLAS. (And would defer to Julian and Matthew about which to choose between these.) Any opinions, objections? -n

3 6

1.10 release again.
by Charles R Harris 07 Apr '15

07 Apr '15

Hi All, I'd like to mark current PR's for inclusion in 1.10. If there is something that you want to have in the release, please mention it here by PR #.I think new enhancement PR's should be considered for 1.11 rather than 1.10, but bug fixes will go in. There is some flexibility, of course, as there are always last minute items that come up when release contents are begin decided. Chuck

3 4

Re: [Numpy-discussion] IDE's for numpy development?
by Suzen, Mehmet 07 Apr '15

07 Apr '15

Hi Chuck, Spider is good. If you are coming from Matlab world. http://spyder-ide.blogspot.co.uk/ I don't think it supports C. But Maybe you are after Eclipse. Best, -m

2 1

Advanced indexing: "fancy" vs. orthogonal
by R Hattersley 05 Apr '15

05 Apr '15

There are two different interpretations in common use of how to handle multi-valued (array/sequence) indexes. The numpy style is to consider all multi-valued indices together which allows arbitrary points to be extracted. The orthogonal style (e.g. as provided by netcdf4-python) is to consider each multi-valued index independently. For example: >>> type(v)<type 'netCDF4.Variable'>>>> v.shape (240, 37, 49)>>> v[(0, 1), (0, 2, 3)].shape (2, 3, 49)>>> np.array(v)[(0, 1), (0, 2, 3)].shape Traceback (most recent call last): File "<stdin>", line 1, in <module>IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (2,) (3,) In a netcdf4-python GitHub issue <https://github.com/Unidata/netcdf4-python/issues/385> the authors of various orthogonal indexing packages have been discussing how to distinguish the two behaviours and have currently settled on a boolean __orthogonal_indexing__ attribute. 1. Is there any appetite for adding that attribute (with the value `False`) to ndarray? 2. As suggested by shoyer <https://github.com/Unidata/netcdf4-python/issues/385#issuecomment-87775034>, is there any appetite for adding an alternative indexer to ndarray where __orthogonal_indexing__ = True? For example: myarray.ix_[(0,1), (0, 2, 3)] Richard

15 38

NPY_SEPARATE_COMPILATION and RELAXED_STRIDES_CHECKING
by Charles R Harris 05 Apr '15

05 Apr '15

Hi All, Just to raise the question if these two options should be removed at some point? The current default value for both is 0, so we have separate compilation and relaxed strides checking by default. Chuck

4 7

GSoC students: please read
by Ralf Gommers 05 Apr '15

05 Apr '15

Hi all, It's great to see that this year there are a lot of students interested in doing a GSoC project with Numpy or Scipy. So far five proposals have been submitted, and it looks like several more are being prepared now. I'd like to give you a bit of advice as well as an idea of what's going to happen in the few weeks. The deadline for submitting applications is 27 March. Don't wait until the last day to submit your proposal! It has happened before that Melange was overloaded and unavailable - the Google program admins will not accept that as an excuse and allow you to submit later. So as soon as your proposal is in good shape, put it in. You can still continue revising it. >From 28 March until 13 April we will continue to interact with you, as we request slots from the PSF and rank the proposals. We don't know how many slots we will get this year, but to give you an impression: for the last two years we got 2 slots. Hopefully we can get more this year, but that's far from certain. Our ranking will be based on a combination of factors: the interaction you've had with potential mentors and the community until now (and continue to have), the quality of your submitted PRs, quality and projected impact of your proposal, your enthusiasm, match with potential mentors, etc. We will also organize a video call (Skype / Google Hangout / ...) with each of you during the first half of April to be able to exchange ideas with a higher communication bandwidth medium than email. Finally a note on mentoring: we will be able to mentor all proposals submitted or suggested until now. Due to the large interest and technical nature of a few topics it has in some cases taken a bit long to provide feedback on draft proposals, however there are no showstoppers in this regard. Please continue improving your proposals and working with your potential mentors. Cheers, Ralf

2 4

Re: [Numpy-discussion] Fix masked arrays to properly edit views
by John Kirkham 04 Apr '15

04 Apr '15

Hey Eric, That's a good point. I remember seeing this behavior before and thought it was a bit odd. Best, John > On Mar 16, 2015, at 2:20 AM, numpy-discussion-request(a)scipy.org wrote: > > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Re: Fix masked arrays to properly edit views (Eric Firing) > 2. Rewrite np.histogram in c? (Robert McGibbon) > 3. numpy.stack -- which function, if any, deserves the name? > (Stephan Hoyer) > 4. Re: Rewrite np.histogram in c? (Jaime Fern?ndez del R?o) > 5. Re: Rewrite np.histogram in c? (Robert McGibbon) > 6. Re: Rewrite np.histogram in c? (Robert McGibbon) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 14 Mar 2015 14:01:04 -1000 > From: Eric Firing <efiring(a)hawaii.edu> > Subject: Re: [Numpy-discussion] Fix masked arrays to properly edit > views > To: numpy-discussion(a)scipy.org > Message-ID: <5504CBC0.1080502(a)hawaii.edu> > Content-Type: text/plain; charset=windows-1252; format=flowed > >> On 2015/03/14 1:02 PM, John Kirkham wrote: >> The sample case of the issue ( >> https://github.com/numpy/numpy/issues/5558 ) is shown below. A proposal >> to address this behavior can be found here ( >> https://github.com/numpy/numpy/pull/5580 ). Please give me your feedback. >> >> >> I tried to change the mask of `a` through a subindexed view, but was >> unable. Using this setup I can reproduce this in the 1.9.1 version of NumPy. >> >> import numpy as np >> >> a = np.arange(6).reshape(2,3) >> a = np.ma.masked_array(a, mask=np.ma.getmaskarray(a), shrink=False) >> >> b = a[1:2,1:2] >> >> c = np.zeros(b.shape, b.dtype) >> c = np.ma.masked_array(c, mask=np.ma.getmaskarray(c), shrink=False) >> c[:] = np.ma.masked >> >> This yields what one would expect for `a`, `b`, and `c` (seen below). >> >> masked_array(data = >> [[0 1 2] >> [3 4 5]], >> mask = >> [[False False False] >> [False False False]], >> fill_value = 999999) >> >> masked_array(data = >> [[4]], >> mask = >> [[False]], >> fill_value = 999999) >> >> masked_array(data = >> [[--]], >> mask = >> [[ True]], >> fill_value = 999999) >> >> Now, it would seem reasonable that to copy data into `b` from `c` one >> can use `__setitem__` (seen below). >> >> b[:] = c >> >> This results in new data and mask for `b`. >> >> masked_array(data = >> [[--]], >> mask = >> [[ True]], >> fill_value = 999999) >> >> This should, in turn, change `a`. However, the mask of `a` remains >> unchanged (seen below). >> >> masked_array(data = >> [[0 1 2] >> [3 0 5]], >> mask = >> [[False False False] >> [False False False]], >> fill_value = 999999) > > I agree that this behavior is wrong. A related oddity is this: > > In [24]: a = np.arange(6).reshape(2,3) > In [25]: a = np.ma.array(a, mask=np.ma.getmaskarray(a), shrink=False) > In [27]: a.sharedmask > True > In [28]: a.unshare_mask() > In [30]: b = a[1:2, 1:2] > In [31]: b[:] = np.ma.masked > In [32]: b.sharedmask > False > In [33]: a > masked_array(data = > [[0 1 2] > [3 -- 5]], > mask = > [[False False False] > [False True False]], > fill_value = 999999) > > It looks like the sharedmask property simply is not being set and > interpreted correctly--a freshly initialized array has sharedmask True; > and after setting it to False, changing the mask of a new view *does* > change the mask in the original. > > Eric > >> >> Best, >> John >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion(a)scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > ------------------------------ > > Message: 2 > Date: Sun, 15 Mar 2015 21:32:49 -0700 > From: Robert McGibbon <rmcgibbo(a)gmail.com> > Subject: [Numpy-discussion] Rewrite np.histogram in c? > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > <CAN4+E8Ff_Ck-9GBRCbSTq6qPiuGxgKeiX3+kKrXn4NM-Lnn6rg(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi, > > Numpy.histogram is implemented in python, and is a little sluggish. This > has been discussed previously on the mailing list, [1, 2]. It came up in a > project that I maintain, where a new feature is bottlenecked by > numpy.histogram, and one developer suggested a faster implementation in > cython [3]. > > Would it make sense to reimplement this function in c? or cython? Is moving > functions like this from python to c to improve performance within the > scope of the development roadmap for numpy? I started implementing this a > little bit in c, [4] but I figured I should check in here first. > > -Robert > > [1] > http://scipy-user.10969.n7.nabble.com/numpy-histogram-is-slow-td17208.html > [2] http://numpy-discussion.10968.n7.nabble.com/Fast-histogram-td9359.html > [3] https://github.com/mdtraj/mdtraj/pull/734 > [4] https://github.com/rmcgibbo/numpy/tree/histogram >

1 0

SciPy 2015 Conference Updates - Call for talks extended to 4/10, registration open, keynotes announced, John Hunter Plotting Contest
by Courtenay Godshall (Enthought) 03 Apr '15

03 Apr '15

---------------------------------------------------------------------------- ----------------------------------------------------------- **LAST CALL FOR SCIPY 2015 TALK AND POSTER SUBMISSIONS - EXTENSION TO 4/10* ---------------------------------------------------------------------------- ----------------------------------------------------------- SciPy 2015 will include 3 major topic tracks and 7 mini-symposia tracks. Submit a proposal on the SciPy 2015 website: <http://scipy2015.scipy.org> http://scipy2015.scipy.org. If you have any questions or comments, feel free to contact us at: <mailto:scipy-organizers@scipy.org> scipy-organizers(a)scipy.org <http://scipy2015.scipy.org> . You can also follow @scipyconf on Twitter or sign up for the mailing list on the website for the latest updates! Major topic tracks include: - Scientific Computing in Python (General track) - Python in Data Science - Quantitative Finance and Computational Social Sciences Mini-symposia will include the applications of Python in: - Astronomy and astrophysics - Computational life and medical sciences - Engineering - Geographic information systems (GIS) - Geophysics - Oceanography and meteorology - Visualization, vision and imaging -------------------------------------------------------------------------- **SCIPY 2015 REGISTRATION IS OPEN** Please register ASAP to help us get a good headcount and open the conference to as many people as we can. PLUS, everyone who registers before May 15 will not only get early bird discounts, but will also be entered in a drawing for a free registration (via refund or extra)! Register on the website at http://scipy2015.scipy.org -------------------------------------------------------------------------- **SCIPY 2015 KEYNOTE SPEAKERS ANNOUNCED** Keynote speakers were just announced and include Wes McKinney, author of Pandas; Chris Wiggins, Chief Data Scientist for The New York Times; and Jake VanderPlas, director of research at the University of Washington's eScience Institute and core contributor to a number of scientific Python libraries including sci-kit learn and AstroML. -------------------------------------------------------------------------- **ENTER THE SCIPY JOHN HUNTER EXCELLENCE IN PLOTTING CONTEST - DUE 4/13** In memory of John Hunter, creator of matplotlib, we are pleased to announce the Third Annual SciPy John Hunter Excellence in Plotting Competition. This open competition aims to highlight the importance of quality plotting to scientific progress and showcase the capabilities of the current generation of plotting software. Participants are invited to submit scientific plots to be judged by a panel. The winning entries will be announced and displayed at the conference. John Hunter's family is graciously sponsoring cash prizes up to $1,000 for the winners. We look forward to exciting submissions that push the boundaries of plotting! See details here: http://scipy2015.scipy.org/ehome/115969/276538/ Entries must be submitted by April 13, 2015 via e-mail to plotting-contest(a)scipy.org -------------------------------------------------------------------------- **CALENDAR AND IMPORTANT DATES** --Sprint, Birds of a Feather, Financial Aid and Talk submissions are open NOW --Apr 10, 2015: Talk and Poster submission deadline --Apr 13, 2015: Plotting contest submissions due --Apr 15, 2015: Financial aid application deadline --Apr 17, 2015: Tutorial schedule announced --May 1, 2015: General conference speakers & schedule announced --May 15, 2015 (or 150 registrants): Early-bird registration ends --Jun 1, 2015: BoF submission deadline --Jul 6-7, 2015: SciPy 2015 Tutorials --Jul 8-10, 2015: SciPy 2015 General Conference --Jul 11-12, 2015: SciPy 2015 Sprints

1 0

IDE's for numpy development?
by Charles R Harris 03 Apr '15

03 Apr '15

Hi All, In a recent exchange Mark Wiebe suggested that the lack of support for numpy development in Visual Studio might limit the number of developers attracted to the project. I'm a vim/console developer myself and make no claim of familiarity with modern development tools, but I wonder if such tools might now be available for Numpy. A quick google search turns up a beta plugin for Visual Studio, <https://pytools.codeplex.com/>, and there is an xcode IDE for the mac that apparently offers some Python support. The two things that I think are required are: 1) support for mixed C, python developement and 2) support for building and testing numpy. I'd be interested in information from anyone with experience in using such an IDE and ideas of how Numpy might make using some of the common IDEs easier. Thoughts? Chuck

8 9

How to Force Storage Order
by Klemm, Michael 02 Apr '15

02 Apr '15

Dear all, I have found a bug in one of my codes and the way it passes a Numpy matrix to MKL's dgemm routine. Up to now I was assuming that the matrixes are using C order. I guess I have to correct this assumption :-). I have found that the numpy.linalg.svd algorithm creates the resulting U, sigma, and V matrixes with Fortran storage. Is there any way to force these kind of algorithms to not change the storage order? That would make passing the matrixes to the native dgemm operation much easier. Cheers, -michael Dr.-Ing. Michael Klemm Senior Application Engineer Software and Services Group Developer Relations Division Phone +49 89 9914 2340 Cell +49 174 2417583 Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

4 4

Adding 'where' to ufunc methods?
by Jaime Fernández del Río 02 Apr '15

02 Apr '15

This question on StackOverflow: http://stackoverflow.com/questions/29394377/minimum-of-numpy-array-ignoring… Got me thinking that I had finally found a use for the 'where' kwarg of ufuncs. Unfortunately it is only provided for the ufunc itself, but not for any of its methods. Is there any fundamental reason these were not implemented back in the day? Any frontal opposition to having them now? Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

4 4