Mailman 3 June 2016 - NumPy-Discussion

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

Design feedback solicitation
by Pavlyk, Oleksandr 14 Jul '16

14 Jul '16

Hi, I am new to this list, so I will start with an introduction. My name is Oleksandr Pavlyk. I now work at Intel Corp. on the Intel Distribution for Python, and previously worked at Wolfram Research for 12 years. My latest project was to write a mirror to numpy.random, named numpy.random_intel. The module uses MKL to sample from different distributions for efficiency. It provides support for different underlying algorithms for basic pseudo-random number generation, i.e. in addition to MT19937, it also provides SFMT19937, MT2203, etc. I recently published a blog about it: https://software.intel.com/en-us/blogs/2016/06/15/faster-random-number-gene… I originally attempted to simply replace numpy.random in the Intel Distribution for Python with the new module, but due to fixed seed backwards incompatibility this results in numerous test failures in numpy, scipy, pandas and other modules. Unlike numpy.random, the new module generates a vector of random numbers at a time, which can be done faster than repeatedly generating the same number of variates one at a time. The source code for the new module is not upstreamed yet, and this email is meant to solicit early community feedback to allow for faster acceptance of the proposed changes. Thank you, Oleksandr

3 4

Accelerate or OpenBLAS for numpy / scipy wheels?
by Matthew Brett 01 Jul '16

01 Jul '16

Hi, I just succeeded in getting an automated dual arch build of numpy and scipy, using OpenBLAS. See the last three build jobs in these two build matrices: https://travis-ci.org/matthew-brett/numpy-wheels/builds/140388119 https://travis-ci.org/matthew-brett/scipy-wheels/builds/140684673 Tests are passing on 32 and 64-bit. I didn't upload these to the usual Rackspace container at wheels.scipy.org to avoid confusion. So, I guess the question now is - should we switch to shipping OpenBLAS wheels for the next release of numpy and scipy? Or should we stick with the Accelerate framework that comes with OSX? In favor of the Accelerate build : faster to build, it's what we've been doing thus far. In favor of OpenBLAS build : allows us to commit to one BLAS / LAPACK library cross platform, when we have the Windows builds working. Faster to fix bugs with good support from main developer. No multiprocessing crashes for Python 2.7. Any thoughts? Cheers, Matthew

7 13

Benchmark regression feeds
by Pauli Virtanen 29 Jun '16

29 Jun '16

Hi, In case someone is interested in getting notifications of performance regressions in the Numpy and Scipy benchmarks, this is available as Atom feeds at: https://pv.github.io/numpy-bench/regressions.xml https://pv.github.io/scipy-bench/regressions.xml -- Pauli Virtanen

2 1

Is numpy.test() supposed to be multithreaded?
by Michael Ward 29 Jun '16

29 Jun '16

Heya, I'm not a numbers guy, but I maintain servers for scientists and researchers who are. Someone pointed out that our numpy installation on a particular server was only using one core. I'm unaware of the who/how the previous version of numpy/OpenBLAS were installed, so I installed them from scratch, and confirmed that the users test code now runs on multiple cores as expected, drastically increasing performance time. Now the user is writing back to say, "my test code is fast now, but numpy.test() is still about three times slower than <some other server we don't manage>". When I watch htop as numpy.test() executes, sure enough, it's using one core. Now I'm not sure if that's the expected behavior or not. Questions: * if numpy.test() is supposed to be using multiple cores, why isn't it, when we've established with other test code that it's now using multiple cores? * if numpy.test() is not supposed to be using multiple cores, what could be the reason that the performance is drastically slower than another server with a comparable CPU, when the user's test code performs comparably? For what it's worth, the users "test" code which does run on multiple cores is as simple as: size=4000 a = np.random.random_sample((size,size)) b = np.random.random_sample((size,size)) x = np.dot(a,b) Whereas this uses only one core: numpy.test() --------------------------- OpenBLAS 0.2.18 was basically just compiled with "make", nothing special to it. Numpy 1.11.0 was installed from source (python setup.py install), using a site.cfg file to point numpy to the new OpenBLAS. Thanks, Mike

5 5

ANN: Bokeh 0.12 Released
by Bryan Van de Ven 28 Jun '16

28 Jun '16

Hi all, On behalf of the Bokeh team, I am pleased to announce the release of version 0.12.0 of Bokeh! This release was a major update, and was focused on areas of layout and styling, new JavaScript APIs for BokehJS, and improvements to the Bokeh Server. But there were many additional improvements in other areas as well. Rather than try to describe all the changes here, I encourage every one to check out the new project blog: https://bokeh.github.io/blog/2016/6/28/release-0-12/ which has details as well as live demonstrations. And as always, see the CHANGELOG and Release Notes for full details. If you are using Anaconda/miniconda, you can install it with conda: conda install bokeh Alternatively, you can also install it with pip: pip install bokeh Full information including details about how to use and obtain BokehJS are at: http://bokeh.pydata.org/en/0.12.0/docs/installation.html Issues, enhancement requests, and pull requests can be made on the Bokeh Github page: https://github.com/bokeh/bokeh Documentation is available at http://bokeh.pydata.org/en/0.12.0 Questions can be directed to the Bokeh mailing list: bokeh(a)continuum.io or the Gitter Chat room: https://gitter.im/bokeh/bokeh Thanks, Bryan Van de Ven Continuum Analytics

1 0

Pip download stats for numpy
by Matthew Brett 26 Jun '16

26 Jun '16

Hi, I just ran a query on pypi downloads [1] using the BigQuery interface to pypi stats [2]. It lists the numpy files downloaded from pypi via a pip install, over the last two weeks, ordered by the number of downloads: 1 100595 numpy-1.11.0.tar.gz 2 97754 numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl 3 38471 numpy-1.8.1-cp27-cp27mu-manylinux1_x86_64.whl 4 20874 numpy-1.11.0-cp27-none-win_amd64.whl 5 20049 numpy-1.11.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 6 17100 numpy-1.10.4-cp27-cp27mu-manylinux1_x86_64.whl 7 15187 numpy-1.10.1.zip 8 14277 numpy-1.11.0-cp35-cp35m-manylinux1_x86_64.whl 9 11538 numpy-1.9.1.tar.gz 10 11272 numpy-1.11.0-cp27-none-win32.whl Of course, it's difficult to know how many of these are from automated builds, such as from travis-ci, but it does look as if manylinux wheels are getting some traction. Cheers, Matthew [1] SELECT COUNT(*) AS downloads, file.filename FROM TABLE_DATE_RANGE( [the-psf:pypi.downloads], TIMESTAMP("20160610"), CURRENT_TIMESTAMP() ) WHERE details.installer.name = 'pip' AND REGEXP_MATCH(file.filename, '^numpy-.*') GROUP BY file.filename ORDER BY downloads DESC LIMIT 1000 [2] https://mail.python.org/pipermail/distutils-sig/2016-May/028986.html

2 1

Numpy 1.11.1 release
by Charles R Harris 26 Jun '16

26 Jun '16

Hi All, I'm pleased to announce the release of Numpy 1.11.1. This release supports Python 2.6 - 2.7, and 3.2 - 3.5 and fixes bugs and regressions found in Numpy 1.11.0 as well as making several build related improvements. Wheels for Linux, Windows, and OSX can be found on PyPI. Sources are available on both PyPI and Sourceforge <https://sourceforge.net/projects/numpy/files/NumPy/1.11.1/>. Thanks to all who were involved in this release, and a special thanks to Matthew Brett for his work on the Linux and Windows wheel infrastructure. The following pull requests have been merged: - 7506 BUG: Make sure numpy imports on python 2.6 when nose is unavailable. - 7530 BUG: Floating exception with invalid axis in np.lexsort. - 7535 BUG: Extend glibc complex trig functions blacklist to glibc < 2.18. - 7551 BUG: Allow graceful recovery for no compiler. - 7558 BUG: Constant padding expected wrong type in constant_values. - 7578 BUG: Fix OverflowError in Python 3.x. in swig interface. - 7590 BLD: Fix configparser.InterpolationSyntaxError. - 7597 BUG: Make np.ma.take work on scalars. - 7608 BUG: linalg.norm(): Don't convert object arrays to float. - 7638 BLD: Correct C compiler customization in system_info.py. - 7654 BUG: ma.median of 1d array should return a scalar. - 7656 BLD: Remove hardcoded Intel compiler flag -xSSE4.2. - 7660 BUG: Temporary fix for str(mvoid) for object field types. - 7665 BUG: Fix incorrect printing of 1D masked arrays. - 7670 BUG: Correct initial index estimate in histogram. - 7671 BUG: Boolean assignment no GIL release when transfer needs API. - 7676 BUG: Fix handling of right edge of final histogram bin. - 7680 BUG: Fix np.clip bug NaN handling for Visual Studio 2015. - 7724 BUG: Fix segfaults in np.random.shuffle. - 7731 MAINT: Change mkl_info.dir_env_var from MKL to MKLROOT. - 7737 BUG: Fix issue on OS X with Python 3.x, npymath.ini not installed. The following developers contributed to this release, developers marked with a '+' are first time contributors. - Allan Haldane - Amit Aronovitch+ - Andrei Kucharavy+ - Charles Harris - Eric Wieser+ - Evgeni Burovski - Loïc Estève+ - Mathieu Lamarre+ - Matthew Brett - Matthias Geier - Nathaniel J. Smith - Nikola Forró+ - Ralf Gommers - Ray Donnelly+ - Robert Kern - Sebastian Berg - Simon Conseil - Simon Gibbons - Sorin Sbarnea+ Chuck

1 0

How best to turn JSON into a CSV or Pandas data frame table?
by David Shi 25 Jun '16

25 Jun '16

Which are the best ways to turn a JSON object into a CSV or Pandas data frame table? Looking forward to hearing from you. Regards. David

1 0

NumPy-Discussion June 2016