Mailman 3 December 2015 - NumPy-Discussion

performance solving system of equations in numpy and MATLAB
by Edward Richards 18 Dec '15

18 Dec '15

I recently did a conceptual experiment to estimate the computational time required to solve an exact expression in contrast to an approximate solution (Helmholtz vs. Helmholtz-Kirchhoff integrals). The exact solution requires a matrix inversion, and in my case the matrix would contain ~15000 rows. On my machine MATLAB seems to perform this matrix inversion with random matrices about 9x faster (20 sec vs 3 mins). I thought the performance would be roughly the same because I presume both rely on the same LAPACK solvers. I will not actually need to solve this problem (even at 20 sec it is prohibitive for broadband simulation), but if I needed to I would reluctantly choose MATLAB . I am simply wondering why there is this performance gap, and if there is a better way to solve this problem in numpy? Thank you, Ned #Python version import numpy as np testA = np.random.randn(15000, 15000) testb = np.random.randn(15000) %time testx = np.linalg.solve(testA, testb) %MATLAB version testA = randn(15000); testb = randn(15000, 1); tic(); testx = testA \ testb; toc();

11 13

Why does np.repeat build a full array?
by Juan Nunez-Iglesias 16 Dec '15

16 Dec '15

Hi, I've recently been using the following pattern to create arrays of a specific repeating value: from numpy.lib.stride_tricks import as_strided value = np.ones((1,), dtype=float) arr = as_strided(value, shape=input_array.shape, strides=(0,)) I can then use arr e.g. to count certain pairs of elements using sparse.coo_matrix. It occurred to me that numpy might have a similar function, and found np.repeat. But it seems that repeat actually creates the full, replicated array, rather than using stride tricks to keep it small. Is there any reason for this? Thanks! Juan.

2 3

Build broken
by Jaime Fernández del Río 16 Dec '15

16 Dec '15

Hi, Travis is repeatedly being unable to complete one of our test builds. It all started after I merged a very simple PR that changed a single word in a docstring, so I have a hard time believing that is the actual cause. Can anyone who actually knows what Travis is doing take a look at the log: https://travis-ci.org/numpy/numpy/builds/96836128 Thanks, Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

4 4

Question about nump.ma.polyfit
by Samuel Dupree 15 Dec '15

15 Dec '15

I'm running Python 2.7.11 from the Anaconda distribution (version 2.4.1) on a MacBook Pro running Mac OS X version 10.11.2 (El Capitan) I'm attempting to use numpy.ma.polyfit to perform a linear least square fit on some data I have. I'm running NumPy version 1.10.1. I've observed that in executing either numpy.polyfit or numpy.ma.polyfit I get the following traceback: /Users/user/anaconda/lib/python2.7/site-packages/numpy/lib/polynomial.py:594: RankWarning: Polyfit may be poorly conditioned warnings.warn(msg, RankWarning) Traceback (most recent call last): File "ComputeEnergy.py", line 132, in <module> coeffs, covar = np.ma.polyfit( xfit, yfit, fit_degree, rcond=rcondv, cov=True ) File "/Users/user/anaconda/lib/python2.7/site-packages/numpy/ma/extras.py", line 1951, in polyfit return np.polyfit(x, y, deg, rcond, full, w, cov) File "/Users/user/anaconda/lib/python2.7/site-packages/numpy/lib/polynomial.py", line 607, in polyfit return c, Vbase * fac ValueError: operands could not be broadcast together with shapes (6,6) (0,) I've attached a stripped down version of the Python program I'm running. Any suggestions? Sam Dupree.

2 1

cross platform build issue: powerpc-e500v2-linux-gnuspe-gcc: error: unrecognized argument in option '-mtune=generic'
by jo 15 Dec '15

15 Dec '15

Hi I am trying to do a cross platform build of numpy (host=x86 target=ppc). This worked for python compiler itself but for numpy I run into following problem. It does select the correct compiler but the options are not correct. please advise... numpy-1.10.2rc2]$ CC=powerpc-e500v2-linux-gnuspe-gcc CXX=powerpc-e500v2-linux-gnuspe-g++ AR=powerpc-e500v2-linux-gnuspe-ar BLAS=None LAPACK=None ATLAS=None python setup.py build bdist_egg --plat-name=powerpc powerpc-e500v2-linux-gnuspe-gcc: _configtest.c powerpc-e500v2-linux-gnuspe-gcc: error: unrecognized argument in option '-mtune=generic' powerpc-e500v2-linux-gnuspe-gcc: note: valid arguments to '-mtune=' are: 401 403 405 405fp 440 440fp 464 464fp 476 476fp 505 601 602 603 603e 604 604e 620 630 740 7400 7450 750 801 821 823 8540 8548 860 970 G3 G4 G5 a2 cell common e300c2 e300c3 e500mc e500mc64 ec603e native power power2 power3 power4 power5 power5+ power6 power6x power7 powerpc powerpc64 rios rios1 rios2 rs64 rsc rsc1 titan powerpc-e500v2-linux-gnuspe-gcc: error: unrecognized argument in option '-mtune=generic' powerpc-e500v2-linux-gnuspe-gcc: note: valid arguments to '-mtune=' are: 401 403 405 405fp 440 440fp 464 464fp 476 476fp 505 601 602 603 603e 604 604e 620 630 740 7400 7450 750 801 821 823 8540 8548 860 970 G3 G4 G5 a2 cell common e300c2 e300c3 e500mc e500mc64 ec603e native power power2 power3 power4 power5 power5+ power6 power6x power7 powerpc powerpc64 rios rios1 rios2 rs64 rsc rsc1 titan powerpc-e500v2-linux-gnuspe-gcc: error: unrecognized argument in option '-mtune=generic' powerpc-e500v2-linux-gnuspe-gcc: note: valid arguments to '-mtune=' are: 401 403 405 405fp 440 440fp 464 464fp 476 476fp 505 601 602 603 603e 604 604e 620 630 740 7400 7450 750 801 821 823 8540 8548 860 970 G3 G4 G5 a2 cell common e300c2 e300c3 e500mc e500mc64 ec603e native power power2 power3 power4 power5 power5+ power6 power6x power7 powerpc powerpc64 rios rios1 rios2 rs64 rsc rsc1 titan powerpc-e500v2-linux-gnuspe-gcc: error: unrecognized argument in option '-mtune=generic' powerpc-e500v2-linux-gnuspe-gcc: note: valid arguments to '-mtune=' are: 401 403 405 405fp 440 440fp 464 464fp 476 476fp 505 601 602 603 603e 604 604e 620 630 740 7400 7450 750 801 821 823 8540 8548 860 970 G3 G4 G5 a2 cell common e300c2 e300c3 e500mc e500mc64 ec603e native power power2 power3 power4 power5 power5+ power6 power6x power7 powerpc powerpc64 rios rios1 rios2 rs64 rsc rsc1 titan failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 266, in <module> setup_package() File "setup.py", line 258, in setup_package setup(**metadata) File "/home/jenkins/Python-2.7.3/tmp/numpy-1.10.2rc2/numpy/distutils/core.py", line 169, in setup return old_setup(**new_attr) File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/home/jenkins/Python-2.7.3/tmp/numpy-1.10.2rc2/numpy/distutils/command/build.py", line 47, in run old_build.run(self) File "/usr/lib64/python2.6/distutils/command/build.py", line 134, in run self.run_command(cmd_name) File "/usr/lib64/python2.6/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/home/jenkins/Python-2.7.3/tmp/numpy-1.10.2rc2/numpy/distutils/command/build_src.py", line 153, in run self.build_sources() File "/home/jenkins/Python-2.7.3/tmp/numpy-1.10.2rc2/numpy/distutils/command/build_src.py", line 164, in build_sources self.build_library_sources(*libname_info) File "/home/jenkins/Python-2.7.3/tmp/numpy-1.10.2rc2/numpy/distutils/command/build_src.py", line 299, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "/home/jenkins/Python-2.7.3/tmp/numpy-1.10.2rc2/numpy/distutils/command/build_src.py", line 387, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 669, in get_mathlib_info raise RuntimeError("Broken toolchain: cannot link a simple C program") RuntimeError: Broken toolchain: cannot link a simple C program Thanks, Jo

1 0

Fwd: Python 3.3 i386 Build
by G Young 15 Dec '15

15 Dec '15

I accidentally subscribed to the defunct discussion mailing list, so my email got rejected the first time I sent to the active mailing list. My question is in the forwarded email below: ---------- Forwarded message ---------- From: G Young <gfyoung17(a)gmail.com> Date: Mon, Dec 14, 2015 at 3:47 PM Subject: re: Python 3.3 i386 Build To: numpy-discussion(a)scipy.org Hello all, I was wondering if anyone else has been running into issues with the Python 3.3 i386 build on Travis. There have been several occasions when the build has strangely failed (and none of the others did) even though there was no plausible way for my changes to break it (see here <https://github.com/numpy/numpy/pull/6831>). Does anyone have an idea as to why this is happening? Thanks! Greg

1 0

NumPy 1.10.2 release
by Charles R Harris 15 Dec '15

15 Dec '15

Hi All, I'm pleased to announce the release of Numpy 1.10.2. This release should take care of the bugs discovered in the 1.10.1 release, some of them severe. Upgrading is strongly advised if you are currently using 1.10.1. Windows binaries and source releases can be found at the usual place on Sourceforge <https://sourceforge.net/projects/numpy/files/NumPy/1.10.2/>. The sources are also available from PyPi <https://pypi.python.org/pypi/numpy/>. Cheers, Chuck

1 0

Numpy intermittent seg fault
by Jacopo Sabbatini 14 Dec '15

14 Dec '15

Hi, I'm experiencing random segmentation faults from numpy. I have generated a core dumped and extracted a stack trace, the following: #0 0x00007f3a8d921d5d in getenv () from /lib64/libc.so.6 #1 0x00007f3a843bde21 in blas_set_parameter () from /opt/apps/sidescananalysis-9.7.1-42-gdd3e068+dev/lib/python2.7/site-packages/numpy/core/../../../../libopenblas.so.0 #2 0x00007f3a843bcd91 in blas_memory_alloc () from /opt/apps/sidescananalysis-9.7.1-42-gdd3e068+dev/lib/python2.7/site-packages/numpy/core/../../../../libopenblas.so.0 #3 0x00007f3a843bd4e5 in blas_thread_server () from /opt/apps/sidescananalysis-9.7.1-42-gdd3e068+dev/lib/python2.7/site-packages/numpy/core/../../../../libopenblas.so.0 #4 0x00007f3a8e09ff18 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f3a8d9ceb2d in clone () from /lib64/libc.so.6 I have experience the segfault from several code paths but they all have the same stack trace. I use conda to run python and numpy. The dump of the packages version is: conda 3.18.9 py27_0 geos 3.3.3 0 matplotlib 1.4.3 np19py27_2 networkx 1.10 py27_0 numpy 1.9.2 py27_2 openblas 0.2.14 3 pandas 0.16.2 np19py27_0 python 2.7.10 2 scikit-image 0.11.3 np19py27_0 scipy 0.15.1 np19py27_0 shapely 1.5.7 py27_0 system 5.8 1 Cheers, Jacopo

3 3

Re: [Numpy-discussion] Memory mapping and NPZ files
by Nathaniel Smith 13 Dec '15

13 Dec '15

On Dec 12, 2015 10:53 AM, "Mathieu Dubois" <mathieu.dubois(a)icm-institute.org> wrote: > > Le 11/12/2015 11:22, Sturla Molden a écrit : >> >> Mathieu Dubois <mathieu.dubois(a)icm-institute.org> wrote: >> >>> The point is precisely that, you can't do memory mapping with Npz files >>> (while it works with Npy files). >> >> The operating system can memory map any file. But as npz-files are >> compressed, you will need to uncompress the contents in your memory mapping >> to make sense of it. > > We agree on that. The goal is to be able to create a np.memmap array from an Npz file. > > >> I would suggest you use PyTables instead of npz-files. >> It allows on the fly compression and uncompression (via blosc) and will >> probably do what you want. > > Yes I know I can use other solutions. The point is that np.load silently ignore the mmap option so I wanted to discuss ways to improve this. I can see a good argument for transitioning to a rule where mmap=False doesn't mmap, mmap=True mmaps if the file is uncompressed and raises an error for compressed files, and mmap="if-possible" gives the current behavior. (It's even possible that the current code would already accept "if-possible" as a alias for True, which would make the transition easier.) Or maybe "never"/"always"/"if-possible" would be better for type consistency reasons, while deprecating the use of bools altogether. But this transition might be a bit more of a hassle, since these definitely won't work on older numpy's. Silently creating a massive temporary file doesn't seem like a great idea to me in any case. Creating a temporary file + mmaping it is essentially equivalent to just loading the data into swappable RAM, except that the swap case is guaranteed not to accidentally leave a massive temp file lying around afterwards. -n

1 0

Re: [Numpy-discussion] FeatureRequest: support for array
by Peter Creasey 13 Dec '15

13 Dec '15

> > > > from itertools import chain > > def fromiter_awesome_edition(iterable): > > elem = next(iterable) > > dtype = whatever_numpy_does_to_infer_dtypes_from_lists(elem) > > return np.fromiter(chain([elem], iterable), dtype=dtype) > > > > I think this would be a huge win for usability. Always getting tripped up by > > the dtype requirement. I can submit a PR if people like this pattern. > > This isn't the semantics of np.array, though -- np.array will look at > the whole input and try to find a common dtype, so this can't be the > implementation for np.array(iter). E.g. try np.array([1, 1.0]) > > I can see an argument for making the dtype= argument to fromiter > optional, with a warning in the docs that it will guess based on the > first element and that you should specify it if you don't want that. > It seems potentially a bit error prone (in the sense that it might > make it easier to end up with code that works great when you test it > but then breaks later when something unexpected happens), but maybe > the usability outweighs that. I don't use fromiter myself so I don't > have a strong opinion. I’m -1 on this, from an occasional user of np.fromiter, also for the np.fromiter([1, 1.5, 2]) ambiguity reason. Pure python does a great job of preventing users from hurting themselves with limited precision arithmetic, however if their application makes them care enough about speed (to be using numpy) and memory (to be using np.fromiter), then it can almost always be assumed that the resulting dtype was important enough to be specified. P

1 0

NumPy-Discussion December 2015