Mailman 3 August 2011 - NumPy-Discussion

Decimal arrays?
by Chris Withers 24 Aug '11

24 Aug '11

Hi All, What's the best type of array to use for decimal values? (ie: where I care about precision and want to avoid any possible rounding errors) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

5 9

NA masks for NumPy are ready to test
by Mark Wiebe 24 Aug '11

24 Aug '11

It's taken a lot of changes to get the NA mask support to its current point, but the code ready for some testing now. You can read the work-in-progress release notes here: https://github.com/m-paradox/numpy/blob/missingdata/doc/release/2.0.0-notes… To try it out, check out the missingdata branch from my github account, here, and build in the standard way: https://github.com/m-paradox/numpy The things most important to test are: * Confirm that existing code still works correctly. I've tested against SciPy and matplotlib. * Confirm that the performance of code not using NA masks is the same or better. * Try to do computations with the NA values, find places they don't work yet, and nominate unimplemented functionality important to you to be next on the development list. The release notes have a preliminary list of implemented/unimplemented functions. * Report any crashes, build problems, or unexpected behaviors. In addition to adding the NA mask, I've also added features and done a few performance changes here and there, like letting reductions like sum take lists of axes instead of being a single axis or all of them. These changes affect various bugs like http://projects.scipy.org/numpy/ticket/1143 and http://projects.scipy.org/numpy/ticket/533. Thanks! Mark Here's a small example run using NAs: >>> import numpy as np >>> np.__version__ '2.0.0.dev-8a5e2a1' >>> a = np.random.rand(3,3,3) >>> a.flags.maskna = True >>> a[np.random.rand(3,3,3) < 0.5] = np.NA >>> a array([[[NA, NA, 0.11511708], [ 0.46661454, 0.47565512, NA], [NA, NA, NA]], [[NA, 0.57860351, NA], [NA, NA, 0.72012669], [ 0.36582123, NA, 0.76289794]], [[ 0.65322748, 0.92794386, NA], [ 0.53745165, 0.97520989, 0.17515083], [ 0.71219688, 0.5184328 , 0.75802805]]]) >>> np.mean(a, axis=-1) array([[NA, NA, NA], [NA, NA, NA], [NA, 0.56260412, 0.66288591]]) >>> np.std(a, axis=-1) array([[NA, NA, NA], [NA, NA, NA], [NA, 0.32710662, 0.10384331]]) >>> np.mean(a, axis=-1, skipna=True) /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2474: RuntimeWarning: invalid value encountered in true_divide um.true_divide(ret, rcount, out=ret, casting='unsafe') array([[ 0.11511708, 0.47113483, nan], [ 0.57860351, 0.72012669, 0.56435958], [ 0.79058567, 0.56260412, 0.66288591]]) >>> np.std(a, axis=-1, skipna=True) /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2707: RuntimeWarning: invalid value encountered in true_divide um.true_divide(arrmean, rcount, out=arrmean, casting='unsafe') /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2730: RuntimeWarning: invalid value encountered in true_divide um.true_divide(ret, rcount, out=ret, casting='unsafe') array([[ 0. , 0.00452029, nan], [ 0. , 0. , 0.19853835], [ 0.13735819, 0.32710662, 0.10384331]]) >>> np.std(a, axis=(1,2), skipna=True) array([ 0.16786895, 0.15498008, 0.23811937])

5 24

ImportError: dynamic module does not define init function (initmultiarray)
by Dominique Orban 24 Aug '11

24 Aug '11

Dear list, I'm embedding Python inside a C program to pull functions from user-supplied Python modules. All is well except when the user-supplied module imports numpy. Requesting a stack trace when an exception occurs reveals the following: --- Traceback (most recent call last): File "/Users/dpo/.virtualenvs/matrox/matrox/curve.py", line 3, in <module> import numpy as np File "/Users/dpo/.virtualenvs/matrox/lib/python2.7/site-packages/numpy/__init__.py", line 137, in <module> import add_newdocs File "/Users/dpo/.virtualenvs/matrox/lib/python2.7/site-packages/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/Users/dpo/.virtualenvs/matrox/lib/python2.7/site-packages/numpy/lib/__init__.py", line 4, in <module> from type_check import * File "/Users/dpo/.virtualenvs/matrox/lib/python2.7/site-packages/numpy/lib/type_check.py", line 8, in <module> import numpy.core.numeric as _nx File "/Users/dpo/.virtualenvs/matrox/lib/python2.7/site-packages/numpy/core/__init__.py", line 5, in <module> import multiarray ImportError: dynamic module does not define init function (initmultiarray) --- (here, "curve.py" is the user-supplied module in question.) The symbol initmultiarray *is* defined in multiarray.so so I'm wondering if anybody has suggestions as to what the problem may be here. A bit of Googling reveals the following: * The 3rd example of Section 31.2.5 of http://www.swig.org/Doc1.3/Python.html says "This error is almost always caused when a bad name is given to the shared object file. For example, if you created a file example.so instead of _example.so you would get this error." * Item #2 in the FAQ at http://biggles.sourceforge.net/doc/1.5/faq says "This is a problem with your module search path. Python is loading [multiarray].so as a module instead of [multiarray].py" But I don't have any multiarray.py. I have other multiarray.so's, but they're not in my search path. And I'm not finding any _multiarray.so with a leading underscore. So I am lead to ask: should multiarray.so really be called _multiarray.so? If not, any idea what the problem is? I'm using Python 2.7.2 compiled as a framework using Homebrew on OSX 10.6.8 and Numpy 1.6.1 installed from PyPi a day or two ago. Thanks much in advance! -- Dominique

2 1

Bug or feature?
by Konrad Hinsen 23 Aug '11

23 Aug '11

Hi everyone, I just stumbled on a behavior in NumPy for which I can't find an explanation in the documentation. I wonder whether this is a bug or an undocumented (or badly documented) feature: -------------------------------------------------------------------------------------- import numpy t = numpy.dtype([("rotation", numpy.float64, (3, 3)), ("translation", numpy.float64, (3,))]) # works a1 = numpy.array([], dtype=t) # doesn't work a2 = numpy.array((), dtype=t) # -> ValueError: size of tuple must match number of fields. -------------------------------------------------------------------------------------- According to my understanding of how numpy.array should work, it shouldn't make a difference if the first argument is a list or a tuple, but in this case there is a difference. Konrad.

4 4

Wrong treatment of byte order?
by Nadav Horesh 23 Aug '11

23 Aug '11

My system is a 64 bit gentoo linux on core i7 machine. Numpy version 1.6.1 and pyton(s) 2.7.2 and 3.2.1 Problem summary: I tried t invert a matrix of explicit little endian byte-order and got an error. The inversion run properly with a native byte order, and I get a wrong answer with not error message when the matrix is set to big-endian. mat is a 3x3 float64 array >> import numpy as N >>> mat.dtype.byteorder '<' >>> N.linalg.inv(mat) # Refuse to ibvert Traceback (most recent call last): File "<pyshell#107>", line 1, in <module> N.linalg.inv(mat) File "/usr/lib64/python2.7/site-packages/numpy/linalg/linalg.py", line 445, in inv return wrap(solve(a, identity(a.shape[0], dtype=a.dtype))) File "/usr/lib64/python2.7/site-packages/numpy/linalg/linalg.py", line 326, in solve results = lapack_routine(n_eq, n_rhs, a, n_eq, pivots, b, n_eq, 0) LapackError: Parameter a has non-native byte order in lapack_lite.dgesv >>> N.linalg.inv(mat.newbyteorder('=')) # OK array([[ 0.09234453, 0.46163744, 0.2713108 ], [ 0.48886135, 0.51230859, 0.2277598 ], [ 0.48303131, 0.82571266, 0.17551993]]) >>> N.linalg.inv(mat.newbyteorder('>')) # WRONG !!! array([[ 2.39051169e-159, -7.70643158e-157, 5.34087235e-160], [ 2.11823992e+305, 2.37224043e+307, -4.31607382e+304], [ -1.26608299e+304, -1.43225563e+306, 7.22233688e+303]]) Nadav

1 0

Segfault for np.lookfor
by Ben Walsh 22 Aug '11

22 Aug '11

Hi My bad. Very sorry about that, guys. There's a patch for this here: https://github.com/walshb/numpy/tree/fix_np_lookfor_segv And I submitted a pull request. I'll add something to the tests too when I have a little more time. Cheers Ben > ------------------------------ > > Message: 3 > Date: Tue, 16 Aug 2011 12:15:22 -0700 > From: Matthew Brett <matthew.brett(a)gmail.com> > Subject: Re: [Numpy-discussion] Segfault for np.lookfor > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > <CAH6Pt5o3bGyJ=Xm1Hes0GMiNPkUYQG_87RAi3ipGAFaM_u69-w(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > >> >> I opened ticket #1937 for this > >> From git-bisect it looks like the culprit is: > > feb8079070b8a659d7eee1b4acbddf470fd8a81d is the first bad commit > commit feb8079070b8a659d7eee1b4acbddf470fd8a81d > Author: Ben Walsh <b(a)wumpster.com> > Date: Sun Jul 10 12:52:52 2011 +0100 > > BUT: Stop _array_find_type trying to make every list element a > subtype of bool. > > Just to remind me, my procedure was: > > <~/tmp/testfor.py> > #!/usr/bin/env python > import sys > from functools import partial > from subprocess import check_call, Popen, PIPE, CalledProcessError > > caller = partial(check_call, shell=True) > popener = partial(Popen, stdout=PIPE, stderr=PIPE, shell=True) > > try: > caller('git clean -fxd') > caller('python setup.py build_ext -i') > except CalledProcessError: > sys.exit(125) # untestable > > proc = popener('python -c "%s"' % > """import sys > import numpy as np > np.lookfor('cos', output=sys.stdout) > """) > > stdout, stderr = proc.communicate() > if 'Segmentation fault' in stderr: > sys.exit(1) # bad > sys.exit(0) # good > </~/tmp/testfor.py> > > Then, I established the v1.6.1 did not have the segfault, and (man git-bisect): > > git co main-master # current upstream master > git bisect start HEAD v1.6.1 -- > git bisect run ~/tmp/testfor.py > > See y'all, > > Matthew >

2 1

numpy segfaults with ctypes
by Angus McMorland 22 Aug '11

22 Aug '11

Hi all, I'm giving this email a new subject, in case that helps it catch the attention of someone who can fix my problem. I currently cannot upgrade numpy from git to any date more recent than 10 July. Git commit feb8079070b8a659d7ee is the first that causes the problem (according to github, the commit was authored by walshb and committed by m-paradox, in case that jogs anyone's memory). I've tried taking a look at the code diff, but I'm afraid I'm just a user, rather than a developer, and it didn't make much sense. My problem is that python segfaults when I run it with the following code: > from ctypes import Structure, c_double > > #-- copied out of an xml2py generated file > class S(Structure): > pass > S._pack_ = 4 > S._fields_ = [ > ('field', c_double * 2), > ] > #-- > > import numpy as np > print np.version.version > s = S() > print "S", np.asarray(s.field) Thanks, Angus -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh

2 2

RGB <-> HSV in numpy?
by He Shiming 22 Aug '11

22 Aug '11

Hi, I'm wondering how to do RGB <-> HSV conversion in numpy. I found a couple solutions through stackoverflow, but somehow they can't be used in my array format. I understand the concept of conversion, but I'm not that familiar with numpy. My source buffer format is 'RGBA' sequence. I can take it into numpy via: numpy.fromstring(data, 'B').astype('I'). So that nd[0::4] becomes the array for the red channel. After color manipulation, I'll convert it back by nd.astype('B').tostring(). How do I run RGB <-> HSV conversion on the nd array? I'd like to keep SV values in the range of 0-1, and H in 0-360. Thank you. -- Best regards, He Shiming

3 5

Can't mix np.newaxis with boolean indexing
by Benjamin Root 21 Aug '11

21 Aug '11

I could have sworn that this use to work: import numpy as np a = np.random.random((100,)) b = (a > 0.5) print a[b, np.newaxis] But instead, I get this error on the latest master: Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: long() argument must be a string or a number, not 'NoneType' Note, the simple work-around would be "a[b][:, np.newaxis]", but I can't imagine why the intuitive syntax would not be valid. Thanks, Ben Root

3 4

SVD does not converge on "clean" matrix
by dhanjal＠telecom-paristech.fr 20 Aug '11

20 Aug '11

Hi all, I get an error message "numpy.linalg.linalg.LinAlgError: SVD did not converge" when calling numpy.linalg.svd on a "clean" matrix of size (1952, 895). The matrix is clean in the sense that it contains no NaN or Inf values. The corresponding npz file is available here: https://docs.google.com/leaf?id=0Bw0NXKxxc40jMWEyNTljMWUtMzBmNS00NGZmLThhZW… Here is some information about my setup: I use Python 2.7.1 on Ubuntu 11.04 with numpy 1.6.1. Furthermore, I thought the problem might be solved by recompiling numpy with my local ATLAS library (version 3.8.3), and this didn't seem to help. On another machine with Python 2.7.1 and numpy 1.5.1 the SVD does converge however it contains 1 NaN singular value and 3 negative singular values of the order -10^-1 (singular values should always be non-negative). I also tried computing the SVD of the matrix using Octave 3.2.4 and Matlab 7.10.0.499 (R2010a) 64-bit (glnxa64) and there were no problems. Any help is greatly appreciated. Thanks in advance, Charanpal

7 9