Mailman 3 January 2010 - NumPy-Discussion

numpy broken by r8077?
by Fernando Perez 22 Jan '10

22 Jan '10

Hi all, This simple-looking change: http://projects.scipy.org/numpy/changeset/8077 is giving me a wholly broken (unimportable) numpy on ubuntu 9.10, 64-bit: uqbar[junk]> python -c 'import numpy' Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/fperez/usr/opt/lib/python2.6/site-packages/numpy/__init__.py", line 136, in <module> import add_newdocs File "/home/fperez/usr/opt/lib/python2.6/site-packages/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/home/fperez/usr/opt/lib/python2.6/site-packages/numpy/lib/__init__.py", line 4, in <module> from type_check import * File "/home/fperez/usr/opt/lib/python2.6/site-packages/numpy/lib/type_check.py", line 8, in <module> import numpy.core.numeric as _nx File "/home/fperez/usr/opt/lib/python2.6/site-packages/numpy/core/__init__.py", line 5, in <module> import multiarray ImportError: /home/fperez/usr/opt/lib/python2.6/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsASCIIString I reverted to 8076 and it's ok now: uqbar[junk]> python -c 'import numpy;numpy.test()' Running unit tests for numpy NumPy version 1.5.0.dev8076 NumPy is installed in /home/fperez/usr/opt/lib/python2.6/site-packages/numpy Python version 2.6.4 (r264:75706, Dec 7 2009, 18:43:55) [GCC 4.4.1] nose version 0.11.1 .......................................................................... ---------------------------------------------------------------------- Ran 2506 tests in 7.083s OK (KNOWNFAIL=5, SKIP=4) The change is very small and was logged as fixing another bug, but it seems something is amiss now. If it was committed like that I imagine it works on some systems, so is it a problem on my end? I'm using the system python (2.6) and it's an otherwise fairly standard ubuntu box, were I only run ipython, numpy, scipy and matplotlib from dev trees and everything else is stock from the distro. Any help much appreciated, f

3 4

Floating exception
by Nils Wagner 22 Jan '10

22 Jan '10

Hi all, I found a strange problem when I try to import numpy python -v >>> import numpy ... dlopen("/data/home/nwagner/local/lib/python2.5/site-packages/numpy/core/multiarray.so", 2); Floating exception Any idea ? Nils

3 5

numpy.test(): invalid value encountered in {isinf, divide, power, ...}
by Darren Dale 21 Jan '10

21 Jan '10

I haven't been following development on the trunk closely, so I apologize if this is a known issue. I didn't see anything relevant when I searched the list. I just updated my checkout of the trunk, cleaned out the old installation and build/, and reinstalled. When I run the test suite (without specifying the verbosity), I get a slew of warnings like: Warning: invalid value encountered in isinf Warning: invalid value encountered in isfinite I checked on both OS X 10.6 and gentoo linux, with similar results. The test suite reports "ok" at the end with 5 known failures and 4 skipped tests. Darren

5 8

MKL with 64bit crashes
by Kashyap Ashwin 21 Jan '10

21 Jan '10

Hello, I compiled numpy-1.3.0 from sources on Ubuntu-hardy, x86-64 (Intel) with MKL. This is my site.cfg: [mkl] # library_dirs = /opt/intel/mkl/10.0.1.014/lib/32/ library_dirs = /opt/intel/mkl/10.2.2.025/lib/em64t include_dirs = /opt/intel/mkl/10.2.2.025/include lapack_libs = mkl_lapack #mkl_libs = mkl_core, guide, mkl_gf_ilp64, mkl_def, mkl_gnu_thread, iomp5, mkl_vml_mc3 mkl_libs = guide, mkl_core, mkl_gnu_thread, iomp5, mkl_gf_ilp64, mkl_mc3, mkl_def In [2]: numpy.test() Running unit tests for numpy NumPy version 1.3.0 NumPy is installed in /opt/********************* Python version 2.5.2 (r252:60911, Jul 22 2009, 15:33:10) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] nose version 0.11.0 .......... MKL ERROR: Parameter 4 was incorrect on entry to DGESV MKL ERROR: Parameter 4 was incorrect on entry to DGESV MKL ERROR: Parameter 4 was incorrect on entry to DGESV MKL ERROR: Parameter 4 was incorrect on entry to DGESV MKL ERROR: Parameter 4 was incorrect on entry to DGESV MKL ERROR: Parameter 4 was incorrect on entry to DGESV .. MKL ERROR: Parameter 4 was incorrect on entry to DGESV FSegmentation fault I am using gcc: gcc -v Using built-in specs. Target: x86_64-linux-gnu Configured with: ../src/configure -v --enable-languages=c,c++,fortran,objc,obj-c++,treelang --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.2 --program-suffix=-4.2 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu4) Anyone having the same issues? Do I have the mkl_libs correctly configured (this seems like a black art!) Thanks, Ashwin

5 6

strange divergence in performance
by Ernest Adrogué 20 Jan '10

20 Jan '10

Hi, I have a function where an array of integers (1-d) is compared element-wise to an integer using the greater-than operator. I noticed that when the integer is 0 it takes about 75% more time than when it's 1 or 2. Is there an explanation? Here is a stripped-down version which does (sort of)show what I say: def filter_array(array, f1, f2, flag=False): if flag: k = 1 else: k = 0 m1 = reduce(np.add, [(array['f1'] == i).astype(int) for i in f1]) > 0 m2 = reduce(np.add, [(array['f2'] == i).astype(int) for i in f2]) > 0 mask = reduce(np.add, (i.astype(int) for i in (m1, m2))) > k return array[mask] Now let's create an array with two fields: a = np.array(zip( np.random.random_integers(0,10,size=5000), np.random.random_integers(0,10,size=5000)), dtype=[('f1',int),('f2',int)]) Now call the function with flag=True and flag=False, and see what happens: In [29]: %timeit filter_array(a, (6,), (0,), flag=False) 1000 loops, best of 3: 536 us per loop In [30]: %timeit filter_array(a, (6,), (0,), flag=True) 1000 loops, best of 3: 245 us per loop In this example the difference seems to be 1:2. In my program is 1:4. I am at a loss about what causes this. Bye.

2 2

dates, np.where finding months
by John [H2O] 20 Jan '10

20 Jan '10

I have an array with the leading column a series of datetime objects. It covers several years. What is the most efficient way to pull out all the 'January' dates? Right now I do this: A = array with column 0 datetime objects January = [i for i in A if i[0].month ==1 ] It works, but I would rather use np.where and get indices and not have to convert my list back into an array. -- View this message in context: http://old.nabble.com/dates%2C-np.where-finding-months-tp27242195p27242195.… Sent from the Numpy-discussion mailing list archive at Nabble.com.

2 4

Efficient orthogonalisation with scipy/numpy
by Gael Varoquaux 20 Jan '10

20 Jan '10

Hi there, Forgive me for turning to the mailing list to do my homework. I am currently optimizing a code, and it turns out that the main bottleneck is the orthogonalisation of a vector 'y' to a set of vectors 'confounds', that I am currently doing with the following code: y = y - np.dot(confounds.T, linalg.lstsq(confounds.T, y)[0]) with np = numpy and linalg = scipy.linalg where scipy calls ATLAS. Most of the time is spent in linalg.lstsq. The length of the vectors is 810, and there are about 10 confounds. Is there a better way of doing this? Cheers, Gaël

6 23

Waf or scons/numscons for a C/Fortran/Cython/Python project -- what's your recommendation?
by Kurt Smith 19 Jan '10

19 Jan '10

My questions here concern those familiar with configure/build/install systems such as distutils, setuptools, scons/numscons or waf (particularly David Cournapeau). I'm creating a tool known as 'fwrap' that has a component that needs to do essentially what f2py does now -- take fortran source code and compile it into a python extension module. It uses Cython to create the extension module, and the current configure/build/install system is a very kludgy monkeypatched Cython.distutils and numpy.distutils setup.py script. The setup.py script works for testing on my system here, but for going prime time, I dread using it. David has made his critiques of distutils known for scientific software, and I agree. What's the best alternative? More specifically: what are the pros/cons between waf and scons/numscons for configure/build/install of a Fortran-C-Cython-Python project? Is scons capable of handling the configure and install stages, or is it only a build system? As I understand it, numscons is called from distutils; distutils handles the configure/install stages. Scons/numscons have more fortran support that waf, from what I can see. The main downside of using scons is that I'd still have to mess around with distutils. It looks like waf has explicit support for all three stages, and could be just what I'm looking for. David has a few threads on the waf-users list about getting fortran working with waf. Has that progressed much? I want to contribute to this, for the benefit of scipy and my project, and to limit duplicated work. From what I gather, the fortran configuration stuff in numscons is separated nicely from the scon-specific stuff :-) Would it be a matter of porting the numscons fortran stuff into waf? Any comments you have on using waf/scons for numerical projects would be welcome! Kurt

7 10

Numpy 1.4 MaskedArray bug?
by stephen.pascoe＠stfc.ac.uk 19 Jan '10

19 Jan '10

We have noticed the MaskedArray implementation in numpy-1.4.0 breaks some of our code. For instance we see the following: in 1.3.0: >>> a = numpy.ma.MaskedArray([[1,2,3],[4,5,6]]) >>> numpy.ma.sum(a, 1) masked_array(data = [ 6 15], mask = False, fill_value = 999999) in 1.4.0 >>> a = numpy.ma.MaskedArray([[1,2,3],[4,5,6]]) >>> numpy.ma.sum(a, 1) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.5/site-packages/numpy-1.4.0-py2.5-linux-x86_64.egg/n umpy/ma/core.py", line 5682, in __call__ return method(*args, **params) File "/usr/lib64/python2.5/site-packages/numpy-1.4.0-py2.5-linux-x86_64.egg/n umpy/ma/core.py", line 4357, in sum newmask = _mask.all(axis=axis) ValueError: axis(=1) out of bounds Also note the "Report Bugs" link on http://numpy.scipy.org is broken (http://numpy.scipy.org/bug-report.html) Thanks, Stephen. --- Stephen Pascoe +44 (0)1235 445980 British Atmospheric Data Centre Rutherford Appleton Laboratory -- Scanned by iCritical.

5 5

Is this expected behavior?
by Russel Howe 19 Jan '10

19 Jan '10

This looks like the difference between memmove and memcpy to me, but I am not sure what the expected behavior of numpy should be. The first shift behaves the way I expect, the second is surprising. I know about numpy.roll. I was hoping for something faster, which this would be if it worked. In [1]: a = (np.random.random((10,10))*10).astype('u1') In [2]: a Out[2]: array([[8, 0, 5, 4, 8, 2, 7, 8, 7, 6], [6, 6, 3, 3, 9, 8, 0, 8, 9, 5], [5, 0, 1, 1, 2, 5, 8, 2, 5, 3], [9, 0, 0, 2, 8, 2, 0, 7, 7, 0], [9, 8, 6, 9, 6, 3, 9, 4, 4, 5], [2, 7, 6, 9, 3, 8, 9, 9, 6, 9], [2, 8, 8, 4, 0, 3, 7, 6, 7, 6], [2, 4, 9, 2, 4, 7, 3, 6, 7, 4], [3, 2, 0, 7, 0, 7, 6, 6, 1, 6], [2, 3, 8, 8, 9, 6, 7, 2, 5, 0]], dtype=uint8) In [3]: a[:, :-1] = a[:, 1:] In [4]: a Out[4]: array([[0, 5, 4, 8, 2, 7, 8, 7, 6, 6], [6, 3, 3, 9, 8, 0, 8, 9, 5, 5], [0, 1, 1, 2, 5, 8, 2, 5, 3, 3], [0, 0, 2, 8, 2, 0, 7, 7, 0, 0], [8, 6, 9, 6, 3, 9, 4, 4, 5, 5], [7, 6, 9, 3, 8, 9, 9, 6, 9, 9], [8, 8, 4, 0, 3, 7, 6, 7, 6, 6], [4, 9, 2, 4, 7, 3, 6, 7, 4, 4], [2, 0, 7, 0, 7, 6, 6, 1, 6, 6], [3, 8, 8, 9, 6, 7, 2, 5, 0, 0]], dtype=uint8) In [5]: a[:, 1:] = a[:, :-1] In [6]: a Out[6]: array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [6, 6, 6, 6, 6, 6, 6, 6, 6, 6], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [8, 8, 8, 8, 8, 8, 8, 8, 8, 8], [7, 7, 7, 7, 7, 7, 7, 7, 7, 7], [8, 8, 8, 8, 8, 8, 8, 8, 8, 8], [4, 4, 4, 4, 4, 4, 4, 4, 4, 4], [2, 2, 2, 2, 2, 2, 2, 2, 2, 2], [3, 3, 3, 3, 3, 3, 3, 3, 3, 3]], dtype=uint8) In [7]: np.__version__ Out[7]: '1.3.0'

4 6