Mailman 3 July 2008 - NumPy-Discussion

indexing (compared to matlab)
by Brian Blais 19 Aug '08

19 Aug '08

Hello, I wanted to do the following thing that I do in Matlab (on a bigger problem), setting the values of a part of a matrix with indexing: >> a=floor(rand(5,5)*10) % make an example matrix to work on a = 2 4 7 9 8 6 9 2 5 2 6 3 5 1 8 1 5 6 1 2 1 2 8 2 9 >> ind=[2,4] ind = 2 4 >> a(ind,ind)=a(ind,ind)+100 a = 2 4 7 9 8 6 109 2 105 2 6 3 5 1 8 1 105 6 101 2 1 2 8 2 9 =========================== In numpy, the same gives: In [11]:a=floor(random.rand(5,5)*10) In [14]:a Out[14]: array([[ 7., 7., 8., 1., 9.], [ 0., 4., 9., 0., 5.], [ 4., 3., 7., 8., 3.], [ 2., 0., 4., 2., 4.], [ 9., 5., 0., 9., 9.]]) In [15]:ind=[1,3] In [20]:a[ind,ind]+=100 In [21]:a Out[21]: array([[ 7., 7., 8., 1., 9.], [ 0., 104., 9., 0., 5.], [ 4., 3., 7., 8., 3.], [ 2., 0., 4., 102., 4.], [ 9., 5., 0., 9., 9.]]) which only replaces 2 values, not all the values in the row,col combinations of [1,1],[1,2],etc...[3,3] like matlab. Is there a preferred way to do this, which I think should be fairly common. If I know that the indices are regular (like a slice) is there a way to do this? thanks, Brian Blais -- Brian Blais bblais(a)bryant.edu http://web.bryant.edu/~bblais

4 7

numpy 1.1.rc2: win32 binaries
by David Cournapeau 10 Aug '08

10 Aug '08

Hi, After some delay, here are the win32 binaries for numpy 1.1.1rc2: http://www.ar.media.kyoto-u.ac.jp/members/david/numpy-1.1.1.dev5559-win32-s… Notes on those binaries: - Based on Atlas 3.8.2 (the 1.1.0 was built against 3.8.0, which had a serious bug wrt dgemm, which is used for numpy.dot. It should solve #844 (problem with numpy.inner) - It is not stricly based on 1.1.rc2, but on 1.1.x trunk. The only difference, though, is a small fix in MANIFEST.in which was broken wrt sdist target and the version of course. If those work out, I will also prepare 2.4 binaries. I am sorry for the delay, cheers, David

7 16

FFT usage / consistency
by Felix Richter 07 Aug '08

07 Aug '08

Hi all, I found myself busy today trying to understand what went wrong in my FFT code. I wrote a minimal example/testing code to check the FFT output against an analytic result and also tried to reverse the transformation to get the original function back. Most curiously, the results depend on whether I first do the fft and then the ifft or the other way round. For the test function, I use the Lorentz function 1/(x^2+1). The exact FT is exp(-|t|)*sqrt(pi/2), the IFT yields the same. 1) First FFT and then IFFT: The real part of FFT oscillates, the imaginary part is not zero, and the magnitudes do not match. All this should not be, but the IFFT reproduces the original function just fine. 2) First IFFT and then FFT: The IFFT is perfect, but the FFT does not reproduce the original function. Could someone please have a look and tell me what I am doing wrong? The code is attached, it also plots the results nicely. Maybe the (corrected) code could be used as an example in the documentation (correct use of (i)fftshift and (i)fftfreq is not trivial!) or as an additional test case. The existing test cases for numpy only seem to check that the fft function can be called, but they do not check consistency of results. I'm using NumPy versions 1.0.4 and 1.1.0 on Linux with fftpack_lite.so (even though fftw3 is installed and configured, but I'll probably ask for that later...) Thanks a lot, Felix

5 5

f2py: undefined symbol: zgemm_
by Harry Mangalam 06 Aug '08

06 Aug '08

Hi All, Using: Python 2.5.2, f2py ver 2_4422, gfortran --version GNU Fortran (GCC) 4.2.3 (Ubuntu 4.2.3-2ubuntu7) on kubuntu 8.04 (Hardy) up-to-date on a Thinkpad T60. After building and using an f2py-generated lib for a while with this command: f2py --opt="-O3" -c -m fd_rrt1d --fcompiler=gnu95 \ --link-lapack_opt *.f I had to regenerate it after some code changes (just commenting out some debugging info). Altho the shared lib gets built correctly, when the calling python is run, I now get this: ./1d.py Traceback (most recent call last): File "./1d.py", line 27, in <module> from fd_rrt1d import * ImportError: /home/hjm/shaka/1D-Mangalam-py/fd_rrt1d.so: undefined symbol: zgemm_ sure enough, nm reports it as undefined: nm fd_rrt1d.so |tail 000065b0 t string_from_pyobj U strlen@@GLIBC_2.0 U strncpy@@GLIBC_2.0 0001df40 b u.1294 00013bee T umatrix1d_dms0_ 000128c3 T umatrix1d_dms1_ 00015458 T umatrix1d_ms0_ U zgemm_ U zgemv_ U zgesv_ but why the recent change? Google and the numpy list seem not to have heard about this one.. -- Harry Mangalam - Research Computing, NACS, E2148, Engineering Gateway, UC Irvine 92697 949 824-0084(o), 949 285-4487(c) -- ..Kick at the darkness til it bleeds daylight. (Lovers in a Dangerous Time) - Bruce Cockburn

1 1

Segfault in PyArray_Item_XDECREF when using recarray object references titles
by Michael Droettboom 06 Aug '08

06 Aug '08

I've run into a segfault that occurs in the array destructor with arrays containing object references with both names and titles. When a field contains both and name and a title, the fields dictionary contains two entries for that field. This means that the array item destructor (which iterates through the fields dictionary) will decref the pointed-to object twice. If the first decref causes the object to be deleted, the second decref has the potential to segfault. It seems the simplest patch is to set the object pointer to NULL after decref'ing, so the second decref will do nothing. However, perhaps there is a way to avoid decref'ing twice in the first place. I've attached a script that exercises the segfault, a gdb backtrace, and a patch. You may need to adjust the number of rows until it is high enough to create a segfault on your system. This is on: RHEL4 Python 2.5.2 Numpy SVN r5497 Cheers, Mike > gdb python GNU gdb Red Hat Linux (6.3.0.0-1.153.el4_6.2rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1". (gdb) run segfault.py Starting program: /wonkabar/data1/usr/bin/python segfault.py [Thread debugging using libthread_db enabled] [New Thread -1208489312 (LWP 30028)] len(dtype) = 1, len(dtype.fields) = 2 {'name': (dtype('object'), 0, 'title'), 'title': (dtype('object'), 0, 'title')} Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1208489312 (LWP 30028)] 0x0097285e in PyArray_Item_XDECREF ( data=0xb7a3e780 "\uffff_\224\uffff `\214\uffff(`\214\uffff0`\214\uffff8`\214\uffff@`\214\uffffH`\214\uffffP`\214\uffffX`\214\uffff``\214\uffffh`\214\uffffp`\214\uffffx`\214\uffff\200`\214\uffff\210`\214\uffff\220`\214\uffff\230`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff", descr=0x9d4680) at numpy/core/src/arrayobject.c:198 198 Py_XDECREF(*temp); (gdb) bt #0 0x0097285e in PyArray_Item_XDECREF ( data=0xb7a3e780 "\uffff_\224\uffff `\214\uffff(`\214\uffff0`\214\uffff8`\214\uffff@`\214\uffffH`\214\uffffP`\214\uffffX`\214\uffff``\214\uffffh`\214\uffffp`\214\uffffx`\214\uffff\200`\214\uffff\210`\214\uffff\220`\214\uffff\230`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff\uffff`\214\uffff", descr=0x9d4680) at numpy/core/src/arrayobject.c:198 #1 0x00991bc7 in PyArray_XDECREF (mp=0xb7ae4f0c) at numpy/core/src/arrayobject.c:211 #2 0x009a579b in array_dealloc (self=0xb7ae4f0c) at numpy/core/src/arrayobject.c:2089 #3 0x0809781f in subtype_dealloc (self=0xb7ae4f0c) at Objects/typeobject.c:709 #4 0x08082a02 in PyDict_SetItem (op=0xb7f56acc, key=0xb7ea7d80, value=0x81379c0) at Objects/dictobject.c:416 #5 0x08085a1e in _PyModule_Clear (m=0xb7f3e0ec) at Objects/moduleobject.c:136 #6 0x080d7138 in PyImport_Cleanup () at Python/import.c:439 #7 0x080e4343 in Py_Finalize () at Python/pythonrun.c:399 #8 0x08056633 in Py_Main (argc=1, argv=0xbff1ca24) at Modules/main.c:545 #9 0x08056323 in main (argc=2, argv=0xbff1ca24) at ./Modules/python.c:23 -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA Index: numpy/core/src/arrayobject.c =================================================================== --- numpy/core/src/arrayobject.c (revision 5497) +++ numpy/core/src/arrayobject.c (working copy) @@ -196,6 +196,7 @@ if (descr->type_num == PyArray_OBJECT) { temp = (PyObject **)data; Py_XDECREF(*temp); + *temp = NULL; } else if PyDescr_HASFIELDS(descr) { PyObject *key, *value, *title=NULL; import numpy as np ROWS = 200000 ra = np.recarray(ROWS, dtype=[(('title', 'name'), 'O')]) print "len(dtype) = %d, len(dtype.fields) = %d" % (len(ra.dtype), len(ra.dtype.fields)) print ra.dtype.fields for i in range(ROWS): ra[i]["name"] = object()

1 2

"import numpy" is slow
by Andrew Dalke 06 Aug '08

06 Aug '08

(Trying again now that I'm subscribed. BTW, there's no link to the subscription page from numpy.scipy.org .) The initial 'import numpy' loads a huge number of modules, even when I don't need them. Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> len(sys.modules) 28 >>> import numpy >>> len(sys.modules) 256 >>> len([s for s in sorted(sys.modules) if 'numpy' in s]) 127 >>> numpy.__version__ '1.1.0' >>> As a result, I assume that's the reason my program's startup cost is quite high. [josiah:~/src/fp] dalke% time python -c 'a=4' 0.014u 0.038s 0:00.05 80.0% 0+0k 0+1io 0pf+0w [josiah:~/src/fp] dalke% time python -c 'import numpy' 0.161u 0.279s 0:00.44 97.7% 0+0k 0+9io 0pf+0w My total runtime is something like 1.4 seconds, and the only thing I'm using NumPy for is to make an array of doubles that I can pass to a C extension. (I could use the array module or ctypes, but figured numpy is more useful for downstream code.) Why does numpy/__init__.py need to import all of these other modules and submodules? Any chance of cutting down on the number, in order to improve startup costs? Andrew dalke(a)dalkescientific.com

17 83

[ANNOUNCE] ETS 3.0.0b1 released!
by Dave Peterson 01 Aug '08

01 Aug '08

Hello, I am very pleased to announce that ETS 3.0.0b1 has just been tagged and released! All ETS sub-projects, including Traits, Chaco, Mayavi, and Envisage, have been tagged and released in one form or another (alpha, beta, final) as part of this effort. All ETS projects have now been registered with PyPi and source tarballs have been uploaded! As a result, for the first time ever, you can now easy_install all of ETS via a simple command like: easy_install ETS Or, alternatively, you can get only specific projects with commands like: easy_install Mayavi Or easy_install Chaco We anticipate the final release of ETS 3.0, and all of its sub-projects at their current version number, in a matter of weeks. Please join us in helping make this final release the most robust release it can be by participating in this beta shakeout period. Discussions about the status of ETS and its sub-projects happens on the enthought-dev mailing list, who's home page is at: https://mail.enthought.com/mailman/listinfo/enthought-dev Thank you to all who have worked so hard on making this release possible! -- Dave Enthought Tool Suite --------------------------- The Enthought Tool Suite (ETS) is a collection of components developed by Enthought and open source participants, which we use every day to construct custom scientific applications. It includes a wide variety of components, including: * an extensible application framework * application building blocks * 2-D and 3-D graphics libraries * scientific and math libraries * developer tools The cornerstone on which these tools rest is the Traits package, which provides explicit type declarations in Python; its features include initialization, validation, delegation, notification, and visualization of typed attributes. More information is available for all the packages within ETS from the Enthought Tool Suite development home page at http://code.enthought.com/projects/tool-suite.php. Testimonials ---------------- "I set out to rebuild an application in one week that had been developed over the last seven years (in C by generations of post-docs). Pyface and Traits were my cornerstones and I knew nothing about Pyface or Wx. It has been a hectic week. But here ... sits in front of me a nice application that does most of what it should. I think this has been a huge success. ... Thanks to the tools Enthought built, and thanks to the friendly support from people on the [enthought-dev] list, I have been able to build what I think is the best application so far. I have built similar applications (controlling cameras for imaging Bose-Einstein condensate) in C+MFC, Matlab, and C+labWindows, each time it has taken me at least four times longer to get to a result I regard as inferior. So I just wanted to say a big "thank you". Thank you to Enthought for providing this great software open-source. Thank you for everybody on the list for your replies." — Gaël Varoquaux, Laboratoire Charles Fabry, Institut d’Optique, Palaiseau, France "I'm currently writing a realtime data acquisition/display application … I'm using Enthought Tool Suite and Traits, and Chaco for display. IMHO, I think that in five years ETS/Traits will be the most comonly used framework for scientific applications." — Gary Pajer, Department of Chemistry, Biochemistry and Physics, Rider University, Lawrenceville NJ

2 1

patches to reduce import costs
by Andrew Dalke 01 Aug '08

01 Aug '08

I don't see a place to submit patches. Is there a patch manager for numpy? Here's a patch to defer importing 'tempfile' until needed. I previously mentioned one other place that didn't need tempfile. With this there is no 'import tempfile' during 'import numpy' This improves startup by about 7% --- numpy/lib/_datasource.py (revision 5576) +++ numpy/lib/_datasource.py (working copy) @@ -35,7 +35,6 @@ __docformat__ = "restructuredtext en" import os -import tempfile from shutil import rmtree from urlparse import urlparse @@ -131,6 +130,7 @@ self._destpath = os.path.abspath(destpath) self._istmpdest = False else: + import tempfile self._destpath = tempfile.mkdtemp() self._istmpdest = True Andrew dalke(a)dalkescientific.com

6 6

Numpy 1.1.1 release notes.
by Charles R Harris 01 Aug '08

01 Aug '08

Hi All, I've attached draft release notes for Numpy 1.1.1. If you have anything to add or correct, let me know. Chuck

3 4

distutils and inplace build: is numpy supposed to work ?
by David Cournapeau 01 Aug '08

01 Aug '08

Hi, I wanted to know if numpy was supposed to work when built in place through the -i option of distutils. The reason why I am asking it that I would like to support it in numscons, and I cannot make it work when using distutils. Importing numpy works in the source tree, but most tests fail because of some missing imports; I have a lots of those: ====================================================================== ERROR: Check that matrix type is preserved. ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/media/src/dsp/numpy/trunk/numpy/linalg/tests/test_linalg.py", line 69, in test_matrix_a_and_b self.do(a, b) File "/usr/media/src/dsp/numpy/trunk/numpy/linalg/tests/test_linalg.py", line 99, in do assert_almost_equal(a, dot(multiply(u, s), vt)) File "/usr/media/src/dsp/numpy/trunk/numpy/linalg/tests/test_linalg.py", line 22, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "numpy/testing/utils.py", line 171, in assert_almost_equal from numpy.core import ndarray File "core/__init__.py", line 27, in <module> __all__ += numeric.__all__ NameError: name 'numeric' is not defined Is this expected, or am I doing something wrong ? cheers, David

3 2