Mailman 3 October 2009 - NumPy-Discussion

[review] Easy win to improve numpy import times by 30 %
by David Cournapeau 09 Oct '09

09 Oct '09

Hi, This branch improves numpy import times quite significantly on slow machines: http://github.com/cournape/numpy/tree/noinspect One of the main culprit is ma, because of inspect (inspect is extremely slow to import; as a data point, python -c "import inspect" takes 67 ms vs python -c "" taking 22 ms, and python -c "import numpy" taking 158 ms on my machine). Since inspect is used in quite a few places, and that we only use it to extract arguments from a function, I added a small numpy.lib.inspect module, and change the import in numpy.ma. I copied the inspect module of python 2.4.4 to ensure maximum compatibility. This speed up the import times from 158 ms to 108 ms ~ 30 % speed improvement. On recent machines, the speedup is less impressive, but still in the 20 % range. I think it largely worths it, and will integrate this unless someone is strongly against it or see a problem with the approach, cheers, David

3 2

PyArray_FROM_OF from Cython
by David Warde-Farley 09 Oct '09

09 Oct '09

I'm trying to use PyArray_FROM_OF from Cython and the generated C code keeps crashing. Dag said on the Cython list that he wasn't sure what was going on, so maybe someone here will have an idea. The line that gdb says is crashing is: #0 0x00e48287 in __pyx_pf_3_vq_vq (__pyx_self=0x0, __pyx_args=0xca2d8, __pyx_kwds=0x0) at _vq_rewrite.c:1025 1025 __pyx_t_1 = PyArray_FROM_OF(((PyObject *)__pyx_v_obs), __pyx_v_flags); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 90; __pyx_clineno = __LINE__; goto __pyx_L1_error;} obs and obs_a are both cdef'd np.ndarrays, and the former (obs) is passed in as an argument. I define flags as cdef int flags = np.NPY_CONTIGUOUS | np.NPY_ALIGNED | np.NPY_NOTSWAPPED and then the line that crashes is obs_a = np.PyArray_FROM_OF(obs, flags) Does anyone know what I'm doing wrong? (I know I could use np.ascontiguous, but as far as I can tell this _should_ work) David

3 4

byteswapping a complex scalar
by Michael Droettboom 09 Oct '09

09 Oct '09

I'm noticing an inconsistency as to how complex numbers are byteswapped as arrays vs. scalars, and wondering if I'm doing something wrong. >>> x = np.array([-1j], '<c8') >>> x.tostring().encode('hex') '00000000000080bf' # This is a little-endian representation, in the order (real, imag) # When I swap the whole array, it swaps each of the (real, imag) parts separately >>> y = x.byteswap() >>> y.tostring().encode('hex') '00000000bf800000' # and this round-trips fine >>> z = np.fromstring(y.tostring(), dtype='>c8') >>> assert z[0] == -1j >>> # When I swap the scalar, it seems to swap the entire 8 bytes >>> y = x[0].byteswap() >>> y.tostring().encode('hex') 'bf80000000000000' # ...and this doesn't round-trip >>> z = np.fromstring(y.tostring(), dtype='>c8') >>> assert z[0] == -1j Traceback (most recent call last): File "<stdin>", line 1, in <module> AssertionError >>> Any thoughts? Mike -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

3 6

NumPy SVN broken
by Stéfan van der Walt 09 Oct '09

09 Oct '09

Hi all, The current SVN HEAD of NumPy is broken and should not be used. Extensions compiled against this version may (will) segfault. Travis, if you could have a look at the side-effects caused by r7050, that would be great. I meant to figure out what was wrong, but seeing that this is a 3000 line patch, I'm not confident I can find the problem easily. Regards Stéfan P.S. The new functionality is great, but I don't think we're going to be able to convince David to release without documenting and testing those changes to the C API.

8 29

Building a new copy of NumPy
by Gökhan Sever 08 Oct '09

08 Oct '09

Hello, I checked-out the latest trunk and make a new installation of NumPy. My question: Is it a known behaviour that this action will result with re-building other packages that are dependent on NumPy. In my case, I had to re-built matplotlib, and now scipy. Here is the error message that I am getting while I try to import a scipy module: I[1]: run lab4.py --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) RuntimeError: FATAL: module compiled aslittle endian, but detected different endianness at runtime --------------------------------------------------------------------------- ImportError Traceback (most recent call last) /home/gsever/AtSc450/labs/04_thermals/lab4.py in <module>() 2 3 import numpy as np ----> 4 from scipy import stats 5 6 /home/gsever/Desktop/python-repo/scipy/scipy/stats/__init__.py in <module>() 5 from info import __doc__ 6 ----> 7 from stats import * 8 from distributions import * 9 from rv import * /home/gsever/Desktop/python-repo/scipy/scipy/stats/stats.py in <module>() 196 # Scipy imports. 197 from numpy import array, asarray, dot, ma, zeros, sum --> 198 import scipy.special as special 199 import scipy.linalg as linalg 200 import numpy as np /home/gsever/Desktop/python-repo/scipy/scipy/special/__init__.py in <module>() 6 #from special_version import special_version as __version__ 7 ----> 8 from basic import * 9 import specfun 10 import orthogonal /home/gsever/Desktop/python-repo/scipy/scipy/special/basic.py in <module>() 6 7 from numpy import * ----> 8 from _cephes import * 9 import types 10 import specfun ImportError: numpy.core.multiarray failed to import WARNING: Failure executing file: <lab4.py> -- Gökhan

3 3

genfromtxt - the return
by Pierre GM 08 Oct '09

08 Oct '09

All, Could you try r7449 ? I introduced some mechanisms to keep track of invalid lines (where the number of columns don't match what's expected). By default, a warning is emitted and these lines are skipped, but an optional argument gives the possibility to raise an exception instead. Now, I need more tests about wrong converters. I'm trying to optimize the upgrade mechanism (there are too many intertwined loops for my taste now), I'll keep you posted. Meanwhile, if you could come with more cases of failure, please send them my way. Cheers P.

4 12

Questions about masked arrays
by Gökhan Sever 07 Oct '09

07 Oct '09

Hello, I have a sample masked array data as shown below. 1-) When I list the whole array I see the fill value correctly. However below that line, when I do access the 5th element, fill_value flies upto 1e+20. What might be wrong here? I[5]: c.data['Air_Temp'] O[5]: masked_array(data = [13.1509 13.1309 13.1278 13.1542 -- 13.1539 13.1387 -- -- -- 13.1107 13.1351 13.2073 13.2562 13.3533 13.3889 13.4067 13.2938 13.1962 13.1248 13.0411 12.9534 12.8354 12.7392 12.6725], mask = [False False False False True False False True True True False False False False False False False False False False False False False False False], fill_value = 999999.9999) I[6]: c.data['Air_Temp'][4] O[6]: masked_array(data = --, mask = True, fill_value = 1e+20) 2-) What is wrong with the arccos calculation? Should not that result the same as with cos(d) result? I[9]: d = c.data['Air_Temp'][4] I[11]: cos(d) O[11]: masked_array(data = --, mask = True, fill_value = 1e+20) I[12]: arccos(d) O[12]: masked_array(data = 1.57079632679, mask = False, fill_value = 1e+20) Any ideas? -- Gökhan

4 19

SVN + Python 2.5.4 (32b) + MacOS 10.6.1
by Pierre GM 07 Oct '09

07 Oct '09

All, I need to test the numpy SVN on a 10.6.1 mac, but using Python 2.5.4 (32b) instead of the 2.6.1 (64b). The sources get compiled OK (apparently, find the build here: http://pastebin.com/m147a2909 ) but numpy fails to import: File ".../.virtualenvs/default25/lib/python2.5/site-packages/numpy/ __init__.py", line 130, in <module> import add_newdocs File ".../.virtualenvs/default25/lib/python2.5/site-packages/numpy/ add_newdocs.py", line 9, in <module> from lib import add_newdoc File ".../.virtualenvs/default25/lib/python2.5/site-packages/numpy/ lib/__init__.py", line 4, in <module> from type_check import * File ".../.virtualenvs/default25/lib/python2.5/site-packages/numpy/ lib/type_check.py", line 8, in <module> import numpy.core.numeric as _nx File ".../.virtualenvs/default25/lib/python2.5/site-packages/numpy/ core/__init__.py", line 8, in <module> import numerictypes as nt File ".../.virtualenvs/default25/lib/python2.5/site-packages/numpy/ core/numerictypes.py", line 737, in <module> _typestr[key] = empty((1,),key).dtype.str[1:] ValueError: array is too big. Obviously, I'm messing between 32b and 64b, but can't figure where. Any help/hint will be deeply appreciated Cheers P. FYI: Python 2.5.4 (r254:67916, Jul 7 2009, 23:51:24) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin] CFLAGS="-arch i386 -arch x86_64" FFLAGS="-arch i386 -arch x86_64"

1 0

numpy.linalg.eig memory issue with libatlas?
by Johan Grönqvist 07 Oct '09

07 Oct '09

[I am resending this as the previous attempt seems to have failed] Hello List, I am looking at memory errors when using numpy.linalg.eig(). Short version: I had memory errors in numpy.linalg.eig(), and I have reasons (valgrind) to believe these are due to writing to incorrect memory addresses in the diagonalization routine zgeev, called by numpy.linalg.eig(). I realized that I had recently installed atlas, and now had several lapack-like libraries, so I uninstalled atlas, and the issues seemed to go away. My question is: Could it be that some lapack/blas/atlas package I use is incompatible with the numpy I use, and if so, is there a method to diagnose this in a more reliable way? Longer version: The system used is an updated debian testing (squeeze), on amd64. My program uses numpy, matplotlib, and a module compiled using cython. I started getting errors from my program this week. Pdb and print-statements tell me that the errors arise around the point where I call numpy.linalg.eig(), but not every time. The type of error varies. Most frequently a segmentation fault, but sometimes a matrix dimension mismatch, and sometimes a message related to the python GC. Valgrind tells me that something "impossible" happened, and that this is probably due to invalid writes earlier during the program execution. There seems to be two invalid writes after each program crash, and the log looks like this (it only contains two invalid writes): [...] ==6508== Invalid write of size 8 ==6508== at 0x92D2597: zunmhr_ (in /usr/lib/atlas/liblapack.so.3gf.0) ==6508== by 0x920A42B: zlaqr3_ (in /usr/lib/atlas/liblapack.so.3gf.0) ==6508== by 0x9205D11: zlaqr0_ (in /usr/lib/atlas/liblapack.so.3gf.0) ==6508== by 0x91B0C4D: zhseqr_ (in /usr/lib/atlas/liblapack.so.3gf.0) ==6508== by 0x911CA15: zgeev_ (in /usr/lib/atlas/liblapack.so.3gf.0) ==6508== by 0x881B81B: lapack_lite_zgeev (lapack_litemodule.c:590) ==6508== by 0x4911D4: PyEval_EvalFrameEx (ceval.c:3612) ==6508== by 0x491CE1: PyEval_EvalFrameEx (ceval.c:3698) ==6508== by 0x4924CC: PyEval_EvalCodeEx (ceval.c:2875) ==6508== by 0x490F17: PyEval_EvalFrameEx (ceval.c:3708) ==6508== by 0x4924CC: PyEval_EvalCodeEx (ceval.c:2875) ==6508== by 0x4DC991: function_call (funcobject.c:517) ==6508== Address 0x67ab118 is not stack'd, malloc'd or (recently) free'd ==6508== ==6508== Invalid write of size 8 ==6508== at 0x92D25A8: zunmhr_ (in /usr/lib/atlas/liblapack.so.3gf.0) ==6508== by 0x920A42B: zlaqr3_ (in /usr/lib/atlas/liblapack.so.3gf.0) ==6508== by 0x9205D11: zlaqr0_ (in /usr/lib/atlas/liblapack.so.3gf.0) ==6508== by 0x91B0C4D: zhseqr_ (in /usr/lib/atlas/liblapack.so.3gf.0) ==6508== by 0x911CA15: zgeev_ (in /usr/lib/atlas/liblapack.so.3gf.0) ==6508== by 0x881B81B: lapack_lite_zgeev (lapack_litemodule.c:590) ==6508== by 0x4911D4: PyEval_EvalFrameEx (ceval.c:3612) ==6508== by 0x491CE1: PyEval_EvalFrameEx (ceval.c:3698) ==6508== by 0x4924CC: PyEval_EvalCodeEx (ceval.c:2875) ==6508== by 0x490F17: PyEval_EvalFrameEx (ceval.c:3708) ==6508== by 0x4924CC: PyEval_EvalCodeEx (ceval.c:2875) ==6508== by 0x4DC991: function_call (funcobject.c:517) ==6508== Address 0x67ab110 is not stack'd, malloc'd or (recently) free'd [...] valgrind: m_mallocfree.c:248 (get_bszB_as_is): Assertion 'bszB_lo == bszB_hi' failed. valgrind: Heap block lo/hi size mismatch: lo = 96, hi = 0. This is probably caused by your program erroneously writing past the end of a heap block and corrupting heap metadata. If you fix any invalid writes reported by Memcheck, this assertion failure will probably go away. Please try that before reporting this as a bug. [...] Today I looked in my package installation logs to see what had changed recently, and I noticed that I installed atlas (debian package libatlas3gf-common) recently. I uninstalled that package, and now the same program seems to have no memory errors. The packages I removed from the system today were libarpack2 libfltk1.1 libftgl2 libgraphicsmagick++3 libgraphicsmagick3 libibverbs1 libopenmpi1.3 libqrupdate1 octave3.2-common octave3.2-emacsen libatlas3gf-base octave3.2 My interpretation is that I had several packages available containing the diagonalization functionality, but that they differed subtly in their interfaces. My recent installation of atlas made numpy use (the incompatible) atlas instead of its previous choice, and removal of atlas restored the situation to the state of last week. Now for the questions: Is this a reasonable hypothesis? Is it known? Can it be investigated more precisely by comparing versions somehow? Regards / johan

1 0

vectorize() broken on Python2.6
by George Trojan 07 Oct '09

07 Oct '09

f2py generated wrappers cannot be vectorized with numpy1.3.0 and Python2.6.2. The reason is change to Python's getargs.c. Vectorize, or rather _get_nargs() defined in lib/function_base.py tries to determine the number of arguments from error message generated while the interpreter parses function invocation without any arguments. The messages (in getargs.c) have changed, for example: Required argument 'a' (pos 1) not found Since the message no longer contains information how many arguments a function takes, a fix is not obvious. Is there a solution coming soon? I posted a message on comp.lang.python few days ago. Sturda Molden generated bug report http://projects.scipy.org/numpy/ticket/1247. However the change he suggests does not fix the problem. I am tempted to apply a temporary workaround for my current needs: The __init__ method in vectorize would accept an additional argument, interface=None. That argument would be a Python code stub, prepared manually (though it could easily be generated by f2py). This stub would be used by _get_nargs() when the original object does not contain attribute 'func_code'. Example: Fortran code integer function f3(a, b, c) integer, intent(in) :: a, b integer, optional, intent(in) :: c if (present(c)) then f3 = a - b else f3 = a + b endif end function f3 Interface def f3_iface(a, b, c=None): pass Call vf3 = numpy.vectorize(ftest.f3, f3_iface) Are there any drawbacks to this approach? George

1 0