Mailman 3 March 2013 - NumPy-Discussion

(@Pat Marion) Re: Yes, this one again "ImportError: No module named multiarray"
by Aron Ahmadia March 13, 2013

March 13, 2013

Pat Marion at Kitware did some work on this, I'm pinging him on the thread. A On Tue, Mar 12, 2013 at 2:05 PM, Dinesh B Vadhia <dineshbvadhia(a)hotmail.com>wrote: > ** > Does that mean numpy won't work with freeze/create_executable type of > tools or is there a workaround? > > > *From:* Aron Ahmadia <aron(a)ahmadia.net> > *Sent:* Tuesday, March 12, 2013 6:17 AM > *To:* Discussion of Numerical Python <numpy-discussion(a)scipy.org> > *Subject:* Re: [Numpy-discussion] Yes,this one again "ImportError: No > module named multiarray" > > multiarray is an extension module that lives within numpy/core, that is, > when, "import multiarray" is called, (and it's the first imported extension > module in numpy), multiarray.ext (ext being dll on Windows I guess), gets > dynamically loaded. > > "No module named multiarray" is indicating problems with your freeze > setup. Most of these tools don't support locally imported extension > modules. > > Does this help you get oriented on your problem? > > A > > > On Tue, Mar 12, 2013 at 1:01 PM, Dinesh B Vadhia < > dineshbvadhia(a)hotmail.com> wrote: > >> ** >> I've been using Numpy/Scipy for >5 years so know a little on how to get >> around them. Recently, I've needed to either freeze or create executables >> with tools such as PyInstaller, Cython, Py2exe and others on both Windows >> (XP 32-bit, 7 64-bit) and Ubuntu (12.04) Linux (64-bit). The test >> program (which runs perfectly with the Python interpreter) is very simple: >> >> import numpy >> >> def main(): >> print numpy.array([12, 23, 34, 45, 56, 67, 78, 89, 90]) >> return >> >> if __name__ == '__main__': >> main() >> >> The software versions are Python 2.7.3, Numpy 1.7.0, and Scipy 0.11. The >> "import numpy" causes an "ImportError: No module named multiarray". After >> endless Googling, I am none the wiser about what (really) causes the >> ImportError let alone what the solution is. The Traceback, similar to >> others found on the web, is: >> >> Traceback (most recent call last): >> File "test.py", ... >> File "C:\Python27\lib\site-packages\numpy\__init__.py", line 154, in >> <module> >> import add_newdocs >> File "C:\Python27\lib\site-packages\numpy\add_newdocs.py", line 9, in >> <module> >> from numpy.lib import add_newdoc >> File "C:\Python27\lib\site-packages\numpy\lib\__init__.py", line 4, in >> <module> >> from type_check import * >> File "C:\Python27\lib\site-packages\numpy\lib\type_check.py", line 8, >> in <module> >> import numpy.core.numeric as _nx >> File "C:\Python27\lib\site-packages\numpy\core\__init__.py", line 5, in >> <module> >> import multiarray >> ImportError: No module named multiarray. >> >> Could someone shed some light on this - please? Thx. >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion(a)scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >

3 2

Performance of einsum?
by Jaakko Luttinen March 13, 2013

March 13, 2013

Hi, I was wondering if someone could provide some intuition on the performance of einsum? I have found that sometimes it is extremely efficient but sometimes it is several orders of magnitudes slower compared to some other approaches, for instance, using multiple dot-calls. My intuition is that the computation time of einsum is linear with respect to the size of the "index space", that is, the product of the maximums of the indices. So, for instance computing the dot product of three matrices A*B*C would not be efficient as einsum('ij,jk,kl->il', A, B, C) because there are four indices i=1,...,I, j=1,...,J, k=1,...,K and l=1,...,L so the total computation time is O(I*J*K*L) which is much worse than with two dot products O(I*J*K+J*K*L), or with two einsum-calls for Y=A*B and Y*C. On the other hand, computing einsum('ij,ij,ij->i', A, B, C) would be "efficient" because the computation time is only O(I*J). Is this intuition roughly correct or how could I intuitively understand when the usage of einsum is bad? Best regards, Jaakko

1 0

Bug in einsum?
by Jaakko Luttinen March 13, 2013

March 13, 2013

Hi, I have encountered a very weird behaviour with einsum. I try to compute something like R*A*R', where * denotes a kind of "matrix multiplication". However, for particular shapes of R and A, the results are extremely bad. I compare two einsum results: First, I compute in two einsum calls as (R*A)*R'. Second, I compute the whole result in one einsum call. However, the results are significantly different for some shapes. My test: import numpy as np for D in range(30): A = np.random.randn(100,D,D) R = np.random.randn(D,D) Y1 = np.einsum('...ik,...kj->...ij', R, A) Y1 = np.einsum('...ik,...kj->...ij', Y1, R.T) Y2 = np.einsum('...ik,...kl,...lj->...ij', R, A, R.T) print("D=%d" % D, np.allclose(Y1,Y2), np.linalg.norm(Y1-Y2)) Output: D=0 True 0.0 D=1 True 0.0 D=2 True 8.40339658678e-15 D=3 True 8.09995399928e-15 D=4 True 3.59428803435e-14 D=5 False 34.755610184 D=6 False 28.3576558351 D=7 False 41.5402690906 D=8 True 2.31709582841e-13 D=9 False 36.0161112799 D=10 True 4.76237746912e-13 D=11 True 4.57944440782e-13 D=12 True 4.90302218301e-13 D=13 True 6.96175851271e-13 D=14 True 1.10067181384e-12 D=15 True 1.29095933163e-12 D=16 True 1.3466837332e-12 D=17 True 1.52265065763e-12 D=18 True 2.05407923852e-12 D=19 True 2.33327630748e-12 D=20 True 2.96849358082e-12 D=21 True 3.31063706175e-12 D=22 True 4.28163620455e-12 D=23 True 3.58951880681e-12 D=24 True 4.69973694769e-12 D=25 True 5.47385264567e-12 D=26 True 5.49643316347e-12 D=27 True 6.75132988402e-12 D=28 True 7.86435437892e-12 D=29 True 7.85453681029e-12 So, for D={5,6,7,9}, allclose returns False and the error norm is HUGE. It doesn't seem like just some small numerical inaccuracy because the error norm is so large. I don't know which one is correct (Y1 or Y2) but at least either one is wrong in my opinion. I ran the same test several times, and each time same values of D fail. If I change the shapes somehow, the failing values of D might change too, but I usually have several failing values. I'm running the latest version from github (commit bd7104cef4) under Python 3.2.3. With NumPy 1.6.1 under Python 2.7.3 the test crashes and Python exits printing "Floating point exception". This seems so weird to me that I wonder if I'm just doing something stupid.. Thanks a lot for any help! Jaakko

1 0

Vectorize and ufunc attribute
by T J March 13, 2013

March 13, 2013

Prior to 1.7, I had working compatibility code such as the following: if has_good_functions: # http://projects.scipy.org/numpy/ticket/1096 from numpy import logaddexp, logaddexp2 else: logaddexp = vectorize(_logaddexp, otypes=[numpy.float64]) logaddexp2 = vectorize(_logaddexp2, otypes=[numpy.float64]) # Run these at least once so that .ufunc.reduce exists logaddexp([1.,2.,3.],[1.,2.,3.]) logaddexp2([1.,2.,3.],[1.,2.,3.]) # And then make reduce available at the top level logaddexp.reduce = logaddexp.ufunc.reduce logaddexp2.reduce = logaddexp2.ufunc.reduce The point was that I wanted to treat the output of vectorize as a hacky drop-in replacement for a ufunc. In 1.7, I discovered that vectorize had changed (https://github.com/numpy/numpy/pull/290), and now there is no longer a ufunc attribute at all. Should this be added back in? Besides hackish drop-in replacements, I see value in to being able to call reduce, accumulate, etc (when possible) on the output of vectorize().

2 2

Request code review of numpy.i changes
by Bill Spotz March 12, 2013

March 12, 2013

https://github.com/wfspotz/numpy/compare/numpy-swig ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz(a)sandia.gov **

2 1

unclear output format for numpy.argmax()
by soumen ganguly March 12, 2013

March 12, 2013

Hello, There are some doubts that i have regarding the argmax() method of numpy.As described in reference doc's of numpy,argmax(axis=None,out=None) returns the indices of the maximum value along the given axis(In this case 0 is default). So, i tried to implement the method to a 2d array with elements say,[[1,2,3],[4,5,6]] along the axis 1.The output to this code is [2,2] and when i implement it along the axis 0,it outputs [1,1,1].I dont see the connection to this output with the scope of argmax method. I would appreciate a detailed insight to the argmax method.

2 1

Casting and promotion rules (e.g. int + uint64 => float)
by Sergio Callegari March 11, 2013

March 11, 2013

Hi, I have noticed that numpy introduces some unexpected type casts, that are in some cases problematic. A very weird cast is int + uint64 -> float for instance, consider the following snippet: import numpy as np a=np.uint64(1) a+1 -> 2.0 this cast is quite different from what other programming languages (e.g., C) would do in this case, so it already comes unexpected. Furthermore, an int64 (or an uint64) is too large to fit into a float, hence this automatic conversion also results in data loss! For instance consider: a=np.uint64(18446744073709551614) a+np.uint64(1) -> 18446744073709551615 # CORRECT! a+1 -> 1.8446744073709552e+19 # Actually 1.84467440737095516160e+19 - LOSS OF DATA in fact np.uint64(a+1) -> 0 Weird, isn't it? Another issue is that variables unexpectedly change type with accumulation operators a=np.uint64(1) a+=1 now a is float I believe that some casting/promotion rules should be revised, since they now lead to difficult to catch, intermittent errors. In case this cannot be done immediately, I suggest at least documenting these promotions, providing examples on how to code many conventional tasks. E.g., incrementing an integer of unknown size b=a+type(a)(1) I have also reported this in https://github.com/numpy/numpy/issues/3118 Thanks!

2 2

Numpy 1.7.0 with Intel MKL 11.0.2.146
by QT March 11, 2013

March 11, 2013

Dear all, I'm at my wits end. I've followed Intel's own instructions<http://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl>on how to compile Numpy with Intel MKL. Everything compiled and linked fine and I've installed it locally in my user folder...There is one nasty problem. When one calls the numpy library to do some computation, it does not use all of the available threads. I have 8 "cores" on my machine and it only uses 4 of them. The MKL_NUM_THREADS environmental variable can be set to tune the number of threads but setting it to 8 does not change anything. Indeed, setting it to 3 does limit the threads to 3....What is going on? As a comparison, the numpy (version 1.4.1, installed from yum, which uses BLAS+ATLAS) uses all 8 threads. I do not get this. You can run this test program python -mtimeit -s'import numpy as np; a = np.random.randn(1e3,1e3)' 'np.dot(a, a)' There is one saving grace, the local numpy built with MKL is much faster than the system's numpy. I hope someone can help me. Searching the internet has been fruitless. Best, Quyen My site.cfg for numpy (1.7.0) [mkl] library_dirs = /opt/intel/mkl/lib/intel64 include_dirs = /opt/intel/mkl/include mkl_libs = mkl_rt lapack_libs = I've edited line 37 of numpy/distutils/intelcompiler.py self.cc_exe = 'icc -O3 -fPIC -fp-model strict -fomit-frame-pointer -openmp -parallel -DMKL_ILP64' Also line 54 of numpy/distutils/fcompiler/intel.py return ['-i8 -xhost -openmp -fp-model strict'] My .bash_profile also contains the lines: source /opt/intel/bin/compilervars.sh intel64 source /opt/intel/mkl/bin/mklvars.sh intel64 The above is needed to set the LD_LIBRARY_PATH so that Python can source the intel dynamic library when numpy is called.

2 3

Compile time flag for numpy
by Sebastian Berg March 9, 2013

March 9, 2013

Hey, how would I go about making a compile time flag for numpy to use as a macro? The reason is: https://github.com/numpy/numpy/pull/2735 so that it would be possible to compile numpy differently for debugging if software depending on numpy is broken by this change. Regards, Sebastian

1 1

aligned / unaligned structured dtype behavior (was: GSOC 2013)
by Kurt Smith March 8, 2013

March 8, 2013

On Wed, Mar 6, 2013 at 12:12 PM, Kurt Smith <kwmsmith(a)gmail.com> wrote: > On Wed, Mar 6, 2013 at 4:29 AM, Francesc Alted <francesc(a)continuum.io> wrote: >> >> I would not run too much. The example above takes 9 bytes to host the >> structure, while a `aligned=True` will take 16 bytes. I'd rather let >> the default as it is, and in case performance is critical, you can >> always copy the unaligned field to a new (homogeneous) array. > > Yes, I can absolutely see the case you're making here, and I made my > "vote" with the understanding that `aligned=False` will almost > certainly stay the default. Adding 'aligned=True' is simple for me to > do, so no harm done. > > My case is based on what's the least surprising behavior: C structs / > all C compilers, the builtin `struct` module, and ctypes `Structure` > subclasses all use padding to ensure aligned fields by default. You > can turn this off to get packed structures, but the default behavior > in these other places is alignment, which is why I was surprised when > I first saw that NumPy structured dtypes are packed by default. > Some surprises with aligned / unaligned arrays: #----------------------------- import numpy as np packed_dt = np.dtype((('a', 'u1'), ('b', 'u8')), align=False) aligned_dt = np.dtype((('a', 'u1'), ('b', 'u8')), align=True) packed_arr = np.ones((10**6,), dtype=packed_dt) aligned_arr = np.ones((10**6,), dtype=aligned_dt) print "all(packed_arr['a'] == aligned_arr['a'])", np.all(packed_arr['a'] == aligned_arr['a']) # True print "all(packed_arr['b'] == aligned_arr['b'])", np.all(packed_arr['b'] == aligned_arr['b']) # True print "all(packed_arr == aligned_arr)", np.all(packed_arr == aligned_arr) # False (!!) #----------------------------- I can understand what's likely going on under the covers that makes these arrays not compare equal, but I'd expect that if all columns of two structured arrays are everywhere equal, then the arrays themselves would be everywhere equal. Bug? And regarding performance, doing simple timings shows a 30%-ish slowdown for unaligned operations: In [36]: %timeit packed_arr['b']**2 100 loops, best of 3: 2.48 ms per loop In [37]: %timeit aligned_arr['b']**2 1000 loops, best of 3: 1.9 ms per loop Whereas summing shows just a 10%-ish slowdown: In [38]: %timeit packed_arr['b'].sum() 1000 loops, best of 3: 1.29 ms per loop In [39]: %timeit aligned_arr['b'].sum() 1000 loops, best of 3: 1.14 ms per loop

4 9