Mailman 3 November 2010 - NumPy-Discussion

Slow element-by-element access?
by Bruce Sherwood 22 Nov '10

22 Nov '10

A colleague showed me a program using Numeric with Python 2.5 which ran much faster than the same program using numpy with Python 2.7. I distilled this down to a simple test case, characterized by a "for" loop in which he does an element-by-element calculation involving arrays: from numpy import arange # or from Numeric import arange from time import clock # Numeric 0.24 seconds; 15 times as fast as numpy # numpy 3.6 seconds N = 100000 a = arange(N) b = arange(N) t = clock() for i in range(1,N-1): pass tpass = clock()-t t = clock() for i in range(1,N-1): b[i] = a[i]-t*(a[i+1]-2*a[i]+a[i-1])+a[i]*a[i]*t t = clock()-t print t-tpass His calculation b[i] = a[i]-t*(a[i+1]-2*a[i]+a[i-1])+a[i]*a[i]*t is 15 times faster with Numeric than with numpy. It is of course the case that he should have done a single array calculation rather than use a "for" loop, and I've explained that to him. The array calculation runs at about the same speed in numpy as in Numeric. Nevertheless, I'm surprised that the element-by-element calculation is so very slow, and surely there are situations where individual element access is important in a numpy calculation. Is this a known issue? Does it matter? I was unable to find any discussion of this. Bruce Sherwood

4 4

Where did the github numpy repository go?
by Charles R Harris 21 Nov '10

21 Nov '10

I keep getting page does not exist. Chuck

3 2

Yet more merges in maintenance/1.5.x
by Pauli Virtanen 20 Nov '10

20 Nov '10

Hi, There are more unnecessary merge commits in the git history. Before you do "git push", check git log --oneline --graph git log --oneline --graph upstream/maintenance/1.5.x.. so that you don't push unnecessary stuff. (Also, don't use "git pull".) $ git log --oneline --graph upstream/maintenance/1.5.x * cc8d516 BUG: Fix exception handling for python 3k. * 15b875a Merge branch 'maintenance/1.5.x' of github.com:numpy/numpy into |\ | * de969d7 REL: set version to 1.5.1. * | 3d429f1 Merge branch 'maintenance/1.5.x' of github.com:numpy/numpy in |\ \ | |/ | * 3304791 REL: set version number to 1.5.1rc2 and set released=True. | * e885a7a TST: mark longdouble tests as knownfail on OS X PPC. | * 516aa4a TST: mark longdouble tests for spacing/nextafter as knownfail o | * 08ccb3a TST: silence ldexp overflow warning. | * 8346ba0 BUG: fix issue with incorrect Fortran arch flags. Closes #1399. | * c0bd3df TST: core: disable C99 complex tests also on Solaris if it seem | * c6504f5 TST: core: mark test_ldexp_overflow as known failure on Python | * 082956b TST: core: fix test_fromfile_tofile_seeks to work on Windows (c * | f819262 Merge branch 'maintenance/1.5.x' of github.com:numpy/numpy in |\ \ | |/ | * 29c38c6 REL: set released=False again, and minor fix in paver script. | * 97cb28e REL: set version number to 1.5.1rc1, released=True. | * 4b18dc8 REL: Set start and end tags for the Changelog. | * 2aa0317 REL: make the OS X installer naming scheme correspond to what i | * 3bc3174 BUG: on Windows the sysconfig module does not contain CFLAGS in | * c9f2514 REL: add a note on #1399 to the 1.5.1 release notes. * | c035b13 DOC: recommend to turn on deprecation warnings for Python >= 2. |/ * 427d3fc Merge branch 'maintenance/1.5.x' of github.com:numpy/numpy into |\ | * b55eacd BUG: core: implement a long-int loop for ldexp, for cases where | * 0e792e6 BUG: core: adjust ComplexWarning location frame up by one, so t | * 4e177d3 BUG: get fortran arch flags from C arch flags if available. Clo | * c8e1315 BUG: DOC: fix invalid vdot documentation (cherry picked from co

2 2

problem with numpy/cython on python3, ok with python2
by Darren Dale 20 Nov '10

20 Nov '10

I just installed numpy for both python2 and 3 from an up-to-date checkout of the 1.5.x branch. I am attempting to cythonize the following code with cython-0.13: --- cimport numpy as np import numpy as np def test(): cdef np.ndarray[np.float64_t, ndim=1] ret ret_arr = np.zeros((20,), dtype=np.float64) ret = ret_arr --- I have this setup.py file: --- from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext import numpy setup( cmdclass = {'build_ext': build_ext}, ext_modules = [ Extension( "test_open", ["test_open.pyx"], include_dirs=[numpy.get_include()] ) ] ) --- When I run "python setup.py build_ext --inplace", everything is fine. When I run "python3 setup.py build_ext --inplace", I get an error: running build_ext cythoning test_open.pyx to test_open.c Error converting Pyrex file to C: ------------------------------------------------------------ ... # For use in situations where ndarray can't replace PyArrayObject*, # like PyArrayObject**. pass ctypedef class numpy.ndarray [object PyArrayObject]: cdef __cythonbufferdefaults__ = {"mode": "strided"} ^ ------------------------------------------------------------ /home/darren/.local/lib/python3.1/site-packages/Cython/Includes/numpy.pxd:173:49: "mode" is not a buffer option Error converting Pyrex file to C: ------------------------------------------------------------ ... cimport numpy as np import numpy as np def test(): cdef np.ndarray[np.float64_t, ndim=1] ret ^ ------------------------------------------------------------ /home/darren/temp/test/test_open.pyx:6:8: 'ndarray' is not a type identifier building 'test_open' extension gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/home/darren/.local/lib/python3.1/site-packages/numpy/core/include -I/usr/include/python3.1 -c test_open.c -o build/temp.linux-x86_64-3.1/test_open.o test_open.c:1: error: #error Do not use this file, it is the result of a failed Cython compilation. error: command 'gcc' failed with exit status 1 Is this a bug, or is there a problem with my example? Thanks, Darren

2 1

changing 'nan' string for np.nan in np.savetxt
by Maria Liukis 20 Nov '10

20 Nov '10

Hello everybody, I was wondering if there is an elegant way of overwriting 'nan' string representation with 'NaN' when saving numpy array containing np.nan values with numpy.savetxt() function. numpy.set_printoptions(nanstr='NaN') setting (which has default value of 'NaN' already) does not seem to have any effect when writing array to the file, only when printing to the screen: $ python Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> print np.__version__ 1.3.0 >>> np.get_printoptions() {'infstr': 'Inf', 'threshold': 1000, 'suppress': False, 'linewidth': 75, 'edgeitems': 3, 'precision': 8, 'nanstr': 'NaN'} >>> >>> a = np.array([1, 2, 3, np.nan]) >>> a array([ 1., 2., 3., NaN]) >>> np.savetxt('testa.txt', a) >>> $ cat testa.txt 1.000000000000000000e+00 2.000000000000000000e+00 3.000000000000000000e+00 nan $ Many thanks, Masha

1 0

seeking advice on a fast string->array conversion
by Darren Dale 19 Nov '10

19 Nov '10

I am wrapping up a small package to parse a particular ascii-encoded file format generated by a program we use heavily here at the lab. (In the unlikely event that you work at a synchrotron, and use Certified Scientific's "spec" program, and are actually interested, the code is currently available at https://github.com/darrendale/praxes/tree/specformat/praxes/io/spec/ .) I have been benchmarking the project against another python package developed by a colleague, which is an extension module written in pure C. My python/cython project takes about twice as long to parse and index a file (~0.8 seconds for 100MB), which is acceptable. However, actually converting ascii strings to numpy arrays, which is done using numpy.fromstring, takes a factor of 10 longer than the extension module. So I am wondering about the performance of np.fromstring: import time import numpy as np s = b'1 ' * 2048 *1200 d = time.time() x = np.fromstring(s) print time.time() - d

4 11

Advise for numerical programming content (New python user)
by Sachin Kumar Sharma 19 Nov '10

19 Nov '10

Users, I am an average Fortran user. I am new to python and I am currently evaluating options and functionalities of numerical programming and related 2d and 3d graphic outputs with python. Kindly share your experience in scientific programming with python like how do you like it, comparison with Fortran and C++. Which version of python + numpy+scipy are compatible with each other or if any other numerical analysis package is available (I am working on windows environment.) Does graphic output like maps, histogram, crossplot, tornado charts is good enough with basic installation or needs some additional packages? Your feedback is valuable for me to start. Thanks & Regards Sachin ************************************************************************ Sachin Kumar Sharma Senior Geomodeler - Samarang Project (IPM) Field Development & Production Services (DCS) Schlumberger Sdn. Bhd., 7th Floor, West Wing, Rohas Perkasa, No. 8 Jalan Perak, Kuala Lumpur, 50450, Malaysia Mobile: +60 12 2196443 * Email: ssharma84(a)exchange.slb.com<mailto:ssharma84@bombay.oilfield.slb.com> sachin_sharma(a)petronas.com.my

4 4

Test failures on 2.6
by T J 18 Nov '10

18 Nov '10

Hi, I'm getting a couple of test failures with Python 2.6, Numpy 1.2.0, Nose 0.10.4: nose version 0.10.4 ..........................................................................................................................................................................................................................................................................................................................................................................................................................................F................K........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./share/home/me/usr/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:68: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(c.readlines(), ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./share/home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1315: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(store._mask, True) /home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1322: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(store._mask, True) /home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1989: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(test.mask, [0,1,0,0,0,0,0,0,0,0]) ...............................................E................................................................................................................................................................................ ====================================================================== ERROR: Tests the min/max functions with explicit outputs ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py", line 653, in test_minmax_funcs_with_output result = npfunc(xm,axis=0,out=nout) File "/home/me/usr/lib/python2.6/site-packages/numpy/core/fromnumeric.py", line 1525, in amin return amin(axis, out) File "/home/me/usr/lib/python2.6/site-packages/numpy/ma/core.py", line 2978, in min np.putmask(out, newmask, np.nan) ValueError: cannot convert float NaN to integer ====================================================================== FAIL: test_umath.TestComplexFunctions.test_against_cmath ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/me/usr/lib/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/home/me/usr/lib/python2.6/site-packages/numpy/core/tests/test_umath.py", line 268, in test_against_cmath assert abs(a - b) < atol, "%s %s: %s; cmath: %s"%(fname,p,a,b) AssertionError: arcsin 2: (1.57079632679-1.31695789692j); cmath: (1.57079632679+1.31695789692j) ---------------------------------------------------------------------- Ran 1726 tests in 8.856s FAILED (KNOWNFAIL=1, errors=1, failures=1) <nose.result.TextTestResult run=1726 errors=1 failures=1>

2 1

Re: [Numpy-discussion] Test failures on 2.6
by W Bradley Knox 18 Nov '10

18 Nov '10

I'm having almost exactly the same problem, but with Python 2.6.1, Numpy 1.2.1, and Nose 0.11.3. Nobody responded to TJ the first time around, so any advice would be greatly appreciated. Thanks, Brad ---------------------------------------------------------------------- From: T J <tjhnson <at> gmail.com> Subject: Test failures on 2.6 Newsgroups: gmane.comp.python.numeric.general Date: 2008-10-05 20:53:22 GMT (2 years, 6 weeks, 1 day, 13 hours and 32 minutes ago) Hi, I'm getting a couple of test failures with Python 2.6, Numpy 1.2.0, Nose 0.10.4: nose version 0.10.4 ..........................................................................................................................................................................................................................................................................................................................................................................................................................................F................K.................................................................................................................................................................................................................................................................................................................................................................... .............................................................................................................................................................................................. ......./share/home/me/usr/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:68: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(c.readlines(), ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./share/home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1315: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(store._mask, True) /home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1322: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(store._mask, True) /home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1989: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(test.mask, [0,1,0,0,0,0,0,0,0,0]) ...............................................E................................................................................................................................................................................ ====================================================================== ERROR: Tests the min/max functions with explicit outputs ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py", line 653, in test_minmax_funcs_with_output result = npfunc(xm,axis=0,out=nout) File "/home/me/usr/lib/python2.6/site-packages/numpy/core/fromnumeric.py", line 1525, in amin return amin(axis, out) File "/home/me/usr/lib/python2.6/site-packages/numpy/ma/core.py", line 2978, in min np.putmask(out, newmask, np.nan) ValueError: cannot convert float NaN to integer ====================================================================== FAIL: test_umath.TestComplexFunctions.test_against_cmath ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/me/usr/lib/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/home/me/usr/lib/python2.6/site-packages/numpy/core/tests/test_umath.py", line 268, in test_against_cmath assert abs(a - b) < atol, "%s %s: %s; cmath: %s"%(fname,p,a,b) AssertionError: arcsin 2: (1.57079632679-1.31695789692j); cmath: (1.57079632679+1.31695789692j) ---------------------------------------------------------------------- Ran 1726 tests in 8.856s FAILED (KNOWNFAIL=1, errors=1, failures=1) <nose.result.TextTestResult run=1726 errors=1 failures=1>

1 0

How to import input data to make ndarray for batch processing?
by Venkat 18 Nov '10

18 Nov '10

Hi All, I am new to Numpy (also Scipy). I am trying to reshape my text data which is in one single column (10,000 rows). I want the data to be in 100x100 array form. I have many files to convert like this. All of them are having file names like 0, 1, 2, ....500. with out any extension. Actually, I renamed actual files so that I can import them in Matlab for batch processing. Since Matlab also new for me, I thought I will try Numpy first. Can any body help me in writing the script to do this for making batch processing. Thanks in advance, Venkat -- ******************************************************************************* D.Venkat Research Scholar Dept of Physics IISc, Bangalore India-560 012 ********************************************************************************

6 5