Mailman 3 January 2007 - NumPy-Discussion

C-API: datatype for arrays of indices
by Martin Wiechert Jan. 24, 2007

Jan. 24, 2007

Hi list, does anybody know, what datatype arrays C-API functions like PyArray_SearchSorted PyArray_ArgSort return? Can I rely on them being npy_intp? Thanks, Martin

2 2

numpy.test error on G5 PPC
by Vincent Nijs Jan. 24, 2007

Jan. 24, 2007

Mac OS X 10.4.8 G5 PPC Numpy svn: 2602 Target: powerpc-apple-darwin8 Configured with: /private/var/tmp/gcc/gcc-5367.obj~1/src/configure --disable-checking -enable-werror --prefix=/usr --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^.-]*$/s/$/-4.0/ --with-gxx-include-dir=/include/c++/4.0.0 --with-slibdir=/usr/lib --build=powerpc-apple-darwin8 --host=powerpc-apple-darwin8 --target=powerpc-apple-darwin8 Thread model: posix gcc version 4.0.1 (Apple Computer, Inc. build 5367) ====================================================================== FAIL: check_large_types (numpy.core.tests.test_scalarmath.test_power) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packag es/numpy/core/tests/test_scalarmath.py", line 46, in check_large_types assert b == 6765201, "error with %r: got %r" % (t,b) AssertionError: error with <type 'numpy.float128'>: got 0.0 ====================================================================== FAIL: check_types (numpy.core.tests.test_scalarmath.test_types) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packag es/numpy/core/tests/test_scalarmath.py", line 19, in check_types assert a == 1, "error with %r: got %r" % (atype,a) AssertionError: error with <type 'numpy.float128'>: got 1.0 ---------------------------------------------------------------------- Ran 526 tests in 1.133s FAILED (failures=2)

2 1

Re: [Numpy-discussion] Misc Pyrex questions
by Francesc Altet Jan. 23, 2007

Jan. 23, 2007

El 2007-01-22 23:59:03 GMT, en/na Pierre GM va escriure: > Other example: what's more efficient ? > myvar = <ndarray>numpy.empty(shape,dtype) > or > myvar = PyArray_EMPTY(dims, NPY_TYPE) Ok. The second one indeed, but possibly less than you realize (just a 25%, see some benchmarks below). The main point is, as always, avoiding premature optimization. You should first port your code to Pyrex, and then concentrate on the hot points. If some python calls are critical for you, then, go ahead and choose the C call. > But elsewhere in the manual is given the example of a loop using `range`, when > one should use some explicit interval, and my understanding was that using > python expressions was not as efficient as having more proper C expressions. > Is this the case here ? Do I have to reimplement __getitem__ on arrays, or > could I just keep on using the current approach ? Special methods (like __getitem__) of Pyrex extensions performs exactly the same than a __getitem__ made in pure C extensions. So, you don't have be worried about that. A benchmark that I've made (I was curious too ;), and that I'm attaching, proves this. Here is the run on a pretty old machine: $ python run_bench.py ******************** NumPy times ********************************* time for __len__ (numpy)--> 0.203 sum (numpy)--> 49987.2991813 time for __getitem__ (numpy)--> 0.314 ******************** Pyrex times ********************************* time for __len__ (pyrex)--> 0.198 sum (pyrex)--> 49987.2991813 time for __getitem__ (pyrex)--> 0.172 ********* Comparing NumPy creation times (python and C) ********** time for creating an empty array (python)--> 3.305 time for creating an empty array (C)--> 2.664 In this case, the __getitem__ of Pyrex seems to perform better than the __getitem__ of the ndarray object written in C (almost a 2x, in fact). However, this is probably an ilusion, as the ndarray __getitem__ will do far more work than the Pyrex one. OTOH, the __len__ method is far more simple, and can be taken as the demonstration that the overhead of calling special methods in Pyrex from Python is similar to C counterparts. Finally, the difference of overhead in using a Python or a C call for creating an empty array is shown in the last part of the benchmark. All in all, a 25% of difference is not that much. Cheers, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth

1 1

Misc Pyrex questions
by Pierre GM Jan. 22, 2007

Jan. 22, 2007

Dear All, I started playing with Pyrex this week-end, to see how easy it would be to port some subclasses to C. A good thing is that it's not as bad as I was dreading. I do have a lot of question, however: - Can I create a subclass w/ Pyrex ? If yes, how ? I haven't been able to find any example in the numerous tutorials I browsed through, and my naive attempt to: cdef NewSubclass(ndarray) didn't work at all. - I want to create an output ndarray from a given an input sequence/ndarray. I'm using PyArray_FROM_OTF, which asks me for a dtype. How can I guess the dtype of the input sequence ? In Python, a numpy.asarray(obj) would do the trick, the dtype would be set to the maximum possible. - What'd be a more pyrex-esque way to code the resize and reshape functions ? - Is there a standard way to deal w/ __getitem__ ? especially when the item is not an integer (but a slice or a sequence) ? Thanks a lot in advance for your help. P. PS: Sorry for the cross-posting

2 2

setmember1d memory leak?
by Per B. Sederberg Jan. 22, 2007

Jan. 22, 2007

Hi Everybody: I was having a massive memory leak problem in some simulations I coded that I traced down to the setmember1d method. After running for about an hour, my program, which should have been taking up .6% of the RAM was taking up 100% of my 4GB of RAM and would eventually get killed by the system. After replacing that method with the following inefficient code, the memory problems disappeared and I can run for days without my memory use increasing beyond .6%: def ismember(a,b): ainb = zeros(len(a),dtype=bool) for item in b: ainb = ainb | (a==item) return ainb Here's the information about my setup: % uname -a Linux 2.4.21-27.0.4.ELsmp #1 SMP Sat Apr 16 18:53:14 EDT 2005 x86_64 x86_64 x86_64 GNU/Linux % python Python 2.4.4 (#1, Jan 21 2007, 12:09:48) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-49)] on linux2 Type "help", "copyright", "credits" or "license" for more information. %>>> import numpy %>>> numpy.version.version '1.0.1' %>>> It is an AMD opteron machine running RedHat Enterprise with an older version of GCC. The same error occurred with an older version of python (the April 2006 release with version 2.4.3). Does anyone have any idea of what might be occurring in setmember1d in combination with this setup that would cause such a massive memory leak? Thanks, Per

2 3

array is not broadcastable to correct shape
by Keith Goodman Jan. 22, 2007

Jan. 22, 2007

What is the simplest way to make the sum of the positive elements in a Nx1 matrix equal 1? I end up converting the matrix to an array, doing the normalization, then converting the array back to a matrix. >> m = M.randn(10,1) >> m[m > 0,:] = m[m > 0,:] / m[m > 0,:].sum() --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) <ipython console> ValueError: array is not broadcastable to correct shape >> >> a = M.asarray(m) >> a[a > 0,:] = a[a > 0,:] / a[a > 0,:].sum() >> a[a > 0,:].sum() 1.0 >> m = M.asmatrix(a)

1 0

f2py
by Sturla Molden Jan. 22, 2007

Jan. 22, 2007

I have a question regarding f2py: Can it map NumPy arrays to Fortran 95 'dope vectors'? I can find many examples on interfacing Fortran 77 with NumPy, but not Fortran 95. This is of course very different as Fortran 77 are always contiguous, whereas Fortran 95 arrays are not. Thus a Fortran 95 compiler will pass a pointer to a struct referred to as a 'dope vector', which by the way is very similar to a NumPy ndarray. But the ABI for a 'dope vector' is implementation (compiler) dependent. So what does f2py do? Can it use Charm? Sturla Molden

1 0

gcc vs Intel compiler
by rex Jan. 22, 2007

Jan. 22, 2007

Has anyone done any benchmarks to compare Numpy's speed with these two compilers on a Core 2 Duo? Thanks, -rex

1 0

testoob and numpy
by Jarrod Millman Jan. 21, 2007

Jan. 21, 2007

Hello, Is anyone using testoob with numpy (http://testoob.sourceforge.net/)? I am trying to figure out how to use it and would appreciate any pointers to get me started. My goal is to provide some documentation on the wiki about how to do this. It looks like Pearu had it working not too long ago: http://projects.scipy.org/scipy/numpy/changeset/2425 There is also a mention of testoob in the DISTUTILS.txt file. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

1 0

ANN: MaskedArray as a subclass of ndarray - followup
by Pierre GM Jan. 21, 2007

Jan. 21, 2007

All, I've updated this famous reimplementation of maskedarray I keep ranting about. A new feature has been introduced : hard_mask. When a masked array is created with the flag hard_mask=True, the mask can only grow, not shrink. In other terms, masked values cannot be unmasked. The flag hard_mask is set to False by default. You can toggle the behavior with the `harden_mask` and `soften_mask` methods. >>> import maskedarray as MA >>> x=MA.array([1,2,3],mask=[1,0,0], hard_mask=True) >>> x[0]=999 >>> print x [-- 2 3] >>> x.soften_mask() >>> x[0]=999 >>> print x [999 2 3] I also put the file `timer_comparison.py`, that runs some unittests with each implementation (numpy.core.ma and maskedarray), and outputs the minimum times. On my machine, there doesn't seem to be a lot of differences, maskedarray being slightly faster. However, I'm not sure whether I can really trust the results. What would be the best way to compare the relative performances of numpy.core.ma and maskedarray ? Should I run tests on a function basis ? Thanks in advance for any idea/suggestion. P.

10 15