Mailman 3 July 2011 - NumPy-Discussion

Array vectorization in numpy
by Carlos Becker 29 Jul '11

29 Jul '11

Hi, I started with numpy a few days ago. I was timing some array operations and found that numpy takes 3 or 4 times longer than Matlab on a simple array-minus-scalar operation. This looks as if there is a lack of vectorization, even though this is just a guess. I hope this is not reposting. I tried searching the mailing list database but did not find anything related specifically to a problem like this one. Here there is the python test code: -------------------------------------------- from datetime import datetime import numpy as np def test(): m = np.ones([2000,2000],float) N = 100 t1 = datetime.now() for x in range(N): k = m - 0.5 t2 = datetime.now() print (t2 - t1).total_seconds() / N -------------------------------------------- And matlab: -------------------------------------------- m = rand(2000,2000); N = 100; tic; for I=1:N k = m - 0.5; end toc / N -------------------------------------------- I have the impression that the speed boost with Matlab is not related to matlab optimizations, since singe-runs also render similar timings. I tried compiling ATLAS for SSE2 and didn't observe any difference. Any clues? Thanks, Carlos

14 37

c-api return two arrays
by Yoshi Rokuko 29 Jul '11

29 Jul '11

hey, i have an algorithm that computes two matrices like that: A(i,k) = (x(i,k) + y(i,k))/norm B(i,k) = (x(i,k) - y(i,k))/norm it would be convenient to have the method like that: >>> A, B = mod.meth(C, prob=.95) is ith possible to return two arrays? best regards

3 4

dtype repr change?
by Matthew Brett 29 Jul '11

29 Jul '11

Hi, I see that (current trunk): In [9]: np.ones((1,), dtype=bool) Out[9]: array([ True], dtype='bool') - whereas (1.5.1): In [2]: np.ones((1,), dtype=bool) Out[2]: array([ True], dtype=bool) That is breaking quite a few doctests. What is the reason for the change? Something to do with more planned dtypes? Thanks a lot, Matthew

10 26

Numpy Help
by DIPO ELEGBEDE 29 Jul '11

29 Jul '11

Hi All, I am fresh on this list and would be looking forward to as much help as I can get. I am hoping to develop ino helping others too after a short while. Kindly help me with this task. I would appreciate if you can point me to an example or brief explanation. I have a 4 by 4 matrix filled with 0s, 1s and 2s. I want to loop through the whole matrix to get the fields with 1s and 2s only and then count how many ones and how many twos. Please help. -- Dipo Elegbede,OCA 08033299270,08077682428

2 2

boolean array indexing and broadcasting rules
by Mark Wiebe 29 Jul '11

29 Jul '11

As part of supporting the NA mask, I've rewritten boolean indexing. Here's a timing comparison of my version versus a previous version: In [2]: np.__version__ Out[2]: '1.4.1' In [3]: a = np.zeros((1000,1000)) In [4]: mask = np.random.rand(1000,1000) > 0.5 In [5]: timeit a[mask] = 1.5 10 loops, best of 3: 71.5 ms per loop In [2]: np.__version__ Out[2]: '2.0.0.dev-a1e98d4' In [3]: a = np.zeros((1000,1000)) In [4]: mask = np.random.rand(1000,1000) > 0.5 In [5]: timeit a[mask] = 1.5 100 loops, best of 3: 12.6 ms per loop That's a 5.6 times speedup. Unfortunately, it turns out that the old code didn't use NumPy broadcasting rules. This change found a bug in the hardmask code of numpy.ma, so I think there's definitely a case to be made that it is improvement, but at the same time it looks like the NumPy and SciPy polynomial code is using this behavior. I get test errors of the following form in Scipy: ====================================================================== ERROR: test_orthogonal_eval.TestPolys.test_sh_chebyu ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/mwiebe/virtualenvs/dev/lib/python2.7/site-packages/nose-1.0.0-py2.7.egg/nose/case.py", line 187, in runTest self.test(*self.arg) File "/home/mwiebe/virtualenvs/dev/lib/python2.7/site-packages/scipy/special/tests/test_orthogonal_eval.py", line 111, in test_sh_chebyu param_ranges=[], x_range=[0, 1]) File "/home/mwiebe/virtualenvs/dev/lib/python2.7/site-packages/scipy/special/tests/test_orthogonal_eval.py", line 66, in check_poly ds.check() File "/home/mwiebe/virtualenvs/dev/lib/python2.7/site-packages/scipy/special/_testutils.py", line 165, in check data = data[param_mask] ValueError: operands could not be broadcast together with shapes (100,3) (100) What are people's thoughts on this predicament? Cheers, Mark

2 1

Issues with adding new dtypes - customizing ndarray attributes
by Martin Ling 29 Jul '11

29 Jul '11

Hi, I'd like to kick off some discussion on general issues I've encountered while developing the quaternion dtype (see other thread, and the code at: https://github.com/martinling/numpy_quaternion) The basic issue is that the attributes of ndarray cannot be adapted to the dtype of a given array. Indeed, their behaviour can't be changed at all without patching numpy itself. There are essentially four cases of the problem: 1. Attributes which do the wrong thing even though there is a mechanism that should let them do the right thing, e.g: >>> a = array([quaternion(1,2,3,4), quaternion(5,6,7,8)]) >>> conjugate(a) # correct, calls conjugate ufunc I defined array([quaternion(1, -2, -3, -4), quaternion(5, -6, -7, -8)], dtype=quaternion) >>> a.conjugate() # incorrect, why doesn't it do the same? array([quaternion(1, 2, 3, 4), quaternion(5, 6, 7, 8)], dtype=quaternion) >>> min(a) # works, calls min ufunc I defined quaternion(1, 2, 3, 4) >>> a.min() # fails, why doesn't it do the same? ValueError: No cast function available. 2. Attributes that do the wrong thing with no mechanism to override them: >>> array([q.real for q in a]) array([ 1., 5.]) >>> a.real # would like this to return the same, can't make it do so array([quaternion(1, 2, 3, 4), quaternion(5, 6, 7, 8)], dtype=quaternion) 3. Attributes that don't exist and could be added to suit the dtype: >>> array([q.y for q in a]) array([ 3., 7.]) >>> a.y # would like this to return the same, can't make it do so AttributeError: 'numpy.ndarray' object has no attribute 'y' 4. Attributes that already exist and make no sense for some dtypes: >>> sa = array(['foo', 'bar', 'baz']) >>> sa.imag # why can I even do this? array(['', '', ''], dtype='|S3') We had ѕome discussion about this at the SciPy conference sprints and the consensus seemed to be that allowing dtypes to customize the attributes of ndarrays would be a good thing. This would also be useful for struct arrays, datetime arrays, etc. What do people think? Martin

2 1

C-API: PyTypeObject* for NumPy scalar types
by Johan Råde 29 Jul '11

29 Jul '11

How do I get the PyTypeObject* for a NumPy scalar type such as np.uint8? (The reason I'm asking is the following: I'm writing a C++ extension module. The Python interface to the module has a function f that takes a NumPy scalar type as an argument, for instance f(np.uint8). Then the corresponding C++ function receives a PyObject* and needs to decide which type object it points to.) --Johan

1 1

ticket 1793
by Paul Anton Letnes 29 Jul '11

29 Jul '11

Hi! In my quest for a bug-free numpy I have, I think, fixed ticket 1793. https://github.com/numpy/numpy/pull/123 Enjoy - feedback welcome, of course. Cheers, Paul.

3 2

Can I index array starting with 1?
by Jeremy Conlin 29 Jul '11

29 Jul '11

I have a need to index my array(s) starting with a 1 instead of a 0. The reason for this is to be consistent with the documentation of a format I'm accessing. I know I can just '-1' from what the documentation says, but that can get cumbersome. Is there a magic flag I can pass to a numpy array (maybe when it is created) so I can index starting at 1 instead of the Python default? Thanks, Jeremy

4 7

Regression in choose()
by Ed Schofield 28 Jul '11

28 Jul '11

Hi all, I have been investigation the limitation of the choose() method (and function) to 32 elements. This is a regression in recent versions of NumPy. I have tested choose() in the following NumPy versions: 1.0.4: fine 1.1.1: bug 1.2.1: fine 1.3.0: bug 1.4.x: bug 1.5.x: bug 1.6.x: bug Numeric 24.3: fine (To run the tests on versions of NumPy prior to 1.4.x I used Python 2.4.3. For the other tests I used Python 2.7.) Here 'bug' means the choose() function has the 32-element limitation. I have been helping an organization to port a large old Numeric-using codebase to NumPy, and the choose() limitation in recent NumPy versions is throwing a spanner in the works. The codebase is currently using both NumPy and Numeric side-by-side, with Numeric only being used for its choose() function, with a few dozen lines like this: a = numpy.array(Numeric.choose(b, c)) Here is a simple example that triggers the bug. It is a simple extension of the example from the choose() docstring: ---------------- import numpy as np choices = [[0, 1, 2, 3], [10, 11, 12, 13], [20, 21, 22, 23], [30, 31, 32, 33]] np.choose([2, 3, 1, 0], choices * 8) ---------------- A side note: the exception message (defined in core/src/multiarray/iterators.c) is also slightly inconsistent with the actual behaviour: Traceback (most recent call last): File "chooser.py", line 6, in <module> np.choose([2, 3, 1, 0], choices * 8) File "/usr/lib64/python2.7/site-packages/numpy/core/fromnumeric.py", line 277, in choose return _wrapit(a, 'choose', choices, out=out, mode=mode) File "/usr/lib64/python2.7/site-packages/numpy/core/fromnumeric.py", line 37, in _wrapit result = getattr(asarray(obj),method)(*args, **kwds) ValueError: Need between 2 and (32) array objects (inclusive). The actual behaviour is that choose() passes with 31 objects but fails with 32 objects, so this should read "exclusive" rather than "inclusive". (And why the parentheses around 32?) Does anyone know what changed between 1.2.1 and 1.3.0 that introduced the 32-element limitation to choose(), and whether we might be able to lift this limitation again for future NumPy versions? I have a couple of days to work on a patch ... if someone can advise me how to approach this. Best wishes, Ed -- Dr. Edward Schofield Python Charmers +61 (0)405 676 229 http://pythoncharmers.com

3 3