Mailman 3 October 2009 - NumPy-Discussion

user defined types
by Artem Serebriyskiy 19 Oct '09

19 Oct '09

Hello! Would you please give me some examples of open source projects which use the implementation of user defined types for numpy library? (implementation on the C-API level)

2 1

Multiple string formatting while writing an array into a file
by Gökhan Sever 19 Oct '09

19 Oct '09

Hello, I have a relatively simple question which I couldn't figure out myself yet. I have an array that I am writing into a file using the following savetxt method. np.savetxt(fid, output_array, fmt='%12.4f', delimiter='') However, I have made some changes on the code and I require to write after 7th element of the array as integer instead of 12.4 formatted float. The change below doesn't help me to solve the problem since I get a "ValueError: setting an array element with a sequence." np.savetxt(fid, (output_array[:7], output_array[7:]), fmt=('%12.4f', '%12d'), delimiter='') What would be the right approach to fix this issue? Thanks. -- Gökhan

1 1

numpy build/installation problems ?
by josef.pktd＠gmail.com 19 Oct '09

19 Oct '09

I wanted to finally upgrade my numpy, so I can build scipy trunk again, but I get test failures with numpy. And running the tests of the previously compiled version of scipy crashes in signaltools. Is this a problem with my build (the usual official MingW on WindowsXP), or are there still ABI problems in numpy trunk? I did the build twice with (I think) clean directories and get the same result. Thanks, Josef Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.__file__ 'C:\\Josef\\_progs\\Subversion\\numpy-trunk\\dist\\numpy-1.4.0.dev7539.win32\\Pr ograms\\Python25\\Lib\\site-packages\\numpy\\__init__.py' >>> numpy.test() Running unit tests for numpy NumPy version 1.4.0.dev7539 NumPy is installed in C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.de v7539.win32\Programs\Python25\Lib\site-packages\numpy Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Int el)] nose version 0.11.1 ................................. \u03a3 .[[' abc' ''] ['12345' 'MixedCase'] ['123 345' 'UPPER']] ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ............................K................................................... ..FF.......................FFFF................................................. ................................................................................ ................................................................................ ................................................................................ ................................................................................ ............................................................................C:\J osef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Programs\Pytho n25\Lib\site-packages\numpy\lib\io.py:1332: ConversionWarning: Some errors were detected ! Line #2 (got 4 columns instead of 5) Line #12 (got 4 columns instead of 5) Line #22 (got 4 columns instead of 5) Line #32 (got 4 columns instead of 5) Line #42 (got 4 columns instead of 5) warnings.warn(errmsg, ConversionWarning) .C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Programs\ Python25\Lib\site-packages\numpy\lib\io.py:1332: ConversionWarning: Some errors were detected ! Line #2 (got 4 columns instead of 2) Line #12 (got 4 columns instead of 2) Line #22 (got 4 columns instead of 2) Line #32 (got 4 columns instead of 2) Line #42 (got 4 columns instead of 2) warnings.warn(errmsg, ConversionWarning) ................E.E.EE....................K........K............F............... ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ..S............................................................................. ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ............................. ====================================================================== ERROR: Test giving usecols with a comma-separated string ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Pr ograms\Python25\Lib\site-packages\numpy\lib\tests\test_io.py", line 747, in test _usecols_as_css names="a, b, c", usecols="a, c") File "\Programs\Python25\Lib\site-packages\numpy\lib\io.py", line 1099, in gen fromtxt the docstring of the `genfromtxt` function. AttributeError: 'tuple' object has no attribute 'index' ====================================================================== ERROR: Test usecols with named columns ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Pr ograms\Python25\Lib\site-packages\numpy\lib\tests\test_io.py", line 773, in test _usecols_with_named_columns usecols=('a', 'c'), **kwargs) File "\Programs\Python25\Lib\site-packages\numpy\lib\io.py", line 1099, in gen fromtxt the docstring of the `genfromtxt` function. AttributeError: 'tuple' object has no attribute 'index' ====================================================================== ERROR: Test with missing and filling values ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Pr ograms\Python25\Lib\site-packages\numpy\lib\tests\test_io.py", line 861, in test _user_filling_values test = np.genfromtxt(StringIO.StringIO(data), **kwargs) File "\Programs\Python25\Lib\site-packages\numpy\lib\io.py", line 1127, in gen fromtxt See Also AttributeError: 'tuple' object has no attribute 'index' ====================================================================== ERROR: test_user_missing_values (test_io.TestFromTxt) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Pr ograms\Python25\Lib\site-packages\numpy\lib\tests\test_io.py", line 845, in test _user_missing_values **basekwargs) File "\Programs\Python25\Lib\site-packages\numpy\lib\io.py", line 1481, in maf romtxt File "\Programs\Python25\Lib\site-packages\numpy\lib\io.py", line 1127, in gen fromtxt See Also AttributeError: 'tuple' object has no attribute 'index' ====================================================================== FAIL: test_umath.test_hypot_special_values(1.#QNAN, 1.#INF) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Pr ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath.py", line 211, in assert_hypot_isinf assert np.isinf(ncu.hypot(x, y)) AssertionError ====================================================================== FAIL: test_umath.test_hypot_special_values(1.#INF, 1.#QNAN) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Pr ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath.py", line 211, in assert_hypot_isinf assert np.isinf(ncu.hypot(x, y)) AssertionError ====================================================================== FAIL: test_umath.test_arctan2_special_values(nan, 2.3561944901923448) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Pr ograms\Python25\Lib\site-packages\numpy\testing\utils.py", line 449, in assert_a lmost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal ACTUAL: nan DESIRED: 2.3561944901923448 ====================================================================== FAIL: test_umath.test_arctan2_special_values(nan, -2.3561944901923448) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Pr ograms\Python25\Lib\site-packages\numpy\testing\utils.py", line 449, in assert_a lmost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal ACTUAL: nan DESIRED: -2.3561944901923448 ====================================================================== FAIL: test_umath.test_arctan2_special_values(nan, 0.78539816339744828) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Pr ograms\Python25\Lib\site-packages\numpy\testing\utils.py", line 449, in assert_a lmost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal ACTUAL: nan DESIRED: 0.78539816339744828 ====================================================================== FAIL: test_umath.test_arctan2_special_values(nan, -0.78539816339744828) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Pr ograms\Python25\Lib\site-packages\numpy\testing\utils.py", line 449, in assert_a lmost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal ACTUAL: nan DESIRED: -0.78539816339744828 ====================================================================== FAIL: test_doctests (test_polynomial.TestDocs) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Pr ograms\Python25\Lib\site-packages\numpy\lib\tests\test_polynomial.py", line 90, in test_doctests return rundocs() File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Pr ograms\Python25\Lib\site-packages\numpy\testing\utils.py", line 951, in rundocs raise AssertionError("Some doctests failed:\n%s" % "\n".join(msg)) AssertionError: Some doctests failed: ********************************************************************** File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7539.win32\Prog rams\Python25\Lib\site-packages\numpy\lib\tests\test_polynomial.py", line 20, in test_polynomial Failed example: print poly1d([100e-90, 1.234567e-9j+3, -1234.999e8]) Expected: 2 1e-88 x + (3 + 1.235e-09j) x - 1.235e+11 Got: 2 1e-088 x + (3 + 1.235e-009j) x - 1.235e+011 ---------------------------------------------------------------------- Ran 2140 tests in 13.281s FAILED (KNOWNFAIL=3, SKIP=1, errors=4, failures=7) <nose.result.TextTestResult run=2140 errors=4 failures=7> >>> import scipy >>> scipy.__file__ 'c:\\josef\\eclipsegworkspace\\scipy-trunk-work\\scipytrunkcopy\\scipy\\__init__ .pyc' >>> scipy.test() Running unit tests for scipy NumPy version 1.4.0.dev7539 NumPy is installed in C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.de v7539.win32\Programs\Python25\Lib\site-packages\numpy SciPy version 0.8.0.dev SciPy is installed in c:\josef\eclipsegworkspace\scipy-trunk-work\scipytrunkcopy \scipy Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Int el)] nose version 0.11.1 E..... ... crash with verbose the last tests are test_rank1 (test_signaltools.TestCorrelateComplexSingle) ... ok test_rank3 (test_signaltools.TestCorrelateComplexSingle) ... ok test_rank1 (test_signaltools.TestCorrelateDouble) ... ok test_rank3 (test_signaltools.TestCorrelateDouble) ... ok test_rank1 (test_signaltools.TestCorrelateExtended) ... ok test_rank3 (test_signaltools.TestCorrelateExtended) ... ok test_rank1 (test_signaltools.TestCorrelateObject) ... ok test_rank3 (test_signaltools.TestCorrelateObject) ... ok test_rank1 (test_signaltools.TestCorrelateSingle) ... ok test_rank3 (test_signaltools.TestCorrelateSingle) ... ok test_signaltools.TestDecimate.test_basic ... crash

3 4

fortran vs numpy on mac/linux - gcc performance?
by Robin 19 Oct '09

19 Oct '09

Hi, I have been looking at moving some of my bottleneck functions to fortran with f2py. To get started I tried some simple things, and was surprised they performend so much better than the number builtins - which I assumed would be c and would be quite fast. On my Macbook pro laptop (Intel core 2 duo) I got the following results. Numpy is built with xcode gcc 4.0.1 and gfortran is 4.2.3 - fortran code for shuffle and bincount below: In [1]: x = np.random.random_integers(0,1023,1000000).astype(int) In [2]: import ftest In [3]: timeit np.bincount(x) 100 loops, best of 3: 3.97 ms per loop In [4]: timeit ftest.bincount(x,1024) 1000 loops, best of 3: 1.15 ms per loop In [5]: timeit np.random.shuffle(x) 1 loops, best of 3: 605 ms per loop In [6]: timeit ftest.shuffle(x) 10 loops, best of 3: 139 ms per loop So fortran was about 4 times faster for these loops - similarly faster than cython as well. So I was really happy as these are two of my biggest bottlenecks, but when I moved a linux workstation I got different results. Here with gcc/gfortran 4.3.3 : In [3]: x = np.random.random_integers(0,1023,1000000).astype(int) In [4]: timeit np.bincount(x) 100 loops, best of 3: 8.18 ms per loop In [5]: timeit ftest.bincount(x,1024) 100 loops, best of 3: 8.25 ms per loop In [6]: In [7]: timeit np.random.shuffle(x) 1 loops, best of 3: 379 ms per loop In [8]: timeit ftest.shuffle(x) 10 loops, best of 3: 172 ms per loop So shuffle is a bit faster, but bincount is now the same as fortran. The only thing I can think is that it is due to much better performance of the more recent c compiler. I think this would also explain why f2py extension was performing so much better than cython on the mac. So my question is - is there a way to build numpy with a more recent compiler on leopard? (I guess I could upgrade to snow leopard now) - Could I make the numpy install use gcc-4.2 from xcode or would it break stuff? Could I use gcc 4.3.3 from macports? It would be great to get a 4x speed up on all numpy c loops! (already just these two functions I use a lot would make a big difference). Cheers Robin

1 1

Subclassing record array
by Loïc BERTHE 18 Oct '09

18 Oct '09

Hi, I would like to create my own class of record array to deal with units. Here is the code I used, inspired from http://docs.scipy.org/doc/numpy-1.3.x/user/basics.subclassing.html#slightly… : [code] from numpy import * class BlocArray(rec.ndarray): """ Recarray with units and pretty print """ fmt_dict = {'S' : '%10s', 'f' : '%10.6G', 'i': '%10d'} def __new__(cls, data, titles=None, units=None): # guess format for each column data2 = [] for line in zip(*data) : try : data2.append(cast[int](line)) # integers except ValueError : try : data2.append(cast[float](line)) # reals except ValueError : data2.append(cast[str](line)) # characters # create the array dt = dtype(zip(titres, [line.dtype for line in data2])) obj = rec.array(data2, dtype=dt).view(cls) # add custom attributes obj.units = units or [] obj._fmt = " ".join(obj.fmt_dict[d[1][1]] for d in dt.descr) + '\n' obj._head = "%10s "*len(dt.names) % dt.names +'\n' obj._head += "%10s "*len(dt.names) % tuple('(%s)' % u for u in units) +'\n' # Finally, we must return the newly created object: return obj titles = ['Name', 'Nb', 'Price'] units = ['/', '/', 'Eur'] data = [['fish', '1', '12.25'], ['egg', '6', '0.85'], ['TV', 1, '125']] bloc = BlocArray(data, titles=titles, units=units) In [544]: bloc Out[544]: Name Nb Price (/) (/) (Eur) fish 1 12.25 egg 6 0.85 TV 1 125 [/code] It's almost working, but I have some isues : - I can't access data through indexing In [563]: bloc['Price'] /home/loic/Python/numpy/test.py in <genexpr>((r,)) 50 51 def __repr__(self): ---> 52 return self._head + ''.join(self._fmt % tuple(r) for r in self) TypeError: 'numpy.float64' object is not iterable So I think that overloading the __repr__ method is not that easy - I can't access data through attributes now : In [564]: bloc.Nb AttributeError: 'BlocArray' object has no attribute 'Nb' - I can't use 'T' as field in theses array as the T method is already here as a shortcut for transpose Have you any hints to make this work ? -- LB

3 2

double-precision sqrt?
by Adam Ginsburg 18 Oct '09

18 Oct '09

Hi folks, I'm trying to write a ray-tracing code for which high precision is required. I also "need" to use square roots. However, math.sqrt and numpy.sqrt seem to only use single-precision floats. Is there a simple way to make sqrt use higher precision? Alternately, am I simply being obtuse? Thanks, Adam Example code: from scipy.optimize.minpack import fsolve from numpy import sqrt sqrt(float64(1.034324523462345)) # 1.0170174646791199 f=lambda x: x**2-float64(1.034324523462345)**2 f(sqrt(float64(1.034324523462345))) # -0.03550269637326231 fsolve(f,1.01) # 1.0343245234623459 f(fsolve(f,1.01)) # 1.7763568394002505e-15 fsolve(f,1.01) - sqrt(float64(1.034324523462345)) # 0.017307058783226026

5 6

intersect1d for N input arrays
by Martin Spacek 17 Oct '09

17 Oct '09

I have a list of many arrays (in my case each is unique, ie has no repeated elements), and I'd like to extract the intersection of all of them, all in one go. I'm running numpy 1.3.0, but looking at today's rev of numpy.lib.arraysetops (http://svn.scipy.org/svn/numpy/trunk/numpy/lib/arraysetops.py), I see intersect1d has changed. Just a note: the example used in the docstring implies that the two arrays need to be the same length, which isn't the case. Maybe it would be good to change the example to two arrays of different lengths. intersect1d takes exactly 2 arrays. I've modified it a little to take the intersection of any number of 1D arrays (of any length), in a list or tuple. It seems to work fine, but could use more testing. Here it is with most of the docs stripped. Feel free to use it, although I suppose for symmetry, many of the other functions in arraysetops.py would also have to be modified to work with N arrays: def intersect1d(arrays, assume_unique=False): """Find the intersection of any number of 1D arrays. Return the sorted, unique values that are in all of the input arrays. Adapted from numpy.lib.arraysetops.intersect1d""" N = len(arrays) arrays = list(arrays) # allow assignment if not assume_unique: for i, arr in enumerate(arrays): arrays[i] = np.unique(arr) aux = np.concatenate(arrays) # one long 1D array aux.sort() # sorted shift = N-1 return aux[aux[shift:] == aux[:-shift]]

2 2

genfromtxt documentation : review needed
by Pierre GM 16 Oct '09

16 Oct '09

All, Here's a first draft for the documentation of np.genfromtxt. It took me longer than I thought, but that way I uncovered and fix some bugs. Please send me your comments/reviews/etc I count especially on our documentation specialist to let me know where to put it. Thx in advance P.

2 2

Google Groups archive?
by David Warde-Farley 16 Oct '09

16 Oct '09

Does anyone know what happened to the Google Groups archive of this list? when I try to access it, I see: Cannot find numpy-discussion The group named numpy-discussion has been removed because it violated Google's Terms Of Service. This seems exceedingly odd. Does anyone know _how_ we violated the ToS? David

4 3

extension questions: f2py and cython
by Robin 15 Oct '09

15 Oct '09

Hi, Sent this last week but checking the archives it appears not to have got through. Hopefully this will work... I am looking at moving some of my code to fortran to use with f2py. To get started I used this simple example: SUBROUTINE bincount (x,c,n,m) IMPLICIT NONE INTEGER, INTENT(IN) :: n,m INTEGER, DIMENSION(n), INTENT(IN) :: x INTEGER, DIMENSION(0:m-1), INTENT(OUT) :: c INTEGER :: i DO i = 1, n c(x(i)) = c(x(i)) + 1 END DO END It performs well: In [1]: x = np.random.random_integers(0,1023,1000000).astype(int) In [4]: timeit test.bincount(x,1024) 1000 loops, best of 3: 1.16 ms per loop In [5]: timeit np.bincount(x) 100 loops, best of 3: 4 ms per loop I'm guessing most of the benefit comes from less checking + not having to find the maximum value (which I pass as parameter m). But I have some questions. It seems to work as is, but I don't set c to zeros anywhere. Can I assume arrays created by f2py are zero? Is this the recommended way to use f2py with arrays? (I initially tried using assumed arrays with DIMENSION(:) but it couldn't get it to work). Also I'm quite new to fortran - what would be the advantages, if any, of using a FORALL instead of DO in a situation like this? I guess with 1D arrays it doesn't make a copy since ordering is not a problem, but if it was 2D arrays am I right in thinking that if I passed in a C order array it would automatically make a copy to F order. What about the return - will I get a number array in F order, or will it automatically be copied to C order? (I guess I will see but I haven't got onto trying that yet). What if I wanted to keep all the array creation in numpy - ie call it as the fortran subroutine bincount(x,c) and have c modified in place? Should I be using !f2py comments? I wasn't clear if these are needed - it seems to work as is but could they give any improvement? For comparison I tried the same thing in cython - after a couple of iterations with not typing things properly I ended up with: import numpy as np cimport numpy as np cimport cython @cython.boundscheck(False) def bincount(np.ndarray[np.int_t, ndim=1] x not None,int m): cdef int n = x.shape[0] cdef unsigned int i cdef np.ndarray[np.int_t, ndim=1] c = np.zeros(m,dtype=np.int) for i from 0 <= i < n: c[<unsigned int>x[i]] += 1 return c which now performs a bit better than np.bincount, but still significantly slower than the fortran. Is this to be expected or am I missing something in the cython? In [14]: timeit ctest.bincount(x,1024) 100 loops, best of 3: 3.31 ms per loop Cheers Robin

3 5