Mailman 3 November 2009 - NumPy-Discussion

converting discrete data to unique integers
by David Warde-Farley 04 Nov '09

04 Nov '09

Hi, Suppose I have an array 'd' In [75]: d Out[75]: array(['parrot', 'parrot', 'dog', 'cat', 'parrot', 'dog', 'parrot', 'cat', 'dog', 'dog', 'dog', 'cat', 'cat', 'dog', 'cat', 'parrot', 'cat', 'cat', 'dog', 'parrot', 'parrot', 'parrot', 'cat', 'dog', 'parrot', 'dog', 'dog', 'dog', 'dog', 'parrot', 'parrot', 'cat', 'dog', 'parrot', 'cat', 'parrot', 'cat', 'dog', 'parrot', 'cat', 'parrot', 'cat', 'parrot', 'parrot', 'parrot', 'parrot', 'dog', 'cat', 'parrot', 'cat'], dtype='|S6') I'd like to map every unique element (these could be strings, objects, or already ints) to a unique integer between 0 and len(unique(d)) - 1. The solution I've come up with is In [76]: uniqueind, vectorind = (d == unique(d)[:, newaxis]).nonzero() In [77]: myints = uniqueind[argsort(vectorind)] But I wonder if there's a better way to do this. Anyone ever run into this problem before? David

5 6

Automatic string length in recarray
by Thomas Robitaille 04 Nov '09

04 Nov '09

Hi, I'm having trouble with creating np.string_ fields in recarrays. If I create a recarray using np.rec.fromrecords([(1,'hello'),(2,'world')],names=['a','b']) the result looks fine: rec.array([(1, 'hello'), (2, 'world')], dtype=[('a', '<i8'), ('b', '| S5')]) But if I want to specify the data types: np.rec.fromrecords([(1,'hello'),(2,'world')],dtype=[('a',np.int8), ('b',np.str)]) the string field is set to a length of zero: rec.array([(1, ''), (2, '')], dtype=[('a', '|i1'), ('b', '|S0')]) I need to specify datatypes for all numerical types since I care about int8/16/32, etc, but I would like to benefit from the auto string length detection that works if I don't specify datatypes. I tried replacing np.str by None but no luck. I know I can specify '|S5' for example, but I don't know in advance what the string length should be set to. Is there a way to solve this problem without manually examining the data that is being passed to rec.fromrecords? Thanks for any help, Thomas

4 6

f2py-users list not working
by Robin 04 Nov '09

04 Nov '09

I just tried to send the message below to f2py-users - f2py-users(a)cens.ioc.ee, but delivery failed. Not sure where else to report this so hopefully here is ok. Cheers Robin ---------- Forwarded message ---------- From: Mail Delivery Subsystem <mailer-daemon(a)googlemail.com> Date: Tue, Nov 3, 2009 at 9:40 PM Subject: Delivery Status Notification (Failure) To: robince(a)gmail.com Delivery to the following recipient failed permanently: f2py-users(a)cens.ioc.ee Technical details of permanent failure: Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 550 550 Unrouteable address (state 14). ----- Original message ----- Subject: writing module for 32/64 bit From: Robin <robince(a)gmail.com> To: f2py-users(a)cens.ioc.ee Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi, I would like to write some subroutines in fortran involving integers and distribute them in a small package. Users might be on 32 bit or 64 bit. Is there an easy or recommended way to approach this? Ideally I would like to work with the native integer type - but if I use 'integer' in fortran it is always 32 bit and f2py converts input aways when numpy is 64 bit. If I use integer*8 in the code then its fine for 64 bit, but on 32 bit platforms, both f2py has to convert and its not the native integer size. What I (think) I'd like to do is be able to use the native platform integer type, like numpy does, and then not worry about. I found there are options like -fdefault-int-8 for gfortran, but when I add that stuff breaks (bus error) since I guess f2py doesn't know about it and is converting and passing 32 bits anyway. Is there any way around this, or do I need to maintain 2 different versions with different fortran code for 32bit/64bit? Or is it possible to acheive something like this with preprossor #ifdefs? (not sure how this works with fortran, or if f2py would be aware of it). Cheers Robin

3 5

arrays comparison issue
by Jean-Luc Menut 04 Nov '09

04 Nov '09

Hello all, If if define 2 variables a and b by doing the following : on [5]: a Out[5]: array([ 1.7]) In [6]: b=array([0.8])+array([0.9]) In [7]: b Out[7]: array([ 1.7]) if I test the equality of a and b, instead to obatin True, I have : In [8]: a==b Out[8]: array([False], dtype=bool) I know it not related only to numpy but also to python. How do you tackle this problem ? (I also tried on IDL and it works well : I guess it truncate the number). Thanks , Jean-Luc

3 3

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 38, Issue 11
by Rick White 04 Nov '09

04 Nov '09

The difference between IDL and numpy is that IDL uses single precision floats by default while numpy uses doubles. If you try it with doubles in IDL, you will see that it also returns false. As David Cournapeau said, you should not expect different floating point arithmetic operations to give exactly the same result. It's just luck if they do. E.g., in IDL you'll find that 0.1+0.6 is not equal to 0.7. That's because 0.1 and 0.6 are not exactly representable as floats. On Nov 4, 2009, Jean-Luc Menut wrote: > Hello all, > > > > If if define 2 variables a and b by doing the following : > > on [5]: a > Out[5]: array([ 1.7]) > > In [6]: b=array([0.8])+array([0.9]) > > In [7]: b > Out[7]: array([ 1.7]) > > > if I test the equality of a and b, instead to obatin True, I have : > In [8]: a==b > Out[8]: array([False], dtype=bool) > > I know it not related only to numpy but also to python. > How do you tackle this problem ? > > (I also tried on IDL and it works well : I guess it truncate the > number). > > Thanks , > > Jean-Luc

1 0

strange performance on mac 2.5/2.6 32/64 bit
by Robin 04 Nov '09

04 Nov '09

Hi, I'm not sure if this is of much interest but it's been really puzzling me so I thought I'd ask. In an earlier post I described how I was surprised a simple f2py wrapped fortran bincount was 4x faster than np.bincount - but that differential only seemed to be on my mac; on moving to linux they both took more or less the same time. I'm trying to work out if it is worth moving some of my bottlenecks to fortran (most of which are np builtins). So far it looks like it is - but only on my mac and only 32bit (see below). Well the only explanation I thought was that the gcc-4.0 used to build numpy on a mac didn't perform so well, so after upgrading to snow leopard I've been trying to look at this again. I was hoping I could get the equivalent performance on my mac, like on linux, which would result in the np c stuff being a couple of times faster. So far, with Python 2.6.3 in 64 bit - numpy seems to be significantly slower and my fortran code _much_ slower - even from the same compiler. Can anyone help me understand what is going on? I have only been able to build 32 bit numpy against 2.5.4 with apple gcc-4.0 and 64 bit numpy against 2.6.3 universal with gcc-4.2. I haven't been able to get a numpy I can import on 2.6.3 in 32 bit mode ( http://projects.scipy.org/numpy/ticket/1221 ). Here are the results for python.org 32 bit 2.5.4, numpy compiled with apple gcc 4.0, f2py using att gfortran 4.2: In [2]: timeit x = np.random.random_integers(0,1023,100000000).astype(int) 1 loops, best of 3: 2.86 s per loop In [3]: x = np.random.random_integers(0,1023,100000000).astype(int) In [4]: timeit np.bincount(x) 1 loops, best of 3: 435 ms per loop In [6]: timeit gf42.bincount(x,1024) 10 loops, best of 3: 129 ms per loop In [7]: np.__version__ Out[7]: '1.4.0.dev7618' And for self-built (apple gcc 4.2) 64 bit 2.6.3, numpy compiled with apple gcc 4.2, f2py using the same att gfortran 4.2: In [3]: timeit x = np.random.random_integers(0,1023,100000000).astype(int) 1 loops, best of 3: 3.91 s per loop # 37% slower than 32bit In [4]: x = np.random.random_integers(0,1023,100000000).astype(int) In [5]: timeit np.bincount(x) 1 loops, best of 3: 582 ms per loop # 34 % slower than 32 bit In [8]: timeit gf42_64.bincount(x,1024) 1 loops, best of 3: 803 ms per loop # 522% slower than 32 bit So why is there this big difference in performance? I'd really like to know why the fortran compiled with the same compiler is so much slower in 64 bit mode. As far as I can tell the flags used are the same. Also why is numpy slower. I was surprised the I was able to import the 64 bit universal module built with f2py from 2.6 inside 32 bit 3.5 and there it was quick again - so it seems the x64_64 code generated by the fortran compiler is much slower (rather than any wrappers or such). I tried using some more recent gfortrans from macports - but could only use them to build modules against the 64 bit python/numpy since I couldn't find a way to get f2py to force 32 bit output. But the performance was more or less the same (always several times slower the 32 bit att gfortran). Any advice appreciated. Cheers Robin -------- subroutine bincount (x,c,n,m) implicit none integer, intent(in) :: n,m integer, dimension(0:n-1), intent(in) :: x integer, dimension(0:m-1), intent(out) :: c integer :: i c = 0 do i = 0, n-1 c(x(i)) = c(x(i)) + 1 end do end

2 3

(apparent) infinite loop in LAPACK/ATLAS
by David Warde-Farley 04 Nov '09

04 Nov '09

Hi all (mostly David C. since he probably knows all this horrible stuff), I noticed on my new laptop (with an Atom N280 in it) that when I run numpy.test() about the 34th test would loop, seemingly forever. Finding this a little odd, I tried an SVD on a small matrix and observed the same behaviour, narrowed it down with gdb to what seems to be an infinite loop involving dlamc1_ and dlamc3_, which I gather are routines for converting from row-major to column-major. I should mention that I built ATLAS (3.9.7, because that's the tarball I had around) and LAPACK (3.1.1) from source. Oh, and this is on Ubuntu 9.10, Karmic Koala, using gcc 4.4.1 (with the exception of the ATLAS kernels which are using gcc 4.2.4 for the reasons stated in the install document). I came upon this mention of a similar problem ( https://projects.coin-or.org/BuildTools/ticket/66 ), so far. I'm going to try using the flag they suggest but it'll be a while before I have a verdict on whether it's fixed or not, so I'm soliciting any other ideas as to the cause of/solution to such an issue. Thanks in advance, David

2 2

fft trouble
by Gabriel 04 Nov '09

04 Nov '09

Hi it`s my first time. I have a trouble whit fft. For example: s = sin(wt) f = abs(fft(s)) In f I have the module the fft(s) which should be equal to 1 for w but it's 1000 for w. All values is multiplied by 1000. Someone know why? Thank's -- Gabriel Antes de imprimir, pensá en tu responsabilidad y compromiso con el MEDIO AMBIENTE.

2 2

f2py error with -DF2PY_REPORT_ATEXIT
by Robin 03 Nov '09

03 Nov '09

Hi, When I try to build a fortran module with f2py from '1.4.0.dev7618' with gfortran 4.2.3 from att.com and apple gcc 4.2 on snow leopard I get the following error when I try to use -DF2PY_REPORT_ATEXIT: In [1]: import gfint --------------------------------------------------------------------------- ImportError Traceback (most recent call last) /Users/robince/fortran/<ipython console> in <module>() ImportError: dlopen(./gfint.so, 2): Symbol not found: _on_exit Referenced from: /Users/robince/fortran/gfint.so Expected in: dynamic lookup Is numpy trac the place for f2py tickets? Or is there an obvious fix. Inspecting gfint.so shows the same symbols for both architectures, and _on_exit is listed there with a U which I guess means undefined. Cheers Robin

1 1

Ignorance question
by David Goldsmith 03 Nov '09

03 Nov '09

I Googled scipy brownian and the top hit was the doc for numpy.random.wald, but said doc has a "tone" that suggests there are more "sophisticated" ways to generate a random Brownian signal? Or is wald indeed SotA? Thanks! DG

4 5