From irving at naml.us  Sun Feb  1 02:20:25 2009
From: irving at naml.us (Geoffrey Irving)
Date: Sat, 31 Jan 2009 23:20:25 -0800
Subject: [Numpy-discussion] minor improvment to ones
In-Reply-To: <gluunc$6t5$1@ger.gmane.org>
References: <gluunc$6t5$1@ger.gmane.org>
Message-ID: <7f9d599f0901312320k39d6be4bx4038c259ec75ce29@mail.gmail.com>

On Fri, Jan 30, 2009 at 5:18 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
> A nit, but it would be nice if 'ones' could fill with a value other than 1.
>
> Maybe an optional val= keyword?

You can use the "tile" function for this.  "tile(3,3)" creates an
array of 3 3's.

Geoffrey


From raik.gruenberg at crg.es  Sun Feb  1 06:02:04 2009
From: raik.gruenberg at crg.es (Raik Gruenberg)
Date: Sun, 01 Feb 2009 12:02:04 +0100
Subject: [Numpy-discussion] puzzle: generate index with many ranges
In-Reply-To: <5A3AA485-6FDB-4F02-96C7-D77DCFBAEFF6@stsci.edu>
References: <mailman.5.1233338404.13748.numpy-discussion@scipy.org>
	<5A3AA485-6FDB-4F02-96C7-D77DCFBAEFF6@stsci.edu>
Message-ID: <4985812C.4050008@crg.es>

Beautiful! That should do the trick. Now let's see how this performs against the 
list comprehension...

Thanks a lot!
Raik

Rick White wrote:
> Here's a technique that works:
> 
> Python 2.4.2 (#5, Nov 21 2005, 23:08:11)
> [GCC 4.0.0 20041026 (Apple Computer, Inc. build 4061)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import numpy as np
>  >>> a = np.array([0,4,0,11])
>  >>> b = np.array([-1,11,4,15])
>  >>> rangelen = b-a+1
>  >>> cumlen = rangelen.cumsum()
>  >>> c = np.arange(cumlen[-1],dtype=np.int32)
>  >>> c += np.repeat(a[1:]-c[cumlen[0:-1]], rangelen[1:])
>  >>> print c
> [ 4  5  6  7  8  9 10 11  0  1  2  3  4 11 12 13 14 15]
> 
> The basic idea is that the difference of your desired output from a  
> simple range is an array with a bunch of constant values appended  
> together, and that is what repeat() does.  I'm assuming that you'll  
> never have b < a.  Notice the slight ugliness of prepending the  
> elements at the beginning so that the cumsum starts with zero.   
> (Maybe there is a cleaner way to do that.)
> 
> This does create a second array (via the repeat) that is the same  
> length as the result.  If that uses too much memory, you could break  
> up the repeat and update of c into segments using a loop.  (You  
> wouldn't need a loop for every a,b element -- do a bunch in each  
> iteration.)
> 
> -- Rick
> 
> Raik Gruenberg wrote:
> 
>> Hi there,
>>
>> perhaps someone has a bright idea for this one:
>>
>> I want to concatenate ranges of numbers into a single array (for  
>> indexing). So I
>> have generated an array "a" with starting positions, for example:
>>
>> a = [4, 0, 11]
>>
>> I have an array b with stop positions:
>>
>> b = [11, 4, 15]
>>
>> and I would like to generate an index array that takes 4..11, then  
>> 0..4, then
>> 11..15.
>>
>> In reality, a and b have 10000+ elements and the arrays to be  
>> "sliced" are very
>> large so I want to avoid any for loops etc. Any idea how this could  
>> be done? I
>> thought some combination of *repeat* and adding of *arange* should  
>> do the trick
>> but just cannot nail it down.
>>
>> Thanks in advance for any hints!
>>
>> Greetings,
>> Raik
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 
> 

-- 
________________________________

Dr. Raik Gruenberg
http://www.raiks.de/contact.html
________________________________


From sebastian.walter at gmail.com  Sun Feb  1 06:40:56 2009
From: sebastian.walter at gmail.com (Sebastian Walter)
Date: Sun, 1 Feb 2009 12:40:56 +0100
Subject: [Numpy-discussion] using numpy functions on an array of objects
In-Reply-To: <3d375d730901311524l70a05603vd06bea11d30e5675@mail.gmail.com>
References: <ec9f80fa0901301003g5e20e60fo1ed03bb2252edbcb@mail.gmail.com>
	<49835272.1060704@noaa.gov>
	<3d375d730901301305r6eb3cf11p5f4e8a0641bbf65c@mail.gmail.com>
	<ec9f80fa0901310830w535cb7a7qdb3d29926ca2e4fe@mail.gmail.com>
	<3d375d730901311524l70a05603vd06bea11d30e5675@mail.gmail.com>
Message-ID: <ec9f80fa0902010340r76dc33f8l82088d717de2d1c@mail.gmail.com>

On Sun, Feb 1, 2009 at 12:24 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Sat, Jan 31, 2009 at 10:30, Sebastian Walter
> <sebastian.walter at gmail.com> wrote:
>> Wouldn't it be nice to have numpy a little more generic?
>> All that would be needed was a little check of the arguments.
>>
>> If I do:
>> numpy.trace(4)
>> shouldn't numpy be smart enough to regard the 4 as a 1x1 array?
>
> Why? It's not a 1x1 array. It's a scalar. If you want a 1x1 array,
> give it a 1x1 array.
>
>> numpy.sin(4) works!
>
> Yes, numpy.sin() operates on scalars in addition to arrays.
>
>> and if
>> x = my_class(4)
>>
>> wouldn't it be nice if
>>
>> numpy.trace(x)
>> would call
>> x.trace() ?
>>
>> numpy.sin(my_class(4)) works!
>>
>> Wouldn't it be nice if numpy worked a little more consistent.
>> Is this worth a ticket? Or am I missing something here?
>
> numpy.sin() is a ufunc. Unary ufuncs will call the method of the same
> name on objects in an object array (or the scalar itself if given an
> object scalar). For example:
>
> In [8]: class MyClass(object):
>   ...:     def __init__(self, x):
>   ...:         self.x = x
>   ...:     def __repr__(self):
>   ...:         return 'MyClass(%r)' % (self.x,)
>   ...:     def sin(self):
>   ...:         return MyClass(self.x+1)
>   ...:
>   ...:
>
> In [9]: sin(MyClass(4))
> Out[9]: MyClass(5)
>
> In [10]: sin([MyClass(4), MyClass(5)])
> Out[10]: array([MyClass(5), MyClass(6)], dtype=object)
>
>
> You'll notice that numpy.sin() does not try to call the list.sin()
> method when given the list. It interprets it as an object array, and
> calls the MyClass.sin() method on each of the elements.
>
> numpy.trace() is not an unary ufunc. It's just a function that
> operates on (N>=2)-D arrays. You simply couldn't apply the same rules
> as numpy.sin(). Otherwise, it would try to call the .trace() method on
> each of the objects in your container, and obviously you can't
> implement trace that way.
>
> Having numpy.trace(x) simply call x.trace() would not be making numpy
> more consistent.
>
> Now, that said, the implementation of numpy.trace(x, *args) is
> actually simply asarray(x).trace(*args). That should probably be
> asanyarray(x) in order to allow ndarray subclasses. But this only
> works because ndarray.trace() already exists. Making every function in
> numpy check for a method first is just not going to happen.

Ok I see. I understand your reasoning.
Nonetheless, I didn't suggest that trace() and sin() are the same,
because they are not. I just wanted to express that they should act
the same
if the object is of *unknown type*.

I mean, numpy.sin(MyClass(3)) works.
In the worst of all possible worlds, numpy would raise an exception
because MyClass is *not an array or a scalar*.
But it doesn't. And that is really cool! It's awesome that numpy works
on arbitrary type for the most part.

In contrast, if trace(X) encounters an unknown type it simply raises
an exception. It could as well try *in the very end*
to call X.trace().
I.e. *not* "numpy check for a method first" but *numpy check for a
method as last resort*.

That wouldn't do any harm, would it? And it is not a major effort to
add those simple checks to dot(), trace(), inv() .
I could provide a patch.

But if that is deemed "not going to happen" for whatever reasons. Is
there a good workaround?
I.e. if I do
import numpy
import mypackage

can I overwrite the functions of numpy? I mean, that is quite a hack,
but is the next *best* option.

The reason for that need is, that I am writing a Python module to
compute higher order derivatives of functions that are given as
an algorithm on matrix operations:

e.g. we want to compute the Hessian of the function

def f(X,Y,Z):
    """ X is (N,M) array, Y is (M,K) array, Z is  (K, L) array"""
     V = numpy.dot(X,Y)
     W = numpy.dot(Y,Z)
     return numpy.trace(V*W)

To do that, I generalized the real numbers to the field of truncated
Taylor polynomials of scalars and
real matrices to truncated Taylor polynomials of matrices.
The theory is explained on
http://en.wikipedia.org/wiki/Automatic_differentiation

You can have a look at the unit test, e.g. at def test_2x2dot2x2_reverse() on
http://github.com/b45ch1/algopy/blob/bd7154e2e7a7e6e1931addc0d9ec0604d488d73f/unit_tests/matrix_reverse.py
Mtc is a class of Matrix Taylor Coefficients. The class Function is
used to build the computational graph of function nodes.

I think it would be a nice addition to everyone who is doing
scientific programming with numpy because derivatives are often
required
and divided differences suck for anything but first order on small
algorithms and especially if one wants to differentiate solutions of
ODEs or PDEs.
If trace, dot, inv, etc. don't work that way, ppl would have to define
two versions of the function f to make it work.


best regards,
Sebastian Walter






>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From nmb at wartburg.edu  Sun Feb  1 16:30:52 2009
From: nmb at wartburg.edu (Neil Martinsen-Burrell)
Date: Sun, 1 Feb 2009 21:30:52 +0000 (UTC)
Subject: [Numpy-discussion] example reading binary Fortran file
References: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com
Message-ID: <>

David Froger <david.froger.info <at> gmail.com> writes:

> Hy,My question is about reading Fortran binary file (oh no this question
> again...)

I've posted this before, but I finally got it cleaned up for the Cookbook.  
For this purpose I use a subclass of file that has methods for reading
unformatted Fortran data.  See
http://www.scipy.org/Cookbook/FortranIO/FortranFile.  I'd gladly see this in
numpy or scipy somewhere, but I'm not sure where it belongs.

> program makeArray

> implicit none
> integer,parameter:: nx=10,ny=20
> real(4),dimension(nx,ny):: ux,uy,p
> integer :: i,j
> open(11,file='uxuyp.bin',form='unformatted')
> do i = 1,nx
>   do j = 1,ny?? 
>     ux(i,j) = real(i*j)?? 
>     uy(i,j) = real(i)/real(j)?? 
>     p (i,j)? = real(i) + real(j)
>   enddo
> enddo
> write(11) ux,uy
> write(11) p
> close(11)
> end program makeArray

When I run the above program compiled with gfortran on my Intel Mac, 
I can read it back with::

>>> import numpy as np
>>> from fortranfile import FortranFile
>>> f=FortranFile('uxuyp.bin', endian='<')
>>> uxuy = f.readReals(prec='f') # 'f' for default reals
>>> len(uxuy)
400
>>> ux = np.array(uxuy[:200]).reshape((20,10)).T
>>> uy = np.array(uxuy[200:]).reshape((20,10)).T
>>> p = f.readReals('f').reshape((20,10)).T
>>> ux
array([[   1.,    2.,    3.,    4.,    5.,    6.,    7.,    8.,    9.,
          10.,   11.,   12.,   13.,   14.,   15.,   16.,   17.,   18.,
          19.,   20.],
       [   2.,    4.,    6.,    8.,   10.,   12.,   14.,   16.,   18.,
          20.,   22.,   24.,   26.,   28.,   30.,   32.,   34.,   36.,
          38.,   40.],
       [   3.,    6.,    9.,   12.,   15.,   18.,   21.,   24.,   27.,
          30.,   33.,   36.,   39.,   42.,   45.,   48.,   51.,   54.,
          57.,   60.],
       [   4.,    8.,   12.,   16.,   20.,   24.,   28.,   32.,   36.,
          40.,   44.,   48.,   52.,   56.,   60.,   64.,   68.,   72.,
          76.,   80.],
       [   5.,   10.,   15.,   20.,   25.,   30.,   35.,   40.,   45.,
          50.,   55.,   60.,   65.,   70.,   75.,   80.,   85.,   90.,
          95.,  100.],
       [   6.,   12.,   18.,   24.,   30.,   36.,   42.,   48.,   54.,
          60.,   66.,   72.,   78.,   84.,   90.,   96.,  102.,  108.,
         114.,  120.],
       [   7.,Proxy-Connection: keep-alive
Cache-Control: max-age=0

 14.,   21.,   28.,   35.,   42.,   49.,   56.,   63.,
          70.,   77.,   84.,   91.,   98.,  105.,  112.,  119.,  126.,
         133.,  140.],
       [   8.,   16.,   24.,   32.,   40.,   48.,   56.,   64.,   72.,
          80.,   88.,   96.,  104.,  112.,  120.,  128.,  136.,  144.,
         152.,  160.],
       [   9.,   18.,   27.,   36.,   45.,   54.,   63.,   72.,   81.,
          90.,   99.,  108.,  117.,  126.,  135.,  144.,  153.,  162.,
         171.,  180.],
       [  10.,   20.,   30.,   40.,   50.,   60.,   70.,   80.,   90.,
         100.,  110.,  120.,  130.,  140.,  150.,  160.,  170.,  180.,
         190.,  200.]])
>>> uy
array([[  1.        ,   0.5       ,   0.33333334,   0.25      ,
          0.2       ,   0.16666667,   0.14285715,   0.125     ,
          0.11111111,   0.1       ,   0.09090909,   0.08333334,
          0.07692308,   0.07142857,   0.06666667,   0.0625    ,
          0.05882353,   0.05555556,   0.05263158,   0.05      ],
       [  2.        ,   1.        ,   0.66666669,   0.5       ,
          0.40000001,   0.33333334,   0.2857143 ,   0.25      ,
          0.22222222,   0.2       ,   0.18181819,   0.16666667,
          0.15384616,   0.14285715,   0.13333334,   0.125     ,
          0.11764706,   0.11111111,   0.10526316,   0.1       ],
       [  3.        ,   1.5       ,   1.        ,   0.75      ,
          0.60000002,   0.5       ,   0.42857143,   0.375     ,
          0.33333334,   0.30000001,   0.27272728,   0.25      ,
          0.23076923,   0.21428572,   0.2       ,   0.1875    ,
          0.17647059,   0.16666667,   0.15789473,   0.15000001],
       [  4.        ,   2.        ,   1.33333337,   1.        ,
          0.80000001,   0.66666669,   0.5714286 ,   0.5       ,
          0.44444445,   0.40000001,   0.36363637,   0.33333334,
          0.30769232,   0.2857143 ,   0.26666668,   0.25      ,
          0.23529412,   0.22222222,   0.21052632,   0.2       ],
       [  5.        ,   2.5       ,   1.66666663,   1.25      ,
          1.        ,   0.83333331,   0.71428573,   0.625     ,
          0.55555558,   0.5       ,   0.45454547,   0.41666666,
          0.38461539,   0.35714287,   0.33333334,   0.3125    ,
          0.29411766,   0.27777779,   0.2631579 ,   0.25      ],
       [  6.        ,   3.        ,   2.        ,   1.5       ,
          1.20000005,   1.        ,   0.85714287,   0.75      ,
          0.66666669,   0.60000002,   0.54545456,   0.5       ,
          0.46153846,   0.42857143,   0.40000001,   0.375     ,
          0.35294119,   0.33333334,   0.31578946,   0.30000001],
       [  7.        ,   3.5       ,   2.33333325,   1.75      ,
          1.39999998,   1.16666663,   1.        ,   0.875     ,
          0.77777779,   0.69999999,   0.63636363,   0.58333331,
          0.53846157,   0.5       ,   0.46666667,   0.4375    ,
          0.41176471,   0.3888889 ,   0.36842105,   0.34999999],
       [  8.        ,   4.        ,   2.66666675,   2.        ,
          1.60000002,   1.33333337,   1.14285719,   1.        ,
          0.8888889 ,   0.80000001,   0.72727275,   0.66666669,
          0.61538464,   0.5714286 ,   0.53333336,   0.5       ,
          0.47058824,   0.44444445,   0.42105263,   0.40000001],
       [  9.        ,   4.5       ,   3.        ,   2.25      ,
          1.79999995,   1.5       ,   1.28571427,   1.125     ,
          1.        ,   0.89999998,   0.81818181,   0.75      ,
          0.69230771,   0.64285713,   0.60000002,   0.5625    ,
          0.52941179,   0.5       ,   0.47368422,   0.44999999],
       [ 10.        ,   5.        ,   3.33333325,   2.5       ,
          2.        ,   1.66666663,   1.42857146,   1.25      ,
          1.11111116,   1.        ,   0.90909094,   0.83333331,
          0.76923078,   0.71428573,   0.66666669,   0.625     ,
          0.58823532,   0.55555558,   0.52631581,   0.5       ]])
>>> p
array([[  2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,  11.,  12.,
         13.,  14.,  15.,  16.,  17.,  18.,  19.,  20.,  21.],
       [  3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,  11.,  12.,  13.,
         14.,  15.,  16.,  17.,  18.,  19.,  20.,  21.,  22.],
       [  4.,   5.,   6.,   7.,   8.,   9.,  10.,  11.,  12.,  13.,  14.,
         15.,  16.,  17.,  18.,  19.,  20.,  21.,  22.,  23.],
       [  5.,   6.,   7.,   8.,   9.,  10.,  11.,  12.,  13.,  14.,  15.,
         16.,  17.,  18.,  19.,  20.,  21.,  22.,  23.,  24.],
       [  6.,   7.,   8.,   9.,  10.,  11.,  12.,  13.,  14.,  15.,  16.,
         17.,  18.,  19.,  20.,  21.,  22.,  23.,  24.,  25.],
       [  7.,   8.,   9.,  10.,  11.,  12.,  13.,  14.,  15.,  16.,  17.,
         18.,  19.,  20.,  21.,  22.,  23.,  24.,  25.,  26.],
       [  8.,   9.,  10.,  11.,  12.,  13.,  14.,  15.,  16.,  17.,  18.,
         19.,  20.,  21.,  22.,  23.,  24.,  25.,  26.,  27.],
       [  9.,  10.,  11.,  12.,  13.,  14.,  15.,  16.,  17.,  18.,  19.,
         20.,  21.,  22.,  23.,  24.,  25.,  26.,  27.,  28.],
       [ 10.,  11.,  12.,  13.,  14.,  15.,  16.,  17.,  18.,  19.,  20.,
         21.,  22.,  23.,  24.,  25.,  26.,  27.,  28.,  29.],
       [ 11.,  12.,  13.,  14.,  15.,  16.,  17.,  18.,  19.,  20.,  21.,
         22.,  23.,  24.,  25.,  26.,  27.,  28.,  29.,  30.]])

Note that you have to provide the shape information for ux and uy because
fortran writes them together as a stream of 400 numbers.

-Neil




From dsdale24 at gmail.com  Sun Feb  1 18:32:41 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 1 Feb 2009 18:32:41 -0500
Subject: [Numpy-discussion] question about ufuncs
Message-ID: <a08e5f80902011532q3a3f4badqa0b722b8e481ca@mail.gmail.com>

I've been playing with __array_wrap__ to make quantities with units play
well with numpy's ufuncs. For example, __array_wrap__ makes it is possible
to do the following:

>>> numpy.sqrt([1.,4.,9.]*m**2)
array([1.,2.,3.])*m

Is there an analog to __array_wrap__ for preprocessing arrays on their way
*into* a ufunc? For example, it would be nice if one could do something
like:

numpy.sin([1,2,3]*arcseconds)

where we have the opportunity to inspect the context, convert the Quantity
to units of radians, and then actually call the ufunc. Is this possible, or
does one have to reimplement such functions?

Thanks,
Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090201/d8911fd4/attachment.html>

From pgmdevlist at gmail.com  Sun Feb  1 19:33:45 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 1 Feb 2009 19:33:45 -0500
Subject: [Numpy-discussion] question about ufuncs
In-Reply-To: <a08e5f80902011532q3a3f4badqa0b722b8e481ca@mail.gmail.com>
References: <a08e5f80902011532q3a3f4badqa0b722b8e481ca@mail.gmail.com>
Message-ID: <48A25BC8-FA0A-44EE-89E2-F6C9939E8A22@gmail.com>


On Feb 1, 2009, at 6:32 PM, Darren Dale wrote:
>
>
> Is there an analog to __array_wrap__ for preprocessing arrays on  
> their way *into* a ufunc? For example, it would be nice if one could  
> do something like:
>
> numpy.sin([1,2,3]*arcseconds)
>
> where we have the opportunity to inspect the context, convert the  
> Quantity to units of radians, and then actually call the ufunc. Is  
> this possible, or does one have to reimplement such functions?

Just an idea: look at the code for numpy.ma ufuncs (in numpy.ma.core).  
By defining a few classes for unary, binary and domained functions,  
you could probably do what you want, without having to recode all the  
functions by hand.
Another idea would be to define some specific __mul__ or __rmul__  
rules for your units, so that the list would be transformed into a  
UnitArray...


From dsdale24 at gmail.com  Sun Feb  1 19:39:04 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 1 Feb 2009 19:39:04 -0500
Subject: [Numpy-discussion] question about ufuncs
In-Reply-To: <48A25BC8-FA0A-44EE-89E2-F6C9939E8A22@gmail.com>
References: <a08e5f80902011532q3a3f4badqa0b722b8e481ca@mail.gmail.com>
	<48A25BC8-FA0A-44EE-89E2-F6C9939E8A22@gmail.com>
Message-ID: <a08e5f80902011639g6cbf656bj1817947841d3f5a8@mail.gmail.com>

On Sun, Feb 1, 2009 at 7:33 PM, Pierre GM <pgmdevlist at gmail.com> wrote:

>
> On Feb 1, 2009, at 6:32 PM, Darren Dale wrote:
> >
> >
> > Is there an analog to __array_wrap__ for preprocessing arrays on
> > their way *into* a ufunc? For example, it would be nice if one could
> > do something like:
> >
> > numpy.sin([1,2,3]*arcseconds)
> >
> > where we have the opportunity to inspect the context, convert the
> > Quantity to units of radians, and then actually call the ufunc. Is
> > this possible, or does one have to reimplement such functions?
>
> Just an idea: look at the code for numpy.ma ufuncs (in numpy.ma.core).
> By defining a few classes for unary, binary and domained functions,
> you could probably do what you want, without having to recode all the
> functions by hand.
> Another idea would be to define some specific __mul__ or __rmul__
> rules for your units, so that the list would be transformed into a
> UnitArray...


I have pretty good implementations of the arithmetic operators, so
([1,2,3]*m)*([4,5,6]*J) already works. numpy.multiply and numpy.sqrt needed
help with array_wrap. I'll study your stuff in ma, thanks for the pointer.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090201/87b3dfb9/attachment.html>

From mattdm at mattdm.org  Sun Feb  1 21:58:18 2009
From: mattdm at mattdm.org (Matthew Miller)
Date: Sun, 1 Feb 2009 21:58:18 -0500
Subject: [Numpy-discussion] numpy.load and gzip file handles
Message-ID: <20090202025818.GA21605@jadzia.bu.edu>

Hi everyone.

I'd like to log the state of my program as it progresses. Using the
numpy.save / numpy.load functions on the same filehandle repeatedly works
very well for this -- but ends up making a file which very quickly grows to
gigabytes. The data compresses well, though, so I thought I'd use Python's
built-in gzip module underneath. This works great for saving -- but when it
comes time to play back, there's an issue:

  >>> import numpy
  >>> import gzip
  >>> f=open("test.gz")
  >>> g=gzip.GzipFile(None,"rb",9,f)
  >>> g
  <gzip open file 'test.gz', mode 'r' at 0xbaad50 0xc0ab90>
  >>> numpy.load(g)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib64/python2.5/site-packages/numpy/lib/io.py", line 133, in load
      fid.seek(-N,1) # back-up
  TypeError: seek() takes exactly 2 arguments (3 given)

Turns out you can't rewind gzip file handles in Python. Oops. The offending
code is that which distinguishes between npy and npz files. Could there
maybe be something added to just trust me that it's an npy?

Or better yet, is there something I'm doing wrong / overlooking?

Thanks!



-- 
Matthew Miller           mattdm at mattdm.org          <http://mattdm.org/>


From stefan at sun.ac.za  Mon Feb  2 01:01:54 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 2 Feb 2009 08:01:54 +0200
Subject: [Numpy-discussion] numpy.load and gzip file handles
In-Reply-To: <20090202025818.GA21605@jadzia.bu.edu>
References: <20090202025818.GA21605@jadzia.bu.edu>
Message-ID: <9457e7c80902012201q4187f42fv89a69f13834d5cbe@mail.gmail.com>

2009/2/2 Matthew Miller <mattdm at mattdm.org>:
> I'd like to log the state of my program as it progresses. Using the
> numpy.save / numpy.load functions on the same filehandle repeatedly works
> very well for this -- but ends up making a file which very quickly grows to
> gigabytes. The data compresses well, though, so I thought I'd use Python's
> built-in gzip module underneath. This works great for saving -- but when it
> comes time to play back, there's an issue:
>
>  >>> import numpy
>  >>> import gzip
>  >>> f=open("test.gz")
>  >>> g=gzip.GzipFile(None,"rb",9,f)
>  >>> g
>  <gzip open file 'test.gz', mode 'r' at 0xbaad50 0xc0ab90>
>  >>> numpy.load(g)
>  Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "/usr/lib64/python2.5/site-packages/numpy/lib/io.py", line 133, in load
>      fid.seek(-N,1) # back-up
>  TypeError: seek() takes exactly 2 arguments (3 given)

The GzipFile in Python 2.5 does not support the 2nd ("whence")
argument.  The solution may be to use this wrapper from the EffBot:

http://effbot.org/librarybook/gzip-example-2.py

In order to "back-port" that functionality.

Regards
St?fan


From mattdm at mattdm.org  Mon Feb  2 01:10:10 2009
From: mattdm at mattdm.org (Matthew Miller)
Date: Mon, 2 Feb 2009 01:10:10 -0500
Subject: [Numpy-discussion] numpy.load and gzip file handles
In-Reply-To: <9457e7c80902012201q4187f42fv89a69f13834d5cbe@mail.gmail.com>
References: <20090202025818.GA21605@jadzia.bu.edu>
	<9457e7c80902012201q4187f42fv89a69f13834d5cbe@mail.gmail.com>
Message-ID: <20090202061010.GA8806@jadzia.bu.edu>

On Mon, Feb 02, 2009 at 08:01:54AM +0200, St?fan van der Walt wrote:
> The GzipFile in Python 2.5 does not support the 2nd ("whence")
> argument.  The solution may be to use this wrapper from the EffBot:
> http://effbot.org/librarybook/gzip-example-2.py
> In order to "back-port" that functionality.

Unless I'm misunderstanding, even with the wrapper one can't actually seek
backwards, which is what the numpy code wants to do.

In the meantime, I'm just using numpy.lib.format.read_array() directly.


-- 
Matthew Miller           mattdm at mattdm.org          <http://mattdm.org/>


From nwagner at iam.uni-stuttgart.de  Mon Feb  2 07:33:43 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 02 Feb 2009 13:33:43 +0100
Subject: [Numpy-discussion] Fortran binary files and numpy/scipy
Message-ID: <web-116566714@uni-stuttgart.de>

Hi all,

How can I import FORTRAN binary files using numpy ?

In FORTRAN I can do

       OPEN(10,FILE='test.mat',FORM='unformatted')

  100  CONTINUE
         READ(10,END=999) IROW, ICOL, VALUE
       GOTO 100

  999  CONTINUE
       END

And In Python/numpy ?

....


Any pointer would be appreciated.

Thanks in advance
                      Nils
 


From matthieu.brucher at gmail.com  Mon Feb  2 07:39:32 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 2 Feb 2009 13:39:32 +0100
Subject: [Numpy-discussion] Fortran binary files and numpy/scipy
In-Reply-To: <web-116566714@uni-stuttgart.de>
References: <web-116566714@uni-stuttgart.de>
Message-ID: <e76aa17f0902020439y50b29c2g8b3ea469bd54e9db@mail.gmail.com>

Hi,

There was a discussion about this last week. You can find it int he archives ;)

Matthieu

2009/2/2 Nils Wagner <nwagner at iam.uni-stuttgart.de>:
> Hi all,
>
> How can I import FORTRAN binary files using numpy ?
>
> In FORTRAN I can do
>
>       OPEN(10,FILE='test.mat',FORM='unformatted')
>
>  100  CONTINUE
>         READ(10,END=999) IROW, ICOL, VALUE
>       GOTO 100
>
>  999  CONTINUE
>       END
>
> And In Python/numpy ?
>
> ....
>
>
> Any pointer would be appreciated.
>
> Thanks in advance
>                      Nils
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From nwagner at iam.uni-stuttgart.de  Mon Feb  2 09:22:42 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 02 Feb 2009 15:22:42 +0100
Subject: [Numpy-discussion] Fortran binary files and numpy/scipy
In-Reply-To: <e76aa17f0902020439y50b29c2g8b3ea469bd54e9db@mail.gmail.com>
References: <web-116566714@uni-stuttgart.de>
	<e76aa17f0902020439y50b29c2g8b3ea469bd54e9db@mail.gmail.com>
Message-ID: <web-116573065@uni-stuttgart.de>

On Mon, 2 Feb 2009 13:39:32 +0100
  Matthieu Brucher <matthieu.brucher at gmail.com> wrote:
> Hi,
> 
> There was a discussion about this last week. You can 
>find it int he archives ;)
> 
> Matthieu
  
Hi Matthieu,

Sorry but I missed that.
Anyway I have some trouble with my short example.

g77 -c binary_fortran.f
g77 -o io binary_fortran.o
./io

  11 254  254.
  12 253  126.
  13 252  84.
  14 251  62.
  15 250  50.
  16 249  41.
  17 248  35.
  18 247  30.
  19 246  27.
  20 245  24.

python -i read_fortran.py

>>> a
array([(16, 1090921693195, 254.0), (16, 16, 
5.3686493512014268e-312),
        (4638566878703255552, 16, 
7.9050503334599447e-323),
        (1082331758605, 4635611391447793664, 
7.9050503334599447e-323),
        (16, 1078036791310, 62.0), (16, 16, 
5.3049894774872906e-312),
        (4632233691727265792, 16, 
7.9050503334599447e-323),
        (1069446856720, 4630967054332067840, 
7.9050503334599447e-323),
        (16, 1065151889425, 35.0), (16, 16, 
5.2413296037731544e-312),
        (4629137466983448576, 16, 
7.9050503334599447e-323),
        (1056561954835, 4628293042053316608, 
7.9050503334599447e-323),
        (16, 1052266987540, 24.0)],
       dtype=[('irow', '<i8'), ('icol', '<i8'), ('value', 
'<f8')])

How can I fix the problem ?

Nils
-------------- next part --------------
A non-text attachment was scrubbed...
Name: binary_fortran.f
Type: text/x-fortran
Size: 433 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090202/d3744895/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: read_fortran.py
Type: text/x-python
Size: 141 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090202/d3744895/attachment.py>

From Mike.Colonno at spacex.com  Mon Feb  2 10:28:10 2009
From: Mike.Colonno at spacex.com (Mike Colonno)
Date: Mon, 2 Feb 2009 07:28:10 -0800
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <657E769D35612D4CAB1FEAACB0A92BB503094E5F3F@MAIL2.spacex.corp>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<497FC559.4000801@ar.media.kyoto-u.ac.jp>
	<9cca9e840901271931s1706de0egd28382e4125644a@mail.gmail.com>
	<49801FF8.8040805@ar.media.kyoto-u.ac.jp>
	<9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>
	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
	<789d27b10901300432q28226381x829575e8b1fa23fe@mail.gmail.com>
	<9cca9e840901300731y69ee6f49ra758ba4f8ccced69@mail.gmail.com>,
	<9cca9e840901311048q21afab1cv7351403f8c95f3af@mail.gmail.com>,
	<657E769D35612D4CAB1FEAACB0A92BB503094E5F3C@MAIL2.spacex.corp>
	<657E769D35612D4CAB1FEAACB0A92BB503094E5F3F@MAIL2.spacex.corp>
Message-ID: <657E769D35612D4CAB1FEAACB0A92BB5030984055E@MAIL2.spacex.corp>

            Hi folks ~

            Any thoughts on the below? I n searching the web I found some other references to " ImportError: DLL load failed: Invalid access to memory location" but none specific to Numpy. As an aside: will there be a Windows x64 binary distributed with the next release of Numpy / Scipy? Does anyone have a working installer now? It may be easier to just wait for an official release and use an existing binary in the meantime vs. banging my head against this build.

>>>from numpy import *
Traceback (most recent call last):
  File "C:\<string>", line 1, in <module>
  File "C:\Python26\Lib\site-packages\numpy\__init__.py", line 130, in <module>
    import add_newdocs
  File "C:\Python26\Lib\site-packages\numpy\add_newdocs.py", line 9, in <module>
    from lib import add_newdoc
  File "C:\Python26\Lib\site-packages\numpy\lib\__init__.py", line 161, in <module>
    from polynomial import *
  File "C:\Python26\Lib\site-packages\numpy\lib\polynomial.py", line 18, in <module>
    from numpy.linalg import eigvals, lstsq
  File "C:\Python26\Lib\site-packages\numpy\linalg\__init__.py", line 47, in <module>
    from linalg import *
  File "C:\Python26\Lib\site-packages\numpy\linalg\linalg.py", line 22, in <module>
    from numpy.linalg import lapack_lite
ImportError: DLL load failed: Invalid access to memory location.

   Thanks for the help,

   ~Mike C.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090202/a5d7ff52/attachment.html>

From cournape at gmail.com  Mon Feb  2 10:55:36 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 3 Feb 2009 00:55:36 +0900
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <657E769D35612D4CAB1FEAACB0A92BB5030984055E@MAIL2.spacex.corp>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<9cca9e840901280818l2106e710q16599bf1423c3caf@mail.gmail.com>
	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
	<789d27b10901300432q28226381x829575e8b1fa23fe@mail.gmail.com>
	<9cca9e840901300731y69ee6f49ra758ba4f8ccced69@mail.gmail.com>
	<9cca9e840901311048q21afab1cv7351403f8c95f3af@mail.gmail.com>
	<657E769D35612D4CAB1FEAACB0A92BB503094E5F3C@MAIL2.spacex.corp>
	<657E769D35612D4CAB1FEAACB0A92BB503094E5F3F@MAIL2.spacex.corp>
	<657E769D35612D4CAB1FEAACB0A92BB5030984055E@MAIL2.spacex.corp>
Message-ID: <5b8d13220902020755i10eda0f3y5b255ec2578bddea@mail.gmail.com>

On Tue, Feb 3, 2009 at 12:28 AM, Mike Colonno <Mike.Colonno at spacex.com> wrote:
>             Hi folks ~
>
>
>
>             Any thoughts on the below?

It is hard to say, but I suspect a problem in your BLAS/LAPACK because
it fails on importing lapack_lite. Your error message is so generic
that it does not give any useful information.

I have worked quite a bit last december on 64 bits support using the
free compilers mingw - it worked OK for the C code, but I have not
managed yet to build a numpy with a full BLAS/LAPACK, which required a
fortran compiler. Since I can't have access to a fortran compiler on
that platform, I can't fix the remaining problems. And nobody stepped
in to fix the problems either,

David


From rmay31 at gmail.com  Mon Feb  2 11:17:13 2009
From: rmay31 at gmail.com (Ryan May)
Date: Mon, 02 Feb 2009 10:17:13 -0600
Subject: [Numpy-discussion] Fortran binary files and numpy/scipy
In-Reply-To: <web-116573065@uni-stuttgart.de>
References: <web-116566714@uni-stuttgart.de>	<e76aa17f0902020439y50b29c2g8b3ea469bd54e9db@mail.gmail.com>
	<web-116573065@uni-stuttgart.de>
Message-ID: <49871C89.4000800@gmail.com>

Nils Wagner wrote:
> On Mon, 2 Feb 2009 13:39:32 +0100
>  Matthieu Brucher <matthieu.brucher at gmail.com> wrote:
>> Hi,
>>
>> There was a discussion about this last week. You can find it int he
>> archives ;)
>>
>> Matthieu
>  
> Hi Matthieu,
> 
> Sorry but I missed that.
> Anyway I have some trouble with my short example.
> 
> g77 -c binary_fortran.f
> g77 -o io binary_fortran.o
> ./io
> 
>  11 254  254.
>  12 253  126.
>  13 252  84.
>  14 251  62.
>  15 250  50.
>  16 249  41.
>  17 248  35.
>  18 247  30.
>  19 246  27.
>  20 245  24.
> 
> python -i read_fortran.py
> 
>>>> a
> array([(16, 1090921693195, 254.0), (16, 16, 5.3686493512014268e-312),
>        (4638566878703255552, 16, 7.9050503334599447e-323),
>        (1082331758605, 4635611391447793664, 7.9050503334599447e-323),
>        (16, 1078036791310, 62.0), (16, 16, 5.3049894774872906e-312),
>        (4632233691727265792, 16, 7.9050503334599447e-323),
>        (1069446856720, 4630967054332067840, 7.9050503334599447e-323),
>        (16, 1065151889425, 35.0), (16, 16, 5.2413296037731544e-312),
>        (4629137466983448576, 16, 7.9050503334599447e-323),
>        (1056561954835, 4628293042053316608, 7.9050503334599447e-323),
>        (16, 1052266987540, 24.0)],
>       dtype=[('irow', '<i8'), ('icol', '<i8'), ('value', '<f8')])
> 
> How can I fix the problem ?
> 

Every write statement in fortran first writes out the number of bytes that will
follow, *then* the actual data.  So, for instance, the first write to file in
your program will write the bytes corresponding to these values:

16 X(1) Y(1) Z(1)

The 16 comes from the size of 2 ints and 1 double.  Since you're always writing
out the 3 values, and they're always the same size, try adding another integer
column as the first field in your array.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From hanni.ali at gmail.com  Mon Feb  2 12:23:51 2009
From: hanni.ali at gmail.com (Hanni Ali)
Date: Mon, 2 Feb 2009 13:23:51 -0400
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <5b8d13220902020755i10eda0f3y5b255ec2578bddea@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<5b8d13220901281636y2fe1b55dt3cb59dd46b43b1d0@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
	<789d27b10901300432q28226381x829575e8b1fa23fe@mail.gmail.com>
	<9cca9e840901300731y69ee6f49ra758ba4f8ccced69@mail.gmail.com>
	<9cca9e840901311048q21afab1cv7351403f8c95f3af@mail.gmail.com>
	<657E769D35612D4CAB1FEAACB0A92BB503094E5F3C@MAIL2.spacex.corp>
	<657E769D35612D4CAB1FEAACB0A92BB503094E5F3F@MAIL2.spacex.corp>
	<657E769D35612D4CAB1FEAACB0A92BB5030984055E@MAIL2.spacex.corp>
	<5b8d13220902020755i10eda0f3y5b255ec2578bddea@mail.gmail.com>
Message-ID: <789d27b10902020923h7099680awafa35905b30fb79a@mail.gmail.com>

Hi David,

I used free trials of the Intel and PGI compilers to try to compile an
external BLAS/LAPACK in conjunction with VS 2008. I also had no problems
getting the C code to compile, but couldn't get linking to work succesfully
with fortran stuff. I would not be surprised if we could get a licence from
Intel or PGI to provide a pre-compiled exe.

Hanni

2009/2/2 David Cournapeau <cournape at gmail.com>

> On Tue, Feb 3, 2009 at 12:28 AM, Mike Colonno <Mike.Colonno at spacex.com>
> wrote:
> >             Hi folks ~
> >
> >
> >
> >             Any thoughts on the below?
>
> It is hard to say, but I suspect a problem in your BLAS/LAPACK because
> it fails on importing lapack_lite. Your error message is so generic
> that it does not give any useful information.
>
> I have worked quite a bit last december on 64 bits support using the
> free compilers mingw - it worked OK for the C code, but I have not
> managed yet to build a numpy with a full BLAS/LAPACK, which required a
> fortran compiler. Since I can't have access to a fortran compiler on
> that platform, I can't fix the remaining problems. And nobody stepped
> in to fix the problems either,
>
> David
>  _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090202/3b0f00ac/attachment.html>

From cournape at gmail.com  Mon Feb  2 12:53:22 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 3 Feb 2009 02:53:22 +0900
Subject: [Numpy-discussion] Building on WinXP 64-bit, Intel Compilers
In-Reply-To: <789d27b10902020923h7099680awafa35905b30fb79a@mail.gmail.com>
References: <9cca9e840901271705s21803633u67be3ed8bad54835@mail.gmail.com>
	<9cca9e840901290757y10350efftfbdb8e4204c16795@mail.gmail.com>
	<789d27b10901300432q28226381x829575e8b1fa23fe@mail.gmail.com>
	<9cca9e840901300731y69ee6f49ra758ba4f8ccced69@mail.gmail.com>
	<9cca9e840901311048q21afab1cv7351403f8c95f3af@mail.gmail.com>
	<657E769D35612D4CAB1FEAACB0A92BB503094E5F3C@MAIL2.spacex.corp>
	<657E769D35612D4CAB1FEAACB0A92BB503094E5F3F@MAIL2.spacex.corp>
	<657E769D35612D4CAB1FEAACB0A92BB5030984055E@MAIL2.spacex.corp>
	<5b8d13220902020755i10eda0f3y5b255ec2578bddea@mail.gmail.com>
	<789d27b10902020923h7099680awafa35905b30fb79a@mail.gmail.com>
Message-ID: <5b8d13220902020953s1bd6e830n9debe5ae79dcfe50@mail.gmail.com>

On Tue, Feb 3, 2009 at 2:23 AM, Hanni Ali <hanni.ali at gmail.com> wrote:
> Hi David,
>
> I used free trials of the Intel and PGI compilers to try to compile an
> external BLAS/LAPACK in conjunction with VS 2008. I also had no problems
> getting the C code to compile, but couldn't get linking to work succesfully
> with fortran stuff. I would not be surprised if we could get a licence from
> Intel or PGI to provide a pre-compiled exe.

I hope to be able to announce some good news on that front very soon :)

David


From nwagner at iam.uni-stuttgart.de  Mon Feb  2 14:18:25 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 02 Feb 2009 20:18:25 +0100
Subject: [Numpy-discussion] Fortran binary files and numpy/scipy
In-Reply-To: <49871C89.4000800@gmail.com>
References: <web-116566714@uni-stuttgart.de>
	<e76aa17f0902020439y50b29c2g8b3ea469bd54e9db@mail.gmail.com>
	<web-116573065@uni-stuttgart.de> <49871C89.4000800@gmail.com>
Message-ID: <web-116584825@uni-stuttgart.de>

On Mon, 02 Feb 2009 10:17:13 -0600
  Ryan May <rmay31 at gmail.com> wrote:
> Nils Wagner wrote:
>> On Mon, 2 Feb 2009 13:39:32 +0100
>>  Matthieu Brucher <matthieu.brucher at gmail.com> wrote:
>>> Hi,
>>>
>>> There was a discussion about this last week. You can 
>>>find it int he
>>> archives ;)
>>>
>>> Matthieu
>>  
>> Hi Matthieu,
>> 
>> Sorry but I missed that.
>> Anyway I have some trouble with my short example.
>> 
>> g77 -c binary_fortran.f
>> g77 -o io binary_fortran.o
>> ./io
>> 
>>  11 254  254.
>>  12 253  126.
>>  13 252  84.
>>  14 251  62.
>>  15 250  50.
>>  16 249  41.
>>  17 248  35.
>>  18 247  30.
>>  19 246  27.
>>  20 245  24.
>> 
>> python -i read_fortran.py
>> 
>>>>> a
>> array([(16, 1090921693195, 254.0), (16, 16, 
>>5.3686493512014268e-312),
>>        (4638566878703255552, 16, 
>>7.9050503334599447e-323),
>>        (1082331758605, 4635611391447793664, 
>>7.9050503334599447e-323),
>>        (16, 1078036791310, 62.0), (16, 16, 
>>5.3049894774872906e-312),
>>        (4632233691727265792, 16, 
>>7.9050503334599447e-323),
>>        (1069446856720, 4630967054332067840, 
>>7.9050503334599447e-323),
>>        (16, 1065151889425, 35.0), (16, 16, 
>>5.2413296037731544e-312),
>>        (4629137466983448576, 16, 
>>7.9050503334599447e-323),
>>        (1056561954835, 4628293042053316608, 
>>7.9050503334599447e-323),
>>        (16, 1052266987540, 24.0)],
>>       dtype=[('irow', '<i8'), ('icol', '<i8'), ('value', 
>>'<f8')])
>> 
>> How can I fix the problem ?
>> 
> 
> Every write statement in fortran first writes out the 
>number of bytes that will
> follow, *then* the actual data.  So, for instance, the 
>first write to file in
> your program will write the bytes corresponding to these 
>values:
> 
> 16 X(1) Y(1) Z(1)
> 
> The 16 comes from the size of 2 ints and 1 double. 
> Since you're always writing
> out the 3 values, and they're always the same size, try 
>adding another integer
> column as the first field in your array.
> 
> Ryan
> 
> -- 
> Ryan May
> Graduate Research Assistant
> School of Meteorology
> University of Oklahoma
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
  
Hi Ryan,

I have modified the python script.

import numpy as np
fname = open("bin.dat",'rb')
dt = 
np.dtype([('isize',int),('irow',int),('icol',int),('value',float)])
a  = np.fromfile(fname,dtype=dt)


>>> a
array([(16, 1090921693195, 4643140847074803712, 
7.9050503334599447e-323),
        (16, 1086626725900, 4638566878703255552, 
7.9050503334599447e-323),
        (16, 1082331758605, 4635611391447793664, 
7.9050503334599447e-323),
        (16, 1078036791310, 4633922541587529728, 
7.9050503334599447e-323),
        (16, 1073741824015, 4632233691727265792, 
7.9050503334599447e-323),
        (16, 1069446856720, 4630967054332067840, 
7.9050503334599447e-323),
        (16, 1065151889425, 4630122629401935872, 
7.9050503334599447e-323),
        (16, 1060856922130, 4629137466983448576, 
7.9050503334599447e-323),
        (16, 1056561954835, 4628293042053316608, 
7.9050503334599447e-323),
        (16, 1052266987540, 4627448617123184640, 
7.9050503334599447e-323)],
       dtype=[('isize', '<i8'), ('irow', '<i8'), ('icol', 
'<i8'), ('value', '<f8')])

Is this a 64-bit problem ?

Nils


From rmay31 at gmail.com  Mon Feb  2 14:38:14 2009
From: rmay31 at gmail.com (Ryan May)
Date: Mon, 02 Feb 2009 13:38:14 -0600
Subject: [Numpy-discussion] Fortran binary files and numpy/scipy
In-Reply-To: <web-116584825@uni-stuttgart.de>
References: <web-116566714@uni-stuttgart.de>	<e76aa17f0902020439y50b29c2g8b3ea469bd54e9db@mail.gmail.com>	<web-116573065@uni-stuttgart.de>
	<49871C89.4000800@gmail.com> <web-116584825@uni-stuttgart.de>
Message-ID: <49874BA6.3050806@gmail.com>

Nils Wagner wrote:
> On Mon, 02 Feb 2009 10:17:13 -0600
>   Ryan May <rmay31 at gmail.com> wrote:
>> Every write statement in fortran first writes out the 
>> number of bytes that will
>> follow, *then* the actual data.  So, for instance, the 
>> first write to file in
>> your program will write the bytes corresponding to these 
>> values:
>>
>> 16 X(1) Y(1) Z(1)
>>
>> The 16 comes from the size of 2 ints and 1 double. 
>> Since you're always writing
>> out the 3 values, and they're always the same size, try 
>> adding another integer
>> column as the first field in your array.
>>
>> Ryan

> Hi Ryan,
> 
> I have modified the python script.
> 
> import numpy as np
> fname = open("bin.dat",'rb')
> dt = 
> np.dtype([('isize',int),('irow',int),('icol',int),('value',float)])
> a  = np.fromfile(fname,dtype=dt)
> 
> 
>>>> a
> array([(16, 1090921693195, 4643140847074803712, 
> 7.9050503334599447e-323),
>         (16, 1086626725900, 4638566878703255552, 
> 7.9050503334599447e-323),
>         (16, 1082331758605, 4635611391447793664, 
> 7.9050503334599447e-323),
>         (16, 1078036791310, 4633922541587529728, 
> 7.9050503334599447e-323),
>         (16, 1073741824015, 4632233691727265792, 
> 7.9050503334599447e-323),
>         (16, 1069446856720, 4630967054332067840, 
> 7.9050503334599447e-323),
>         (16, 1065151889425, 4630122629401935872, 
> 7.9050503334599447e-323),
>         (16, 1060856922130, 4629137466983448576, 
> 7.9050503334599447e-323),
>         (16, 1056561954835, 4628293042053316608, 
> 7.9050503334599447e-323),
>         (16, 1052266987540, 4627448617123184640, 
> 7.9050503334599447e-323)],
>        dtype=[('isize', '<i8'), ('irow', '<i8'), ('icol', 
> '<i8'), ('value', '<f8')])
> 
> Is this a 64-bit problem ?
> 

I don't know if it's a 64-bit problem per-se, so much as a disagreement between
fortran and numpy.  Numpy is making the size of the integer fields 8 bytes, while
in Fortran, they're only 4 bytes.  When constructing your dtype, use np.int32 or
'<i4' for your type for the integer fields, and see if that fixes it.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From nwagner at iam.uni-stuttgart.de  Mon Feb  2 14:58:17 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 02 Feb 2009 20:58:17 +0100
Subject: [Numpy-discussion] Fortran binary files and numpy/scipy
In-Reply-To: <49874BA6.3050806@gmail.com>
References: <web-116566714@uni-stuttgart.de>
	<e76aa17f0902020439y50b29c2g8b3ea469bd54e9db@mail.gmail.com>
	<web-116573065@uni-stuttgart.de> <49871C89.4000800@gmail.com>
	<web-116584825@uni-stuttgart.de> <49874BA6.3050806@gmail.com>
Message-ID: <web-116585563@uni-stuttgart.de>

On Mon, 02 Feb 2009 13:38:14 -0600
  Ryan May <rmay31 at gmail.com> wrote:
> Nils Wagner wrote:
>> On Mon, 02 Feb 2009 10:17:13 -0600
>>   Ryan May <rmay31 at gmail.com> wrote:
>>> Every write statement in fortran first writes out the 
>>> number of bytes that will
>>> follow, *then* the actual data.  So, for instance, the 
>>> first write to file in
>>> your program will write the bytes corresponding to these 
>>> values:
>>>
>>> 16 X(1) Y(1) Z(1)
>>>
>>> The 16 comes from the size of 2 ints and 1 double. 
>>> Since you're always writing
>>> out the 3 values, and they're always the same size, try 
>>> adding another integer
>>> column as the first field in your array.
>>>
>>> Ryan
> 
>> Hi Ryan,
>> 
>> I have modified the python script.
>> 
>> import numpy as np
>> fname = open("bin.dat",'rb')
>> dt = 
>> np.dtype([('isize',int),('irow',int),('icol',int),('value',float)])
>> a  = np.fromfile(fname,dtype=dt)
>> 
>> 
>>>>> a
>> array([(16, 1090921693195, 4643140847074803712, 
>> 7.9050503334599447e-323),
>>         (16, 1086626725900, 4638566878703255552, 
>> 7.9050503334599447e-323),
>>         (16, 1082331758605, 4635611391447793664, 
>> 7.9050503334599447e-323),
>>         (16, 1078036791310, 4633922541587529728, 
>> 7.9050503334599447e-323),
>>         (16, 1073741824015, 4632233691727265792, 
>> 7.9050503334599447e-323),
>>         (16, 1069446856720, 4630967054332067840, 
>> 7.9050503334599447e-323),
>>         (16, 1065151889425, 4630122629401935872, 
>> 7.9050503334599447e-323),
>>         (16, 1060856922130, 4629137466983448576, 
>> 7.9050503334599447e-323),
>>         (16, 1056561954835, 4628293042053316608, 
>> 7.9050503334599447e-323),
>>         (16, 1052266987540, 4627448617123184640, 
>> 7.9050503334599447e-323)],
>>        dtype=[('isize', '<i8'), ('irow', '<i8'), 
>>('icol', 
>> '<i8'), ('value', '<f8')])
>> 
>> Is this a 64-bit problem ?
>> 
> 
> I don't know if it's a 64-bit problem per-se, so much as 
>a disagreement between
> fortran and numpy.  Numpy is making the size of the 
>integer fields 8 bytes, while
> in Fortran, they're only 4 bytes.  When constructing 
>your dtype, use np.int32 or
> '<i4' for your type for the integer fields, and see if 
>that fixes it.
> 

dt = 
np.dtype([('isize','int32'),('irow','int32'),('icol','int32'),('value','float')])


>>> a
array([(16, 0, 11, 1.2549267404367662e-321),
        (1081065472, 16, 0, 7.9050503334599447e-323),
        (12, 253, 0, 3.4485523805914514e-313),
        (0, 16, 0, 5.3474293932967148e-312),
        (0, 1079312384, 16, 3.3951932655444357e-313), (0, 
14, 251, 62.0),
        (16, 0, 16, 3.1829936864479085e-313),
        (250, 0, 1078525952, 7.9050503334599447e-323),
        (16, 0, 16, 1.2302234581447039e-321),
        (1078231040, 16, 0, 7.9050503334599447e-323),
        (17, 248, 0, 3.4484552433329538e-313),
        (0, 16, 0, 5.2413296037731544e-312),
        (0, 1077805056, 16, 3.3951932655444357e-313), (0, 
19, 246, 27.0),
        (16, 0, 16, 4.2439915819305446e-313),
        (245, 0, 1077411840, 7.9050503334599447e-323)],
       dtype=[('isize', '<i4'), ('irow', '<i4'), ('icol', 
'<i4'), ('value', '<f8')])

Any idea ?

  Nils


From rmay31 at gmail.com  Mon Feb  2 15:07:35 2009
From: rmay31 at gmail.com (Ryan May)
Date: Mon, 02 Feb 2009 14:07:35 -0600
Subject: [Numpy-discussion] Fortran binary files and numpy/scipy
In-Reply-To: <web-116585563@uni-stuttgart.de>
References: <web-116566714@uni-stuttgart.de>	<e76aa17f0902020439y50b29c2g8b3ea469bd54e9db@mail.gmail.com>	<web-116573065@uni-stuttgart.de>
	<49871C89.4000800@gmail.com>	<web-116584825@uni-stuttgart.de>
	<49874BA6.3050806@gmail.com> <web-116585563@uni-stuttgart.de>
Message-ID: <49875287.6030805@gmail.com>

Nils Wagner wrote:
>>> Is this a 64-bit problem ?
>>>
>> I don't know if it's a 64-bit problem per-se, so much as 
>> a disagreement between
>> fortran and numpy.  Numpy is making the size of the 
>> integer fields 8 bytes, while
>> in Fortran, they're only 4 bytes.  When constructing 
>> your dtype, use np.int32 or
>> '<i4' for your type for the integer fields, and see if 
>> that fixes it.
>>
> 
> dt = 
> np.dtype([('isize','int32'),('irow','int32'),('icol','int32'),('value','float')])
> 
> 
>>>> a
> array([(16, 0, 11, 1.2549267404367662e-321),
>         (1081065472, 16, 0, 7.9050503334599447e-323),
>         (12, 253, 0, 3.4485523805914514e-313),
>         (0, 16, 0, 5.3474293932967148e-312),
>         (0, 1079312384, 16, 3.3951932655444357e-313), (0, 
> 14, 251, 62.0),
>         (16, 0, 16, 3.1829936864479085e-313),
>         (250, 0, 1078525952, 7.9050503334599447e-323),
>         (16, 0, 16, 1.2302234581447039e-321),
>         (1078231040, 16, 0, 7.9050503334599447e-323),
>         (17, 248, 0, 3.4484552433329538e-313),
>         (0, 16, 0, 5.2413296037731544e-312),
>         (0, 1077805056, 16, 3.3951932655444357e-313), (0, 
> 19, 246, 27.0),
>         (16, 0, 16, 4.2439915819305446e-313),
>         (245, 0, 1077411840, 7.9050503334599447e-323)],
>        dtype=[('isize', '<i4'), ('irow', '<i4'), ('icol', 
> '<i4'), ('value', '<f8')])
> 

Maybe on 64-bit machines, the number of bytes is 64-bits instead of 32 (see the
fact that the first 12 bytes of the file are 16 0 11.  Try:

dt =
np.dtype([('isize','int64'),('irow','int32'),('icol','int32'),('value','float')])

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From nwagner at iam.uni-stuttgart.de  Mon Feb  2 15:57:36 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 02 Feb 2009 21:57:36 +0100
Subject: [Numpy-discussion] Fortran binary files and numpy/scipy
In-Reply-To: <49875287.6030805@gmail.com>
References: <web-116566714@uni-stuttgart.de>
	<e76aa17f0902020439y50b29c2g8b3ea469bd54e9db@mail.gmail.com>
	<web-116573065@uni-stuttgart.de> <49871C89.4000800@gmail.com>
	<web-116584825@uni-stuttgart.de> <49874BA6.3050806@gmail.com>
	<web-116585563@uni-stuttgart.de> <49875287.6030805@gmail.com>
Message-ID: <web-116586882@uni-stuttgart.de>

On Mon, 02 Feb 2009 14:07:35 -0600
  Ryan May <rmay31 at gmail.com> wrote:
> Nils Wagner wrote:
>>>> Is this a 64-bit problem ?
>>>>
>>> I don't know if it's a 64-bit problem per-se, so much as 
>>> a disagreement between
>>> fortran and numpy.  Numpy is making the size of the 
>>> integer fields 8 bytes, while
>>> in Fortran, they're only 4 bytes.  When constructing 
>>> your dtype, use np.int32 or
>>> '<i4' for your type for the integer fields, and see if 
>>> that fixes it.
>>>
>> 
>> dt = 
>> np.dtype([('isize','int32'),('irow','int32'),('icol','int32'),('value','float')])
>> 
>> 
>>>>> a
>> array([(16, 0, 11, 1.2549267404367662e-321),
>>         (1081065472, 16, 0, 7.9050503334599447e-323),
>>         (12, 253, 0, 3.4485523805914514e-313),
>>         (0, 16, 0, 5.3474293932967148e-312),
>>         (0, 1079312384, 16, 3.3951932655444357e-313), 
>>(0, 
>> 14, 251, 62.0),
>>         (16, 0, 16, 3.1829936864479085e-313),
>>         (250, 0, 1078525952, 7.9050503334599447e-323),
>>         (16, 0, 16, 1.2302234581447039e-321),
>>         (1078231040, 16, 0, 7.9050503334599447e-323),
>>         (17, 248, 0, 3.4484552433329538e-313),
>>         (0, 16, 0, 5.2413296037731544e-312),
>>         (0, 1077805056, 16, 3.3951932655444357e-313), 
>>(0, 
>> 19, 246, 27.0),
>>         (16, 0, 16, 4.2439915819305446e-313),
>>         (245, 0, 1077411840, 7.9050503334599447e-323)],
>>        dtype=[('isize', '<i4'), ('irow', '<i4'), 
>>('icol', 
>> '<i4'), ('value', '<f8')])
>> 
> 
> Maybe on 64-bit machines, the number of bytes is 64-bits 
>instead of 32 (see the
> fact that the first 12 bytes of the file are 16 0 11. 
> Try:
> 
> dt =
> np.dtype([('isize','int64'),('irow','int32'),('icol','int32'),('value','float')])
> 
> Ryan
> 

Strange

>>> a
array([(16, 11, 254, 254.0), (16, 16, 0, 
5.3686493512014268e-312),
        (4638566878703255552, 16, 0, 
7.9050503334599447e-323),
        (1082331758605, 0, 1079312384, 
7.9050503334599447e-323),
        (16, 14, 251, 62.0), (16, 16, 0, 
5.3049894774872906e-312),
        (4632233691727265792, 16, 0, 
7.9050503334599447e-323),
        (1069446856720, 0, 1078231040, 
7.9050503334599447e-323),
        (16, 17, 248, 35.0), (16, 16, 0, 
5.2413296037731544e-312),
        (4629137466983448576, 16, 0, 
7.9050503334599447e-323),
        (1056561954835, 0, 1077608448, 
7.9050503334599447e-323),
        (16, 20, 245, 24.0)],
       dtype=[('isize', '<i8'), ('irow', '<i4'), ('icol', 
'<i4'), ('value', '<f8')])
  


From rmay31 at gmail.com  Mon Feb  2 16:46:27 2009
From: rmay31 at gmail.com (Ryan May)
Date: Mon, 02 Feb 2009 15:46:27 -0600
Subject: [Numpy-discussion] Fortran binary files and numpy/scipy
In-Reply-To: <web-116586882@uni-stuttgart.de>
References: <web-116566714@uni-stuttgart.de>	<e76aa17f0902020439y50b29c2g8b3ea469bd54e9db@mail.gmail.com>	<web-116573065@uni-stuttgart.de>
	<49871C89.4000800@gmail.com>	<web-116584825@uni-stuttgart.de>
	<49874BA6.3050806@gmail.com>	<web-116585563@uni-stuttgart.de>
	<49875287.6030805@gmail.com> <web-116586882@uni-stuttgart.de>
Message-ID: <498769B3.5020307@gmail.com>

Nils Wagner wrote:
> On Mon, 02 Feb 2009 14:07:35 -0600
>   Ryan May <rmay31 at gmail.com> wrote:
>> Nils Wagner wrote:
>>>>> Is this a 64-bit problem ?
>>>>>
>>>> I don't know if it's a 64-bit problem per-se, so much as 
>>>> a disagreement between
>>>> fortran and numpy.  Numpy is making the size of the 
>>>> integer fields 8 bytes, while
>>>> in Fortran, they're only 4 bytes.  When constructing 
>>>> your dtype, use np.int32 or
>>>> '<i4' for your type for the integer fields, and see if 
>>>> that fixes it.
>>>>
>>> dt = 
>>> np.dtype([('isize','int32'),('irow','int32'),('icol','int32'),('value','float')])
>>>
>>>
>>>>>> a
>>> array([(16, 0, 11, 1.2549267404367662e-321),
>>>         (1081065472, 16, 0, 7.9050503334599447e-323),
>>>         (12, 253, 0, 3.4485523805914514e-313),
>>>         (0, 16, 0, 5.3474293932967148e-312),
>>>         (0, 1079312384, 16, 3.3951932655444357e-313), 
>>> (0, 
>>> 14, 251, 62.0),
>>>         (16, 0, 16, 3.1829936864479085e-313),
>>>         (250, 0, 1078525952, 7.9050503334599447e-323),
>>>         (16, 0, 16, 1.2302234581447039e-321),
>>>         (1078231040, 16, 0, 7.9050503334599447e-323),
>>>         (17, 248, 0, 3.4484552433329538e-313),
>>>         (0, 16, 0, 5.2413296037731544e-312),
>>>         (0, 1077805056, 16, 3.3951932655444357e-313), 
>>> (0, 
>>> 19, 246, 27.0),
>>>         (16, 0, 16, 4.2439915819305446e-313),
>>>         (245, 0, 1077411840, 7.9050503334599447e-323)],
>>>        dtype=[('isize', '<i4'), ('irow', '<i4'), 
>>> ('icol', 
>>> '<i4'), ('value', '<f8')])
>>>
>> Maybe on 64-bit machines, the number of bytes is 64-bits 
>> instead of 32 (see the
>> fact that the first 12 bytes of the file are 16 0 11. 
>> Try:
>>
>> dt =
>> np.dtype([('isize','int64'),('irow','int32'),('icol','int32'),('value','float')])
>>
>> Ryan
>>
> 
> Strange
> 
>>>> a
> array([(16, 11, 254, 254.0), (16, 16, 0, 
> 5.3686493512014268e-312),
>         (4638566878703255552, 16, 0, 
> 7.9050503334599447e-323),
>         (1082331758605, 0, 1079312384, 
> 7.9050503334599447e-323),
>         (16, 14, 251, 62.0), (16, 16, 0, 
> 5.3049894774872906e-312),
>         (4632233691727265792, 16, 0, 
> 7.9050503334599447e-323),
>         (1069446856720, 0, 1078231040, 
> 7.9050503334599447e-323),
>         (16, 17, 248, 35.0), (16, 16, 0, 
> 5.2413296037731544e-312),
>         (4629137466983448576, 16, 0, 
> 7.9050503334599447e-323),
>         (1056561954835, 0, 1077608448, 
> 7.9050503334599447e-323),
>         (16, 20, 245, 24.0)],
>        dtype=[('isize', '<i8'), ('irow', '<i4'), ('icol', 
> '<i4'), ('value', '<f8')])
>   

Apparently I was slightly off on the details (it's been awhile since I had to
deal with this nonsense).  The number of bytes written is written before *and*
after writing your actual data.  So the following should work:

dtype=[('isize', '<i8'), ('irow', '<i4'), ('icol', '<i4'), ('value', '<f8'),
'isize2', '<i8'])

Weird.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From Mike.Colonno at spacex.com  Mon Feb  2 18:56:22 2009
From: Mike.Colonno at spacex.com (Mike Colonno)
Date: Mon, 2 Feb 2009 15:56:22 -0800
Subject: [Numpy-discussion] f2py: VS version on Windows
In-Reply-To: <498769B3.5020307@gmail.com>
References: <web-116566714@uni-stuttgart.de>
	<e76aa17f0902020439y50b29c2g8b3ea469bd54e9db@mail.gmail.com>
	<web-116573065@uni-stuttgart.de>	<49871C89.4000800@gmail.com>
	<web-116584825@uni-stuttgart.de>	<49874BA6.3050806@gmail.com>
	<web-116585563@uni-stuttgart.de>	<49875287.6030805@gmail.com>
	<web-116586882@uni-stuttgart.de> <498769B3.5020307@gmail.com>
Message-ID: <657E769D35612D4CAB1FEAACB0A92BB503098E574A@MAIL2.spacex.corp>

            I'm trying to test out f2py in Windows (python 2.5.4 32-bit for now + most recent Numpy). I'd like to use the Intel compilers, but msvc is fine if needed. I get the output below about which I have a question re: the warning about VS version. I have VS 2008 currently which should have no trouble making binaries compatible with older version of VS(?) Is there any way around this error with VS > 2003?



            Thanks,

            ~Mike C.





C:\Python25\Lib\site-packages\numpy\f2py\docs>C:\Python25\Scripts\f2py.py -c --f

compiler=intel -m hello hello.f

Ignoring "Python was built with Visual Studio 2003;

extensions must be built with a compiler than can generate compatible binaries.

Visual Studio 2003 was not found on this system. If you have Cygwin installed,

you can try compiling with MingW32, by passing "-c mingw32" to setup.py." (one s

hould fix me in fcompiler/compaq.py)

running build

running config_cc

unifing config_cc, config, build_clib, build_ext, build commands --compiler opti

ons

running config_fc

unifing config_fc, config, build_clib, build_ext, build commands --fcompiler opt

ions

running build_src

building extension "hello" sources

f2py options: []

f2py:> c:\docume~1\mike\locals~1\temp\tmptd0t5g\src.win32-2.5\hellomodule.c

creating c:\docume~1\mike\locals~1\temp\tmptd0t5g

creating c:\docume~1\mike\locals~1\temp\tmptd0t5g\src.win32-2.5

Reading fortran codes...

        Reading file 'hello.f' (format:fix,strict)

Post-processing...

        Block: hello

                        Block: foo

Post-processing (stage 2)...

Building modules...

        Building module "hello"...

                Constructing wrapper function "foo"...

                  foo(a)

        Wrote C/API module "hello" to file "c:\docume~1\mike\locals~1\temp\tmptd

0t5g\src.win32-2.5/hellomodule.c"

  adding 'c:\docume~1\mike\locals~1\temp\tmptd0t5g\src.win32-2.5\fortranobject.c

' to sources.

  adding 'c:\docume~1\mike\locals~1\temp\tmptd0t5g\src.win32-2.5' to include_dir

s.

copying C:\Python25\lib\site-packages\numpy\f2py\src\fortranobject.c -> c:\docum

e~1\mike\locals~1\temp\tmptd0t5g\src.win32-2.5

copying C:\Python25\lib\site-packages\numpy\f2py\src\fortranobject.h -> c:\docum

e~1\mike\locals~1\temp\tmptd0t5g\src.win32-2.5

running build_ext

No module named msvccompiler in numpy.distutils; trying from distutils

error: Python was built with Visual Studio 2003;

extensions must be built with a compiler than can generate compatible binaries.

Visual Studio 2003 was not found on this system. If you have Cygwin installed,

you can try compiling with MingW32, by passing "-c mingw32" to setup.py.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090202/d4b2a6ba/attachment.html>

From patrickmarshwx at gmail.com  Mon Feb  2 19:01:51 2009
From: patrickmarshwx at gmail.com (Patrick Marsh)
Date: Mon, 2 Feb 2009 18:01:51 -0600
Subject: [Numpy-discussion] f2py: VS version on Windows
In-Reply-To: <657E769D35612D4CAB1FEAACB0A92BB503098E574A@MAIL2.spacex.corp>
References: <web-116566714@uni-stuttgart.de> <web-116573065@uni-stuttgart.de>
	<49871C89.4000800@gmail.com> <web-116584825@uni-stuttgart.de>
	<49874BA6.3050806@gmail.com> <web-116585563@uni-stuttgart.de>
	<49875287.6030805@gmail.com> <web-116586882@uni-stuttgart.de>
	<498769B3.5020307@gmail.com>
	<657E769D35612D4CAB1FEAACB0A92BB503098E574A@MAIL2.spacex.corp>
Message-ID: <e9962b830902021601x7e4f42ccw6987289b1f973d67@mail.gmail.com>

Hi Mike,

Python extensions must be built using the same compiler as was used to
build the Python binary.  Python 2.5.4 was built using MSVC2003 and so
extensions for it must be built using the same compiler. The exception
to this rule is that extensions built using mingw32 (and msys) will
work with most, if not all, windows Python binaries.

-Patrick






On Mon, Feb 2, 2009 at 5:56 PM, Mike Colonno <Mike.Colonno at spacex.com> wrote:
>             I'm trying to test out f2py in Windows (python 2.5.4 32-bit for
> now + most recent Numpy). I'd like to use the Intel compilers, but msvc is
> fine if needed. I get the output below about which I have a question re: the
> warning about VS version. I have VS 2008 currently which should have no
> trouble making binaries compatible with older version of VS(?) Is there any
> way around this error with VS > 2003?
>
>
>
>             Thanks,
>
>             ~Mike C.
>
>
>
>
>
> C:\Python25\Lib\site-packages\numpy\f2py\docs>C:\Python25\Scripts\f2py.py -c
> --f
>
> compiler=intel -m hello hello.f
>
> Ignoring "Python was built with Visual Studio 2003;
>
> extensions must be built with a compiler than can generate compatible
> binaries.
>
> Visual Studio 2003 was not found on this system. If you have Cygwin
> installed,
>
> you can try compiling with MingW32, by passing "-c mingw32" to setup.py."
> (one s
>
> hould fix me in fcompiler/compaq.py)
>
> running build
>
> running config_cc
>
> unifing config_cc, config, build_clib, build_ext, build commands --compiler
> opti
>
> ons
>
> running config_fc
>
> unifing config_fc, config, build_clib, build_ext, build commands --fcompiler
> opt
>
> ions
>
> running build_src
>
> building extension "hello" sources
>
> f2py options: []
>
> f2py:> c:\docume~1\mike\locals~1\temp\tmptd0t5g\src.win32-2.5\hellomodule.c
>
> creating c:\docume~1\mike\locals~1\temp\tmptd0t5g
>
> creating c:\docume~1\mike\locals~1\temp\tmptd0t5g\src.win32-2.5
>
> Reading fortran codes...
>
>         Reading file 'hello.f' (format:fix,strict)
>
> Post-processing...
>
>         Block: hello
>
>                         Block: foo
>
> Post-processing (stage 2)...
>
> Building modules...
>
>         Building module "hello"...
>
>                 Constructing wrapper function "foo"...
>
>                   foo(a)
>
>         Wrote C/API module "hello" to file
> "c:\docume~1\mike\locals~1\temp\tmptd
>
> 0t5g\src.win32-2.5/hellomodule.c"
>
>   adding
> 'c:\docume~1\mike\locals~1\temp\tmptd0t5g\src.win32-2.5\fortranobject.c
>
> ' to sources.
>
>   adding 'c:\docume~1\mike\locals~1\temp\tmptd0t5g\src.win32-2.5' to
> include_dir
>
> s.
>
> copying C:\Python25\lib\site-packages\numpy\f2py\src\fortranobject.c ->
> c:\docum
>
> e~1\mike\locals~1\temp\tmptd0t5g\src.win32-2.5
>
> copying C:\Python25\lib\site-packages\numpy\f2py\src\fortranobject.h ->
> c:\docum
>
> e~1\mike\locals~1\temp\tmptd0t5g\src.win32-2.5
>
> running build_ext
>
> No module named msvccompiler in numpy.distutils; trying from distutils
>
> error: Python was built with Visual Studio 2003;
>
> extensions must be built with a compiler than can generate compatible
> binaries.
>
> Visual Studio 2003 was not found on this system. If you have Cygwin
> installed,
>
> you can try compiling with MingW32, by passing "-c mingw32" to setup.py.
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>



-- 
Patrick Marsh
Graduate Research Assistant
School of Meteorology
University of Oklahoma
http://www.patricktmarsh.com


From mtrumpis at berkeley.edu  Mon Feb  2 19:21:53 2009
From: mtrumpis at berkeley.edu (mtrumpis at berkeley.edu)
Date: Mon, 2 Feb 2009 16:21:53 -0800 (PST)
Subject: [Numpy-discussion] SVD errors
Message-ID: <34928.128.32.52.185.1233620513.squirrel@calmail.berkeley.edu>

Hello list.. I've run into two SVD errors over the last few days. Both
errors are identical in numpy/scipy.

I've submitted a ticket for the 1st problem (numpy ticket #990). Summary
is: some builds of the lapack_lite module linking against system LAPACK
(not the bundled dlapack_lite.o, etc) give a "LinAlgError: SVD did not
converge" exception on my matrix. This error does occur using Mac's
Accelerate framework LAPACK, and a coworker's Ubuntu LAPACK version. It
does not seem to happen using ATLAS LAPACK (nor using Octave/Matlab on
said Ubuntu)

Just today I've come across a negative singular value cropping up in an
SVD of a different matrix. This error does occur on my ATLAS LAPACK based
numpy, as well as on the Ubuntu setup. And once again, it does not happen
in Octave/Matlab.

I'm using numpy 1.3.0.dev6336 -- don't know what the Ubuntu box is running.

Here are some npy files for the two different cases:

https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/noconverge_operator.npy
https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/negsval_operator.npy

Mike



From robert.kern at gmail.com  Mon Feb  2 19:27:05 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 2 Feb 2009 18:27:05 -0600
Subject: [Numpy-discussion] SVD errors
In-Reply-To: <34928.128.32.52.185.1233620513.squirrel@calmail.berkeley.edu>
References: <34928.128.32.52.185.1233620513.squirrel@calmail.berkeley.edu>
Message-ID: <3d375d730902021627x2ef54411j5bb9504927ff831@mail.gmail.com>

On Mon, Feb 2, 2009 at 18:21,  <mtrumpis at berkeley.edu> wrote:
> Hello list.. I've run into two SVD errors over the last few days. Both
> errors are identical in numpy/scipy.
>
> I've submitted a ticket for the 1st problem (numpy ticket #990). Summary
> is: some builds of the lapack_lite module linking against system LAPACK
> (not the bundled dlapack_lite.o, etc) give a "LinAlgError: SVD did not
> converge" exception on my matrix. This error does occur using Mac's
> Accelerate framework LAPACK, and a coworker's Ubuntu LAPACK version. It
> does not seem to happen using ATLAS LAPACK (nor using Octave/Matlab on
> said Ubuntu)

These are almost certainly issues with the particular implementations
of LAPACK that you are using. I don't think there is anything we can
do from numpy or scipy to change this.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From lou_boog2000 at yahoo.com  Mon Feb  2 19:40:04 2009
From: lou_boog2000 at yahoo.com (Lou Pecora)
Date: Mon, 2 Feb 2009 16:40:04 -0800 (PST)
Subject: [Numpy-discussion] SVD errors
In-Reply-To: <34928.128.32.52.185.1233620513.squirrel@calmail.berkeley.edu>
Message-ID: <459961.27309.qm@web34408.mail.mud.yahoo.com>

I ran into this problem a year or so ago. I suspect my messages to the list are in the archives somewhere. It is a known problem and involves a hard-coded maximum number of iterations in the SVD code.  The problem is on the LaPack side.  You can go in and change it, but then you have to recompile everything and rebuild Numpy, etc. etc. Not sure how easy/hard this is.  I avoided it. 

What I found that worked for me (depends on your numerical situation) is to take the original matrix you are trying to decompose, say A, and examine, instead, the SVD of A^T A.  Then the singular values of that matrix are the square of the singular values of A.  This worked for me, but my original matrix was square.  Maybe that helped.  Don't know. It's worth a try.

-- Lou Pecora,   my views are my own.


--- On Mon, 2/2/09, mtrumpis at berkeley.edu <mtrumpis at berkeley.edu> wrote:

> From: mtrumpis at berkeley.edu <mtrumpis at berkeley.edu>
> Subject: [Numpy-discussion] SVD errors
> To: numpy-discussion at scipy.org
> Date: Monday, February 2, 2009, 7:21 PM
> Hello list.. I've run into two SVD errors over the last
> few days. Both
> errors are identical in numpy/scipy.
> 
> I've submitted a ticket for the 1st problem (numpy
> ticket #990). Summary
> is: some builds of the lapack_lite module linking against
> system LAPACK
> (not the bundled dlapack_lite.o, etc) give a
> "LinAlgError: SVD did not
> converge" exception on my matrix. This error does
> occur using Mac's
> Accelerate framework LAPACK, and a coworker's Ubuntu
> LAPACK version. It
> does not seem to happen using ATLAS LAPACK (nor using
> Octave/Matlab on
> said Ubuntu)
> 
> Just today I've come across a negative singular value
> cropping up in an
> SVD of a different matrix. This error does occur on my
> ATLAS LAPACK based
> numpy, as well as on the Ubuntu setup. And once again, it
> does not happen
> in Octave/Matlab.
> 
> I'm using numpy 1.3.0.dev6336 -- don't know what
> the Ubuntu box is running.
> 
> Here are some npy files for the two different cases:
> 
> https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/noconverge_operator.npy
> https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/negsval_operator.npy
> 
> Mike
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion


      


From sturla at molden.no  Mon Feb  2 20:55:58 2009
From: sturla at molden.no (Sturla Molden)
Date: Tue, 3 Feb 2009 02:55:58 +0100 (CET)
Subject: [Numpy-discussion] f2py: VS version on Windows
In-Reply-To: <e9962b830902021601x7e4f42ccw6987289b1f973d67@mail.gmail.com>
References: <web-116566714@uni-stuttgart.de> <web-116573065@uni-stuttgart.de>
	<49871C89.4000800@gmail.com> <web-116584825@uni-stuttgart.de>
	<49874BA6.3050806@gmail.com> <web-116585563@uni-stuttgart.de>
	<49875287.6030805@gmail.com> <web-116586882@uni-stuttgart.de>
	<498769B3.5020307@gmail.com>
	<657E769D35612D4CAB1FEAACB0A92BB503098E574A@MAIL2.spacex.corp>
	<e9962b830902021601x7e4f42ccw6987289b1f973d67@mail.gmail.com>
Message-ID: <263280a61eb7f65d0aee5504fa956201.squirrel@webmail.uio.no>


> Python extensions must be built using the same compiler as was used to
> build the Python binary.  Python 2.5.4 was built using MSVC2003 and so
> extensions for it must be built using the same compiler. The exception
> to this rule is that extensions built using mingw32 (and msys) will
> work with most, if not all, windows Python binaries.


This is NOT correct.

What you cannot do, is safely share CRT objects between different CRTs.

You cannot malloc() some memory with msvcrt.dll, and subsequently free()
the memory with msvcr71.dll. Similarly, you cannot fopen() a FILE* with
msvcr71.dll and fread() with msvcrt.dll. Applications that do this will
sooner or later fail in mysterious ways.

Thus to be on the safe side, you should link the same CRT as Python and
other native extensions use. That is, msvcr71.dll for Python 2.5.

MSVC2003 usually does this by default, but there are exceptions. For
example, Visual C++ 2003 Toolkit links against a static version of
msvcr71. GCC (mingw)  will by default link with msvcrt.dll. It will link
with the same CRT as Python if you link with -lmsvcr71.

You can create link libraries against this CRT for most compilers, you
just need a .def file. Thus, you can use all compilers.

If your extension DO NOT share its CRT objects with Python, it does not
matter what compiler you use.

IANAL, but there is a licensing issue one needs to be aware of: You are
not allowed to redistribute msvcr71.dll except if you own a VS2003 license
and have used that compiler. This is important if you use py2exe to create
a Windows executable. This is one example where GCC cannot be used. But it
is a legal issue, not a technical one. On the other hand, I don't think
Microsoft has ever percecuted anyone for infringment of their msvcr71.dll
copyright. They just want you to develop for their OS. But if you are
going to use py2exe, you have to sort this problem out, or simply
redistribute a full Python distro instead. In that case, PSF has built
Python with a licensed VC2003 compiler, and you are just redustributing
their binary installer (which is ok).


Sturla Molden





From sturla at molden.no  Mon Feb  2 21:03:31 2009
From: sturla at molden.no (Sturla Molden)
Date: Tue, 3 Feb 2009 03:03:31 +0100 (CET)
Subject: [Numpy-discussion] f2py: VS version on Windows
In-Reply-To: <657E769D35612D4CAB1FEAACB0A92BB503098E574A@MAIL2.spacex.corp>
References: <web-116566714@uni-stuttgart.de>
	<e76aa17f0902020439y50b29c2g8b3ea469bd54e9db@mail.gmail.com>
	<web-116573065@uni-stuttgart.de> <49871C89.4000800@gmail.com>
	<web-116584825@uni-stuttgart.de> <49874BA6.3050806@gmail.com>
	<web-116585563@uni-stuttgart.de> <49875287.6030805@gmail.com>
	<web-116586882@uni-stuttgart.de> <498769B3.5020307@gmail.com>
	<657E769D35612D4CAB1FEAACB0A92BB503098E574A@MAIL2.spacex.corp>
Message-ID: <bdb73a060bf61af1c466a3865ca8e031.squirrel@webmail.uio.no>


>             I'm trying to test out f2py in Windows (python 2.5.4 32-bit
> for now + most recent Numpy). I'd like to use the Intel
> compilers, but msvc is fine if needed. I get the output below
> about which I have a question re: the warning about VS
> version. I have VS 2008 currently which should have no trouble
> making binaries compatible with older version of VS(?) Is
> there any way around this error with VS > 2003?

Shortly speaking: you must for Python 2.5 link with msvcr71.dll. The
version of the MSVC compiler is unimportant if you can force it to use
this CRT. Check your link libraries, and see if you find one for this DLL.
Otherwise, you need to create a .def file for this DLL and create the link
library from that. And make sure you don't link with other CRT versions.

S.M.




From patrickmarshwx at gmail.com  Mon Feb  2 21:12:10 2009
From: patrickmarshwx at gmail.com (Patrick Marsh)
Date: Mon, 2 Feb 2009 20:12:10 -0600
Subject: [Numpy-discussion] f2py: VS version on Windows
In-Reply-To: <bdb73a060bf61af1c466a3865ca8e031.squirrel@webmail.uio.no>
References: <web-116566714@uni-stuttgart.de> <49871C89.4000800@gmail.com>
	<web-116584825@uni-stuttgart.de> <49874BA6.3050806@gmail.com>
	<web-116585563@uni-stuttgart.de> <49875287.6030805@gmail.com>
	<web-116586882@uni-stuttgart.de> <498769B3.5020307@gmail.com>
	<657E769D35612D4CAB1FEAACB0A92BB503098E574A@MAIL2.spacex.corp>
	<bdb73a060bf61af1c466a3865ca8e031.squirrel@webmail.uio.no>
Message-ID: <e9962b830902021812m3e60ad54l7d38de808475ca19@mail.gmail.com>

You learn something new every day.  It turns out I had previously been
given wrong information and I apologize for passing that along to the
list.

-Patrick



On Mon, Feb 2, 2009 at 8:03 PM, Sturla Molden <sturla at molden.no> wrote:
>
>>             I'm trying to test out f2py in Windows (python 2.5.4 32-bit
>> for now + most recent Numpy). I'd like to use the Intel
>> compilers, but msvc is fine if needed. I get the output below
>> about which I have a question re: the warning about VS
>> version. I have VS 2008 currently which should have no trouble
>> making binaries compatible with older version of VS(?) Is
>> there any way around this error with VS > 2003?
>
> Shortly speaking: you must for Python 2.5 link with msvcr71.dll. The
> version of the MSVC compiler is unimportant if you can force it to use
> this CRT. Check your link libraries, and see if you find one for this DLL.
> Otherwise, you need to create a .def file for this DLL and create the link
> library from that. And make sure you don't link with other CRT versions.
>
> S.M.
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Patrick Marsh
Graduate Research Assistant
School of Meteorology
University of Oklahoma
http://www.patricktmarsh.com


From cournape at gmail.com  Mon Feb  2 22:25:40 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 3 Feb 2009 12:25:40 +0900
Subject: [Numpy-discussion] f2py: VS version on Windows
In-Reply-To: <263280a61eb7f65d0aee5504fa956201.squirrel@webmail.uio.no>
References: <web-116566714@uni-stuttgart.de> <web-116584825@uni-stuttgart.de>
	<49874BA6.3050806@gmail.com> <web-116585563@uni-stuttgart.de>
	<49875287.6030805@gmail.com> <web-116586882@uni-stuttgart.de>
	<498769B3.5020307@gmail.com>
	<657E769D35612D4CAB1FEAACB0A92BB503098E574A@MAIL2.spacex.corp>
	<e9962b830902021601x7e4f42ccw6987289b1f973d67@mail.gmail.com>
	<263280a61eb7f65d0aee5504fa956201.squirrel@webmail.uio.no>
Message-ID: <5b8d13220902021925w696472f2h12d0925640b3e433@mail.gmail.com>

On Tue, Feb 3, 2009 at 10:55 AM, Sturla Molden <sturla at molden.no> wrote:
>
>> Python extensions must be built using the same compiler as was used to
>> build the Python binary.  Python 2.5.4 was built using MSVC2003 and so
>> extensions for it must be built using the same compiler. The exception
>> to this rule is that extensions built using mingw32 (and msys) will
>> work with most, if not all, windows Python binaries.
>
>
> This is NOT correct.

Although it is technically true, that's relatively irrelevant for
practical matters: it is not currently possible to build numpy with VS
2008 and 7.1 CRT, and you have to build numpy with the same compiler
as the one used by python if you use MS compilers.

David


From strawman at astraw.com  Tue Feb  3 03:18:49 2009
From: strawman at astraw.com (Andrew Straw)
Date: Tue, 03 Feb 2009 00:18:49 -0800
Subject: [Numpy-discussion] N-D array interface page is out of date
In-Reply-To: <49791FA1.3020803@astraw.com>
References: <49791FA1.3020803@astraw.com>
Message-ID: <4987FDE9.5030303@astraw.com>

Regarding http://numpy.scipy.org/array_interface.shtml :

I just noticed that this out of date page is now featuring in recent
discussions about the future of Numpy in Ubuntu:
https://bugs.launchpad.net/ubuntu/+source/python-numpy/+bug/309215

Can someone with appropriate permissions fix the page or give me the
appropriate permissions so I can do it? I think even deleting the page
is better than keeping it as-is.

-Andrew

Andrew Straw wrote:
> Hi, I just noticed that the N-D array interface page is outdated and
> doesn't mention the buffer interface that is standard with Python 2.6
> and Python 3.0:
> 
> http://numpy.scipy.org/array_interface.shtml
> 
> This page is linked to from http://numpy.scipy.org/
> 
> I suggest, at the minimum, modifying the page with really annoying
> blinking red letters at the top (or other suitable warning) that this is
> deprecated at that people should use
> http://www.python.org/dev/peps/pep-3118/ instead.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From pav at iki.fi  Tue Feb  3 03:46:12 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 3 Feb 2009 08:46:12 +0000 (UTC)
Subject: [Numpy-discussion] SVD errors
References: <34928.128.32.52.185.1233620513.squirrel@calmail.berkeley.edu>
	<3d375d730902021627x2ef54411j5bb9504927ff831@mail.gmail.com>
Message-ID: <gm908k$d55$1@ger.gmane.org>

Mon, 02 Feb 2009 18:27:05 -0600, Robert Kern wrote:
> On Mon, Feb 2, 2009 at 18:21,  <mtrumpis at berkeley.edu> wrote:
>> Hello list.. I've run into two SVD errors over the last few days. Both
>> errors are identical in numpy/scipy.
>>
>> I've submitted a ticket for the 1st problem (numpy ticket #990).
>> Summary is: some builds of the lapack_lite module linking against
>> system LAPACK (not the bundled dlapack_lite.o, etc) give a
>> "LinAlgError: SVD did not converge" exception on my matrix. This error
>> does occur using Mac's Accelerate framework LAPACK, and a coworker's
>> Ubuntu LAPACK version. It does not seem to happen using ATLAS LAPACK
>> (nor using Octave/Matlab on said Ubuntu)
> 
> These are almost certainly issues with the particular implementations of
> LAPACK that you are using. I don't think there is anything we can do
> from numpy or scipy to change this.

Yes, this is almost certainly a LAPACK problem. If in doubt, you can test 
it with the following F90 program (be sure to link it against the same 
LAPACK as Numpy). Save the matrices with 'np.savetxt("foo.txt", x.ravel
())' and run './test < foo.txt'.

----
program test
    implicit none
    integer, parameter :: N = 128
    double precision :: A(N*N), S(N), U(N,N), Vh(N,N)
    double precision, allocatable :: WORK(:)
    double precision :: tmp
    integer :: IWORK(8*N)
    integer :: INFO = 0, LWORK

    read(*,*) A
    A = reshape(transpose(reshape(A, (/N, N/))), (/ N*N /))
    call dgesdd('A', N, N, A, N, S, U, N, Vh, N, tmp, -1, IWORK, INFO)
    LWORK = tmp
    if (info .ne. 0) stop 'lwork query failed'
    write(*,*) 'lwork:', lwork
    allocate(WORK(LWORK))
    call dgesdd('A', N, N, A, N, S, U, N, Vh, N, WORK, LWORK, IWORK, INFO)
    write(*,*) 'info:', INFO
    write(*,*) 'min(S):', minval(S)
    if (INFO .ne. 0) then
        write(*,*) ' -> SVD failed to converge'
    end if
    if (minval(S) .lt. 0) then
        write(*,*) ' -> negative singular value'
    end if
end program
----



From ondrej at certik.cz  Tue Feb  3 04:03:15 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 3 Feb 2009 01:03:15 -0800
Subject: [Numpy-discussion] porting NumPy to Python 3
In-Reply-To: <f14846790901290211q3bffba43oce25657ed04779dc@mail.gmail.com>
References: <f14846790901290211q3bffba43oce25657ed04779dc@mail.gmail.com>
Message-ID: <85b5c3130902030103w69180a6bm4143050b4b82bf1f@mail.gmail.com>

Hi James,

On Thu, Jan 29, 2009 at 2:11 AM, James Watson <watson.jim at gmail.com> wrote:
> Hi,
>
> I am interested in contributing to the port of NumPy to Python 3.  Who
> I should coordinate effort with?
>
> I have started at the Python end of the problem (as opposed to
> http://www.scipy.org/Python3k), e.g. I have several patches to get
> 2to3 to work on NumPy's Python source code.

I am sorry that noone has replied to your email. Could you please
upload your patches somewhere?

Ondrej


From faltet at pytables.org  Tue Feb  3 04:15:16 2009
From: faltet at pytables.org (Francesc Alted)
Date: Tue, 3 Feb 2009 10:15:16 +0100
Subject: [Numpy-discussion] example reading binary Fortran file
In-Reply-To: <dcba39620901301453n21b9221di25c5267c8ec66bcf@mail.gmail.com>
References: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>
	<6df541a5c8d6ecf26b6e38f404958401.squirrel@webmail.uio.no>
	<dcba39620901301453n21b9221di25c5267c8ec66bcf@mail.gmail.com>
Message-ID: <200902031015.17490.faltet@pytables.org>

A Friday 30 January 2009, David Froger escrigu?:
> ok for f2py!
>
> > Otherwise, you will have to figure out how your Fortran program
> > writes the file. I.e. what padding, metainformation, etc. that are
> > used. If you switch Fortran compiler, or even compiler version from
> > the same vendor, you must start over again.
>
> In my experience, I never had this kind of problem. I just have to
> convert files between big/little endian with uswap
> (http://linux.die.net/man/1/uswap), but I never see, in my
> experience, a Fortran program writting data differently given the
> compilator.
>
> >For my own work, I just makes sure NEVER to do any I/O in Fortran!
> > It is asking for trouble. I leave the I/O to Python or C, where it
> > belongs. That way I know what data are written and what data are
> > read.
>
> Unfortunately., binary files are mandatory in the contex I work. I
> use a scientific code written in Fortran to compute fluid dynamics.
> Typically the simulation is run on supercalculator and generate giga
> and giga of datum, so we must use the binary format, which recquire
> less memory for the stockage. Then, I like to post-trait the datum
> using Python and Gnuplot.py.
>
> That's why I'm looking for a performantant, easy and 'standard'  way
> to read binary Fortran files. (I think many people have the same
> need).

If you need to compact your datafiles to a maximum, you may want to 
write your data with the HDF5 library [1] that, besides using a binary 
format, it allows on-the-flight compression.  HDF5 is a fairly growing 
standard in scientific computing and it has wrappers for the most 
important languages like C, Fortran, Java and, of course, Python ;-)
In particular, the available HDF5 interfaces to Python allows to 
read/write HDF5 native files very easily.  Also, many computational 
environments, like Matlab, Octave or IDL do support HDF5 files.

[1] http://www.hdfgroup.org/HDF5/

Cheers,

-- 
Francesc Alted


From ndbecker2 at gmail.com  Tue Feb  3 07:24:40 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 03 Feb 2009 07:24:40 -0500
Subject: [Numpy-discussion] from_function
Message-ID: <gm9d28$slc$1@ger.gmane.org>

I've been using something I wrote:

coef_from_function (function, delta, size)
which does (c++ code):

  double center = double(size-1)/2;
  for (int i = 0; i < size; ++i)
    coef[i] = call<value_t> (func, double(i - center) * delta);

I thought to translate this to np.fromfunction.  It seems fromfunction is 
not as flexible, it uses only a fixed integer grid?




From stefan at sun.ac.za  Tue Feb  3 08:06:47 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 3 Feb 2009 15:06:47 +0200
Subject: [Numpy-discussion] N-D array interface page is out of date
In-Reply-To: <4987FDE9.5030303@astraw.com>
References: <49791FA1.3020803@astraw.com> <4987FDE9.5030303@astraw.com>
Message-ID: <9457e7c80902030506l7094e8d3x33996b861f61bff8@mail.gmail.com>

2009/2/3 Andrew Straw <strawman at astraw.com>:
> Can someone with appropriate permissions fix the page or give me the
> appropriate permissions so I can do it? I think even deleting the page
> is better than keeping it as-is.

Who all has editing access to this page?  Is it hosted on scipy.org?

St?fan


From pgmdevlist at gmail.com  Tue Feb  3 09:49:00 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 3 Feb 2009 09:49:00 -0500
Subject: [Numpy-discussion] Numpy 1.3 release date ?
Message-ID: <D955A90A-2121-4D34-A8AE-99292D86FC7F@gmail.com>

All,
When can we expect numpy 1.3 to be released ?
Sincerely,
P.


From Mike.Colonno at spacex.com  Tue Feb  3 10:52:36 2009
From: Mike.Colonno at spacex.com (Mike Colonno)
Date: Tue, 3 Feb 2009 07:52:36 -0800
Subject: [Numpy-discussion] f2py: VS version on Windows
In-Reply-To: <5b8d13220902021925w696472f2h12d0925640b3e433@mail.gmail.com>
References: <web-116566714@uni-stuttgart.de>
	<web-116584825@uni-stuttgart.de>	<49874BA6.3050806@gmail.com>
	<web-116585563@uni-stuttgart.de>	<49875287.6030805@gmail.com>
	<web-116586882@uni-stuttgart.de>	<498769B3.5020307@gmail.com>
	<657E769D35612D4CAB1FEAACB0A92BB503098E574A@MAIL2.spacex.corp>
	<e9962b830902021601x7e4f42ccw6987289b1f973d67@mail.gmail.com>
	<263280a61eb7f65d0aee5504fa956201.squirrel@webmail.uio.no>
	<5b8d13220902021925w696472f2h12d0925640b3e433@mail.gmail.com>
Message-ID: <657E769D35612D4CAB1FEAACB0A92BB503098E5931@MAIL2.spacex.corp>

   	Thanks to all for clearing this up. I have been bouncing this issue off the folks at Intel and they allege that Intel C++ should be able to do this independent of the version of VS used originally (I am skeptical). I am still getting some MKL-related missing symbol errors that we are clearing up and I will post anything useful that I discover. 

	~Mike C.

-----Original Message-----
From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of David Cournapeau
Sent: Monday, February 02, 2009 7:26 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] f2py: VS version on Windows

On Tue, Feb 3, 2009 at 10:55 AM, Sturla Molden <sturla at molden.no> wrote:
>
>> Python extensions must be built using the same compiler as was used to
>> build the Python binary.  Python 2.5.4 was built using MSVC2003 and so
>> extensions for it must be built using the same compiler. The exception
>> to this rule is that extensions built using mingw32 (and msys) will
>> work with most, if not all, windows Python binaries.
>
>
> This is NOT correct.

Although it is technically true, that's relatively irrelevant for
practical matters: it is not currently possible to build numpy with VS
2008 and 7.1 CRT, and you have to build numpy with the same compiler
as the one used by python if you use MS compilers.

David
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion


From rmay31 at gmail.com  Tue Feb  3 11:24:42 2009
From: rmay31 at gmail.com (Ryan May)
Date: Tue, 03 Feb 2009 10:24:42 -0600
Subject: [Numpy-discussion] genloadtxt question
Message-ID: <49886FCA.50407@gmail.com>

Pierre,

Should the following work?

import numpy as np
from StringIO import StringIO

converter = {'date':lambda s: datetime.strptime(s,'%Y-%m-%d %H:%M:%SZ')}
data = np.ndfromtxt(StringIO('2009-02-03 12:00:00Z,72214.0'), delimiter=',',
names=['date','stid'], dtype=None, converters=converter)

Right now, it's giving me the following:

Traceback (most recent call last):
  File "check_oban.py", line 15, in <module>
    converters=converter)
  File "/home/rmay/.local/lib64/python2.5/site-packages/numpy/lib/io.py", line
993, in ndfromtxt
    return genfromtxt(fname, **kwargs)
  File "/home/rmay/.local/lib64/python2.5/site-packages/numpy/lib/io.py", line
842, in genfromtxt
    locked=True)
  File "/home/rmay/.local/lib64/python2.5/site-packages/numpy/lib/_iotools.py",
line 472, in update
    self.type = self._getsubdtype(func('0'))
  File "check_oban.py", line 9, in <lambda>
    lambda s: datetime.strptime(s,'%Y-%m-%d %H:%M:%SZ').replace(tzinfo=UTC)}
  File "/usr/lib64/python2.5/_strptime.py", line 330, in strptime
    (data_string, format))
ValueError: time data did not match format:  data=0  fmt=%Y-%m-%d %H:%M:%SZ

Which comes from a part of the code in updating converters where it passes the
string '0' to the converter.  Are the converters expected to handle what amounts
to bad input even though the file itself has no such problems? Specifying the
dtype doesn't appear to help either.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From pgmdevlist at gmail.com  Tue Feb  3 12:17:01 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 3 Feb 2009 12:17:01 -0500
Subject: [Numpy-discussion] genloadtxt question
In-Reply-To: <49886FCA.50407@gmail.com>
References: <49886FCA.50407@gmail.com>
Message-ID: <D8DA6BB9-D908-467D-87B7-8171EF0BB5BF@gmail.com>


On Feb 3, 2009, at 11:24 AM, Ryan May wrote:

> Pierre,
>
> Should the following work?
>
> import numpy as np
> from StringIO import StringIO
>
> converter = {'date':lambda s: datetime.strptime(s,'%Y-%m-%d %H:%M: 
> %SZ')}
> data = np.ndfromtxt(StringIO('2009-02-03 12:00:00Z,72214.0'),  
> delimiter=',',
> names=['date','stid'], dtype=None, converters=converter)

Well, yes, it should work. That's indeed a problem with the  
getsubdtype method of the converter.
The problem is that we need to estimate the datatype of the output of  
the converter. In most cases, trying to convert '0' works properly,  
not in yours however. In r6338, I force the type to object if  
converting '0' does not work. That's a patch till the next corner  
case...



From simpson at math.toronto.edu  Tue Feb  3 12:19:45 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Tue, 3 Feb 2009 12:19:45 -0500
Subject: [Numpy-discussion] array vector elementwise multiplication
Message-ID: <6A3525EC-A389-4DDF-A68C-BB8455417916@math.toronto.edu>

I have an M x N matrix A and two vectors, an M dimensional vector x  
and an N dimensional vector y.  I would like to be able to do two  
things.

1.  Multiply, elementwise, every column of A by x

2.  Multiply, elementwise, every row of A by y.

What's the "quick" way to do this in numpy?

-gideon



From rmay31 at gmail.com  Tue Feb  3 12:27:19 2009
From: rmay31 at gmail.com (Ryan May)
Date: Tue, 03 Feb 2009 11:27:19 -0600
Subject: [Numpy-discussion] genloadtxt question
In-Reply-To: <D8DA6BB9-D908-467D-87B7-8171EF0BB5BF@gmail.com>
References: <49886FCA.50407@gmail.com>
	<D8DA6BB9-D908-467D-87B7-8171EF0BB5BF@gmail.com>
Message-ID: <49887E77.3050409@gmail.com>

Pierre GM wrote:
> On Feb 3, 2009, at 11:24 AM, Ryan May wrote:
> 
>> Pierre,
>>
>> Should the following work?
>>
>> import numpy as np
>> from StringIO import StringIO
>>
>> converter = {'date':lambda s: datetime.strptime(s,'%Y-%m-%d %H:%M: 
>> %SZ')}
>> data = np.ndfromtxt(StringIO('2009-02-03 12:00:00Z,72214.0'),  
>> delimiter=',',
>> names=['date','stid'], dtype=None, converters=converter)
> 
> Well, yes, it should work. That's indeed a problem with the  
> getsubdtype method of the converter.
> The problem is that we need to estimate the datatype of the output of  
> the converter. In most cases, trying to convert '0' works properly,  
> not in yours however. In r6338, I force the type to object if  
> converting '0' does not work. That's a patch till the next corner  
> case...

Thanks for the quick patch!  And yeah, I can't think of any better behavior. It's
 actually what I ended up doing in my conversion function, so, if nothing else,
it removes the user from having to write that kind of boilerplate code.

Thanks,

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From david.froger.info at gmail.com  Tue Feb  3 13:30:46 2009
From: david.froger.info at gmail.com (David Froger)
Date: Tue, 3 Feb 2009 19:30:46 +0100
Subject: [Numpy-discussion] example reading binary Fortran file
In-Reply-To: <200902031015.17490.faltet@pytables.org>
References: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>
	<6df541a5c8d6ecf26b6e38f404958401.squirrel@webmail.uio.no>
	<dcba39620901301453n21b9221di25c5267c8ec66bcf@mail.gmail.com>
	<200902031015.17490.faltet@pytables.org>
Message-ID: <dcba39620902031030u1c80bfa0u28e915dd80b963f3@mail.gmail.com>

Thanks a lot Fransesc and Neil, yours messages really help me.

I'll look at these solutions attentively.

Here is what I write recently, but I begin to understand it's effectively
not portable...

def fread(fileObject,*arrayAttributs):
    """ Reading in a binary (=unformatted) Fortran file

    Let's call 'record' the list of arrays written with one write in the
Fortran file.

    Call one fread per write.

    Parameter :
        * fileObject, eg: fileObject = open("data.bin",'rb')
        * arrayAttributs = ( (shape1,dtype1,readorskip1),
(shape2,dtype2,readorskip2) ...)
             * shape: eg: (100,200)
             * dtype: eg: '<f4' (little endian, simple precision), '>f8'
(big endian, double precision)
             * readorskip = [0|1]
                  * 1: the array is read and return
                  * 0: the size of the array is skipped to read the array
after, the array isn't returned

    Exemples (with write ux,uy,p in the Fortran code)
        * f = open("uxuyp.bin",'rb')
          nx,ny = 100,200
          p = readFortran( f, ((nx,ny),'<f8',0), ((nx,ny),'<f8',0),
((nx,ny),'<f8',1) )
          f.close
    """
    import numpy,os,utils

    # compute the recordSize to be read
    recordBytes = 8
    for (shape,dtype,read) in arrayAttributs:
        dtype = numpy.dtype(dtype)
        # number of elements to be read in this array
        count = 1
        for size in shape:
            count *= size
        # size of the record
        recordBytes += count*dtype.itemsize

    # the fileSize
    fileSize = os.stat(fileObject.name)[6]

    # compare recordSize and fileSize
    if recordBytes > fileSize:
        import logging
        logging.error('To much data to be read in %r',fileObject.name)
        logging.error('File Size: %r',fileSize)
        logging.error('To be read: %r',recordBytes)
        NoneList = []
        for (shape,dtype,read) in arrayAttributs:
            if read: NoneList.append(None)
        return NoneList

    # skip the four bytes in the beginning of the record
    fileObject.seek(4,1)

    # read the arrays in record
    arrays = []
    for (shape,dtype,read) in arrayAttributs:
        # number of elements to be read in this array
        count=1
        for size in shape:
            count *= size
        if read:
            array = numpy.fromfile(fileObject, count=count,
dtype=dtype).reshape(shape, order='F')
            arrays.append(array)
        else:
            dtype = numpy.dtype(dtype)
            arrayBytes = count*dtype.itemsize
            fileObject.seek(arrayBytes,1)

    # skip the four bytes at the end of the record
    fileObject.seek(4,1)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090203/6c355486/attachment.html>

From david.froger.info at gmail.com  Tue Feb  3 13:34:15 2009
From: david.froger.info at gmail.com (David Froger)
Date: Tue, 3 Feb 2009 19:34:15 +0100
Subject: [Numpy-discussion] example reading binary Fortran file
In-Reply-To: <dcba39620902031030u1c80bfa0u28e915dd80b963f3@mail.gmail.com>
References: <dcba39620901300803r63ff5843yafd0d01a33a535c8@mail.gmail.com>
	<6df541a5c8d6ecf26b6e38f404958401.squirrel@webmail.uio.no>
	<dcba39620901301453n21b9221di25c5267c8ec66bcf@mail.gmail.com>
	<200902031015.17490.faltet@pytables.org>
	<dcba39620902031030u1c80bfa0u28e915dd80b963f3@mail.gmail.com>
Message-ID: <dcba39620902031034u4e784c94sa7b0f33910c2e0d1@mail.gmail.com>

the last line was missing :

     return arrays
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090203/45c012e3/attachment.html>

From rmay31 at gmail.com  Tue Feb  3 15:54:37 2009
From: rmay31 at gmail.com (Ryan May)
Date: Tue, 03 Feb 2009 14:54:37 -0600
Subject: [Numpy-discussion] Operations on masked items
Message-ID: <4988AF0D.60504@gmail.com>

Pierre,

I know you did some preliminary work on helping to make sure that doing
operations on masked arrays doesn't change the underlying data.  I ran into the
following today.

import numpy as np
a = np.ma.array([1,2,3], mask=[False, True, False])
b = a * 10
c = 10 * a
print b.data # Prints [10 2 30] Good!
print c.data # Prints [10 10 30] Oops.

I tracked it down to __call__ on the _MaskedBinaryOperation class.  If there's a
mask on the data, you use:

	result = np.where(m, da, self.f(da, db, *args, **kwargs))

You can see that if a (and hence da) is a scalar, your masked values end up with
the value of the scalar.  If this is getting too hairy to handle not touching
data, I understand.  I just thought I should point out the inconsistency here.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From rmay31 at gmail.com  Tue Feb  3 16:00:28 2009
From: rmay31 at gmail.com (Ryan May)
Date: Tue, 03 Feb 2009 15:00:28 -0600
Subject: [Numpy-discussion] Operations on masked items
In-Reply-To: <4988AF0D.60504@gmail.com>
References: <4988AF0D.60504@gmail.com>
Message-ID: <4988B06C.3020504@gmail.com>

Ryan May wrote:
> Pierre,
> 
> I know you did some preliminary work on helping to make sure that doing
> operations on masked arrays doesn't change the underlying data.  I ran into the
> following today.
> 
> import numpy as np
> a = np.ma.array([1,2,3], mask=[False, True, False])
> b = a * 10
> c = 10 * a
> print b.data # Prints [10 2 30] Good!
> print c.data # Prints [10 10 30] Oops.
> 
> I tracked it down to __call__ on the _MaskedBinaryOperation class.  If there's a
> mask on the data, you use:
> 
> 	result = np.where(m, da, self.f(da, db, *args, **kwargs))
> 
> You can see that if a (and hence da) is a scalar, your masked values end up with
> the value of the scalar.  If this is getting too hairy to handle not touching
> data, I understand.  I just thought I should point out the inconsistency here.

Well, I guess I hit send too soon.  Here's one easy solution (consistent with
what you did for __radd__), change the code for __rmul__ to do:

	return multiply(self, other)

instead of:

	return multiply(other, self)

That fixes it for me, and I don't see how it would break anything.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From pgmdevlist at gmail.com  Tue Feb  3 16:19:16 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 3 Feb 2009 16:19:16 -0500
Subject: [Numpy-discussion] Operations on masked items
In-Reply-To: <4988B06C.3020504@gmail.com>
References: <4988AF0D.60504@gmail.com> <4988B06C.3020504@gmail.com>
Message-ID: <D8782881-F6EF-44BD-B197-15C72AABB163@gmail.com>


On Feb 3, 2009, at 4:00 PM, Ryan May wrote:
>
> Well, I guess I hit send too soon.  Here's one easy solution  
> (consistent with
> what you did for __radd__), change the code for __rmul__ to do:
>
> 	return multiply(self, other)
>
> instead of:
>
> 	return multiply(other, self)
>
> That fixes it for me, and I don't see how it would break anything.

Good call, but once again: "Thou shalt not put trust in ye masked  
values [1]".

 >>> a = np.ma.array([1,2,3],mask=[0,1,0])
 >>> b = np.ma.array([10, 20, 30], mask=[0,1,0])
 >>> (a*b).data
array([10,  2, 90])
 >>> (b*a).data
array([10, 20, 90])

So yes, __mul__ is not commutative when you deal w/ masked arrays (at  
least, when you try to access the data under a mask). Nothing I can  
do. Remember that preventing the underlying data to be modified is  
NEVER guaranteed...

[1] Epistle of Paul (Dubois).


From rmay31 at gmail.com  Tue Feb  3 18:00:42 2009
From: rmay31 at gmail.com (Ryan May)
Date: Tue, 03 Feb 2009 17:00:42 -0600
Subject: [Numpy-discussion] Operations on masked items
In-Reply-To: <D8782881-F6EF-44BD-B197-15C72AABB163@gmail.com>
References: <4988AF0D.60504@gmail.com> <4988B06C.3020504@gmail.com>
	<D8782881-F6EF-44BD-B197-15C72AABB163@gmail.com>
Message-ID: <4988CC9A.3060308@gmail.com>

Pierre GM wrote:
> On Feb 3, 2009, at 4:00 PM, Ryan May wrote:
>> Well, I guess I hit send too soon.  Here's one easy solution  
>> (consistent with
>> what you did for __radd__), change the code for __rmul__ to do:
>>
>> 	return multiply(self, other)
>>
>> instead of:
>>
>> 	return multiply(other, self)
>>
>> That fixes it for me, and I don't see how it would break anything.
> 
> Good call, but once again: "Thou shalt not put trust in ye masked  
> values [1]".
> 
>  >>> a = np.ma.array([1,2,3],mask=[0,1,0])
>  >>> b = np.ma.array([10, 20, 30], mask=[0,1,0])
>  >>> (a*b).data
> array([10,  2, 90])
>  >>> (b*a).data
> array([10, 20, 90])
> 
> So yes, __mul__ is not commutative when you deal w/ masked arrays (at  
> least, when you try to access the data under a mask). Nothing I can  
> do. Remember that preventing the underlying data to be modified is  
> NEVER guaranteed...

Fair enough.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From ellisonbg.net at gmail.com  Tue Feb  3 19:00:05 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 3 Feb 2009 16:00:05 -0800
Subject: [Numpy-discussion] Few minor issues with numscons
Message-ID: <6ce0ac130902031600m7a056b09rfd3069da3e8df028@mail.gmail.com>

David,

I am trying to use numscons to build a project and am running into
some problems:

Two smaller issues and one show stopper.  First, the smaller ones:

* The web presense of numscons is currently very confusing.  There are
a couple of locations with info about it, but the most prominent ones
appear to be quite outdated:

http://projects.scipy.org/scipy/numpy/wiki/NumScons

...refers to...

https://code.launchpad.net/numpy.scons.support

Which is no longer being used and doesn't have any links to the most
recent versions  I had to hunt for a while to find the develoment
repo, which is on github and the most recent release, which is now at
pypi.

It is probably a good idea to update these locations so as not to
confuse new folks (or old folks too).

* The scons/scons-local subdir is not installed when running python
setup.py install.  I had to use python setupegg.py to get this to
install in the right place.

I will send a different email in a second about the show stopper as it
is a bigger topic related to a numpy bug.

Cheers,

Brian


From ellisonbg.net at gmail.com  Tue Feb  3 19:12:52 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 3 Feb 2009 16:12:52 -0800
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
Message-ID: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>

I am trying to use numscons to build a project and have run into a
show stopper.  I am using:

OS X 10.5
The builtin Python 2.5.2

Here is what I see upon running python setup.py scons:

scons: Reading SConscript files ...
DistutilsPlatformError: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.3"
but "10.5" during configure:
  File "/Users/bgranger/Library/Python/2.5/src/numscons/tests/examples/checkers/SConstruct",
line 2:
    GetInitEnvironment(ARGUMENTS).DistutilsSConscript('SConscript')
  File "/Users/bgranger/Library/Python/2.5/site-packages/numscons-0.9.4-py2.5.egg/numscons/core/numpyenv.py",
line 108:
[this goes on for a while]

This bug is one that I am familiar with.  Here is a sketch:

* numpy.distutils sets MACOSX_DEPLOYMENT_TARGET=10.3 if
MACOSX_DEPLOYMENT_TARGET is not set in the environment.

* But, the built-in Python on OS X 10.5 has
MACOSX_DEPLOYMENT_TARGET=10.5.  When Python is built, it saves this
info in a file.

* When called distutils checks to make sure that the current value of
MACOSX_DEPLOYMENT_TARGET matches the one that was used to build
Python.

Hence the mismatch.  I am pretty sure that the offending code is in:

numpy.distutils.fcompiler.gnu.get_flags_linker_so

I think I know how to fix this and will get started on it, but I
wanted to see if anyone else had any experience with this or knew
another way around this.

Cheers,

Brian


From robert.kern at gmail.com  Tue Feb  3 19:17:34 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 3 Feb 2009 18:17:34 -0600
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
Message-ID: <3d375d730902031617t43726c8as3b9e8d46583f2152@mail.gmail.com>

On Tue, Feb 3, 2009 at 18:12, Brian Granger <ellisonbg.net at gmail.com> wrote:
> I am trying to use numscons to build a project and have run into a
> show stopper.  I am using:
>
> OS X 10.5
> The builtin Python 2.5.2
>
> Here is what I see upon running python setup.py scons:
>
> scons: Reading SConscript files ...
> DistutilsPlatformError: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.3"
> but "10.5" during configure:
>  File "/Users/bgranger/Library/Python/2.5/src/numscons/tests/examples/checkers/SConstruct",
> line 2:
>    GetInitEnvironment(ARGUMENTS).DistutilsSConscript('SConscript')
>  File "/Users/bgranger/Library/Python/2.5/site-packages/numscons-0.9.4-py2.5.egg/numscons/core/numpyenv.py",
> line 108:
> [this goes on for a while]
>
> This bug is one that I am familiar with.  Here is a sketch:
>
> * numpy.distutils sets MACOSX_DEPLOYMENT_TARGET=10.3 if
> MACOSX_DEPLOYMENT_TARGET is not set in the environment.
>
> * But, the built-in Python on OS X 10.5 has
> MACOSX_DEPLOYMENT_TARGET=10.5.  When Python is built, it saves this
> info in a file.
>
> * When called distutils checks to make sure that the current value of
> MACOSX_DEPLOYMENT_TARGET matches the one that was used to build
> Python.
>
> Hence the mismatch.  I am pretty sure that the offending code is in:
>
> numpy.distutils.fcompiler.gnu.get_flags_linker_so
>
> I think I know how to fix this and will get started on it, but I
> wanted to see if anyone else had any experience with this or knew
> another way around this.

Well, the workaround is to set MACOSX_DEPLOYMENT_TARGET=10.5 in your
environment.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ellisonbg.net at gmail.com  Tue Feb  3 19:20:54 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 3 Feb 2009 16:20:54 -0800
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <3d375d730902031617t43726c8as3b9e8d46583f2152@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
	<3d375d730902031617t43726c8as3b9e8d46583f2152@mail.gmail.com>
Message-ID: <6ce0ac130902031620t782b2387hbd2de5e8e0ced580@mail.gmail.com>

Robert,

Thanks.

Yes, I just saw that this will work.  When I fixed this in Cython a
while back this workaround wouldn't work.  Would you still consider
this a bug?  The logic to fix it is fairly simply.

Brian

On Tue, Feb 3, 2009 at 4:17 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Tue, Feb 3, 2009 at 18:12, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> I am trying to use numscons to build a project and have run into a
>> show stopper.  I am using:
>>
>> OS X 10.5
>> The builtin Python 2.5.2
>>
>> Here is what I see upon running python setup.py scons:
>>
>> scons: Reading SConscript files ...
>> DistutilsPlatformError: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.3"
>> but "10.5" during configure:
>>  File "/Users/bgranger/Library/Python/2.5/src/numscons/tests/examples/checkers/SConstruct",
>> line 2:
>>    GetInitEnvironment(ARGUMENTS).DistutilsSConscript('SConscript')
>>  File "/Users/bgranger/Library/Python/2.5/site-packages/numscons-0.9.4-py2.5.egg/numscons/core/numpyenv.py",
>> line 108:
>> [this goes on for a while]
>>
>> This bug is one that I am familiar with.  Here is a sketch:
>>
>> * numpy.distutils sets MACOSX_DEPLOYMENT_TARGET=10.3 if
>> MACOSX_DEPLOYMENT_TARGET is not set in the environment.
>>
>> * But, the built-in Python on OS X 10.5 has
>> MACOSX_DEPLOYMENT_TARGET=10.5.  When Python is built, it saves this
>> info in a file.
>>
>> * When called distutils checks to make sure that the current value of
>> MACOSX_DEPLOYMENT_TARGET matches the one that was used to build
>> Python.
>>
>> Hence the mismatch.  I am pretty sure that the offending code is in:
>>
>> numpy.distutils.fcompiler.gnu.get_flags_linker_so
>>
>> I think I know how to fix this and will get started on it, but I
>> wanted to see if anyone else had any experience with this or knew
>> another way around this.
>
> Well, the workaround is to set MACOSX_DEPLOYMENT_TARGET=10.5 in your
> environment.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From robert.kern at gmail.com  Tue Feb  3 19:23:42 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 3 Feb 2009 18:23:42 -0600
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <6ce0ac130902031620t782b2387hbd2de5e8e0ced580@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
	<3d375d730902031617t43726c8as3b9e8d46583f2152@mail.gmail.com>
	<6ce0ac130902031620t782b2387hbd2de5e8e0ced580@mail.gmail.com>
Message-ID: <3d375d730902031623p6171dd09gbcc9bf75a793c8b1@mail.gmail.com>

On Tue, Feb 3, 2009 at 18:20, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Robert,
>
> Thanks.
>
> Yes, I just saw that this will work.  When I fixed this in Cython a
> while back this workaround wouldn't work.  Would you still consider
> this a bug?  The logic to fix it is fairly simply.

What is the fix you are thinking of?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From Catherine.M.Moroney at jpl.nasa.gov  Tue Feb  3 19:27:12 2009
From: Catherine.M.Moroney at jpl.nasa.gov (Catherine Moroney)
Date: Tue, 3 Feb 2009 16:27:12 -0800
Subject: [Numpy-discussion] reading binary Fortran
Message-ID: <8674E2D1-D119-4885-9686-36EC666CE9EF@jpl.nasa.gov>

I've noticed a lot of discussion on how to read binary files
written out from Fortran, and nobody seems to have mentioned
how to modify your Fortran code so it writes out a file that
can be read with numpy.fromfile() in a single line.

For example, to write out a NLINE x NSMP array of floats in Fortran
in a line by line fashion, the code below works just fine.

open(iunit,file=flname,form='unformatted',access='direct', &
      recl=nsmp*4,status='replace',action='write')
do iline=1,nline
    write(iunit,rec=iline) (data(iline,ismp),ismp=1,nsmp)
end do
close(iunit)

It's more code than simply "write(iunit) data", but it has the advantage
of being easily imported into python, matlab and other software packages
that can read flat binary data.  To import a file written out in
this fashion into numpy, a single call to numpy.fromfile(flname)
works with no need to define datatypes or the like.

Apologies if I'm missing some reason why the above doesn't work or
is not preferable to the "write(iunit) data".

Catherine




From ellisonbg.net at gmail.com  Tue Feb  3 19:34:48 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 3 Feb 2009 16:34:48 -0800
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <3d375d730902031623p6171dd09gbcc9bf75a793c8b1@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
	<3d375d730902031617t43726c8as3b9e8d46583f2152@mail.gmail.com>
	<6ce0ac130902031620t782b2387hbd2de5e8e0ced580@mail.gmail.com>
	<3d375d730902031623p6171dd09gbcc9bf75a793c8b1@mail.gmail.com>
Message-ID: <6ce0ac130902031634ye40a829gbac28740fa281231@mail.gmail.com>

> What is the fix you are thinking of?

This is how Cython currently handles this logic.  This would have to
be modified to include the additional case of a user setting
MACOSX_DEPLOYMENT_TARGET in their environment, but that logic is
already in numpy.distutils.fcompiler.gnu.get_flags_linker_so

This is really just a special case for 1) OS X 10.5 and 2) built-in Python.

# MACOSX_DEPLOYMENT_TARGET can be set to 10.3 in most cases.
# But for the built-in Python 2.5.1 on Leopard, it needs to be set for 10.5.
# This looks like a bug that will be fixed in 2.5.2. If Apple updates their
# Python to 2.5.2, this fix should be OK.
import distutils.sysconfig as sc
python_prefix = sc.get_config_var('prefix')
leopard_python_prefix =
'/System/Library/Frameworks/Python.framework/Versions/2.5'
full_version = "%s.%s.%s" % sys.version_info[:3]
if python_prefix == leopard_python_prefix and full_version == '2.5.1':
    os.environ["MACOSX_DEPLOYMENT_TARGET"] = "10.5"
else:
    os.environ["MACOSX_DEPLOYMENT_TARGET"] = "10.3"


From robert.kern at gmail.com  Tue Feb  3 19:42:31 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 3 Feb 2009 18:42:31 -0600
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <6ce0ac130902031634ye40a829gbac28740fa281231@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
	<3d375d730902031617t43726c8as3b9e8d46583f2152@mail.gmail.com>
	<6ce0ac130902031620t782b2387hbd2de5e8e0ced580@mail.gmail.com>
	<3d375d730902031623p6171dd09gbcc9bf75a793c8b1@mail.gmail.com>
	<6ce0ac130902031634ye40a829gbac28740fa281231@mail.gmail.com>
Message-ID: <3d375d730902031642o65baa95ciebde9b5cc8f8b800@mail.gmail.com>

On Tue, Feb 3, 2009 at 18:34, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> What is the fix you are thinking of?
>
> This is how Cython currently handles this logic.  This would have to
> be modified to include the additional case of a user setting
> MACOSX_DEPLOYMENT_TARGET in their environment, but that logic is
> already in numpy.distutils.fcompiler.gnu.get_flags_linker_so
>
> This is really just a special case for 1) OS X 10.5 and 2) built-in Python.
>
> # MACOSX_DEPLOYMENT_TARGET can be set to 10.3 in most cases.
> # But for the built-in Python 2.5.1 on Leopard, it needs to be set for 10.5.
> # This looks like a bug that will be fixed in 2.5.2. If Apple updates their
> # Python to 2.5.2, this fix should be OK.
> import distutils.sysconfig as sc
> python_prefix = sc.get_config_var('prefix')
> leopard_python_prefix =
> '/System/Library/Frameworks/Python.framework/Versions/2.5'
> full_version = "%s.%s.%s" % sys.version_info[:3]
> if python_prefix == leopard_python_prefix and full_version == '2.5.1':
>    os.environ["MACOSX_DEPLOYMENT_TARGET"] = "10.5"
> else:
>    os.environ["MACOSX_DEPLOYMENT_TARGET"] = "10.3"

Hmm, that's still going to break for any custom build that decides to
build Python with a specific MACOSX_DEPLOYMENT_TARGET. If you're going
to fix it at all, it should default to the value in the Makefile that
sysconfig is going to check against. The relevant code to copy is in
sysconfig._init_posix().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ellisonbg.net at gmail.com  Tue Feb  3 19:53:14 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 3 Feb 2009 16:53:14 -0800
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <3d375d730902031642o65baa95ciebde9b5cc8f8b800@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
	<3d375d730902031617t43726c8as3b9e8d46583f2152@mail.gmail.com>
	<6ce0ac130902031620t782b2387hbd2de5e8e0ced580@mail.gmail.com>
	<3d375d730902031623p6171dd09gbcc9bf75a793c8b1@mail.gmail.com>
	<6ce0ac130902031634ye40a829gbac28740fa281231@mail.gmail.com>
	<3d375d730902031642o65baa95ciebde9b5cc8f8b800@mail.gmail.com>
Message-ID: <6ce0ac130902031653x2a304265t41b3c4f70bc257ef@mail.gmail.com>

> Hmm, that's still going to break for any custom build that decides to
> build Python with a specific MACOSX_DEPLOYMENT_TARGET. If you're going
> to fix it at all, it should default to the value in the Makefile that
> sysconfig is going to check against. The relevant code to copy is in
> sysconfig._init_posix().

Yes, I agree that sysconfig._init_posix() has the proper logic for
this.  This logic should also be applied to Cython as well probably.

Would you say that the proper fix then is to inspect the Makefile and
set MACOSX_DEPLOYMENT_TARGET to the valued used to build Python
itself.  Or should we still try to set it to 10.3 in some cases (like
the current numpy.distutils does) or look at the environment as well?

Cheers,

Brian


From robert.kern at gmail.com  Tue Feb  3 19:58:51 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 3 Feb 2009 18:58:51 -0600
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <6ce0ac130902031653x2a304265t41b3c4f70bc257ef@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
	<3d375d730902031617t43726c8as3b9e8d46583f2152@mail.gmail.com>
	<6ce0ac130902031620t782b2387hbd2de5e8e0ced580@mail.gmail.com>
	<3d375d730902031623p6171dd09gbcc9bf75a793c8b1@mail.gmail.com>
	<6ce0ac130902031634ye40a829gbac28740fa281231@mail.gmail.com>
	<3d375d730902031642o65baa95ciebde9b5cc8f8b800@mail.gmail.com>
	<6ce0ac130902031653x2a304265t41b3c4f70bc257ef@mail.gmail.com>
Message-ID: <3d375d730902031658l2b727ad2t3cfbaff32865e013@mail.gmail.com>

On Tue, Feb 3, 2009 at 18:53, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> Hmm, that's still going to break for any custom build that decides to
>> build Python with a specific MACOSX_DEPLOYMENT_TARGET. If you're going
>> to fix it at all, it should default to the value in the Makefile that
>> sysconfig is going to check against. The relevant code to copy is in
>> sysconfig._init_posix().
>
> Yes, I agree that sysconfig._init_posix() has the proper logic for
> this.  This logic should also be applied to Cython as well probably.
>
> Would you say that the proper fix then is to inspect the Makefile and
> set MACOSX_DEPLOYMENT_TARGET to the valued used to build Python
> itself.  Or should we still try to set it to 10.3 in some cases (like
> the current numpy.distutils does) or look at the environment as well?

1) Trust the environment variable if given and let distutils raise its
error message (why not raise it ourselves? distutils' error message
and explanation is already out in THE GOOGLE.)

2) Otherwise, use the value in the Makefile if it's there.

3) If it's not even in the Makefile for whatever reason, go with 10.3.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Tue Feb  3 20:09:52 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 3 Feb 2009 18:09:52 -0700
Subject: [Numpy-discussion] array vector elementwise multiplication
In-Reply-To: <6A3525EC-A389-4DDF-A68C-BB8455417916@math.toronto.edu>
References: <6A3525EC-A389-4DDF-A68C-BB8455417916@math.toronto.edu>
Message-ID: <e06186140902031709t3df29ddetf659a5797ef7a4c1@mail.gmail.com>

On Tue, Feb 3, 2009 at 10:19 AM, Gideon Simpson <simpson at math.toronto.edu>wrote:

> I have an M x N matrix A and two vectors, an M dimensional vector x
> and an N dimensional vector y.  I would like to be able to do two
> things.
>
> 1.  Multiply, elementwise, every column of A by x
>
> 2.  Multiply, elementwise, every row of A by y.
>
> What's the "quick" way to do this in numpy?
>

In [1]: M = ones((3,3))

In [2]: x = arange(3)

In [3]: M*x
Out[3]:
array([[ 0.,  1.,  2.],
       [ 0.,  1.,  2.],
       [ 0.,  1.,  2.]])

In [4]: x[:,newaxis]*M
Out[4]:
array([[ 0.,  0.,  0.],
       [ 1.,  1.,  1.],
       [ 2.,  2.,  2.]])

 Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090203/940a466d/attachment.html>

From cournape at gmail.com  Tue Feb  3 20:13:02 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 4 Feb 2009 10:13:02 +0900
Subject: [Numpy-discussion] Few minor issues with numscons
In-Reply-To: <6ce0ac130902031600m7a056b09rfd3069da3e8df028@mail.gmail.com>
References: <6ce0ac130902031600m7a056b09rfd3069da3e8df028@mail.gmail.com>
Message-ID: <5b8d13220902031713n68524e14g1608ca9156972d7d@mail.gmail.com>

Hi Brian,

On Wed, Feb 4, 2009 at 9:00 AM, Brian Granger <ellisonbg.net at gmail.com> wrote:
> David,
>
> I am trying to use numscons to build a project and am running into
> some problems:
>
> Two smaller issues and one show stopper.  First, the smaller ones:
>
> * The web presense of numscons is currently very confusing.  There are
> a couple of locations with info about it, but the most prominent ones
> appear to be quite outdated:
>
> http://projects.scipy.org/scipy/numpy/wiki/NumScons
>
> ...refers to...
>
> https://code.launchpad.net/numpy.scons.support
>
> Which is no longer being used and doesn't have any links to the most
> recent versions  I had to hunt for a while to find the develoment
> repo, which is on github and the most recent release, which is now at
> pypi.

The releases are on Pypi for quite some time. I converted the repo to
git and put it on github, but I have not really worked on numscons for
several months now for lack of time ( and because numscons it mostly
"done" and the main limitations of numscons are not fixable without
fixing some fairly major scons limitations).

Basically, the repo on github is only the convertion from bzr, without
any new features.

> It is probably a good idea to update these locations so as not to
> confuse new folks (or old folks too).
>
> * The scons/scons-local subdir is not installed when running python
> setup.py install.  I had to use python setupegg.py to get this to
> install in the right place.

That's strange, you are not the first one to mention this bug, but I
never had this problem myself - I myself never use setupegg except
when creating eggs for pypi.

cheers,

David


From cournape at gmail.com  Tue Feb  3 21:04:25 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 4 Feb 2009 11:04:25 +0900
Subject: [Numpy-discussion] Numpy 1.3 release date ?
In-Reply-To: <D955A90A-2121-4D34-A8AE-99292D86FC7F@gmail.com>
References: <D955A90A-2121-4D34-A8AE-99292D86FC7F@gmail.com>
Message-ID: <5b8d13220902031804w6cb1bf98q68a5ae627700076a@mail.gmail.com>

On Tue, Feb 3, 2009 at 11:49 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> All,
> When can we expect numpy 1.3 to be released ?

Looking at the log from 1.2.x, the main committers were Charles
Harris, you and me for (that makes ~ 80 % of the commits) for *numpy*.
Then, there is the doc itself, which has seen major work I believe,
although I have not followed much.

Talking about the things I have worked on for 1.3, I did not have the
time to finish everything. The things which I would like to be done
before a 1.3.0 release are:
 - fix formatting issues that Pauli and me worked on (for locale
independence and all): I believe it is mostly a matter of merging the
changes in the trunk, and testing. Pauli, do you agree ?
 - make sure 1.3.0 builds and runs fine on 2.6, in particular on windows.
 - see if one can support windows x64.

I think official 2.6 support (with binaries for the platforms where we
support binaries), x64 support and everything which has been done
already would be enough to make a release.

David


From ellisonbg.net at gmail.com  Wed Feb  4 00:21:48 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 3 Feb 2009 21:21:48 -0800
Subject: [Numpy-discussion] Few minor issues with numscons
In-Reply-To: <5b8d13220902031713n68524e14g1608ca9156972d7d@mail.gmail.com>
References: <6ce0ac130902031600m7a056b09rfd3069da3e8df028@mail.gmail.com>
	<5b8d13220902031713n68524e14g1608ca9156972d7d@mail.gmail.com>
Message-ID: <6ce0ac130902032121kf6dc4b1rba48401d5e5ba3@mail.gmail.com>

> The releases are on Pypi for quite some time. I converted the repo to
> git and put it on github, but I have not really worked on numscons for
> several months now for lack of time ( and because numscons it mostly
> "done" and the main limitations of numscons are not fixable without
> fixing some fairly major scons limitations).
>
> Basically, the repo on github is only the convertion from bzr, without
> any new features.

Do you have plans to continue to maintain it though?

One other things I forgot to mention.  I first tried the head of your
git repo and numpy complained that the version of numscons (0.10) was
too *old*.  It wanted the version to be greater than somehting like
0.9.1, and clearly it was, so it looks like there is a bug in the
numscons version parsing in numpy.distutils.

>> * The scons/scons-local subdir is not installed when running python
>> setup.py install.  I had to use python setupegg.py to get this to
>> install in the right place.
>
> That's strange, you are not the first one to mention this bug, but I
> never had this problem myself - I myself never use setupegg except
> when creating eggs for pypi.

OK, I will have a look at this further to see what the cause is.

Thanks for a great package that solve a really painful set of issues!

Brian

> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From ellisonbg.net at gmail.com  Wed Feb  4 00:22:25 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 3 Feb 2009 21:22:25 -0800
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <3d375d730902031658l2b727ad2t3cfbaff32865e013@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
	<3d375d730902031617t43726c8as3b9e8d46583f2152@mail.gmail.com>
	<6ce0ac130902031620t782b2387hbd2de5e8e0ced580@mail.gmail.com>
	<3d375d730902031623p6171dd09gbcc9bf75a793c8b1@mail.gmail.com>
	<6ce0ac130902031634ye40a829gbac28740fa281231@mail.gmail.com>
	<3d375d730902031642o65baa95ciebde9b5cc8f8b800@mail.gmail.com>
	<6ce0ac130902031653x2a304265t41b3c4f70bc257ef@mail.gmail.com>
	<3d375d730902031658l2b727ad2t3cfbaff32865e013@mail.gmail.com>
Message-ID: <6ce0ac130902032122i1ba31be3y9c2d034ee0a1d2fc@mail.gmail.com>

> 1) Trust the environment variable if given and let distutils raise its
> error message (why not raise it ourselves? distutils' error message
> and explanation is already out in THE GOOGLE.)
>
> 2) Otherwise, use the value in the Makefile if it's there.
>
> 3) If it's not even in the Makefile for whatever reason, go with 10.3.

Sounds good, do you want to me work up a patch?

Brian

> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From robert.kern at gmail.com  Wed Feb  4 00:32:01 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 3 Feb 2009 23:32:01 -0600
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <6ce0ac130902032122i1ba31be3y9c2d034ee0a1d2fc@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
	<3d375d730902031617t43726c8as3b9e8d46583f2152@mail.gmail.com>
	<6ce0ac130902031620t782b2387hbd2de5e8e0ced580@mail.gmail.com>
	<3d375d730902031623p6171dd09gbcc9bf75a793c8b1@mail.gmail.com>
	<6ce0ac130902031634ye40a829gbac28740fa281231@mail.gmail.com>
	<3d375d730902031642o65baa95ciebde9b5cc8f8b800@mail.gmail.com>
	<6ce0ac130902031653x2a304265t41b3c4f70bc257ef@mail.gmail.com>
	<3d375d730902031658l2b727ad2t3cfbaff32865e013@mail.gmail.com>
	<6ce0ac130902032122i1ba31be3y9c2d034ee0a1d2fc@mail.gmail.com>
Message-ID: <3d375d730902032132l33417bedia7dacf08f4e7996e@mail.gmail.com>

On Tue, Feb 3, 2009 at 23:22, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> 1) Trust the environment variable if given and let distutils raise its
>> error message (why not raise it ourselves? distutils' error message
>> and explanation is already out in THE GOOGLE.)
>>
>> 2) Otherwise, use the value in the Makefile if it's there.
>>
>> 3) If it's not even in the Makefile for whatever reason, go with 10.3.
>
> Sounds good, do you want to me work up a patch?

Yes, please.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From scott.sinclair.za at gmail.com  Wed Feb  4 01:06:31 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Wed, 4 Feb 2009 08:06:31 +0200
Subject: [Numpy-discussion] Numpy 1.3 release date ?
In-Reply-To: <5b8d13220902031804w6cb1bf98q68a5ae627700076a@mail.gmail.com>
References: <D955A90A-2121-4D34-A8AE-99292D86FC7F@gmail.com>
	<5b8d13220902031804w6cb1bf98q68a5ae627700076a@mail.gmail.com>
Message-ID: <6a17e9ee0902032206n2c0f4d30o70dcb02e18a4aa6d@mail.gmail.com>

> 2009/2/4 David Cournapeau <cournape at gmail.com>:
> On Tue, Feb 3, 2009 at 11:49 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
>> All,
>> When can we expect numpy 1.3 to be released ?
>
> I think official 2.6 support (with binaries for the platforms where we
> support binaries), x64 support and everything which has been done
> already would be enough to make a release.

There are a bunch of documentation patches that should to be reviewed
and applied to SVN before the release (especially those marked 'Needs
review' or better).

http://docs.scipy.org/numpy/patch/

Cheers,
Scott


From nadavh at visionsense.com  Wed Feb  4 05:55:06 2009
From: nadavh at visionsense.com (Nadav Horesh)
Date: Wed, 04 Feb 2009 12:55:06 +0200
Subject: [Numpy-discussion] Error building numpy documentation
Message-ID: <1233744906.25601.9.camel@nadav.envision.co.il>

I just dowloads the latest numpy's svn version and tried to build its
documentation with

$ make latex 

on the doc subdirectory, and got the following error message:

writing... Sphinx error:
too many nesting section levels for LaTeX, at heading:
numpy.ma.MaskedArray.__lt__
make: *** [latex] Error 1

Machine: 64 bit gentoo linux running texlive2007

Any ideas?

  Nadav


From pav at iki.fi  Wed Feb  4 04:04:38 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 4 Feb 2009 09:04:38 +0000 (UTC)
Subject: [Numpy-discussion] Numpy 1.3 release date ?
References: <D955A90A-2121-4D34-A8AE-99292D86FC7F@gmail.com>
	<5b8d13220902031804w6cb1bf98q68a5ae627700076a@mail.gmail.com>
Message-ID: <gmbln6$ic6$1@ger.gmane.org>

Wed, 04 Feb 2009 11:04:25 +0900, David Cournapeau wrote:
[clip]
> Talking about the things I have worked on for 1.3, I did not have the
> time to finish everything. The things which I would like to be done
> before a 1.3.0 release are:
>  - fix formatting issues that Pauli and me worked on (for locale
> independence and all): I believe it is mostly a matter of merging the
> changes in the trunk, and testing. Pauli, do you agree ?

Agree, I think it should be OK, and IIRC tests passed also on Windows 
(well, Wine to be accurate). I don't recall anything more that would be 
required to do.

It might be nice also to address this wart: 
http://scipy.org/scipy/numpy/ticket/883 for Numpy 1.3. But it's a 
separate issue from the locale and formatting problems.

Pauli



From david at ar.media.kyoto-u.ac.jp  Wed Feb  4 03:57:53 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 04 Feb 2009 17:57:53 +0900
Subject: [Numpy-discussion] Few minor issues with numscons
In-Reply-To: <6ce0ac130902032121kf6dc4b1rba48401d5e5ba3@mail.gmail.com>
References: <6ce0ac130902031600m7a056b09rfd3069da3e8df028@mail.gmail.com>	<5b8d13220902031713n68524e14g1608ca9156972d7d@mail.gmail.com>
	<6ce0ac130902032121kf6dc4b1rba48401d5e5ba3@mail.gmail.com>
Message-ID: <49895891.5020705@ar.media.kyoto-u.ac.jp>

Brian Granger wrote:
>> The releases are on Pypi for quite some time. I converted the repo to
>> git and put it on github, but I have not really worked on numscons for
>> several months now for lack of time ( and because numscons it mostly
>> "done" and the main limitations of numscons are not fixable without
>> fixing some fairly major scons limitations).
>>
>> Basically, the repo on github is only the convertion from bzr, without
>> any new features.
>>     
>
> Do you have plans to continue to maintain it though?
>   

I can't say for sure; I am still quite interested in working on
build/distributions problems in numpy and more generally python. But in
my mind, this work only really makes sense if it becomes integrated to
numpy.distutils and even can be used by arbitrary python extension as
some point. And that's just not realistic at this point because of some
fundamental scons limitations (I would like scons to be callable from a
pure python scripts, as a library). So I spent quite some time on scons
itself  (and more recently waf, which started as a scons fork).

There is also the fundamental time problem, but we all are in the same
boat on this one :)

> One other things I forgot to mention.  I first tried the head of your
> git repo and numpy complained that the version of numscons (0.10) was
> too *old*.  It wanted the version to be greater than somehting like
> 0.9.1, and clearly it was, so it looks like there is a bug in the
> numscons version parsing in numpy.distutils.
>   

Can be a stupid bug in the version check. It should be ok with numpy
trunk, though.

David


From william at resolversystems.com  Wed Feb  4 05:11:21 2009
From: william at resolversystems.com (William Reade)
Date: Wed, 04 Feb 2009 10:11:21 +0000
Subject: [Numpy-discussion] Resolver One -- .NET spreadsheet with NumPy --
	beta testers?
Message-ID: <498969C9.1070306@resolversystems.com>

Hi all

We're about to release the first public beta of Resolver One, our 
Pythonic spreadsheet, with (rather basic) NumPy integration. It requires 
32-bit Windows and .NET 2.0SP1 to run, so it may not be appropriate for 
everyone, but -- if anyone is interested in playing with it -- it would 
be really useful to have some real live NumPy users telling us what we 
could improve. If anyone here would like to try it out, please let me 
know, and I'll arrange beta access.

Cheers
William
http://resolversystems.com/


From scott.sinclair.za at gmail.com  Wed Feb  4 05:15:14 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Wed, 4 Feb 2009 12:15:14 +0200
Subject: [Numpy-discussion] Error building numpy documentation
In-Reply-To: <1233744906.25601.9.camel@nadav.envision.co.il>
References: <1233744906.25601.9.camel@nadav.envision.co.il>
Message-ID: <6a17e9ee0902040215g619e605fkb1d6a401484c854c@mail.gmail.com>

> 2009/2/4 Nadav Horesh <nadavh at visionsense.com>:
> I just dowloads the latest numpy's svn version and tried to build its
> documentation with
>
> $ make latex
>
> on the doc subdirectory, and got the following error message:
>
> writing... Sphinx error:
> too many nesting section levels for LaTeX, at heading:
> numpy.ma.MaskedArray.__lt__
> make: *** [latex] Error 1
>
> Machine: 64 bit gentoo linux running texlive2007
>
> Any ideas?

No ideas, but this has been reported before:

http://projects.scipy.org/pipermail/numpy-discussion/2009-January/039917.html

I've filed a ticket:

http://scipy.org/scipy/numpy/ticket/998

Cheers,
Scott


From david at ar.media.kyoto-u.ac.jp  Wed Feb  4 05:22:56 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 04 Feb 2009 19:22:56 +0900
Subject: [Numpy-discussion] Numpy 1.3 release date ?
In-Reply-To: <6a17e9ee0902032206n2c0f4d30o70dcb02e18a4aa6d@mail.gmail.com>
References: <D955A90A-2121-4D34-A8AE-99292D86FC7F@gmail.com>	<5b8d13220902031804w6cb1bf98q68a5ae627700076a@mail.gmail.com>
	<6a17e9ee0902032206n2c0f4d30o70dcb02e18a4aa6d@mail.gmail.com>
Message-ID: <49896C80.9070904@ar.media.kyoto-u.ac.jp>

Scott Sinclair wrote:
>
> There are a bunch of documentation patches that should to be reviewed
> and applied to SVN before the release (especially those marked 'Needs
> review' or better).
>
> http://docs.scipy.org/numpy/patch/

In my mind, documentations fixes are much less important than code, not
because documentation matters less, but because it can be handled at the
last moment much more easily - there is little chance that a doc change
breaks on windows only, for example.

David


From scott.sinclair.za at gmail.com  Wed Feb  4 06:04:11 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Wed, 4 Feb 2009 13:04:11 +0200
Subject: [Numpy-discussion] Numpy 1.3 release date ?
In-Reply-To: <49896C80.9070904@ar.media.kyoto-u.ac.jp>
References: <D955A90A-2121-4D34-A8AE-99292D86FC7F@gmail.com>
	<5b8d13220902031804w6cb1bf98q68a5ae627700076a@mail.gmail.com>
	<6a17e9ee0902032206n2c0f4d30o70dcb02e18a4aa6d@mail.gmail.com>
	<49896C80.9070904@ar.media.kyoto-u.ac.jp>
Message-ID: <6a17e9ee0902040304o35b696cfgc75ba7ba3005dc5a@mail.gmail.com>

> 2009/2/4 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
> Scott Sinclair wrote:
>>
>> There are a bunch of documentation patches that should to be reviewed
>> and applied to SVN before the release (especially those marked 'Needs
>> review' or better).
>>
>> http://docs.scipy.org/numpy/patch/
>
> In my mind, documentations fixes are much less important than code, not
> because documentation matters less, but because it can be handled at the
> last moment much more easily - there is little chance that a doc change
> breaks on windows only, for example.

Sure. Just trying to encourage a few more reviews and documentation
contributions.

Cheers,
Scott


From josef.pktd at gmail.com  Wed Feb  4 11:00:36 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 4 Feb 2009 11:00:36 -0500
Subject: [Numpy-discussion] poly1d left versus right multiplication with np
	numbers
Message-ID: <1cd32cbb0902040800j4436b68dl690c90f7cf1e4c2@mail.gmail.com>

I just had a hard to find bug in my program. poly1d  treats numpy
scalars differently than python numbers when left or right
multiplication is used.

Essentially, if the first term is the numpy scalar, multiplied by a
polynomial, then the result is an np.array.
If the order is reversed, then the result is an instance of np.poly1d.
The return types are also the same for numpy arrays, which is at least
understandable, although a warning would be good)

When using plain (python) numbers, then both left and right
multiplication of the number with the polynomial returns a polynomial.

Is this a bug or a feature? I didn't see it mentioned in the docs.

My problem for debugging was that in the examples I used python
numbers while the program used numpy scalars, and it took me a while
to figure out that this is the source of my bugs.

examples below

Josef



>>> polys
[poly1d([1]), poly1d([-1.,  0.]), poly1d([ 1.,  0., -1.]), poly1d([
1.,  0., -3.,  0.]), poly1d([ 1.,  0., -6.,  0.,  3.])]

np.array on left is fine
>>> (polys[2]*np.array(0.5/6.0) + polys[3]*np.array(0.5/24.0))
poly1d([ 0.02083333,  0.08333333, -0.0625    , -0.08333333])
>>> (polys[2]*0.5/6.0 + polys[3]*0.5/24.0)
poly1d([ 0.02083333,  0.08333333, -0.0625    , -0.08333333])
>>> (0.5/6.0*polys[2] + 0.5/24.0*polys[3])
poly1d([ 0.02083333,  0.08333333, -0.0625    , -0.08333333])

problems with np.array on left

>>> (np.array(0.5/6.0)*polys[2] + np.array(0.5/24.0)*polys[3])
Traceback (most recent call last):
  File "<pyshell#722>", line 1, in <module>
    (np.array(0.5/6.0)*polys[2] + np.array(0.5/24.0)*polys[3])
ValueError: shape mismatch: objects cannot be broadcast to a single shape
>>> np.array(0.5/6.0)*polys[2]
array([ 0.08333333,  0.        , -0.08333333])
>>> polys[2]*np.array(0.5/6.0)
poly1d([ 0.08333333,  0.        , -0.08333333])
>>> 0.5/6.0*polys[2]
poly1d([ 0.08333333,  0.        , -0.08333333])
>>> np.array(0.5/6.0)
array(0.083333333333333329)

same with numpy scalar

>>> np.array([0.5/6.0])[0]*polys[2]
array([ 0.08333333,  0.        , -0.08333333])
>>> np.array([0.5/6.0])[0]
0.083333333333333329


From pgmdevlist at gmail.com  Wed Feb  4 11:19:38 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 4 Feb 2009 11:19:38 -0500
Subject: [Numpy-discussion] poly1d left versus right multiplication with
	np numbers
In-Reply-To: <1cd32cbb0902040800j4436b68dl690c90f7cf1e4c2@mail.gmail.com>
References: <1cd32cbb0902040800j4436b68dl690c90f7cf1e4c2@mail.gmail.com>
Message-ID: <5164FDE2-A811-4FE4-A370-5939F85C7268@gmail.com>


On Feb 4, 2009, at 11:00 AM, josef.pktd at gmail.com wrote:

> I just had a hard to find bug in my program. poly1d  treats numpy
> scalars differently than python numbers when left or right
> multiplication is used.
>
> Essentially, if the first term is the numpy scalar, multiplied by a
> polynomial, then the result is an np.array.
> If the order is reversed, then the result is an instance of np.poly1d.
> The return types are also the same for numpy arrays, which is at least
> understandable, although a warning would be good)
>
> When using plain (python) numbers, then both left and right
> multiplication of the number with the polynomial returns a polynomial.
>
> Is this a bug or a feature? I didn't see it mentioned in the docs.

Looks like yet another example of ticket #826:
http://scipy.org/scipy/numpy/ticket/826
This one is getting quite a problem, and I have no idea how to fix  
it... 


From josef.pktd at gmail.com  Wed Feb  4 11:44:11 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 4 Feb 2009 11:44:11 -0500
Subject: [Numpy-discussion] poly1d left versus right multiplication with
	np numbers
In-Reply-To: <5164FDE2-A811-4FE4-A370-5939F85C7268@gmail.com>
References: <1cd32cbb0902040800j4436b68dl690c90f7cf1e4c2@mail.gmail.com>
	<5164FDE2-A811-4FE4-A370-5939F85C7268@gmail.com>
Message-ID: <1cd32cbb0902040844u6305466ej182831b573e893e2@mail.gmail.com>

On Wed, Feb 4, 2009 at 11:19 AM, Pierre GM <pgmdevlist at gmail.com> wrote:
>
> On Feb 4, 2009, at 11:00 AM, josef.pktd at gmail.com wrote:
>
>> I just had a hard to find bug in my program. poly1d  treats numpy
>> scalars differently than python numbers when left or right
>> multiplication is used.
>>
>> Essentially, if the first term is the numpy scalar, multiplied by a
>> polynomial, then the result is an np.array.
>> If the order is reversed, then the result is an instance of np.poly1d.
>> The return types are also the same for numpy arrays, which is at least
>> understandable, although a warning would be good)
>>
>> When using plain (python) numbers, then both left and right
>> multiplication of the number with the polynomial returns a polynomial.
>>
>> Is this a bug or a feature? I didn't see it mentioned in the docs.
>
> Looks like yet another example of ticket #826:
> http://scipy.org/scipy/numpy/ticket/826
> This one is getting quite a problem, and I have no idea how to fix
> it...

Thanks, yes it looks exactly like this ticket. At least, once I know
about it, it is not too difficult to work around, but it costs a lot
of debugging time to figure this out.

Josef


From jh at physics.ucf.edu  Wed Feb  4 12:01:45 2009
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Wed, 04 Feb 2009 12:01:45 -0500
Subject: [Numpy-discussion] Numpy 1.3 release date ?
In-Reply-To: <mailman.5249.1233742520.2878.numpy-discussion@scipy.org>
	(numpy-discussion-request@scipy.org)
References: <mailman.5249.1233742520.2878.numpy-discussion@scipy.org>
Message-ID: <wl8r62e2l1i.fsf@glup.physics.ucf.edu>

Scott Sinclair <scott.sinclair.za at gmail.com> wrote:
>> 2009/2/4 David Cournapeau <cournape at gmail.com>:
>> On Tue, Feb 3, 2009 at 11:49 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
>>> All,
>>> When can we expect numpy 1.3 to be released ?
>>
>> I think official 2.6 support (with binaries for the platforms where we
>> support binaries), x64 support and everything which has been done
>> already would be enough to make a release.

I'd like to see a self-consistent and reasonably recent numpy/scipy
make the Ubuntu 9.04 FeatureFreeze deadline of 19 February.  Is this
possible?  One of the reasons we had discussed doing more frequent
releases was to get consistent and recent packages in the mainstream
Linuxes, and Ubuntu is very popular in our community.

>There are a bunch of documentation patches that should to be reviewed
>and applied to SVN before the release (especially those marked 'Needs
>review' or better).

>http://docs.scipy.org/numpy/patch/

At this point, we're focussing on getting semi-decent drafts of as
many docstrings as possible rather than on polishing the ones we have
as drafts, on the basis of "something is always better than nothing as
long as it's not wrong".  So, please don't hold up a release for
review, just get as many drafts in as possible.

Also, it has become clear that we need separate technical and writing
reviews, as a number of pages marked as "reviewed" are deficient in
one or the other area, probably the one that wasn't the strength of
the reviewer.  That entails discussion and revamping of the site,
which can happen once we have a larger percentage of the docstrings in
draft form.  (Any discussion on the merits of this should take place
on scipy-dev, which is the home of doc discussions.)

--jh--


From watson.jim at gmail.com  Wed Feb  4 12:02:54 2009
From: watson.jim at gmail.com (James Watson)
Date: Wed, 4 Feb 2009 17:02:54 +0000
Subject: [Numpy-discussion] porting NumPy to Python 3
Message-ID: <f14846790902040902w57983eaav83eb2de2d7a19b86@mail.gmail.com>

Hi Ondrej,

To get 2to3 to run without warnings, the following files require minor changes:
- numpy/distutils/fcompiler/intel.py
- numpy/distutils/misc_util.py
- numpy/distutils/command/build_src.py
- numpy/f2py/crackfortran.py
- numpy/lib/function_base.py
- numpy/linalg/lapack_lite/make_lite.py

There are also other (possibly many, still working on this) files that
require syntactic changes to run post 2to3.

Is there anywhere specific I should upload these changes?  Is this
list appropriate?  Is there a developer I can send these and future
patches to?

James.



> Date: Tue, 3 Feb 2009 01:03:15 -0800
> From: Ondrej Certik <ondrej at certik.cz>
> Subject: Re: [Numpy-discussion] porting NumPy to Python 3
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Message-ID:
>        <85b5c3130902030103w69180a6bm4143050b4b82bf1f at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> Hi James,
>
> On Thu, Jan 29, 2009 at 2:11 AM, James Watson <watson.jim at gmail.com> wrote:
>> Hi,
>>
>> I am interested in contributing to the port of NumPy to Python 3.  Who
>> I should coordinate effort with?
>>
>> I have started at the Python end of the problem (as opposed to
>> http://www.scipy.org/Python3k), e.g. I have several patches to get
>> 2to3 to work on NumPy's Python source code.
>
> I am sorry that noone has replied to your email. Could you please
> upload your patches somewhere?
>
> Ondrej


From bpederse at gmail.com  Wed Feb  4 12:09:51 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 4 Feb 2009 09:09:51 -0800
Subject: [Numpy-discussion] genfromtxt view with object dtype
Message-ID: <e183a99d0902040909g638c330bke4e07a46ddbd6cca@mail.gmail.com>

hi, i am using genfromtxt, with a dtype like this:
[('seqid', '|S24'), ('source', '|S16'), ('type', '|S16'), ('start',
'<i4'), ('end', '<i4'), ('score', '<f8'), ('strand', '|S1'), ('phase',
'<i4'), ('attrs', '|O4')]

where i'm having problems with the attrs column which i'd like to be a
dict. i can specify a convertor to parse a string into a dict, and it
is correctly converted to a dict,
but then in io.py it tries to take a view() of that dtype and it gives
the error:

    A = np.genfromtxt(fname, **kwargs)
  File "/usr/lib/python2.5/site-packages/numpy/lib/io.py", line 922,
in genfromtxt
    output = rows.view(dtype)
 TypeError: Cannot change data-type for object array.


is there anyway around this or must that col be kept as a string?
it seems like genfromtxt expects you to specify either a dtype _or_ a
convertor, not both.

thanks,
-brent


From pgmdevlist at gmail.com  Wed Feb  4 12:36:41 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 4 Feb 2009 12:36:41 -0500
Subject: [Numpy-discussion] genfromtxt view with object dtype
In-Reply-To: <e183a99d0902040909g638c330bke4e07a46ddbd6cca@mail.gmail.com>
References: <e183a99d0902040909g638c330bke4e07a46ddbd6cca@mail.gmail.com>
Message-ID: <1C0B2F62-23C4-47CE-BC31-863629FAE543@gmail.com>


On Feb 4, 2009, at 12:09 PM, Brent Pedersen wrote:

> hi, i am using genfromtxt, with a dtype like this:
> [('seqid', '|S24'), ('source', '|S16'), ('type', '|S16'), ('start',
> '<i4'), ('end', '<i4'), ('score', '<f8'), ('strand', '|S1'), ('phase',
> '<i4'), ('attrs', '|O4')]

Brent,
Please post a simple, self-contained example with a few lines of the  
file you want to load.



From charlesr.harris at gmail.com  Wed Feb  4 12:44:56 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 4 Feb 2009 10:44:56 -0700
Subject: [Numpy-discussion] porting NumPy to Python 3
In-Reply-To: <f14846790902040902w57983eaav83eb2de2d7a19b86@mail.gmail.com>
References: <f14846790902040902w57983eaav83eb2de2d7a19b86@mail.gmail.com>
Message-ID: <e06186140902040944w1c71cc05ib954b6bc04a26fec@mail.gmail.com>

On Wed, Feb 4, 2009 at 10:02 AM, James Watson <watson.jim at gmail.com> wrote:

> Hi Ondrej,
>
> To get 2to3 to run without warnings, the following files require minor
> changes:
> - numpy/distutils/fcompiler/intel.py
> - numpy/distutils/misc_util.py
> - numpy/distutils/command/build_src.py
> - numpy/f2py/crackfortran.py
> - numpy/lib/function_base.py
> - numpy/linalg/lapack_lite/make_lite.py
>
> There are also other (possibly many, still working on this) files that
> require syntactic changes to run post 2to3.
>
> Is there anywhere specific I should upload these changes?  Is this
> list appropriate?  Is there a developer I can send these and future
> patches to?
>

Maybe we could add a python 3 milestone so folks could post tickets for
these sorts of things. For the time being you could just post to the list
here with the patches and something in the subject line that identifies it
as a patch for python 3. I think we could make any small changes as long as
they are compatible with python 2.4.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090204/029be6cc/attachment.html>

From charlesr.harris at gmail.com  Wed Feb  4 12:48:31 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 4 Feb 2009 10:48:31 -0700
Subject: [Numpy-discussion] Numpy 1.3 release date ?
In-Reply-To: <wl8r62e2l1i.fsf@glup.physics.ucf.edu>
References: <mailman.5249.1233742520.2878.numpy-discussion@scipy.org>
	<wl8r62e2l1i.fsf@glup.physics.ucf.edu>
Message-ID: <e06186140902040948m1b45bb32i15e49fb0470ad695@mail.gmail.com>

On Wed, Feb 4, 2009 at 10:01 AM, <jh at physics.ucf.edu> wrote:

> Scott Sinclair <scott.sinclair.za at gmail.com> wrote:
> >> 2009/2/4 David Cournapeau <cournape at gmail.com>:
> >> On Tue, Feb 3, 2009 at 11:49 PM, Pierre GM <pgmdevlist at gmail.com>
> wrote:
> >>> All,
> >>> When can we expect numpy 1.3 to be released ?
> >>
> >> I think official 2.6 support (with binaries for the platforms where we
> >> support binaries), x64 support and everything which has been done
> >> already would be enough to make a release.
>
> I'd like to see a self-consistent and reasonably recent numpy/scipy
> make the Ubuntu 9.04 FeatureFreeze deadline of 19 February.  Is this
> possible?  One of the reasons we had discussed doing more frequent
> releases was to get consistent and recent packages in the mainstream
> Linuxes, and Ubuntu is very popular in our community.
>

That deadline is pretty close ;) There has been a dearth of developer time
these last few months, what with work, holidays, and dissertations, and I
don't know when that is going to get better.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090204/7f3d90ea/attachment.html>

From bsouthey at gmail.com  Wed Feb  4 13:13:43 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Wed, 04 Feb 2009 12:13:43 -0600
Subject: [Numpy-discussion] porting NumPy to Python 3
In-Reply-To: <e06186140902040944w1c71cc05ib954b6bc04a26fec@mail.gmail.com>
References: <f14846790902040902w57983eaav83eb2de2d7a19b86@mail.gmail.com>
	<e06186140902040944w1c71cc05ib954b6bc04a26fec@mail.gmail.com>
Message-ID: <4989DAD7.9060507@gmail.com>

Charles R Harris wrote:
>
>
> On Wed, Feb 4, 2009 at 10:02 AM, James Watson <watson.jim at gmail.com 
> <mailto:watson.jim at gmail.com>> wrote:
>
>     Hi Ondrej,
>
>     To get 2to3 to run without warnings, the following files require
>     minor changes:
>     - numpy/distutils/fcompiler/intel.py
>     - numpy/distutils/misc_util.py
>     - numpy/distutils/command/build_src.py
>     - numpy/f2py/crackfortran.py
>     - numpy/lib/function_base.py
>     - numpy/linalg/lapack_lite/make_lite.py
>
>     There are also other (possibly many, still working on this) files that
>     require syntactic changes to run post 2to3.
>
>     Is there anywhere specific I should upload these changes?  Is this
>     list appropriate?  Is there a developer I can send these and future
>     patches to?
>
>
> Maybe we could add a python 3 milestone so folks could post tickets 
> for these sorts of things. For the time being you could just post to 
> the list here with the patches and something in the subject line that 
> identifies it as a patch for python 3. I think we could make any small 
> changes as long as they are compatible with python 2.4.
>
> Chuck
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   
If you follow Guido's recommendation (at the bottom of 
http://docs.python.org/dev/3.0/whatsnew/3.0.html), then we should first 
be compatible with Python 2.6 (about the current stage) . Then 
compatible using Python 2.3 with the -3 flag (warn about Python 3.x 
incompatibilities) and fix warnings like:
 numpy/core/__init__.py:11: DeprecationWarning: the cPickle module has 
been removed in Python 3.0
(Of course you lose speed of cPickle if you blindly change it to use the 
old Pickle in Python 2.x)

Finally use the 2to3 tool to get a Python 3 version.

Bruce




From bsouthey at gmail.com  Wed Feb  4 15:40:24 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Wed, 04 Feb 2009 14:40:24 -0600
Subject: [Numpy-discussion] ImportError: No module named dateutil.parser
Message-ID: <4989FD38.7080009@gmail.com>

Hi,
I just updated to the latest SVN but I get a failure when running the 
tests due to missing dateutil.parser. Should this module exist in Numpy 
or just a inappropriate test?

Bruce

 >>> numpy.test()
Running unit tests for numpy
NumPy version 1.3.0.dev6338
NumPy is installed in /usr/lib64/python2.5/site-packages/numpy
Python version 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) [GCC 4.3.2 
20080917 (Red Hat 4.3.2-4)]
nose version 0.10.3
.......................................................................................................................................................................................................................................................KKKKKKKK..................................................................................................................................................................................................K.....................................................................................................................................................E........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
======================================================================
ERROR: Tests updatemapper
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/usr/lib64/python2.5/site-packages/numpy/lib/tests/test__iotools.py", 
line 133, in test_upgrademapper
    import dateutil.parser
ImportError: No module named dateutil.parser

----------------------------------------------------------------------
Ran 1888 tests in 6.084s

FAILED (KNOWNFAIL=9, errors=1)
<nose.result.TextTestResult run=1888 errors=1 failures=0>




From robert.kern at gmail.com  Wed Feb  4 15:48:31 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 4 Feb 2009 14:48:31 -0600
Subject: [Numpy-discussion] ImportError: No module named dateutil.parser
In-Reply-To: <4989FD38.7080009@gmail.com>
References: <4989FD38.7080009@gmail.com>
Message-ID: <3d375d730902041248l256d1402wc2a684aaa9d60db8@mail.gmail.com>

On Wed, Feb 4, 2009 at 14:40, Bruce Southey <bsouthey at gmail.com> wrote:
> Hi,
> I just updated to the latest SVN but I get a failure when running the
> tests due to missing dateutil.parser. Should this module exist in Numpy
> or just a inappropriate test?

It's a bad test.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From pgmdevlist at gmail.com  Wed Feb  4 15:54:38 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 4 Feb 2009 15:54:38 -0500
Subject: [Numpy-discussion] ImportError: No module named dateutil.parser
In-Reply-To: <4989FD38.7080009@gmail.com>
References: <4989FD38.7080009@gmail.com>
Message-ID: <3433B34A-1F97-40E5-B37C-379A778604EE@gmail.com>


On Feb 4, 2009, at 3:40 PM, Bruce Southey wrote:

> Hi,
> I just updated to the latest SVN but I get a failure when running the
> tests due to missing dateutil.parser. Should this module exist in  
> Numpy
> or just a inappropriate test?

I put the corresponding tests in a try/except ImportError block  
(r6339), that should fix it.


From robert.kern at gmail.com  Wed Feb  4 15:56:24 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 4 Feb 2009 14:56:24 -0600
Subject: [Numpy-discussion] ImportError: No module named dateutil.parser
In-Reply-To: <3433B34A-1F97-40E5-B37C-379A778604EE@gmail.com>
References: <4989FD38.7080009@gmail.com>
	<3433B34A-1F97-40E5-B37C-379A778604EE@gmail.com>
Message-ID: <3d375d730902041256i323140cre3e41ab19075553f@mail.gmail.com>

On Wed, Feb 4, 2009 at 14:54, Pierre GM <pgmdevlist at gmail.com> wrote:
>
> On Feb 4, 2009, at 3:40 PM, Bruce Southey wrote:
>
>> Hi,
>> I just updated to the latest SVN but I get a failure when running the
>> tests due to missing dateutil.parser. Should this module exist in
>> Numpy
>> or just a inappropriate test?
>
> I put the corresponding tests in a try/except ImportError block
> (r6339), that should fix it.

No, rewrite the test to not use external libraries, please. Test the
functionality without needing dateutils.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From bpederse at gmail.com  Wed Feb  4 16:03:16 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 4 Feb 2009 13:03:16 -0800
Subject: [Numpy-discussion] genfromtxt view with object dtype
In-Reply-To: <1C0B2F62-23C4-47CE-BC31-863629FAE543@gmail.com>
References: <e183a99d0902040909g638c330bke4e07a46ddbd6cca@mail.gmail.com>
	<1C0B2F62-23C4-47CE-BC31-863629FAE543@gmail.com>
Message-ID: <e183a99d0902041303v5699bf74i2c96709d40c6f174@mail.gmail.com>

On Wed, Feb 4, 2009 at 9:36 AM, Pierre GM <pgmdevlist at gmail.com> wrote:
>
> On Feb 4, 2009, at 12:09 PM, Brent Pedersen wrote:
>
>> hi, i am using genfromtxt, with a dtype like this:
>> [('seqid', '|S24'), ('source', '|S16'), ('type', '|S16'), ('start',
>> '<i4'), ('end', '<i4'), ('score', '<f8'), ('strand', '|S1'), ('phase',
>> '<i4'), ('attrs', '|O4')]
>
> Brent,
> Please post a simple, self-contained example with a few lines of the
> file you want to load.
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

hi pierre, here is an example.
thanks,
-brent

######################

import numpy as np
from cStringIO import StringIO

gffstr = """\
##gff-version 3
1\tucb\tgene\t2234602\t2234702\t.\t-\t.\tID=grape_1_2234602_2234702;match=EVM_prediction_supercontig_1.248,EVM_prediction_supercontig_1.248.mRNA
1\tucb\tgene\t2300292\t2302123\t.\t+\t.\tID=grape_1_2300292_2302123;match=EVM_prediction_supercontig_244.8
1\tucb\tgene\t2303615\t2303967\t.\t+\t.\tID=grape_1_2303615_2303967;match=EVM_prediction_supercontig_244.8
1\tucb\tgene\t2303616\t2303966\t.\t+\t.\tParent=grape_1_2303615_2303967
1\tucb\tgene\t3596400\t3596503\t.\t-\t.\tID=grape_1_3596400_3596503;match=evm.TU.supercontig_167.27
1\tucb\tgene\t3600651\t3600977\t.\t-\t.\tmatch=evm.model.supercontig_1217.1,evm.model.supercontig_1217.1.mRNA
"""

dtype = {'names' :
                  ('seqid', 'source', 'type', 'start', 'end',
                    'score', 'strand', 'phase', 'attrs') ,
        'formats':
                  ['S24', 'S16', 'S16', 'i4', 'i4', 'f8',
                      'S1', 'i4', 'S128']}

#OK with S128 for attrs
print np.genfromtxt(StringIO(gffstr), dtype = dtype)



def _attr(kvstr):
    pairs = [kv.split("=") for kv in kvstr.split(";")]
    return dict(pairs)

# change S128 to object to have col attrs as dictionary
dtype['formats'][-1] = 'O'
converters = {8: _attr }
#NOT OK
print np.genfromtxt(StringIO(gffstr), dtype = dtype, converters=converters)


From simon.palmer at gmail.com  Wed Feb  4 16:18:33 2009
From: simon.palmer at gmail.com (Simon Palmer)
Date: Wed, 4 Feb 2009 13:18:33 -0800 (PST)
Subject: [Numpy-discussion]  Multiplying a matrix by a vector
Message-ID: <21839828.post@talk.nabble.com>


Bit of a newb question I suspect...

I have a matrix and a vector which has the same number of elements as the
matrix has rows.  I want to multiply each element in a row in the matrix by
the corresponding element in the vector.  I can obviously do this with a
loop, but am guessing there is a more elegant (and faster?) solution.  

Just for clarity...
Matrix
[X1, X2, X3]
[Y1, Y2, Y3]

Vector
[A, B]

Desired result
[X1*A, X2*A, X3*A]
[Y1*B, Y2*B, Y3*B]
-- 
View this message in context: http://www.nabble.com/Multiplying-a-matrix-by-a-vector-tp21839828p21839828.html
Sent from the Numpy-discussion mailing list archive at Nabble.com.



From h5py at alfven.org  Wed Feb  4 16:22:38 2009
From: h5py at alfven.org (Andrew Collette)
Date: Wed, 4 Feb 2009 13:22:38 -0800
Subject: [Numpy-discussion] Array dtype problems
Message-ID: <f125f5e0902041322j26030b45k52c5e48cdc92341c@mail.gmail.com>

Hello,

I'm having an issue with 'array' dtypes; if I do this:

mydtype = numpy.dtype(('i', (4,)))
arr = numpy.empty((100,), dtype=mydtype)

then the array datatype shape is "absorbed" into the shape of the
array.  This simplifies indexing, but is really annoying as it means
that the dtype doesn't "round trip"; I can't even use use astype as it
complains about a shape mismatch.  How can I create an array in a
manner that preserves the dtype array information?  Or is there a way
to take an existing array of the correct shape and "re-cast" it to use
an array dtype?

Thanks,
Andrew Collette


From pgmdevlist at gmail.com  Wed Feb  4 17:12:51 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 4 Feb 2009 17:12:51 -0500
Subject: [Numpy-discussion] ImportError: No module named dateutil.parser
In-Reply-To: <3d375d730902041256i323140cre3e41ab19075553f@mail.gmail.com>
References: <4989FD38.7080009@gmail.com>
	<3433B34A-1F97-40E5-B37C-379A778604EE@gmail.com>
	<3d375d730902041256i323140cre3e41ab19075553f@mail.gmail.com>
Message-ID: <5348E9DF-067B-4D6E-ABE4-9D36F35442BF@gmail.com>


On Feb 4, 2009, at 3:56 PM, Robert Kern wrote:

>
> No, rewrite the test to not use external libraries, please. Test the
> functionality without needing dateutils.

OK then, should be fixed in r6340.


From robert.kern at gmail.com  Wed Feb  4 17:15:21 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 4 Feb 2009 16:15:21 -0600
Subject: [Numpy-discussion] ImportError: No module named dateutil.parser
In-Reply-To: <5348E9DF-067B-4D6E-ABE4-9D36F35442BF@gmail.com>
References: <4989FD38.7080009@gmail.com>
	<3433B34A-1F97-40E5-B37C-379A778604EE@gmail.com>
	<3d375d730902041256i323140cre3e41ab19075553f@mail.gmail.com>
	<5348E9DF-067B-4D6E-ABE4-9D36F35442BF@gmail.com>
Message-ID: <3d375d730902041415p2fbaen9d1182671afce01b@mail.gmail.com>

On Wed, Feb 4, 2009 at 16:12, Pierre GM <pgmdevlist at gmail.com> wrote:
>
> On Feb 4, 2009, at 3:56 PM, Robert Kern wrote:
>
>> No, rewrite the test to not use external libraries, please. Test the
>> functionality without needing dateutils.
>
> OK then, should be fixed in r6340.

Thank you!

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Wed Feb  4 17:48:49 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 04 Feb 2009 14:48:49 -0800
Subject: [Numpy-discussion] Multiplying a matrix by a vector
In-Reply-To: <21839828.post@talk.nabble.com>
References: <21839828.post@talk.nabble.com>
Message-ID: <498A1B51.8030609@noaa.gov>

Simon Palmer wrote:
> I have a matrix and a vector which has the same number of elements as the
> matrix has rows.  I want to multiply each element in a row in the matrix by
> the corresponding element in the vector.

 >>> M = np.arange(6).reshape((2,3))
 >>> M
array([[0, 1, 2],
        [3, 4, 5]])
 >>> v = np.array((4,5)).reshape((-1,1)) # make it a column vector
 >>> v
array([[4],
        [5]])
 >>> M * v
array([[ 0,  4,  8],
        [15, 20, 25]])


you can also do it with np.newaxis:

 >>> v = np.array((4,5))
 >>> M * v[:,np.newaxis]
array([[ 0,  4,  8],
        [15, 20, 25]])



http://www.scipy.org/EricsBroadcastingDoc


If you are really working with a matrix, rather than a 2-d array, you 
may want to look at the np.matrix object.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From pgmdevlist at gmail.com  Wed Feb  4 17:50:38 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 4 Feb 2009 17:50:38 -0500
Subject: [Numpy-discussion] Renaming a field of an object array
Message-ID: <864EDE43-CC49-469F-A2C8-65D947CC0B51@gmail.com>

All,
I'm a tad puzzled by the following behavior (I'm trying to correct a  
bug in genfromtxt):

I'm creating an empty structured ndarray, using np.object as dtype.

 >>> a = np.empty(1,dtype=[('',np.object)])
array([(None,)],
       dtype=[('f0', '|O4')])

Now, I'd like to rename the field:
 >>> a.view([('NAME',np.object)])
TypeError: Cannot change data-type for object array.

I understand why I can't change the *type* of the field, but not why I  
can't change its name that way. What would be an option that wouldn't  
involve creating a new array ?
Thx in advance.



From bpederse at gmail.com  Wed Feb  4 18:25:17 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 4 Feb 2009 15:25:17 -0800
Subject: [Numpy-discussion] Renaming a field of an object array
In-Reply-To: <864EDE43-CC49-469F-A2C8-65D947CC0B51@gmail.com>
References: <864EDE43-CC49-469F-A2C8-65D947CC0B51@gmail.com>
Message-ID: <e183a99d0902041525h414047c3r2604f5c3994c93ad@mail.gmail.com>

On Wed, Feb 4, 2009 at 2:50 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> All,
> I'm a tad puzzled by the following behavior (I'm trying to correct a
> bug in genfromtxt):
>
> I'm creating an empty structured ndarray, using np.object as dtype.
>
>  >>> a = np.empty(1,dtype=[('',np.object)])
> array([(None,)],
>       dtype=[('f0', '|O4')])
>
> Now, I'd like to rename the field:
>  >>> a.view([('NAME',np.object)])
> TypeError: Cannot change data-type for object array.
>
> I understand why I can't change the *type* of the field, but not why I
> can't change its name that way. What would be an option that wouldn't
> involve creating a new array ?
> Thx in advance.
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

hi, i was looking at this as well. the code in arrayobject.c doesnt
match the error string. i changed the code to do what the error string
says and seems thing to work.
i think the if-block below it should also use xor (not changed in this
patch), but i'm not a c programmer so i may be missing something
obvious.

svn diff numpy/core/src/arrayobject.c
Index: numpy/core/src/arrayobject.c
===================================================================
--- numpy/core/src/arrayobject.c        (revision 6338)
+++ numpy/core/src/arrayobject.c        (working copy)
@@ -6506,9 +6506,16 @@
         PyErr_SetString(PyExc_TypeError, "invalid data-type for array");
         return -1;
     }
-    if (PyDataType_FLAGCHK(newtype, NPY_ITEM_HASOBJECT) ||
-        PyDataType_FLAGCHK(newtype, NPY_ITEM_IS_POINTER) ||
-        PyDataType_FLAGCHK(self->descr, NPY_ITEM_HASOBJECT) ||
+    if (PyDataType_FLAGCHK(newtype, NPY_ITEM_HASOBJECT) ^
+            PyDataType_FLAGCHK(self->descr, NPY_ITEM_HASOBJECT)) {
+        PyErr_SetString(PyExc_TypeError,                      \
+                        "Cannot change data-type for object " \
+                        "array.");
+        Py_DECREF(newtype);
+        return -1;
+    }
+
+    if (PyDataType_FLAGCHK(newtype, NPY_ITEM_IS_POINTER) ||
         PyDataType_FLAGCHK(self->descr, NPY_ITEM_IS_POINTER)) {
         PyErr_SetString(PyExc_TypeError,                      \
                         "Cannot change data-type for object " \


From cournape at gmail.com  Wed Feb  4 21:40:02 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 5 Feb 2009 11:40:02 +0900
Subject: [Numpy-discussion] Numpy 1.3 release date ?
In-Reply-To: <e06186140902040948m1b45bb32i15e49fb0470ad695@mail.gmail.com>
References: <mailman.5249.1233742520.2878.numpy-discussion@scipy.org>
	<wl8r62e2l1i.fsf@glup.physics.ucf.edu>
	<e06186140902040948m1b45bb32i15e49fb0470ad695@mail.gmail.com>
Message-ID: <5b8d13220902041840y796c5037ubc82aa431735037f@mail.gmail.com>

On Thu, Feb 5, 2009 at 2:48 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Wed, Feb 4, 2009 at 10:01 AM, <jh at physics.ucf.edu> wrote:
>>
>> Scott Sinclair <scott.sinclair.za at gmail.com> wrote:
>> >> 2009/2/4 David Cournapeau <cournape at gmail.com>:
>> >> On Tue, Feb 3, 2009 at 11:49 PM, Pierre GM <pgmdevlist at gmail.com>
>> >> wrote:
>> >>> All,
>> >>> When can we expect numpy 1.3 to be released ?
>> >>
>> >> I think official 2.6 support (with binaries for the platforms where we
>> >> support binaries), x64 support and everything which has been done
>> >> already would be enough to make a release.
>>
>> I'd like to see a self-consistent and reasonably recent numpy/scipy
>> make the Ubuntu 9.04 FeatureFreeze deadline of 19 February.  Is this
>> possible?  One of the reasons we had discussed doing more frequent
>> releases was to get consistent and recent packages in the mainstream
>> Linuxes, and Ubuntu is very popular in our community.
>
> That deadline is pretty close ;)

I think it is safe to say it will not be possible to release numpy for
that date.

Concerning Ubuntu, it sounds more realistic to make sure it is updated
to 1.2.1, and if possible to release 0.7.0.

David


From pgmdevlist at gmail.com  Wed Feb  4 23:51:34 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 4 Feb 2009 23:51:34 -0500
Subject: [Numpy-discussion] genfromtxt view with object dtype
In-Reply-To: <e183a99d0902041303v5699bf74i2c96709d40c6f174@mail.gmail.com>
References: <e183a99d0902040909g638c330bke4e07a46ddbd6cca@mail.gmail.com>
	<1C0B2F62-23C4-47CE-BC31-863629FAE543@gmail.com>
	<e183a99d0902041303v5699bf74i2c96709d40c6f174@mail.gmail.com>
Message-ID: <CEF9B734-51D7-4BDF-837E-F4995A2DEE8C@gmail.com>

OK, Brent, try r6341.
I fixed genfromtxt for cases like yours (explicit dtype involving a  
np.object).
Note that the fix won't work if the dtype is nested and involves  
np.objects (as we would hit the pb of renaming fields we observed...).
Let me know how it goes.
P.

On Feb 4, 2009, at 4:03 PM, Brent Pedersen wrote:

> On Wed, Feb 4, 2009 at 9:36 AM, Pierre GM <pgmdevlist at gmail.com>  
> wrote:
>>
>> On Feb 4, 2009, at 12:09 PM, Brent Pedersen wrote:
>>
>>> hi, i am using genfromtxt, with a dtype like this:
>>> [('seqid', '|S24'), ('source', '|S16'), ('type', '|S16'), ('start',
>>> '<i4'), ('end', '<i4'), ('score', '<f8'), ('strand', '|S1'),  
>>> ('phase',
>>> '<i4'), ('attrs', '|O4')]
>>
>> Brent,
>> Please post a simple, self-contained example with a few lines of the
>> file you want to load.
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>
> hi pierre, here is an example.
> thanks,
> -brent
>
> ######################
>
> import numpy as np
> from cStringIO import StringIO
>
> gffstr = """\
> ##gff-version 3
> 1\tucb\tgene\t2234602\t2234702\t.\t-\t. 
> \tID 
> = 
> grape_1_2234602_2234702 
> ;match 
> = 
> EVM_prediction_supercontig_1.248,EVM_prediction_supercontig_1.248.mRNA
> 1\tucb\tgene\t2300292\t2302123\t.\t+\t. 
> \tID=grape_1_2300292_2302123;match=EVM_prediction_supercontig_244.8
> 1\tucb\tgene\t2303615\t2303967\t.\t+\t. 
> \tID=grape_1_2303615_2303967;match=EVM_prediction_supercontig_244.8
> 1\tucb\tgene\t2303616\t2303966\t.\t+\t. 
> \tParent=grape_1_2303615_2303967
> 1\tucb\tgene\t3596400\t3596503\t.\t-\t. 
> \tID=grape_1_3596400_3596503;match=evm.TU.supercontig_167.27
> 1\tucb\tgene\t3600651\t3600977\t.\t-\t. 
> \tmatch=evm.model.supercontig_1217.1,evm.model.supercontig_1217.1.mRNA
> """
>
> dtype = {'names' :
>                  ('seqid', 'source', 'type', 'start', 'end',
>                    'score', 'strand', 'phase', 'attrs') ,
>        'formats':
>                  ['S24', 'S16', 'S16', 'i4', 'i4', 'f8',
>                      'S1', 'i4', 'S128']}
>
> #OK with S128 for attrs
> print np.genfromtxt(StringIO(gffstr), dtype = dtype)
>
>
>
> def _attr(kvstr):
>    pairs = [kv.split("=") for kv in kvstr.split(";")]
>    return dict(pairs)
>
> # change S128 to object to have col attrs as dictionary
> dtype['formats'][-1] = 'O'
> converters = {8: _attr }
> #NOT OK
> print np.genfromtxt(StringIO(gffstr), dtype = dtype,  
> converters=converters)
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From bpederse at gmail.com  Thu Feb  5 00:22:31 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 4 Feb 2009 21:22:31 -0800
Subject: [Numpy-discussion] genfromtxt view with object dtype
In-Reply-To: <CEF9B734-51D7-4BDF-837E-F4995A2DEE8C@gmail.com>
References: <e183a99d0902040909g638c330bke4e07a46ddbd6cca@mail.gmail.com>
	<1C0B2F62-23C4-47CE-BC31-863629FAE543@gmail.com>
	<e183a99d0902041303v5699bf74i2c96709d40c6f174@mail.gmail.com>
	<CEF9B734-51D7-4BDF-837E-F4995A2DEE8C@gmail.com>
Message-ID: <e183a99d0902042122r568e63bft74972269fe644e83@mail.gmail.com>

On Wed, Feb 4, 2009 at 8:51 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> OK, Brent, try r6341.
> I fixed genfromtxt for cases like yours (explicit dtype involving a
> np.object).
> Note that the fix won't work if the dtype is nested and involves
> np.objects (as we would hit the pb of renaming fields we observed...).
> Let me know how it goes.
> P.
>

that fixes it. thanks again pierre!
-b




> On Feb 4, 2009, at 4:03 PM, Brent Pedersen wrote:
>
>> On Wed, Feb 4, 2009 at 9:36 AM, Pierre GM <pgmdevlist at gmail.com>
>> wrote:
>>>
>>> On Feb 4, 2009, at 12:09 PM, Brent Pedersen wrote:
>>>
>>>> hi, i am using genfromtxt, with a dtype like this:
>>>> [('seqid', '|S24'), ('source', '|S16'), ('type', '|S16'), ('start',
>>>> '<i4'), ('end', '<i4'), ('score', '<f8'), ('strand', '|S1'),
>>>> ('phase',
>>>> '<i4'), ('attrs', '|O4')]
>>>
>>> Brent,
>>> Please post a simple, self-contained example with a few lines of the
>>> file you want to load.
>>>
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Numpy-discussion at scipy.org
>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>> hi pierre, here is an example.
>> thanks,
>> -brent
>>
>> ######################
>>
>> import numpy as np
>> from cStringIO import StringIO
>>
>> gffstr = """\
>> ##gff-version 3
>> 1\tucb\tgene\t2234602\t2234702\t.\t-\t.
>> \tID
>> =
>> grape_1_2234602_2234702
>> ;match
>> =
>> EVM_prediction_supercontig_1.248,EVM_prediction_supercontig_1.248.mRNA
>> 1\tucb\tgene\t2300292\t2302123\t.\t+\t.
>> \tID=grape_1_2300292_2302123;match=EVM_prediction_supercontig_244.8
>> 1\tucb\tgene\t2303615\t2303967\t.\t+\t.
>> \tID=grape_1_2303615_2303967;match=EVM_prediction_supercontig_244.8
>> 1\tucb\tgene\t2303616\t2303966\t.\t+\t.
>> \tParent=grape_1_2303615_2303967
>> 1\tucb\tgene\t3596400\t3596503\t.\t-\t.
>> \tID=grape_1_3596400_3596503;match=evm.TU.supercontig_167.27
>> 1\tucb\tgene\t3600651\t3600977\t.\t-\t.
>> \tmatch=evm.model.supercontig_1217.1,evm.model.supercontig_1217.1.mRNA
>> """
>>
>> dtype = {'names' :
>>                  ('seqid', 'source', 'type', 'start', 'end',
>>                    'score', 'strand', 'phase', 'attrs') ,
>>        'formats':
>>                  ['S24', 'S16', 'S16', 'i4', 'i4', 'f8',
>>                      'S1', 'i4', 'S128']}
>>
>> #OK with S128 for attrs
>> print np.genfromtxt(StringIO(gffstr), dtype = dtype)
>>
>>
>>
>> def _attr(kvstr):
>>    pairs = [kv.split("=") for kv in kvstr.split(";")]
>>    return dict(pairs)
>>
>> # change S128 to object to have col attrs as dictionary
>> dtype['formats'][-1] = 'O'
>> converters = {8: _attr }
>> #NOT OK
>> print np.genfromtxt(StringIO(gffstr), dtype = dtype,
>> converters=converters)
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From opossumnano at gmail.com  Thu Feb  5 03:14:44 2009
From: opossumnano at gmail.com (Tiziano Zito)
Date: Thu, 5 Feb 2009 09:14:44 +0100
Subject: [Numpy-discussion] Numpy 1.3 release date ?
In-Reply-To: <D955A90A-2121-4D34-A8AE-99292D86FC7F@gmail.com>
References: <D955A90A-2121-4D34-A8AE-99292D86FC7F@gmail.com>
Message-ID: <20090205081443.GA25806@localhost>

what about fixing
http://scipy.org/scipy/scipy/ticket/812 ? this is actually a
scipy-numpy compatibility problem, where numpy is wrong IMO.

it is a one-line fix, I think.

thank you!

tiziano



From trevor at notcows.com  Thu Feb  5 15:45:38 2009
From: trevor at notcows.com (Trevor Clarke)
Date: Thu, 5 Feb 2009 15:45:38 -0500
Subject: [Numpy-discussion] help writing an array subtype
Message-ID: <7bde5d400902051245x754ac906h21887d8ddda39036@mail.gmail.com>

I'm embedded python (and numpy) is a C++ app and I'm trying to access my
array data in numpy but I'm not sure where to start. Due to data sizes, I
don't have access to the entire array contiguously, my app implements a
virtual memory like paging system where variable sized pages of data are
brought into memory on demand. Each page has a contiguous block of data
containing at least the requested piece of the array (if it's more, it's
arranged so the appropriate data can be accessed with strides). I'd like to
load pages when they are accessed in an ndarray and when a new view is
requested. I was thinking of loading and holding the most recently accessed
page of the array on a per view basis. (i.e. if a view is created it can
access a different part of the array from the original array) I've ready
over the documentation on subclassing ndarray but there's a lot of
information there and I'm not quite sure the best place to start. Could
someone point me in the right direction or perhaps offer a better solution?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090205/3d26a68c/attachment.html>

From rmay31 at gmail.com  Thu Feb  5 18:06:49 2009
From: rmay31 at gmail.com (Ryan May)
Date: Thu, 05 Feb 2009 17:06:49 -0600
Subject: [Numpy-discussion] Argsort
Message-ID: <498B7109.8060501@gmail.com>

Hi,

Ok, what am I missing here:

x = np.array([[4,2],[5,3]])
x[x.argsort(1)]

array([[[5, 3],
        [4, 2]],

       [[5, 3],
        [4, 2]]])

I was expecting:

array([[2,4],[3,5]])

Certainly not a 3D array.  What am I doing wrong?

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From oliphant at enthought.com  Thu Feb  5 18:08:49 2009
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 05 Feb 2009 17:08:49 -0600
Subject: [Numpy-discussion] Selection of only a certain number of fields
Message-ID: <498B7181.5000300@enthought.com>


Hi all,

I've been fairly quiet on this list for awhile due to work and family 
schedule, but I think about how things can improve regularly.    One 
feature that's been requested by a few people is the ability to select 
multiple fields from a structured array.

Thus,  suppose *arr* is a structured array with dtype:

[('name', 'S25'),
  ('height', float),
  ('age', int),
  ('gender', 'S8')
]

Then,  newarr = arr[['name', 'age']]  should be a structured array with 
just the name and age fields.

It seems to me that there are two reasonable behaviors here (and 
possibly two different approaches to getting those behaviors):

1) Copy the data into a new array with a new dtype
2) Create a new array that is just a view of the old data with some of 
the fields "hidden"

I lean for having the proposed syntax do #2, but wonder what other 
people think.  

Any opinions and/or suggestions?

-Travis

-- 

Travis Oliphant
Enthought, Inc.
(512) 536-1057
http://www.enthought.com
oliphant at enthought.com



From robert.kern at gmail.com  Thu Feb  5 18:15:56 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 5 Feb 2009 17:15:56 -0600
Subject: [Numpy-discussion] Argsort
In-Reply-To: <498B7109.8060501@gmail.com>
References: <498B7109.8060501@gmail.com>
Message-ID: <3d375d730902051515sb7afa3ah469522cd536ca19@mail.gmail.com>

On Thu, Feb 5, 2009 at 17:06, Ryan May <rmay31 at gmail.com> wrote:
> Hi,
>
> Ok, what am I missing here:
>
> x = np.array([[4,2],[5,3]])
> x[x.argsort(1)]
>
> array([[[5, 3],
>        [4, 2]],
>
>       [[5, 3],
>        [4, 2]]])
>
> I was expecting:
>
> array([[2,4],[3,5]])
>
> Certainly not a 3D array.  What am I doing wrong?

Remember that x[i] applies the index array i to the first axis of x.
In order to apply i to the second axis, you need an argument in the
first position, too, to give the indices that apply to the first axis.
Remember that the arguments will be broadcast against each other.

In [11]: x[ [[0],[1]], x.argsort(1)]
Out[11]:
array([[2, 4],
       [3, 5]])

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From gael.varoquaux at normalesup.org  Thu Feb  5 18:16:18 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 6 Feb 2009 00:16:18 +0100
Subject: [Numpy-discussion] PEP: named axis (was: Selection of only a
	certain number of fields)
In-Reply-To: <498B7181.5000300@enthought.com>
References: <498B7181.5000300@enthought.com>
Message-ID: <20090205231618.GA21014@phare.normalesup.org>

On Thu, Feb 05, 2009 at 05:08:49PM -0600, Travis E. Oliphant wrote:
> I've been fairly quiet on this list for awhile due to work and family 
> schedule, but I think about how things can improve regularly.    One 
> feature that's been requested by a few people is the ability to select 
> multiple fields from a structured array.

Hey Travis,

I have no opinion on the above, as I don't have this use case. However, as
you are talking about implementing something, I jump on the occasion to
suggest another gadget, slightly related: I would like named axis.
Suppose you have a 5D array, I would like to be able to give each axis
names, eg (to chose an example you might be familiar with) ('Frontal',
'Lateral', 'Axial', 'Time', 'Subjects'). And if this could be understood
be numpy operations (say ufuncs and fancy indexing) so that I could do (a
is my 5D array):

>>> b = a.mean(axis='Time')
>>> b.axis
('Frontal', 'Lateral', 'Axial', 'Subjects')

I believe this would make a big difference for people working with
n-dimensional arrays, where n is large.

I do realize this is probably a lot of work, this is why I had been
refraining from mentioning it. I don't feel I can implement this.

Cheers,

Ga?l


From pgmdevlist at gmail.com  Thu Feb  5 19:37:19 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 5 Feb 2009 19:37:19 -0500
Subject: [Numpy-discussion] Selection of only a certain number of fields
In-Reply-To: <498B7181.5000300@enthought.com>
References: <498B7181.5000300@enthought.com>
Message-ID: <3D14D1EA-AC5C-4283-B9FB-137580E1BAB5@gmail.com>


On Feb 5, 2009, at 6:08 PM, Travis E. Oliphant wrote:

>
> Hi all,
>
> I've been fairly quiet on this list for awhile due to work and family
> schedule, but I think about how things can improve regularly.    One
> feature that's been requested by a few people is the ability to select
> multiple fields from a structured array.

>
> [...]

+1 for #2.

Note that we now have a drop_fields function in np.lib.recfunctions, a  
reimplementation of the equivalent function in matplotlib. It works  
along the lines of your proposition #1 (create a new array w/ a new  
dtype and fill it)



From ellisonbg.net at gmail.com  Thu Feb  5 23:00:07 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 5 Feb 2009 20:00:07 -0800
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <3d375d730902032132l33417bedia7dacf08f4e7996e@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
	<3d375d730902031617t43726c8as3b9e8d46583f2152@mail.gmail.com>
	<6ce0ac130902031620t782b2387hbd2de5e8e0ced580@mail.gmail.com>
	<3d375d730902031623p6171dd09gbcc9bf75a793c8b1@mail.gmail.com>
	<6ce0ac130902031634ye40a829gbac28740fa281231@mail.gmail.com>
	<3d375d730902031642o65baa95ciebde9b5cc8f8b800@mail.gmail.com>
	<6ce0ac130902031653x2a304265t41b3c4f70bc257ef@mail.gmail.com>
	<3d375d730902031658l2b727ad2t3cfbaff32865e013@mail.gmail.com>
	<6ce0ac130902032122i1ba31be3y9c2d034ee0a1d2fc@mail.gmail.com>
	<3d375d730902032132l33417bedia7dacf08f4e7996e@mail.gmail.com>
Message-ID: <6ce0ac130902052000x761c47cen46a3815c2b741713@mail.gmail.com>

Robert,

Can you have a look at the following fix and see if it is satisfactory?

http://github.com/ellisonbg/numpy/blob/81360e93968968dc9dcbafd7895da7cec5015a3c/numpy/distutils/fcompiler/gnu.py

Brian


On Tue, Feb 3, 2009 at 9:32 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Tue, Feb 3, 2009 at 23:22, Brian Granger <ellisonbg.net at gmail.com> wrote:
>>> 1) Trust the environment variable if given and let distutils raise its
>>> error message (why not raise it ourselves? distutils' error message
>>> and explanation is already out in THE GOOGLE.)
>>>
>>> 2) Otherwise, use the value in the Makefile if it's there.
>>>
>>> 3) If it's not even in the Makefile for whatever reason, go with 10.3.
>>
>> Sounds good, do you want to me work up a patch?
>
> Yes, please.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From oliphant at enthought.com  Thu Feb  5 23:09:26 2009
From: oliphant at enthought.com (Travis Oliphant)
Date: Thu, 05 Feb 2009 22:09:26 -0600
Subject: [Numpy-discussion] Selection of only a certain number of fields
In-Reply-To: <3D14D1EA-AC5C-4283-B9FB-137580E1BAB5@gmail.com>
References: <498B7181.5000300@enthought.com>
	<3D14D1EA-AC5C-4283-B9FB-137580E1BAB5@gmail.com>
Message-ID: <498BB7F6.6040603@enthought.com>

Pierre GM wrote:
> On Feb 5, 2009, at 6:08 PM, Travis E. Oliphant wrote:
>
>   
>> Hi all,
>>
>> I've been fairly quiet on this list for awhile due to work and family
>> schedule, but I think about how things can improve regularly.    One
>> feature that's been requested by a few people is the ability to select
>> multiple fields from a structured array.
>>     
>
>   
>> [...]
>>     
>
> +1 for #2.
>
> Note that we now have a drop_fields function in np.lib.recfunctions, a  
> reimplementation of the equivalent function in matplotlib. It works  
> along the lines of your proposition #1 (create a new array w/ a new  
> dtype and fill it)
>   

After more thought, I think I was too eager in my suggestion of #2.  
It's actually not really possible to do a view the way I would want it 
to work.   It would be possible to create a data-type with 
hidden-fields, but a copy would be not "get rid of the extra data".

Thus  newarr = arr[['name', 'age']].copy() would be exactly the same 
size as arr because elements are  copied wholesale and each "row" is a 
single element in the NumPy array.    Some infrastructure would have to 
be implemented at a fundamental level to handle partial-element 
manipulation similar at least in spirit to what is needed to handle 
bit-level striding on a fundamental level.  

Also, I don't remember if we resolved how hidden fields would be shown 
in the array interface.  

So, I think that we may be stuck with #1 which at least is consistent 
with the "fancy-indexing" is a copy pattern (and is just syntatic sugar 
for capability you've already implemented in recfunctions).

-Travis



From robert.kern at gmail.com  Thu Feb  5 23:13:31 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 5 Feb 2009 22:13:31 -0600
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <6ce0ac130902052000x761c47cen46a3815c2b741713@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
	<6ce0ac130902031620t782b2387hbd2de5e8e0ced580@mail.gmail.com>
	<3d375d730902031623p6171dd09gbcc9bf75a793c8b1@mail.gmail.com>
	<6ce0ac130902031634ye40a829gbac28740fa281231@mail.gmail.com>
	<3d375d730902031642o65baa95ciebde9b5cc8f8b800@mail.gmail.com>
	<6ce0ac130902031653x2a304265t41b3c4f70bc257ef@mail.gmail.com>
	<3d375d730902031658l2b727ad2t3cfbaff32865e013@mail.gmail.com>
	<6ce0ac130902032122i1ba31be3y9c2d034ee0a1d2fc@mail.gmail.com>
	<3d375d730902032132l33417bedia7dacf08f4e7996e@mail.gmail.com>
	<6ce0ac130902052000x761c47cen46a3815c2b741713@mail.gmail.com>
Message-ID: <3d375d730902052013t49f5fbechb89bb8e1332bfb07@mail.gmail.com>

On Thu, Feb 5, 2009 at 22:00, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Robert,
>
> Can you have a look at the following fix and see if it is satisfactory?
>
> http://github.com/ellisonbg/numpy/blob/81360e93968968dc9dcbafd7895da7cec5015a3c/numpy/distutils/fcompiler/gnu.py

Looks good.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From oliphant at enthought.com  Thu Feb  5 23:17:54 2009
From: oliphant at enthought.com (Travis Oliphant)
Date: Thu, 05 Feb 2009 22:17:54 -0600
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <20090205231618.GA21014@phare.normalesup.org>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
Message-ID: <498BB9F2.5080501@enthought.com>

Gael Varoquaux wrote:
> On Thu, Feb 05, 2009 at 05:08:49PM -0600, Travis E. Oliphant wrote:
>   
>> I've been fairly quiet on this list for awhile due to work and family 
>> schedule, but I think about how things can improve regularly.    One 
>> feature that's been requested by a few people is the ability to select 
>> multiple fields from a structured array.
>>     
>
> Hey Travis,
>
> I have no opinion on the above, as I don't have this use case. However, as
> you are talking about implementing something, I jump on the occasion to
> suggest another gadget, slightly related: I would like named axis.
> Suppose you have a 5D array, I would like to be able to give each axis
> names, eg (to chose an example you might be familiar with) ('Frontal',
> 'Lateral', 'Axial', 'Time', 'Subjects'). And if this could be understood
> be numpy operations (say ufuncs and fancy indexing) so that I could do (a
> is my 5D array):
>
>   
This could be implemented but would require adding information to the 
NumPy array.   I've been thinking for a long time that we ought to add a 
"dictionary" attribute to the NumPy array (i.e. a new member to the 
PyArrayObject data-structure).   A lot of subclasses of NumPy arrays 
just add meta-information that could be stored there. 

Then, it would be a trivial thing to check to see if the dictionary had 
say an "axis_mapping" keyword and if so then do the conversions found 
there.

I think this has been brought up before, though.  What do people think 
about adding a default dictionary to every instance of a NumPy array.

The question that always arises in this context which I don't have good 
answers for is what do you do with the dictionary on the output of 
ufuncs?   One approach is to always return NULL for the dictionary and 
don't try and guess.   A slightly different one is to at least handle 
the case where all inputs have the same dictionary and return a new 
"shallow" copy of that.


-Travis



From ellisonbg.net at gmail.com  Thu Feb  5 23:19:22 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 5 Feb 2009 20:19:22 -0800
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <3d375d730902052013t49f5fbechb89bb8e1332bfb07@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
	<3d375d730902031623p6171dd09gbcc9bf75a793c8b1@mail.gmail.com>
	<6ce0ac130902031634ye40a829gbac28740fa281231@mail.gmail.com>
	<3d375d730902031642o65baa95ciebde9b5cc8f8b800@mail.gmail.com>
	<6ce0ac130902031653x2a304265t41b3c4f70bc257ef@mail.gmail.com>
	<3d375d730902031658l2b727ad2t3cfbaff32865e013@mail.gmail.com>
	<6ce0ac130902032122i1ba31be3y9c2d034ee0a1d2fc@mail.gmail.com>
	<3d375d730902032132l33417bedia7dacf08f4e7996e@mail.gmail.com>
	<6ce0ac130902052000x761c47cen46a3815c2b741713@mail.gmail.com>
	<3d375d730902052013t49f5fbechb89bb8e1332bfb07@mail.gmail.com>
Message-ID: <6ce0ac130902052019ne743034of817f02e31312f1f@mail.gmail.com>

Great, what is the best way of rolling this into numpy?

Brian

On Thu, Feb 5, 2009 at 8:13 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Thu, Feb 5, 2009 at 22:00, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> Robert,
>>
>> Can you have a look at the following fix and see if it is satisfactory?
>>
>> http://github.com/ellisonbg/numpy/blob/81360e93968968dc9dcbafd7895da7cec5015a3c/numpy/distutils/fcompiler/gnu.py
>
> Looks good.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From robert.kern at gmail.com  Thu Feb  5 23:29:42 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 5 Feb 2009 22:29:42 -0600
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <498BB9F2.5080501@enthought.com>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<498BB9F2.5080501@enthought.com>
Message-ID: <3d375d730902052029l21f675d1leecb0d405db5dc23@mail.gmail.com>

On Thu, Feb 5, 2009 at 22:17, Travis Oliphant <oliphant at enthought.com> wrote:
> Gael Varoquaux wrote:
>> On Thu, Feb 05, 2009 at 05:08:49PM -0600, Travis E. Oliphant wrote:
>>
>>> I've been fairly quiet on this list for awhile due to work and family
>>> schedule, but I think about how things can improve regularly.    One
>>> feature that's been requested by a few people is the ability to select
>>> multiple fields from a structured array.
>>>
>>
>> Hey Travis,
>>
>> I have no opinion on the above, as I don't have this use case. However, as
>> you are talking about implementing something, I jump on the occasion to
>> suggest another gadget, slightly related: I would like named axis.
>> Suppose you have a 5D array, I would like to be able to give each axis
>> names, eg (to chose an example you might be familiar with) ('Frontal',
>> 'Lateral', 'Axial', 'Time', 'Subjects'). And if this could be understood
>> be numpy operations (say ufuncs and fancy indexing) so that I could do (a
>> is my 5D array):
>>
>>
> This could be implemented but would require adding information to the
> NumPy array.

More than that, though. Every function and method that takes an axis
or reduces an axis will need to be rewritten. For that reason, I'm -1
on the proposal.

>  I've been thinking for a long time that we ought to add a
> "dictionary" attribute to the NumPy array (i.e. a new member to the
> PyArrayObject data-structure).   A lot of subclasses of NumPy arrays
> just add meta-information that could be stored there.
>
> Then, it would be a trivial thing to check to see if the dictionary had
> say an "axis_mapping" keyword and if so then do the conversions found
> there.
>
> I think this has been brought up before, though.  What do people think
> about adding a default dictionary to every instance of a NumPy array.
>
> The question that always arises in this context which I don't have good
> answers for is what do you do with the dictionary on the output of
> ufuncs?   One approach is to always return NULL for the dictionary and
> don't try and guess.   A slightly different one is to at least handle
> the case where all inputs have the same dictionary and return a new
> "shallow" copy of that.

I'm of the opinion that it should never guess. We have no idea what
semantics are being placed on the dict. Even in the case where all of
the inputs have the same dict, the operation may easily invalidate the
metadata. For example, a reduction on one of these axis-decorated
arrays would make the axis labels incorrect.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Fri Feb  6 01:13:54 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 05 Feb 2009 22:13:54 -0800
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <498BB9F2.5080501@enthought.com>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<498BB9F2.5080501@enthought.com>
Message-ID: <498BD522.8090508@noaa.gov>

Travis Oliphant wrote:
>   What do people think
> about adding a default dictionary to every instance of a NumPy array.

It sound kind of heavyweight to me. I tend to use lots of small arrays 
(to represent an x,y point, for instance). There are enough performance 
issues with that as it stands. Maybe an empty dict isn't much, but it is 
extra.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception


From robert.kern at gmail.com  Fri Feb  6 01:20:37 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 6 Feb 2009 00:20:37 -0600
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <498BD522.8090508@noaa.gov>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<498BB9F2.5080501@enthought.com> <498BD522.8090508@noaa.gov>
Message-ID: <3d375d730902052220i34972e78r2605a4fb376d440f@mail.gmail.com>

On Fri, Feb 6, 2009 at 00:13, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> Travis Oliphant wrote:
>>   What do people think
>> about adding a default dictionary to every instance of a NumPy array.
>
> It sound kind of heavyweight to me. I tend to use lots of small arrays
> (to represent an x,y point, for instance). There are enough performance
> issues with that as it stands. Maybe an empty dict isn't much, but it is
> extra.

I think we can create the dict on demand, so there will be no overhead
except for the space for the pointer in the struct.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From olivier.grisel at ensta.org  Fri Feb  6 02:33:11 2009
From: olivier.grisel at ensta.org (Olivier Grisel)
Date: Fri, 6 Feb 2009 08:33:11 +0100
Subject: [Numpy-discussion] PEP: named axis (was: Selection of only a
	certain number of fields)
In-Reply-To: <20090205231618.GA21014@phare.normalesup.org>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
Message-ID: <db0bec8f0902052333y363715a7y1a93ae4696ef309e@mail.gmail.com>

+1

On Feb 6, 2009 12:16 AM, "Gael Varoquaux" <gael.varoquaux at normalesup.org>
wrote:

On Thu, Feb 05, 2009 at 05:08:49PM -0600, Travis E. Oliphant wrote:
> I've been fairly quiet on this list for awhile due to work and family
> schedule, but I think about how things can improve regularly.    One
> feature that's been requested by a few people is the ability to select
> multiple fields from a structured array.

Hey Travis,

I have no opinion on the above, as I don't have this use case. However, as
you are talking about implementing something, I jump on the occasion to
suggest another gadget, slightly related: I would like named axis.
Suppose you have a 5D array, I would like to be able to give each axis
names, eg (to chose an example you might be familiar with) ('Frontal',
'Lateral', 'Axial', 'Time', 'Subjects'). And if this could be understood
be numpy operations (say ufuncs and fancy indexing) so that I could do (a
is my 5D array):

>>> b = a.mean(axis='Time')
>>> b.axis
('Frontal', 'Lateral', 'Axial', 'Subjects')

I believe this would make a big difference for people working with
n-dimensional arrays, where n is large.

I do realize this is probably a lot of work, this is why I had been
refraining from mentioning it. I don't feel I can implement this.

Cheers,

Ga?l
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090206/32d3aa22/attachment.html>

From faltet at pytables.org  Fri Feb  6 02:49:48 2009
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 6 Feb 2009 08:49:48 +0100
Subject: [Numpy-discussion] Selection of only a certain number of fields
In-Reply-To: <498BB7F6.6040603@enthought.com>
References: <498B7181.5000300@enthought.com>
	<3D14D1EA-AC5C-4283-B9FB-137580E1BAB5@gmail.com>
	<498BB7F6.6040603@enthought.com>
Message-ID: <200902060849.48610.faltet@pytables.org>

A Friday 06 February 2009, Travis Oliphant escrigu?:
> Pierre GM wrote:
> > On Feb 5, 2009, at 6:08 PM, Travis E. Oliphant wrote:
> >> Hi all,
> >>
> >> I've been fairly quiet on this list for awhile due to work and
> >> family schedule, but I think about how things can improve
> >> regularly.    One feature that's been requested by a few people is
> >> the ability to select multiple fields from a structured array.
> >>
> >>
> >>
> >> [...]
> >
> > +1 for #2.
> >
> > Note that we now have a drop_fields function in
> > np.lib.recfunctions, a reimplementation of the equivalent function
> > in matplotlib. It works along the lines of your proposition #1
> > (create a new array w/ a new dtype and fill it)
>
> After more thought, I think I was too eager in my suggestion of #2.
> It's actually not really possible to do a view the way I would want
> it to work.   It would be possible to create a data-type with
> hidden-fields, but a copy would be not "get rid of the extra data".
>
> Thus  newarr = arr[['name', 'age']].copy() would be exactly the same
> size as arr because elements are  copied wholesale and each "row" is
> a single element in the NumPy array.    Some infrastructure would
> have to be implemented at a fundamental level to handle
> partial-element manipulation similar at least in spirit to what is
> needed to handle bit-level striding on a fundamental level.
>
> Also, I don't remember if we resolved how hidden fields would be
> shown in the array interface.
>
> So, I think that we may be stuck with #1 which at least is consistent
> with the "fancy-indexing" is a copy pattern (and is just syntatic
> sugar for capability you've already implemented in recfunctions).

Mmh, I'd also vote for #2 for performance reasons, but as the 
implementation seems quite involved, I suppose that #1 would be great 
too.

Cheers,

-- 
Francesc Alted


From gael.varoquaux at normalesup.org  Fri Feb  6 03:09:38 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 6 Feb 2009 09:09:38 +0100
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <3d375d730902052029l21f675d1leecb0d405db5dc23@mail.gmail.com>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<498BB9F2.5080501@enthought.com>
	<3d375d730902052029l21f675d1leecb0d405db5dc23@mail.gmail.com>
Message-ID: <20090206080938.GA7464@phare.normalesup.org>

On Thu, Feb 05, 2009 at 10:29:42PM -0600, Robert Kern wrote:
> >> I have no opinion on the above, as I don't have this use case. However, as
> >> you are talking about implementing something, I jump on the occasion to
> >> suggest another gadget, slightly related: I would like named axis.
> >> Suppose you have a 5D array, I would like to be able to give each axis
> >> names, eg (to chose an example you might be familiar with) ('Frontal',
> >> 'Lateral', 'Axial', 'Time', 'Subjects'). And if this could be understood
> >> be numpy operations (say ufuncs and fancy indexing) so that I could do (a
> >> is my 5D array):


> > This could be implemented but would require adding information to the
> > NumPy array.

> More than that, though. Every function and method that takes an axis
> or reduces an axis will need to be rewritten. For that reason, I'm -1
> on the proposal.

Yes, this is the reason why this proposition is actually a lot of work.
This is also what makes it interesting. Sticking information on a numpy
array is useful, but it does not achieve something that is not feasible
without the help of numpy. The proposition is about much more than that.

Ga?l


From gael.varoquaux at normalesup.org  Fri Feb  6 03:10:28 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 6 Feb 2009 09:10:28 +0100
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <3d375d730902052220i34972e78r2605a4fb376d440f@mail.gmail.com>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<498BB9F2.5080501@enthought.com> <498BD522.8090508@noaa.gov>
	<3d375d730902052220i34972e78r2605a4fb376d440f@mail.gmail.com>
Message-ID: <20090206081028.GB7464@phare.normalesup.org>

On Fri, Feb 06, 2009 at 12:20:37AM -0600, Robert Kern wrote:
> On Fri, Feb 6, 2009 at 00:13, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> > Travis Oliphant wrote:
> >>   What do people think
> >> about adding a default dictionary to every instance of a NumPy array.

> > It sound kind of heavyweight to me. I tend to use lots of small arrays
> > (to represent an x,y point, for instance). There are enough performance
> > issues with that as it stands. Maybe an empty dict isn't much, but it is
> > extra.

> I think we can create the dict on demand, so there will be no overhead
> except for the space for the pointer in the struct.

I am +1 for the dict created on demand. It seems like a great idea to
help subclassing.

Ga?l


From stefan at sun.ac.za  Fri Feb  6 04:12:31 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 6 Feb 2009 11:12:31 +0200
Subject: [Numpy-discussion] Selection of only a certain number of fields
In-Reply-To: <498BB7F6.6040603@enthought.com>
References: <498B7181.5000300@enthought.com>
	<3D14D1EA-AC5C-4283-B9FB-137580E1BAB5@gmail.com>
	<498BB7F6.6040603@enthought.com>
Message-ID: <9457e7c80902060112t4bf30199h3d41b529c7d53b26@mail.gmail.com>

Hi Travis

2009/2/6 Travis Oliphant <oliphant at enthought.com>:
> Thus  newarr = arr[['name', 'age']].copy() would be exactly the same
> size as arr because elements are  copied wholesale and each "row" is a
> single element in the NumPy array.    Some infrastructure would have to
> be implemented at a fundamental level to handle partial-element
> manipulation similar at least in spirit to what is needed to handle
> bit-level striding on a fundamental level.

I like your suggestion!  Can you think of a way to implement #2 with
the correct copy semantics?  Being able to create a view without
copying is such a big plus that it is worth considering, even at an
implementation cost.

Regards
St?fan


From stefan at sun.ac.za  Fri Feb  6 04:22:20 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 6 Feb 2009 11:22:20 +0200
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <3d375d730902052029l21f675d1leecb0d405db5dc23@mail.gmail.com>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<498BB9F2.5080501@enthought.com>
	<3d375d730902052029l21f675d1leecb0d405db5dc23@mail.gmail.com>
Message-ID: <9457e7c80902060122o365cbc28mf5a0bc22d5bc6f6d@mail.gmail.com>

Hi Robert

2009/2/6 Robert Kern <robert.kern at gmail.com>:
>> This could be implemented but would require adding information to the
>> NumPy array.
>
> More than that, though. Every function and method that takes an axis
> or reduces an axis will need to be rewritten. For that reason, I'm -1
> on the proposal.

Are you -1 on the array dictionary, or on using it to do axis mapping?
 I would imagine that Gael would be happier even if he had to do

axis = x.meta.axis['Lateral']
some_func(x, axis)

> I'm of the opinion that it should never guess. We have no idea what
> semantics are being placed on the dict. Even in the case where all of
> the inputs have the same dict, the operation may easily invalidate the
> metadata. For example, a reduction on one of these axis-decorated
> arrays would make the axis labels incorrect.

That's a good point.  So what would be a sane way of propagating
meta-data?  If we don't want to make any assumptions, it becomes the
user's responsibility to do it manually.

Cheers
St?fan


From dsdale24 at gmail.com  Fri Feb  6 08:48:47 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Fri, 6 Feb 2009 08:48:47 -0500
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <9457e7c80902060122o365cbc28mf5a0bc22d5bc6f6d@mail.gmail.com>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<498BB9F2.5080501@enthought.com>
	<3d375d730902052029l21f675d1leecb0d405db5dc23@mail.gmail.com>
	<9457e7c80902060122o365cbc28mf5a0bc22d5bc6f6d@mail.gmail.com>
Message-ID: <a08e5f80902060548t4adbabecv31a38e6c2caf6921@mail.gmail.com>

On Fri, Feb 6, 2009 at 4:22 AM, St?fan van der Walt <stefan at sun.ac.za>wrote:

> Hi Robert
>
> 2009/2/6 Robert Kern <robert.kern at gmail.com>:
> >> This could be implemented but would require adding information to the
> >> NumPy array.
> >
> > More than that, though. Every function and method that takes an axis
> > or reduces an axis will need to be rewritten. For that reason, I'm -1
> > on the proposal.
>
> Are you -1 on the array dictionary, or on using it to do axis mapping?
>  I would imagine that Gael would be happier even if he had to do
>
> axis = x.meta.axis['Lateral']
> some_func(x, axis)
>
> > I'm of the opinion that it should never guess. We have no idea what
> > semantics are being placed on the dict. Even in the case where all of
> > the inputs have the same dict, the operation may easily invalidate the
> > metadata. For example, a reduction on one of these axis-decorated
> > arrays would make the axis labels incorrect.
>
> That's a good point.  So what would be a sane way of propagating
> meta-data?  If we don't want to make any assumptions, it becomes the
> user's responsibility to do it manually.
>

In which case they end up writing a subclass to propagate only that portion
of the dict that they were using.

I'll add another example where a subclass would be needed to propagate the
metadata: physical quantities. I have a package (nearly ready to submit to
this list for request for comment) that uses a dict subclass to describe the
dimensionality of a quantity, like {m:1, s:-1}. I don't think this metadata
dictionary would be useful for quantities since two arrays may have the same
dimensionality but the propagated dimensionality would be {m:1, s:-1} for
addition, {m:2, s:-2} for multiplication, {} for division.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090206/e6ef4708/attachment.html>

From bsouthey at gmail.com  Fri Feb  6 10:31:47 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Fri, 06 Feb 2009 09:31:47 -0600
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <db0bec8f0902052333y363715a7y1a93ae4696ef309e@mail.gmail.com>
References: <498B7181.5000300@enthought.com>	<20090205231618.GA21014@phare.normalesup.org>
	<db0bec8f0902052333y363715a7y1a93ae4696ef309e@mail.gmail.com>
Message-ID: <498C57E3.6090507@gmail.com>

Hi,
+1 on the idea but how will this work with other numpy methods?

suppose *arr* is a structured array with dtype:

[('name', 'S25'),
  ('height', float),
  ('age', int),
  ('gender', 'S8')
]


Would you be able to first define a list of columns such as
cols=['height', 'age'] 
arr[cols]
This would be a handy feature.

For example, for some compatible array A, would you be able to the 
following with a view

np.linalg.lstsq(A, arr[['age']])
np.linalg.lstsq(A,arr[['height', 'age']]

Bruce





From faltet at pytables.org  Fri Feb  6 10:42:01 2009
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 6 Feb 2009 16:42:01 +0100
Subject: [Numpy-discussion] How to guess the compiler type from distuils?
Message-ID: <200902061642.01694.faltet@pytables.org>

Hi,

We would like to use the numpy.distutils machinery for numexpr and I'd 
like to add different compiler flags depending on the compiler used.  
Anbody knows a simple way to do this with numpy.distutils (ore plain 
distutils)?

Thanks,

-- 
Francesc Alted


From faltet at pytables.org  Fri Feb  6 11:27:45 2009
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 6 Feb 2009 17:27:45 +0100
Subject: [Numpy-discussion] How to guess the compiler type from distuils?
In-Reply-To: <200902061642.01694.faltet@pytables.org>
References: <200902061642.01694.faltet@pytables.org>
Message-ID: <200902061727.45553.faltet@pytables.org>

A Friday 06 February 2009, Francesc Alted escrigu?:
> Hi,
>
> We would like to use the numpy.distutils machinery for numexpr and
> I'd like to add different compiler flags depending on the compiler
> used. Anbody knows a simple way to do this with numpy.distutils (ore
> plain distutils)?

I've figured out how to do it.  For the records, you simply have to 
import this:

from numpy.distutils.command.build_ext import build_ext as 
numpy_build_ext

and then define the next class:

class build_ext(numpy_build_ext):
    def build_extension(self, ext):
        # at this point we know what the C compiler is.
        c = self.compiler
        old_compile_options = None
        # For MS Visual C, we use /O1 instead of the default /Ox,
        # as /Ox takes a long time (~5 mins) to compile.
        # The speed of the code isn't noticeably different.
        if c.compiler_type == 'msvc':
            if not c.initialized:
                c.initialize()
            old_compile_options = c.compile_options[:]
            if '/Ox' in c.compile_options:
                c.compile_options.remove('/Ox')
            c.compile_options.append('/O1')
            ext.extra_compile_args = []
        numpy_build_ext.build_extension(self, ext)
        if old_compile_options is not None:
            self.compiler.compile_options = old_compile_options

where you can taylor compiler options at your will.

Cheers,

-- 
Francesc Alted


From reakinator at gmail.com  Fri Feb  6 11:29:52 2009
From: reakinator at gmail.com (Rich E)
Date: Fri, 6 Feb 2009 17:29:52 +0100
Subject: [Numpy-discussion] default float type of array not accepted by
	SWIG wrapped C functions
In-Reply-To: <875E04698300DB4FA52B4219ABA6039B126E9D05BD@ES02SNLNT.srn.sandia.gov>
References: <a3297bc90901220845y72907465q3800384d212ab9b8@mail.gmail.com>
	<875E04698300DB4FA52B4219ABA6039B126E9D05BD@ES02SNLNT.srn.sandia.gov>
Message-ID: <a3297bc90902060829n1f73099euf8cc4ae6ba6c3da5@mail.gmail.com>

I ended up solving my problem in SWIG, so I might as well post it
here.  I just made my own 'array' and 'zeros' functions with floating
point precision as follows:

%pythoncode %{

from numpy import array as np_array
def array (n, type='float32'):
        return(np_array(n, type))

from numpy import zeros as np_zeros
def zeros (n, type='float32'):
        return(np_zeros(n, type))
%}

Pretty basic, I know.  But it cuts down on alot of unnecessary code.

- Rich


On Thu, Jan 22, 2009 at 6:09 PM, Spotz, William F <wfspotz at sandia.gov> wrote:
> Rich,
>
> Basic python only supports double precision floats, so that is not an option.
>
> NumPy does not have, as far as I know, a way to set the default precision, although it might be a reasonable request.
>
> As for the SWIG interface file, almost anything is possible.  Can you give an example of a function prototype you are wrapping, the %apply directive you use and and example of python code accessing it?
>
> -Bill
> ________________________________________
> From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Rich E [reakinator at gmail.com]
> Sent: Thursday, January 22, 2009 11:45 AM
> To: Discussion of Numerical Python
> Subject: [Numpy-discussion] default float type of array not accepted by SWIG wrapped C functions
>
> Hi all,
>
> I have a SWIG wrapped C library that uses 32bit floating point arrays,
> using the numpy.i typemapping system for passing the arrays.  For
> every array that I make, I have to convert it using astype('float32'),
> else python complains that I tried to pass a double-precision array.
>
> Is there any way to set the default floating point precision to 32bit,
> in python or in the SWIG interface file?
>
> regards,
> Rich
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From Chris.Barker at noaa.gov  Fri Feb  6 12:09:35 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 06 Feb 2009 09:09:35 -0800
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <a08e5f80902060548t4adbabecv31a38e6c2caf6921@mail.gmail.com>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<498BB9F2.5080501@enthought.com>
	<3d375d730902052029l21f675d1leecb0d405db5dc23@mail.gmail.com>
	<9457e7c80902060122o365cbc28mf5a0bc22d5bc6f6d@mail.gmail.com>
	<a08e5f80902060548t4adbabecv31a38e6c2caf6921@mail.gmail.com>
Message-ID: <498C6ECF.1090408@noaa.gov>

Darren Dale wrote:
> I have a package (nearly ready to 
> submit to this list for request for comment) that uses a dict subclass 
> to describe the dimensionality of a quantity, like {m:1, s:-1}.

I'm looking forward to that -- you may have just saved me a bunch of coding!

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From dsdale24 at gmail.com  Fri Feb  6 12:53:09 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Fri, 6 Feb 2009 12:53:09 -0500
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <498C6ECF.1090408@noaa.gov>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<498BB9F2.5080501@enthought.com>
	<3d375d730902052029l21f675d1leecb0d405db5dc23@mail.gmail.com>
	<9457e7c80902060122o365cbc28mf5a0bc22d5bc6f6d@mail.gmail.com>
	<a08e5f80902060548t4adbabecv31a38e6c2caf6921@mail.gmail.com>
	<498C6ECF.1090408@noaa.gov>
Message-ID: <a08e5f80902060953l6ea4fa49m3ba15b09fa0c1afe@mail.gmail.com>

On Fri, Feb 6, 2009 at 12:09 PM, Christopher Barker
<Chris.Barker at noaa.gov>wrote:

> Darren Dale wrote:
> > I have a package (nearly ready to
> > submit to this list for request for comment) that uses a dict subclass
> > to describe the dimensionality of a quantity, like {m:1, s:-1}.
>
> I'm looking forward to that -- you may have just saved me a bunch of
> coding!
>
>
There is an alpha available at packages.python.org/quantities, and a
development site at launchpad.net/python-quantities . There is a link to a
mailing list at the launchpad site as well. I think there is still a couple
weekends worth of effort on documentation, unit testing, and ufuncs before
it is really ready for further discussion here.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090206/d1f98e1e/attachment.html>

From suchindra at gmail.com  Fri Feb  6 14:24:37 2009
From: suchindra at gmail.com (Suchindra Sandhu)
Date: Fri, 6 Feb 2009 14:24:37 -0500
Subject: [Numpy-discussion] numpy.any oddity
Message-ID: <b39d0d980902061124h4cd14c8dw6d26eee1e81405fc@mail.gmail.com>

Hi,

I accidently stumbled upon this odd behavior by numpy.any. The following
code leaks memory -

for i in xrange(10000000):
        print N.any({'whatever': N.arange(10000000)})

Ofcourse, I called "any" on a dict object by accident, but it should not
really leak memory.

I am running numpy version 1.0.4 with python 2.5.2

Cheers,
Suchindra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090206/f37e66cf/attachment.html>

From kbasye1 at jhu.edu  Fri Feb  6 14:00:10 2009
From: kbasye1 at jhu.edu (Ken Basye)
Date: Fri, 06 Feb 2009 14:00:10 -0500
Subject: [Numpy-discussion] Can I fill an existing array from an iterator?
Message-ID: <498C88BA.5010300@jhu.edu>

Hi Folks,
  I wonder if there's a way to fill an existing array from an iterator 
without creating a temporary array.  That is, I'm looking for something 
that has the effect of

  >>> target = np.array(xrange(9), dtype = float)
  >>> target[:] = np.fromiter(repeat(3.14159, 9), dtype=float)

without creating a second array object in the second line.  The closest 
I got was this:

  >>> target[:] = xrange(-9, 0)
  >>> target[:] = tuple(repeat(5.5, 9))

Note that xrange isn't really an iterator, and appears to be handled 
specially.  Emptying the iterator into a tuple works, but
I was hoping for something even more direct.  If this isn't possible, I 
wonder what people would think of adding an optional 'out' argument to 
the 'fromXXX' functions which behaved like the 'out' argument to sum() 
and other functions.

  Thanks,
     Ken


From nwagner at iam.uni-stuttgart.de  Fri Feb  6 15:16:28 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Fri, 06 Feb 2009 21:16:28 +0100
Subject: [Numpy-discussion] xblas and numpy
Message-ID: <web-116756672@uni-stuttgart.de>

Hi all,

Just curious. Is it possible to use xblas with numpy ?
  

http://www.netlib.org/xblas/


Nils


From robert.kern at gmail.com  Fri Feb  6 15:39:45 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 6 Feb 2009 14:39:45 -0600
Subject: [Numpy-discussion] numpy.any oddity
In-Reply-To: <b39d0d980902061124h4cd14c8dw6d26eee1e81405fc@mail.gmail.com>
References: <b39d0d980902061124h4cd14c8dw6d26eee1e81405fc@mail.gmail.com>
Message-ID: <3d375d730902061239p791234e7k8d0f9d04e2a66464@mail.gmail.com>

On Fri, Feb 6, 2009 at 13:24, Suchindra Sandhu <suchindra at gmail.com> wrote:
> Hi,
>
> I accidently stumbled upon this odd behavior by numpy.any. The following
> code leaks memory -
>
> for i in xrange(10000000):
>         print N.any({'whatever': N.arange(10000000)})
>
> Ofcourse, I called "any" on a dict object by accident, but it should not
> really leak memory.
>
> I am running numpy version 1.0.4 with python 2.5.2

Upgrade to a more recent version of numpy. I do not see a leak. I
vaguely recall there being a problem in the 1.0.x series with
object-dtype scalars, which is what numpy.any() will convert a dict
to.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Fri Feb  6 16:02:32 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 6 Feb 2009 15:02:32 -0600
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <498C57E3.6090507@gmail.com>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<db0bec8f0902052333y363715a7y1a93ae4696ef309e@mail.gmail.com>
	<498C57E3.6090507@gmail.com>
Message-ID: <3d375d730902061302n104e7b73jdf70c220f42c9bcb@mail.gmail.com>

On Fri, Feb 6, 2009 at 09:31, Bruce Southey <bsouthey at gmail.com> wrote:
> Hi,
> +1 on the idea but how will this work with other numpy methods?
>
> suppose *arr* is a structured array with dtype:
>
> [('name', 'S25'),
>  ('height', float),
>  ('age', int),
>  ('gender', 'S8')
> ]
>
>
> Would you be able to first define a list of columns such as
> cols=['height', 'age']
> arr[cols]
> This would be a handy feature.

Yes. ndarray.__getitem__() doesn't know anything about where the list
of strings comes from.

> For example, for some compatible array A, would you be able to the
> following with a view
>
> np.linalg.lstsq(A, arr[['age']])
> np.linalg.lstsq(A,arr[['height', 'age']]

No, you'd still get a record array, which lstsq() doesn't know what to do with.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From michael.s.gilbert at gmail.com  Fri Feb  6 16:24:18 2009
From: michael.s.gilbert at gmail.com (Michael S. Gilbert)
Date: Fri, 6 Feb 2009 16:24:18 -0500
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
	mersenne twister key?
Message-ID: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>

In numpy/random/mtrand/randomkit.c on line 159, the initial mersenne twister key (populated from /dev/urandom) gets bit-wise and'ed with 0xffffffff.  I'm just curious as why this is done.  A bit-wise and with all ones should just give you your original quantity back, right?  I don't think there is a problem since the operation doesn't really do anything, and the same thing exists in the mersenne twister reference code, but I am curious as to why it is even there in the first place.  Thanks for any thoughts.

Regards,
Mike


From robert.kern at gmail.com  Fri Feb  6 16:25:35 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 6 Feb 2009 15:25:35 -0600
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
	mersenne twister key?
In-Reply-To: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
Message-ID: <3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>

On Fri, Feb 6, 2009 at 15:24, Michael S. Gilbert
<michael.s.gilbert at gmail.com> wrote:
> In numpy/random/mtrand/randomkit.c on line 159, the initial mersenne twister key (populated from /dev/urandom) gets bit-wise and'ed with 0xffffffff.  I'm just curious as why this is done.  A bit-wise and with all ones should just give you your original quantity back, right?  I don't think there is a problem since the operation doesn't really do anything, and the same thing exists in the mersenne twister reference code, but I am curious as to why it is even there in the first place.  Thanks for any thoughts.

On most 64-bit machines, unsigned longs are 64 bits, so 0xffffffffUL
is only 32 bits of 1s.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From dsdale24 at gmail.com  Fri Feb  6 16:25:42 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Fri, 6 Feb 2009 16:25:42 -0500
Subject: [Numpy-discussion] question about ufuncs
In-Reply-To: <a08e5f80902011639g6cbf656bj1817947841d3f5a8@mail.gmail.com>
References: <a08e5f80902011532q3a3f4badqa0b722b8e481ca@mail.gmail.com>
	<48A25BC8-FA0A-44EE-89E2-F6C9939E8A22@gmail.com>
	<a08e5f80902011639g6cbf656bj1817947841d3f5a8@mail.gmail.com>
Message-ID: <a08e5f80902061325k6077debjf2b5b036133d450e@mail.gmail.com>

On Sun, Feb 1, 2009 at 7:39 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> On Sun, Feb 1, 2009 at 7:33 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
>
>>
>> On Feb 1, 2009, at 6:32 PM, Darren Dale wrote:
>> >
>> >
>> > Is there an analog to __array_wrap__ for preprocessing arrays on
>> > their way *into* a ufunc? For example, it would be nice if one could
>> > do something like:
>> >
>> > numpy.sin([1,2,3]*arcseconds)
>> >
>> > where we have the opportunity to inspect the context, convert the
>> > Quantity to units of radians, and then actually call the ufunc. Is
>> > this possible, or does one have to reimplement such functions?
>>
>> Just an idea: look at the code for numpy.ma ufuncs (in numpy.ma.core).
>> By defining a few classes for unary, binary and domained functions,
>> you could probably do what you want, without having to recode all the
>> functions by hand.
>> Another idea would be to define some specific __mul__ or __rmul__
>> rules for your units, so that the list would be transformed into a
>> UnitArray...
>
>
> I have pretty good implementations of the arithmetic operators, so
> ([1,2,3]*m)*([4,5,6]*J) already works. numpy.multiply and numpy.sqrt needed
> help with array_wrap. I'll study your stuff in ma, thanks for the pointer.
>

I've been looking at how ma implements things like multiply() and
MaskedArray.__mul__. I'm surprised that MaskedArray.__mul__ actually calls
ma.multiply() rather than calling super(MaskedArray,self).__mul__(). Maybe
that is the way ndarray does it, but I don't think this is the right
approach for my quantity subclasses. If I want to make a MaskedQuantity
(someday), MaskedQuantity.__mul__ should be calling
super(MaskedQuantity,self).__mul__(), not reimplementations of
numpy.multiply or ma.multiply, right?

As I understand it, the point of __array_wrap__ is to provide a mechanism
such that ndarray subclasses could work with numpy's built-in ufuncs. In my
case, I had been planning to use the calling ufunc as a key to find the
appropriate way to propagate whatever metadata is associated with the
subclass, for example:

    def __array_wrap__(self, obj, context):
        try:
            result =
super(Quantity,self).__array_wrap__(obj,context).view(type(self))
        except:
            result = obj.view(type(self))

        ufunc, objs, huh = context
        result._dimensionality = self._propagate_dimensionality[ufunc](objs)
        return result

Where self._propagate_dimensionality is a dictionary of functions operating
on Dimensionality objects.

There are some cases where the default numpy function expects certain units
on the way in, like the trig functions, which I think would have to be
reimplemented. But aside from that, is there anything wrong with taking this
approach? It seems to allow quantities to integrate pretty well with the
numpy builtins.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090206/33696421/attachment.html>

From robert.kern at gmail.com  Fri Feb  6 16:30:36 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 6 Feb 2009 15:30:36 -0600
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <9457e7c80902060122o365cbc28mf5a0bc22d5bc6f6d@mail.gmail.com>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<498BB9F2.5080501@enthought.com>
	<3d375d730902052029l21f675d1leecb0d405db5dc23@mail.gmail.com>
	<9457e7c80902060122o365cbc28mf5a0bc22d5bc6f6d@mail.gmail.com>
Message-ID: <3d375d730902061330r29c19ce7sb44d03e8b79616fa@mail.gmail.com>

On Fri, Feb 6, 2009 at 03:22, St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi Robert
>
> 2009/2/6 Robert Kern <robert.kern at gmail.com>:
>>> This could be implemented but would require adding information to the
>>> NumPy array.
>>
>> More than that, though. Every function and method that takes an axis
>> or reduces an axis will need to be rewritten. For that reason, I'm -1
>> on the proposal.
>
> Are you -1 on the array dictionary, or on using it to do axis mapping?

I'm -1 on rewriting every axis= argument to accept strings. I'm +1 on
a generic metadata dict that does not implicitly propagate.

>  I would imagine that Gael would be happier even if he had to do
>
> axis = x.meta.axis['Lateral']
> some_func(x, axis)

That's fine with me.

>> I'm of the opinion that it should never guess. We have no idea what
>> semantics are being placed on the dict. Even in the case where all of
>> the inputs have the same dict, the operation may easily invalidate the
>> metadata. For example, a reduction on one of these axis-decorated
>> arrays would make the axis labels incorrect.
>
> That's a good point.  So what would be a sane way of propagating
> meta-data?  If we don't want to make any assumptions, it becomes the
> user's responsibility to do it manually.

I don't think there is *any* sane way of numpy propagating the user's
metadata. The user must be the one to do it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From rmay31 at gmail.com  Fri Feb  6 16:56:32 2009
From: rmay31 at gmail.com (Ryan May)
Date: Fri, 6 Feb 2009 15:56:32 -0600
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <3d375d730902061330r29c19ce7sb44d03e8b79616fa@mail.gmail.com>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<498BB9F2.5080501@enthought.com>
	<3d375d730902052029l21f675d1leecb0d405db5dc23@mail.gmail.com>
	<9457e7c80902060122o365cbc28mf5a0bc22d5bc6f6d@mail.gmail.com>
	<3d375d730902061330r29c19ce7sb44d03e8b79616fa@mail.gmail.com>
Message-ID: <c7b95a020902061356r3f42c1fck98db9d17dbab468@mail.gmail.com>

On Fri, Feb 6, 2009 at 3:30 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Fri, Feb 6, 2009 at 03:22, St?fan van der Walt <stefan at sun.ac.za>
> wrote:
> > Hi Robert
> >
> > 2009/2/6 Robert Kern <robert.kern at gmail.com>:
> >>> This could be implemented but would require adding information to the
> >>> NumPy array.
> >>
> >> More than that, though. Every function and method that takes an axis
> >> or reduces an axis will need to be rewritten. For that reason, I'm -1
> >> on the proposal.
> >
> > Are you -1 on the array dictionary, or on using it to do axis mapping?
>
> I'm -1 on rewriting every axis= argument to accept strings. I'm +1 on
> a generic metadata dict that does not implicitly propagate.
>
> >  I would imagine that Gael would be happier even if he had to do
> >
> > axis = x.meta.axis['Lateral']
> > some_func(x, axis)
>
> That's fine with me.
>
> >> I'm of the opinion that it should never guess. We have no idea what
> >> semantics are being placed on the dict. Even in the case where all of
> >> the inputs have the same dict, the operation may easily invalidate the
> >> metadata. For example, a reduction on one of these axis-decorated
> >> arrays would make the axis labels incorrect.
> >
> > That's a good point.  So what would be a sane way of propagating
> > meta-data?  If we don't want to make any assumptions, it becomes the
> > user's responsibility to do it manually.
>
> I don't think there is *any* sane way of numpy propagating the user's
> metadata. The user must be the one to do it.
>

I'm +1 on all of what Robert said.  I've considered writing a
subclass/wrapping just so I can make metadata available while passing around
recarrays.  It'd save me a bunch of work.  I don't think there's anything
wrong with making the user propagate the dictionary.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090206/52b8c4d9/attachment.html>

From pgmdevlist at gmail.com  Fri Feb  6 17:18:39 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Fri, 6 Feb 2009 17:18:39 -0500
Subject: [Numpy-discussion] question about ufuncs
In-Reply-To: <a08e5f80902061325k6077debjf2b5b036133d450e@mail.gmail.com>
References: <a08e5f80902011532q3a3f4badqa0b722b8e481ca@mail.gmail.com>
	<48A25BC8-FA0A-44EE-89E2-F6C9939E8A22@gmail.com>
	<a08e5f80902011639g6cbf656bj1817947841d3f5a8@mail.gmail.com>
	<a08e5f80902061325k6077debjf2b5b036133d450e@mail.gmail.com>
Message-ID: <457C5E99-AD7D-4592-B589-836D8530A1FE@gmail.com>


On Feb 6, 2009, at 4:25 PM, Darren Dale wrote:

>
> I've been looking at how ma implements things like multiply() and  
> MaskedArray.__mul__. I'm surprised that MaskedArray.__mul__ actually  
> calls ma.multiply() rather than calling  
> super(MaskedArray,self).__mul__().

There's some under-the-hood machinery to deal with the data, and we  
need to be able to manipulate it *before* the operation takes place.  
The super() approach calls __array_wrap__ on the result, so *after*  
the operation took place, and that's not what we wanted...

> Maybe that is the way ndarray does it, but I don't think this is the  
> right approach for my quantity subclasses. If I want to make a  
> MaskedQuantity (someday), MaskedQuantity.__mul__ should be calling  
> super(MaskedQuantity,self).__mul__(), not reimplementations of  
> numpy.multiply or ma.multiply, right?

You'll end up calling ma.multiply anyway  
(super(MaskedQuantity,self).__mul__ will call MaskedArray.__mul__  
which calls ma.multiply... So yes, I think you can stick to the  
super() approach in your case

>
> There are some cases where the default numpy function expects  
> certain units on the way in, like the trig functions, which I think  
> would have to be reimplemented.

And you can probably define a generic class to deal with that instead  
of reimplementing the functions individually (and we're back to the  
initial advice).


> But aside from that, is there anything wrong with taking this  
> approach? It seems to allow quantities to integrate pretty well with  
> the numpy builtins.

Go and try, the problems (if any) will show up...



From michael.s.gilbert at gmail.com  Fri Feb  6 17:57:23 2009
From: michael.s.gilbert at gmail.com (Michael S. Gilbert)
Date: Fri, 6 Feb 2009 17:57:23 -0500
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
 mersenne twister key?
In-Reply-To: <3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
Message-ID: <20090206175723.45e986fb.michael.s.gilbert@gmail.com>

Ok, so isn't this a slight waste of memory then (a doubling on 64-bit platforms)?  Of course the tradeoff is whether you want to maintain two codebases for 32- and 64-bit or just one.  The advantages of a single codebase probably outweight an increase in memory usage since we're only talking about the difference between 2kB and 4kB, which is fairly insignificant.

Mike

On Fri, 6 Feb 2009 15:25:35 -0600
Robert Kern <robert.kern at gmail.com> wrote:

> On Fri, Feb 6, 2009 at 15:24, Michael S. Gilbert
> <michael.s.gilbert at gmail.com> wrote:
> > In numpy/random/mtrand/randomkit.c on line 159, the initial mersenne twister key (populated from /dev/urandom) gets bit-wise and'ed with 0xffffffff.  I'm just curious as why this is done.  A bit-wise and with all ones should just give you your original quantity back, right?  I don't think there is a problem since the operation doesn't really do anything, and the same thing exists in the mersenne twister reference code, but I am curious as to why it is even there in the first place.  Thanks for any thoughts.
> 
> On most 64-bit machines, unsigned longs are 64 bits, so 0xffffffffUL
> is only 32 bits of 1s.
> 
> -- 
> Robert Kern
> 
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion


From robert.kern at gmail.com  Fri Feb  6 17:56:00 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 6 Feb 2009 16:56:00 -0600
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
	mersenne twister key?
In-Reply-To: <20090206175723.45e986fb.michael.s.gilbert@gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>
Message-ID: <3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>

On Fri, Feb 6, 2009 at 16:57, Michael S. Gilbert
<michael.s.gilbert at gmail.com> wrote:
> Ok, so isn't this a slight waste of memory then (a doubling on 64-bit platforms)?  Of course the tradeoff is whether you want to maintain two codebases for 32- and 64-bit or just one.  The advantages of a single codebase probably outweight an increase in memory usage since we're only talking about the difference between 2kB and 4kB, which is fairly insignificant.

I'm not going to modify the upstream source and risk introducing bugs.

PS: I am on the mailing list. You do not need to Cc: me.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From michael.s.gilbert at gmail.com  Fri Feb  6 18:14:31 2009
From: michael.s.gilbert at gmail.com (Michael S. Gilbert)
Date: Fri, 6 Feb 2009 18:14:31 -0500
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
 mersenne twister key?
In-Reply-To: <3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>
	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
Message-ID: <20090206181431.cdf5e7f0.michael.s.gilbert@gmail.com>

> I'm not going to modify the upstream source and risk introducing bugs.

I agree, its not worth risking it to save 2k of memory.


From dsdale24 at gmail.com  Fri Feb  6 18:11:10 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Fri, 6 Feb 2009 18:11:10 -0500
Subject: [Numpy-discussion] question about ufuncs
In-Reply-To: <457C5E99-AD7D-4592-B589-836D8530A1FE@gmail.com>
References: <a08e5f80902011532q3a3f4badqa0b722b8e481ca@mail.gmail.com>
	<48A25BC8-FA0A-44EE-89E2-F6C9939E8A22@gmail.com>
	<a08e5f80902011639g6cbf656bj1817947841d3f5a8@mail.gmail.com>
	<a08e5f80902061325k6077debjf2b5b036133d450e@mail.gmail.com>
	<457C5E99-AD7D-4592-B589-836D8530A1FE@gmail.com>
Message-ID: <a08e5f80902061511q3ed62530ic89ac747eb721642@mail.gmail.com>

On Fri, Feb 6, 2009 at 5:18 PM, Pierre GM <pgmdevlist at gmail.com> wrote:

>
> On Feb 6, 2009, at 4:25 PM, Darren Dale wrote:
>
> >
> > I've been looking at how ma implements things like multiply() and
> > MaskedArray.__mul__. I'm surprised that MaskedArray.__mul__ actually
> > calls ma.multiply() rather than calling
> > super(MaskedArray,self).__mul__().
>
> There's some under-the-hood machinery to deal with the data, and we
> need to be able to manipulate it *before* the operation takes place.
> The super() approach calls __array_wrap__ on the result, so *after*
> the operation took place, and that's not what we wanted...
>

It looks like there are enough cases where manipulation needs to happen on
the way in that it might be useful to consider a mechanism for doing so. It
could avoid the need for lots of wrappers and decorators down the road.


>
> > Maybe that is the way ndarray does it, but I don't think this is the
> > right approach for my quantity subclasses. If I want to make a
> > MaskedQuantity (someday), MaskedQuantity.__mul__ should be calling
> > super(MaskedQuantity,self).__mul__(), not reimplementations of
> > numpy.multiply or ma.multiply, right?
>
> You'll end up calling ma.multiply anyway
> (super(MaskedQuantity,self).__mul__ will call MaskedArray.__mul__
> which calls ma.multiply... So yes, I think you can stick to the
> super() approach in your case
>
> >
> > There are some cases where the default numpy function expects
> > certain units on the way in, like the trig functions, which I think
> > would have to be reimplemented.
>
> And you can probably define a generic class to deal with that instead
> of reimplementing the functions individually (and we're back to the
> initial advice).
>
>
> > But aside from that, is there anything wrong with taking this
> > approach? It seems to allow quantities to integrate pretty well with
> > the numpy builtins.
>
> Go and try, the problems (if any) will show up...
>

Oh boy...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090206/85e6b5e1/attachment.html>

From michael.s.gilbert at gmail.com  Fri Feb  6 18:18:44 2009
From: michael.s.gilbert at gmail.com (Michael S. Gilbert)
Date: Fri, 6 Feb 2009 18:18:44 -0500
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
 mersenne twister key?
In-Reply-To: <3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>
	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
Message-ID: <20090206181844.9b2070ff.michael.s.gilbert@gmail.com>

> I'm not going to modify the upstream source and risk introducing bugs.

BTW, there is a 64-bit version of the reference mersenne twister 
implementation available [1].

Mike

[1] http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/VERSIONS/C-LANG/mt19937-64.c


From dsdale24 at gmail.com  Fri Feb  6 18:55:39 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Fri, 6 Feb 2009 18:55:39 -0500
Subject: [Numpy-discussion] question about ufuncs
In-Reply-To: <a08e5f80902061511q3ed62530ic89ac747eb721642@mail.gmail.com>
References: <a08e5f80902011532q3a3f4badqa0b722b8e481ca@mail.gmail.com>
	<48A25BC8-FA0A-44EE-89E2-F6C9939E8A22@gmail.com>
	<a08e5f80902011639g6cbf656bj1817947841d3f5a8@mail.gmail.com>
	<a08e5f80902061325k6077debjf2b5b036133d450e@mail.gmail.com>
	<457C5E99-AD7D-4592-B589-836D8530A1FE@gmail.com>
	<a08e5f80902061511q3ed62530ic89ac747eb721642@mail.gmail.com>
Message-ID: <a08e5f80902061555x572ae119qe4422496a4f28312@mail.gmail.com>

On Fri, Feb 6, 2009 at 6:11 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> On Fri, Feb 6, 2009 at 5:18 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
>
>>
>> On Feb 6, 2009, at 4:25 PM, Darren Dale wrote:
>>
>> >
>> > I've been looking at how ma implements things like multiply() and
>> > MaskedArray.__mul__. I'm surprised that MaskedArray.__mul__ actually
>> > calls ma.multiply() rather than calling
>> > super(MaskedArray,self).__mul__().
>>
>> There's some under-the-hood machinery to deal with the data, and we
>> need to be able to manipulate it *before* the operation takes place.
>> The super() approach calls __array_wrap__ on the result, so *after*
>> the operation took place, and that's not what we wanted...
>>
>
> It looks like there are enough cases where manipulation needs to happen on
> the way in that it might be useful to consider a mechanism for doing so. It
> could avoid the need for lots of wrappers and decorators down the road.
>
>
>>
>> > Maybe that is the way ndarray does it, but I don't think this is the
>> > right approach for my quantity subclasses. If I want to make a
>> > MaskedQuantity (someday), MaskedQuantity.__mul__ should be calling
>> > super(MaskedQuantity,self).__mul__(), not reimplementations of
>> > numpy.multiply or ma.multiply, right?
>>
>> You'll end up calling ma.multiply anyway
>> (super(MaskedQuantity,self).__mul__ will call MaskedArray.__mul__
>> which calls ma.multiply... So yes, I think you can stick to the
>> super() approach in your case
>>
>> >
>> > There are some cases where the default numpy function expects
>> > certain units on the way in, like the trig functions, which I think
>> > would have to be reimplemented.
>>
>> And you can probably define a generic class to deal with that instead
>> of reimplementing the functions individually (and we're back to the
>> initial advice).
>>
>>
>> > But aside from that, is there anything wrong with taking this
>> > approach? It seems to allow quantities to integrate pretty well with
>> > the numpy builtins.
>>
>> Go and try, the problems (if any) will show up...
>>
>
> Oh boy...
>

Well, without an analog to __array_wrap__ on the way in, the out parameter
for ufuncs poses a serious downside:

q1=1*m
q2=1*ft
numpy.add(q1,q2,q1)

I can raise an error complaining that you cant add meters and feet, but if I
catch it in __array_wrap__ it is too late to prevent overwriting q1 with
meaningless data. Also:

q1=[[1,1],[1,1]]*m
q2=[[2,2],[2,2]]*m
numpy.multiply(q1[0],q2[0],q1[0])

This will again raise an error, since quantities will not allow you to
attempt to change the units of a view. But numpy.multiply would have already
modified the data.

It seems like integrating quantities with existing numpy and scipy routines
would be greatly simplified if I could take advantage of __array_wrap__, but
I don't think I can use it if I can't raise errors in time to avoid
corrupting data in place. I guess for the purposes of demonstrating the
packages usefulness, for now I should wrap or reimplement numpy's ufuncs and
defer using __array_wrap__ unless it becomes possible for subclasses to
manipulate data on the way in to ufuncs.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090206/0e8dc7e2/attachment.html>

From alan at ajackson.org  Fri Feb  6 22:39:08 2009
From: alan at ajackson.org (Alan Jackson)
Date: Fri, 6 Feb 2009 21:39:08 -0600
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <498BB9F2.5080501@enthought.com>
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<498BB9F2.5080501@enthought.com>
Message-ID: <20090206213908.105ddcb9@ajackson.org>

On Thu, 05 Feb 2009 22:17:54 -0600
Travis Oliphant <oliphant at enthought.com> wrote:

> Gael Varoquaux wrote:
> > On Thu, Feb 05, 2009 at 05:08:49PM -0600, Travis E. Oliphant wrote:
> >   
> >> I've been fairly quiet on this list for awhile due to work and family 
> >> schedule, but I think about how things can improve regularly.    One 
> >> feature that's been requested by a few people is the ability to select 
> >> multiple fields from a structured array.
> >>     
> 
> 
> -Travis

Seems to me that would be getting numpy arrays very close to the
R data frame object - which could be a good thing.


-- 
-----------------------------------------------------------------------
| Alan K. Jackson            | To see a World in a Grain of Sand      |
| alan at ajackson.org          | And a Heaven in a Wild Flower,         |
| www.ajackson.org           | Hold Infinity in the palm of your hand |
| Houston, Texas             | And Eternity in an hour. - Blake       |
-----------------------------------------------------------------------


From ondrej at certik.cz  Sat Feb  7 01:04:00 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 6 Feb 2009 22:04:00 -0800
Subject: [Numpy-discussion] preferred numpy build system
Message-ID: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>

Hi,

I have couple beginners questions about numscons. What is the
preferred build system for numpy now, is it numscons? The README
doesn't mention numscons, so I am a bit confused what the future plan
is.

Also by doing:

$ python setupscons.py install
Running from numpy source directory.
Traceback (most recent call last):
  File "setupscons.py", line 56, in <module>
    raise DistutilsError('\n'.join(msg))
distutils.errors.DistutilsError: You cannot build numpy with scons
without the numscons package
(Failure was: No module named numscons)


so the numscons package needs to be installed -- but it is not in
Debian. So is it supposed to be in Debian? Is numscons supposed to be
a build system for other projects as well? Why not to just send the
needed patches to scons and just use scons?

Thanks,
Ondrej


From nwagner at iam.uni-stuttgart.de  Sat Feb  7 08:03:15 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 07 Feb 2009 14:03:15 +0100
Subject: [Numpy-discussion] ERROR: Test flat on masked_matrices
Message-ID: <web-116767274@uni-stuttgart.de>


======================================================================
ERROR: Test flat on masked_matrices
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/numpy/ma/tests/test_core.py", 
line 1127, in test_flat
     test = ma.array(np.matrix([[1, 2, 3]]), mask=[0, 0, 
1])
NameError: global name 'ma' is not defined

----------------------------------------------------------------------
Ran 1897 tests in 14.713s

FAILED (KNOWNFAIL=9, errors=1)
<nose.result.TextTestResult run=1897 errors=1 failures=0>
  


From neilcrighton at gmail.com  Sat Feb  7 09:27:43 2009
From: neilcrighton at gmail.com (Neil)
Date: Sat, 7 Feb 2009 14:27:43 +0000 (UTC)
Subject: [Numpy-discussion] Selection of only a certain number of fields
References: <498B7181.5000300@enthought.com>
Message-ID: <loom.20090207T142122-739@post.gmane.org>

Travis E. Oliphant <oliphant <at> enthought.com> writes:

> I've been fairly quiet on this list for awhile due to work and family 
> schedule, but I think about how things can improve regularly.    One 
> feature that's been requested by a few people is the ability to select 
> multiple fields from a structured array.
> 
> Thus,  suppose *arr* is a structured array with dtype:
> 
> [('name', 'S25'),
>   ('height', float),
>   ('age', int),
>   ('gender', 'S8')
> ]
> 
> Then,  newarr = arr[['name', 'age']]  should be a structured array with 
> just the name and age fields.
> 

What are some common use cases for this feature?

I use structured arrays quite a lot, but I haven't found myself wanting
something like this. If I do need a subset of a structured array generally I use
something like 

[rec[n] for n in 'name age gender'.split()]

For me that use case doesn't come up very often though. 




From cournape at gmail.com  Sat Feb  7 09:42:52 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 7 Feb 2009 23:42:52 +0900
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
Message-ID: <5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>

Hi Ondrej,

On Sat, Feb 7, 2009 at 3:04 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> I have couple beginners questions about numscons. What is the
> preferred build system for numpy now, is it numscons? The README
> doesn't mention numscons, so I am a bit confused what the future plan
> is.
>

Sorry for the confusion: numscons is NOT the preferred build system.
The current numpy.distutils extensions, as shipped by numpy, is the
preferred one. Numscons is more an experiment, if you want.

> Also by doing:
>
> $ python setupscons.py install
> Running from numpy source directory.
> Traceback (most recent call last):
>  File "setupscons.py", line 56, in <module>
>    raise DistutilsError('\n'.join(msg))
> distutils.errors.DistutilsError: You cannot build numpy with scons
> without the numscons package
> (Failure was: No module named numscons)
>
>
> so the numscons package needs to be installed -- but it is not in
> Debian.

No, it not.

> So is it supposed to be in Debian?

No, I don't think it should be. It is not yet stabilized code wise, so
it does not make much sense to package it.

> Is numscons supposed to be
> a build system for other projects as well? Why not to just send the
> needed patches to scons and just use scons?

Because you cannot just use scons. Numscons is a library build on top
of scons, for the needs of numpy. There also needs to be some hook
from numpy.distutils to use scons (numscons adds a new distutils
command, which is used instead of build to build any compiled
code-based extensions). Most of the changes needed for scons have been
integrated upstream, though, except one or two things.

David


From suchindra at gmail.com  Sat Feb  7 10:08:34 2009
From: suchindra at gmail.com (Suchindra Sandhu)
Date: Sat, 7 Feb 2009 10:08:34 -0500
Subject: [Numpy-discussion] numpy.any oddity
In-Reply-To: <3d375d730902061239p791234e7k8d0f9d04e2a66464@mail.gmail.com>
References: <b39d0d980902061124h4cd14c8dw6d26eee1e81405fc@mail.gmail.com>
	<3d375d730902061239p791234e7k8d0f9d04e2a66464@mail.gmail.com>
Message-ID: <b39d0d980902070708u3f8830a7g694de810411304a0@mail.gmail.com>

Thanks. I tried the latest version and indeed there is no leak.

Cheers,
Suchindra

On Fri, Feb 6, 2009 at 3:39 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Fri, Feb 6, 2009 at 13:24, Suchindra Sandhu <suchindra at gmail.com>
> wrote:
> > Hi,
> >
> > I accidently stumbled upon this odd behavior by numpy.any. The following
> > code leaks memory -
> >
> > for i in xrange(10000000):
> >         print N.any({'whatever': N.arange(10000000)})
> >
> > Ofcourse, I called "any" on a dict object by accident, but it should not
> > really leak memory.
> >
> > I am running numpy version 1.0.4 with python 2.5.2
>
> Upgrade to a more recent version of numpy. I do not see a leak. I
> vaguely recall there being a problem in the 1.0.x series with
> object-dtype scalars, which is what numpy.any() will convert a dict
> to.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090207/6bcbc5b8/attachment.html>

From faltet at pytables.org  Sat Feb  7 13:17:57 2009
From: faltet at pytables.org (Francesc Alted)
Date: Sat, 7 Feb 2009 19:17:57 +0100
Subject: [Numpy-discussion] Selection of only a certain number of fields
In-Reply-To: <loom.20090207T142122-739@post.gmane.org>
References: <498B7181.5000300@enthought.com>
	<loom.20090207T142122-739@post.gmane.org>
Message-ID: <200902071917.57537.faltet@pytables.org>

A Saturday 07 February 2009, Neil escrigu?:
> Travis E. Oliphant <oliphant <at> enthought.com> writes:
> > I've been fairly quiet on this list for awhile due to work and
> > family schedule, but I think about how things can improve
> > regularly.    One feature that's been requested by a few people is
> > the ability to select multiple fields from a structured array.
> >
> > Thus,  suppose *arr* is a structured array with dtype:
> >
> > [('name', 'S25'),
> >   ('height', float),
> >   ('age', int),
> >   ('gender', 'S8')
> > ]
> >
> > Then,  newarr = arr[['name', 'age']]  should be a structured array
> > with just the name and age fields.
>
> What are some common use cases for this feature?
>
> I use structured arrays quite a lot, but I haven't found myself
> wanting something like this. If I do need a subset of a structured
> array generally I use something like
>
> [rec[n] for n in 'name age gender'.split()]

Good point.  However, there are still some very valid reasons for having 
an idiom like:

newarr = arr[['name', 'age']]

returning a record array.

The first one (and most important IMO), is that newarr continues to be 
an structured array (BTW, when changed this name from the original 
record array?), and you can use all the features of these beasts with 
it.  Other reason (albeit a bit secondary) is that its data buffer can 
be shared through the array interface with other applications, or plain 
C code, in a relatively straightforward way.  However, if newarr 
becomes a list (or dictionary), this is simply not possible.

Cheers,

-- 
Francesc Alted


From ondrej at certik.cz  Sat Feb  7 13:21:55 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Sat, 7 Feb 2009 10:21:55 -0800
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
Message-ID: <85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>

Hi David,

> Sorry for the confusion: numscons is NOT the preferred build system.
> The current numpy.distutils extensions, as shipped by numpy, is the
> preferred one. Numscons is more an experiment, if you want.

Ah, I see, thanks for the clarification.

>> So is it supposed to be in Debian?
>
> No, I don't think it should be. It is not yet stabilized code wise, so
> it does not make much sense to package it.

Ok.

>
>> Is numscons supposed to be
>> a build system for other projects as well? Why not to just send the
>> needed patches to scons and just use scons?
>
> Because you cannot just use scons. Numscons is a library build on top
> of scons, for the needs of numpy. There also needs to be some hook
> from numpy.distutils to use scons (numscons adds a new distutils
> command, which is used instead of build to build any compiled
> code-based extensions). Most of the changes needed for scons have been
> integrated upstream, though, except one or two things.

I see. I think it's a bit confusing that one needs to build a new
build system just to build numpy, e.g. that both distutils and scons
are not good enough.

Ondrej


From pgmdevlist at gmail.com  Sat Feb  7 13:52:18 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sat, 7 Feb 2009 13:52:18 -0500
Subject: [Numpy-discussion] ERROR: Test flat on masked_matrices
In-Reply-To: <web-116767274@uni-stuttgart.de>
References: <web-116767274@uni-stuttgart.de>
Message-ID: <D44EFAC2-C35A-4177-8E50-A13D9780FC04@gmail.com>


On Feb 7, 2009, at 8:03 AM, Nils Wagner wrote:

>
> ======================================================================
> ERROR: Test flat on masked_matrices
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/usr/local/lib64/python2.5/site-packages/numpy/ma/tests/ 
> test_core.py",
> line 1127, in test_flat
>     test = ma.array(np.matrix([[1, 2, 3]]), mask=[0, 0,
> 1])
> NameError: global name 'ma' is not defined

Oops, sorry about that...


From ellisonbg.net at gmail.com  Sat Feb  7 13:55:19 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sat, 7 Feb 2009 10:55:19 -0800
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
	<85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
Message-ID: <6ce0ac130902071055k2bed3c2ape8c9b633429e38cf@mail.gmail.com>

> I see. I think it's a bit confusing that one needs to build a new
> build system just to build numpy, e.g. that both distutils and scons
> are not good enough.

I would not say that numscons is a *new* build system.  Rather, I look
at numscons as a glue layer that allows scons to be used within
distutils for building extensions and libraries.
distutils/numpy/distutils does things that scons doesn't and scons
does things that distutils doesn't.  But you definitely need the glue
layer to use both approaches together.

Cheers,

Brian



> Ondrej
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From oliphant at enthought.com  Sat Feb  7 15:45:28 2009
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Sat, 07 Feb 2009 14:45:28 -0600
Subject: [Numpy-discussion] Selection of only a certain number of fields
In-Reply-To: <200902071917.57537.faltet@pytables.org>
References: <498B7181.5000300@enthought.com>	<loom.20090207T142122-739@post.gmane.org>
	<200902071917.57537.faltet@pytables.org>
Message-ID: <498DF2E8.8060500@enthought.com>

Francesc Alted wrote:
> A Saturday 07 February 2009, Neil escrigu?:
>   
>> Travis E. Oliphant <oliphant <at> enthought.com> writes:
>>     
>>> I've been fairly quiet on this list for awhile due to work and
>>> family schedule, but I think about how things can improve
>>> regularly.    One feature that's been requested by a few people is
>>> the ability to select multiple fields from a structured array.
>>>
>>> Thus,  suppose *arr* is a structured array with dtype:
>>>
>>> [('name', 'S25'),
>>>   ('height', float),
>>>   ('age', int),
>>>   ('gender', 'S8')
>>> ]
>>>
>>> Then,  newarr = arr[['name', 'age']]  should be a structured array
>>> with just the name and age fields.
>>>       
>> What are some common use cases for this feature?
>>
>> I use structured arrays quite a lot, but I haven't found myself
>> wanting something like this. If I do need a subset of a structured
>> array generally I use something like
>>
>> [rec[n] for n in 'name age gender'.split()]
>>     
>
> Good point.  However, there are still some very valid reasons for having 
> an idiom like:
>
> newarr = arr[['name', 'age']]
>
> returning a record array.
>
> The first one (and most important IMO), is that newarr continues to be 
> an structured array (BTW, when changed this name from the original 
> record array?), 
To avoid confusion with the "record array" subclass which maps 
attributes to fields, Eric Jones and I have been using this terminology 
for about a year. 

-Travis



-- 

Travis Oliphant
Enthought, Inc.
(512) 536-1057 (office)
(512) 536-1059 (fax)
http://www.enthought.com
oliphant at enthought.com



From stefan at sun.ac.za  Sun Feb  8 04:54:04 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 8 Feb 2009 11:54:04 +0200
Subject: [Numpy-discussion] Behaviour of integer powers
Message-ID: <9457e7c80902080154h7d65e8cbl9aaf451ef2ff389@mail.gmail.com>

Hi all,

Ticket #955 (http://scipy.org/scipy/numpy/ticket/955) touches on the
following issue:

>>> 0.0 ** np.array([-1, 0, 1], dtype=np.int32)
array([ Inf,   1.,   0.])
>>> 0.0 ** np.array([-1, 0, 1], dtype=np.int32)[0]
------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython console>", line 1, in <module>
ZeroDivisionError: 0.0 cannot be raised to a negative power

This is on a 32-bit platform.

As I understand this happens because, in the second case, Python sees
that "-1" is an int and does the power operation.  In other words,
when we raise to the power of an array, the NumPy machinery is
involved, whereas if we raise to np.int32(-1), it is not.  This is due
to the int32 type deriving from the Python int.

I can't think of an easy way to address the problem, but I was hoping
to get some advice from the list.

Thanks
St?fan


From robert.kern at gmail.com  Sun Feb  8 05:09:23 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 8 Feb 2009 04:09:23 -0600
Subject: [Numpy-discussion] Behaviour of integer powers
In-Reply-To: <9457e7c80902080154h7d65e8cbl9aaf451ef2ff389@mail.gmail.com>
References: <9457e7c80902080154h7d65e8cbl9aaf451ef2ff389@mail.gmail.com>
Message-ID: <3d375d730902080209i22224febp218d62345d76474b@mail.gmail.com>

On Sun, Feb 8, 2009 at 03:54, St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi all,
>
> Ticket #955 (http://scipy.org/scipy/numpy/ticket/955) touches on the
> following issue:
>
>>>> 0.0 ** np.array([-1, 0, 1], dtype=np.int32)
> array([ Inf,   1.,   0.])
>>>> 0.0 ** np.array([-1, 0, 1], dtype=np.int32)[0]
> ------------------------------------------------------------
> Traceback (most recent call last):
>  File "<ipython console>", line 1, in <module>
> ZeroDivisionError: 0.0 cannot be raised to a negative power
>
> This is on a 32-bit platform.
>
> As I understand this happens because, in the second case, Python sees
> that "-1" is an int and does the power operation.  In other words,
> when we raise to the power of an array, the NumPy machinery is
> involved, whereas if we raise to np.int32(-1), it is not.  This is due
> to the int32 type deriving from the Python int.
>
> I can't think of an easy way to address the problem, but I was hoping
> to get some advice from the list.

I don't think there is anything we can do to fix this except not to
subclass from int. I think float.__pow__(self, other) checks that
isinstance(other, int) and does its own thing. numpy.int_.__rpow__
will never get called, and that's the only place we can implement our
logic.

We can document the wart and recommend casting the base to a float64
scalar first.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From cournape at gmail.com  Sun Feb  8 06:10:08 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 8 Feb 2009 20:10:08 +0900
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
	<85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
Message-ID: <5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>

On Sun, Feb 8, 2009 at 3:21 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi David,
>
>> Sorry for the confusion: numscons is NOT the preferred build system.
>> The current numpy.distutils extensions, as shipped by numpy, is the
>> preferred one. Numscons is more an experiment, if you want.
>
> Ah, I see, thanks for the clarification.
>
>>> So is it supposed to be in Debian?
>>
>> No, I don't think it should be. It is not yet stabilized code wise, so
>> it does not make much sense to package it.
>
> Ok.
>
>>
>>> Is numscons supposed to be
>>> a build system for other projects as well? Why not to just send the
>>> needed patches to scons and just use scons?
>>
>> Because you cannot just use scons. Numscons is a library build on top
>> of scons, for the needs of numpy. There also needs to be some hook
>> from numpy.distutils to use scons (numscons adds a new distutils
>> command, which is used instead of build to build any compiled
>> code-based extensions). Most of the changes needed for scons have been
>> integrated upstream, though, except one or two things.
>
> I see. I think it's a bit confusing that one needs to build a new
> build system just to build numpy, e.g. that both distutils and scons
> are not good enough.

I don't find it that surprising - numpy and scipy require some
relatively advanced features (mixed language and cross-platform with
support for many toolchains). Within the open source tools, I know
only two which can handle those requirements: scons and cmake. For
example, it would almost impossible to build numpy/scipy with
autoconf.

David


From josef.pktd at gmail.com  Sun Feb  8 08:02:10 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 8 Feb 2009 08:02:10 -0500
Subject: [Numpy-discussion] Behaviour of integer powers
In-Reply-To: <3d375d730902080209i22224febp218d62345d76474b@mail.gmail.com>
References: <9457e7c80902080154h7d65e8cbl9aaf451ef2ff389@mail.gmail.com>
	<3d375d730902080209i22224febp218d62345d76474b@mail.gmail.com>
Message-ID: <1cd32cbb0902080502q492e105dl6115e2ae87b94740@mail.gmail.com>

On Sun, Feb 8, 2009 at 5:09 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Sun, Feb 8, 2009 at 03:54, St?fan van der Walt <stefan at sun.ac.za> wrote:
>> Hi all,
>>
>> Ticket #955 (http://scipy.org/scipy/numpy/ticket/955) touches on the
>> following issue:
>>
>>>>> 0.0 ** np.array([-1, 0, 1], dtype=np.int32)
>> array([ Inf,   1.,   0.])
>>>>> 0.0 ** np.array([-1, 0, 1], dtype=np.int32)[0]
>> ------------------------------------------------------------
>> Traceback (most recent call last):
>>  File "<ipython console>", line 1, in <module>
>> ZeroDivisionError: 0.0 cannot be raised to a negative power
>>
>> This is on a 32-bit platform.
>>
>> As I understand this happens because, in the second case, Python sees
>> that "-1" is an int and does the power operation.  In other words,
>> when we raise to the power of an array, the NumPy machinery is
>> involved, whereas if we raise to np.int32(-1), it is not.  This is due
>> to the int32 type deriving from the Python int.
>>
>> I can't think of an easy way to address the problem, but I was hoping
>> to get some advice from the list.
>
> I don't think there is anything we can do to fix this except not to
> subclass from int. I think float.__pow__(self, other) checks that
> isinstance(other, int) and does its own thing. numpy.int_.__rpow__
> will never get called, and that's the only place we can implement our
> logic.
>
> We can document the wart and recommend casting the base to a float64
> scalar first.
>
> --
> Robert Kern
>

I thought when in doubt about the domain, the useful way around this, is to call
np.power directly. This is how it is used in stats.distributions.

>>> 0.0 ** np.array([-1, 0, 1], dtype=np.int32)[0]
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
ZeroDivisionError: 0.0 cannot be raised to a negative power
>>> np.power(0.0, np.array([-1, 0, 1], dtype=np.int32)[0])
1.#INF
>>> 0.01 ** np.array([-1, 0, 1], dtype=np.int32)[0]
100.0
>>> np.power(np.nan, np.array([-1, 0, 1], dtype=np.int32)[0])
-1.#IND
>>> np.nan ** np.array([-1, 0, 1], dtype=np.int32)[0]
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
ValueError: (33, 'Domain error')


But I just found that nan in the exponent in an array are not propagated:

>>> 0.0 ** np.array([-np.nan, 0, 1], dtype=np.int32)[0]
1.0
>>> np.power(0.0, np.array([-np.nan, 0, 1], dtype=np.int32)[0])
1.0
>>> np.power(0.0, -np.nan)
1.#QNAN
>>> np.power(0.0, np.nan)
-1.#IND

>>> 0.0**np.nan
0.0


From josef.pktd at gmail.com  Sun Feb  8 08:19:08 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 8 Feb 2009 08:19:08 -0500
Subject: [Numpy-discussion] Behaviour of integer powers
In-Reply-To: <1cd32cbb0902080502q492e105dl6115e2ae87b94740@mail.gmail.com>
References: <9457e7c80902080154h7d65e8cbl9aaf451ef2ff389@mail.gmail.com>
	<3d375d730902080209i22224febp218d62345d76474b@mail.gmail.com>
	<1cd32cbb0902080502q492e105dl6115e2ae87b94740@mail.gmail.com>
Message-ID: <1cd32cbb0902080519m283a64cdu4c1504905f85ac3c@mail.gmail.com>

On Sun, Feb 8, 2009 at 8:02 AM,  <josef.pktd at gmail.com> wrote:

> But I just found that nan in the exponent in an array are not propagated:
>
>>>> 0.0 ** np.array([-np.nan, 0, 1], dtype=np.int32)[0]
> 1.0
>>>> np.power(0.0, np.array([-np.nan, 0, 1], dtype=np.int32)[0])
> 1.0

correction:
np.power propagates nans, it's again the casting of nan to zero int,
that tripped me up.

I shouln't cut and paste examples too fast.

>>> np.power(0.0, np.array([-np.nan, 0, 1]))
array([ NaN,   1.,   0.])
>>> np.power(0.0, np.array([-np.nan, 0, 1]))[0]
1.#QNAN

Josef


From neilcrighton at gmail.com  Sun Feb  8 11:09:02 2009
From: neilcrighton at gmail.com (Neil)
Date: Sun, 8 Feb 2009 16:09:02 +0000 (UTC)
Subject: [Numpy-discussion] Selection of only a certain number of fields
References: <498B7181.5000300@enthought.com>
	<loom.20090207T142122-739@post.gmane.org>
	<200902071917.57537.faltet@pytables.org>
Message-ID: <loom.20090208T154447-548@post.gmane.org>

Francesc Alted <faltet <at> pytables.org> writes:

> > What are some common use cases for this feature?
> >
> > I use structured arrays quite a lot, but I haven't found myself
> > wanting something like this. If I do need a subset of a structured
> > array generally I use something like
> >
> > [rec[n] for n in 'name age gender'.split()]
> 
> Good point.  However, there are still some very valid reasons for having 
> an idiom like:
> 
> newarr = arr[['name', 'age']]
> 
> returning a record array.
> 
> The first one (and most important IMO), is that newarr continues to be 
> an structured array (BTW, when changed this name from the original 
> record array?), and you can use all the features of these beasts with 
> it.  Other reason (albeit a bit secondary) is that its data buffer can 
> be shared through the array interface with other applications, or plain 
> C code, in a relatively straightforward way.  However, if newarr 
> becomes a list (or dictionary), this is simply not possible.
> 
> Cheers,
> 

That's not a sample use case ;)

One of the things I love about Python is that it has a small core set of
features and tries to avoid having many ways to do the same thing.  This makes
it extremely easy to learn.  With every new feature, numpy gets a little bit
harder to learn, there's more to document and the code base gets larger and so
harder to maintain.  In those senses, whenever you add a new function/feature to
numpy, it gets a little bit worse.

So I think it would be nice to have some concrete examples of what the new
feature will be useful for, just to show how it outweighs those negatives.  As a
bonus, they'd provide nice examples to put in the documentation :).

Neil

PS.  Thanks for your work on pytables!  I've used it quite a bit, mostly for
reading hdf5 files.




From dsdale24 at gmail.com  Sun Feb  8 11:49:12 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 8 Feb 2009 11:49:12 -0500
Subject: [Numpy-discussion] segfaults when passing ndarray subclass to ufunc
	with out=None
Message-ID: <a08e5f80902080849w3e271032i87600bee3ca76cb8@mail.gmail.com>

I am seeing some really strange behavior when I try to pass an ndarray
subclass and out=None to numpy's ufuncs. This example will reproduce the
problem with svn numpy, the first print statement yields 1 as expected, the
second yields  "<type 'builtin_function_or_method'>" and the third yields a
segmentation fault:

import numpy as np

class MyArray(np.ndarray):

    __array_priority__ = 20

    def __new__(cls):
        return np.asarray(1).view(cls).copy()

    def __repr__(self):
        return 'my_array'

    __str__ = __repr__

    def __mul__(self, other):
        return super(MyArray, self).__mul__(other)

    def __rmul__(self, other):
        return super(MyArray, self).__rmul__(other)

mine = MyArray()
print np.multiply(1, 1, None)
x = np.multiply(mine, mine, None)
print type(x)
print x

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090208/154cef92/attachment.html>

From ondrej at certik.cz  Sun Feb  8 14:39:48 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Sun, 8 Feb 2009 11:39:48 -0800
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
	<85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
	<5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>
Message-ID: <85b5c3130902081139n45e6b72dl770862baa87065c4@mail.gmail.com>

On Sun, Feb 8, 2009 at 3:10 AM, David Cournapeau <cournape at gmail.com> wrote:
> On Sun, Feb 8, 2009 at 3:21 AM, Ondrej Certik <ondrej at certik.cz> wrote:
>> Hi David,
>>
>>> Sorry for the confusion: numscons is NOT the preferred build system.
>>> The current numpy.distutils extensions, as shipped by numpy, is the
>>> preferred one. Numscons is more an experiment, if you want.
>>
>> Ah, I see, thanks for the clarification.
>>
>>>> So is it supposed to be in Debian?
>>>
>>> No, I don't think it should be. It is not yet stabilized code wise, so
>>> it does not make much sense to package it.
>>
>> Ok.
>>
>>>
>>>> Is numscons supposed to be
>>>> a build system for other projects as well? Why not to just send the
>>>> needed patches to scons and just use scons?
>>>
>>> Because you cannot just use scons. Numscons is a library build on top
>>> of scons, for the needs of numpy. There also needs to be some hook
>>> from numpy.distutils to use scons (numscons adds a new distutils
>>> command, which is used instead of build to build any compiled
>>> code-based extensions). Most of the changes needed for scons have been
>>> integrated upstream, though, except one or two things.
>>
>> I see. I think it's a bit confusing that one needs to build a new
>> build system just to build numpy, e.g. that both distutils and scons
>> are not good enough.
>
> I don't find it that surprising - numpy and scipy require some
> relatively advanced features (mixed language and cross-platform with
> support for many toolchains). Within the open source tools, I know
> only two which can handle those requirements: scons and cmake. For
> example, it would almost impossible to build numpy/scipy with
> autoconf.

Yes, I am investigating cmake, it's pretty cool. I wrote some macros
for cython etc. What I like about cmake is that it is cross platform
and it just produces makefiles on linux, or visual studio files (or
whatever) on windows.  When I get more experience with it, I'll post
here.

What I don't like on cmake is that it uses it's own language, instead
of python, on the other hand, so far everything seems to just work.
Contrary to numscons, where it looks almost like a new python program
just to build numpy.

Ondrej


From ellisonbg.net at gmail.com  Sun Feb  8 15:17:54 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sun, 8 Feb 2009 12:17:54 -0800
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
	<85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
	<5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>
Message-ID: <6ce0ac130902081217w2f5a51eag33c704373010a0a9@mail.gmail.com>

> I don't find it that surprising - numpy and scipy require some
> relatively advanced features (mixed language and cross-platform with
> support for many toolchains). Within the open source tools, I know
> only two which can handle those requirements: scons and cmake. For
> example, it would almost impossible to build numpy/scipy with
> autoconf.

I know some autotools/autoconf hackers that could probably get it to
build numpy/scipy, ..., oh what, you want Windows support?  ;-)

Cheers,

Brian

> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From ellisonbg.net at gmail.com  Sun Feb  8 15:26:07 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sun, 8 Feb 2009 12:26:07 -0800
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <85b5c3130902081139n45e6b72dl770862baa87065c4@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
	<85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
	<5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>
	<85b5c3130902081139n45e6b72dl770862baa87065c4@mail.gmail.com>
Message-ID: <6ce0ac130902081226q256722f3s67caeccaccd1309c@mail.gmail.com>

> Yes, I am investigating cmake, it's pretty cool. I wrote some macros
> for cython etc. What I like about cmake is that it is cross platform
> and it just produces makefiles on linux, or visual studio files (or
> whatever) on windows.  When I get more experience with it, I'll post
> here.

Yes, while I haven't played with cmake much, it does look very nice.

> What I don't like on cmake is that it uses it's own language, instead
> of python, on the other hand, so far everything seems to just work.

I too don't like the idea of cmake having its own language.  While I
love to learn new languages, I like to have good reasons and I'm not
sure that building software is a good reason.

I have been playing with Scons and I really do like the fact that it
is Python.  While I haven't tried it yet, I am pretty sure that we
could use IPython to interactively debug a Scons build.  That would be
really nice!

> Contrary to numscons, where it looks almost like a new python program
> just to build numpy.

But hold on, it is not fair to compare cmake with numscons (this is an
apples and oranges thing).  You should compare cmake with *Scons*
itself.  The problem with comparing cmake with numscons is that cmake
can't do what numscons does - i.e., plug into distutils.  There is a
lot of extra things that distutils does other than build extensions
and cmake won't do these things out of the box.  Obviously, someone
could get cmake to do these things, but then you are developing a
complete distutils replacement.  And I think that any distutils
replacement should done in Python.

Cheers,

Brian

> Ondrej
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From ondrej at certik.cz  Sun Feb  8 18:02:36 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Sun, 8 Feb 2009 15:02:36 -0800
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <6ce0ac130902081226q256722f3s67caeccaccd1309c@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
	<85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
	<5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>
	<85b5c3130902081139n45e6b72dl770862baa87065c4@mail.gmail.com>
	<6ce0ac130902081226q256722f3s67caeccaccd1309c@mail.gmail.com>
Message-ID: <85b5c3130902081502g76156d15v7a26adfcc0335f01@mail.gmail.com>

On Sun, Feb 8, 2009 at 12:26 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> Yes, I am investigating cmake, it's pretty cool. I wrote some macros
>> for cython etc. What I like about cmake is that it is cross platform
>> and it just produces makefiles on linux, or visual studio files (or
>> whatever) on windows.  When I get more experience with it, I'll post
>> here.
>
> Yes, while I haven't played with cmake much, it does look very nice.
>
>> What I don't like on cmake is that it uses it's own language, instead
>> of python, on the other hand, so far everything seems to just work.
>
> I too don't like the idea of cmake having its own language.  While I
> love to learn new languages, I like to have good reasons and I'm not
> sure that building software is a good reason.
>
> I have been playing with Scons and I really do like the fact that it
> is Python.  While I haven't tried it yet, I am pretty sure that we
> could use IPython to interactively debug a Scons build.  That would be
> really nice!
>
>> Contrary to numscons, where it looks almost like a new python program
>> just to build numpy.
>
> But hold on, it is not fair to compare cmake with numscons (this is an
> apples and oranges thing).  You should compare cmake with *Scons*
> itself.  The problem with comparing cmake with numscons is that cmake
> can't do what numscons does - i.e., plug into distutils.  There is a
> lot of extra things that distutils does other than build extensions
> and cmake won't do these things out of the box.  Obviously, someone
> could get cmake to do these things, but then you are developing a
> complete distutils replacement.  And I think that any distutils
> replacement should done in Python.

Yes, I agree with what you said. I still want to try cmake though, if
nothing, at least to know where we want to go. Yes, I should compare
cmake with scons.

We can either develop python front end to cmake, or just help improve
scons/numscons etc. I really like that cmake generates native
makefiles and windows visual studio files etc. In any case, whatever
we choose to use, I want to have experience with both scons and cmake.

Ondrej


From cournape at gmail.com  Sun Feb  8 19:58:55 2009
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 9 Feb 2009 09:58:55 +0900
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <85b5c3130902081139n45e6b72dl770862baa87065c4@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
	<85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
	<5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>
	<85b5c3130902081139n45e6b72dl770862baa87065c4@mail.gmail.com>
Message-ID: <5b8d13220902081658l576bc36dodfedd75a5389b23@mail.gmail.com>

On Mon, Feb 9, 2009 at 4:39 AM, Ondrej Certik <ondrej at certik.cz> wrote:

>
> Yes, I am investigating cmake, it's pretty cool. I wrote some macros
> for cython etc. What I like about cmake is that it is cross platform
> and it just produces makefiles on linux, or visual studio files (or
> whatever) on windows.  When I get more experience with it, I'll post
> here.

That's exactly what I don't like about cmake  - it means you can't
produce accurate builds (you need to rerun cmake everytime you change
the configuration or dependencies, whereas this is automatic with
scons/waf). It also have (used to have) very poor documentation unless
you buy the book - but it looks like this is changing.

> What I don't like on cmake is that it uses it's own language, instead
> of python, on the other hand, so far everything seems to just work.
> Contrary to numscons, where it looks almost like a new python program
> just to build numpy.

Again, numscons is just a library on top of scons to support things we
need in numpy, it is not really a new program - it is a separate
package to avoid adding experimental code to numpy itself. Numscons is
~ 3000 LOC, of which 1000 is for the core, 1000 for
blas/lapack/fortran support, and 1000 for tools which are not properly
supported in scons (recent MSVC, python extensions).

I think you would have almost as much work with cmake if not more -
when I started numscons, cmake did not have fortran support (it now
has, although I don't know how far - it does not seem to handle mixed
fortran/C, for example).

If you don't mind fast changing software, you should look at waf: it
is extremely fast, based on python. It also handles a lot of
distribution issues already (tarball generation, compiler selection,
etc...) which scons does not.

David


From cournape at gmail.com  Sun Feb  8 20:03:34 2009
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 9 Feb 2009 10:03:34 +0900
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <6ce0ac130902081217w2f5a51eag33c704373010a0a9@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
	<85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
	<5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>
	<6ce0ac130902081217w2f5a51eag33c704373010a0a9@mail.gmail.com>
Message-ID: <5b8d13220902081703p10c15036xb8b6c02089e77145@mail.gmail.com>

On Mon, Feb 9, 2009 at 5:17 AM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> I don't find it that surprising - numpy and scipy require some
>> relatively advanced features (mixed language and cross-platform with
>> support for many toolchains). Within the open source tools, I know
>> only two which can handle those requirements: scons and cmake. For
>> example, it would almost impossible to build numpy/scipy with
>> autoconf.
>
> I know some autotools/autoconf hackers that could probably get it to
> build numpy/scipy, ..., oh what, you want Windows support?  ;-)

Yes, autotools support for MS compilers is very poor - I think it is
fair to say it is unusable as of today. And one of the goal of
numscons is to make the process easier to use/hack (it is a mixed
success on this front - the build knowledge is much easier to read in
numscons than in distutils IMHO, but it is still too arcane on some
fronts, partly because distutils is way too flexible in areas it
should not).

David


From ondrej at certik.cz  Sun Feb  8 22:33:34 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Sun, 8 Feb 2009 19:33:34 -0800
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <5b8d13220902081658l576bc36dodfedd75a5389b23@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
	<85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
	<5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>
	<85b5c3130902081139n45e6b72dl770862baa87065c4@mail.gmail.com>
	<5b8d13220902081658l576bc36dodfedd75a5389b23@mail.gmail.com>
Message-ID: <85b5c3130902081933k13eda392yf892f632436b8eda@mail.gmail.com>

> That's exactly what I don't like about cmake  - it means you can't
> produce accurate builds (you need to rerun cmake everytime you change
> the configuration or dependencies, whereas this is automatic with
> scons/waf). It also have (used to have) very poor documentation unless
> you buy the book - but it looks like this is changing.

You can always rerun cmake if you want (and make it automatic). Imho
that's not a problem. But maybe it is done in a better way with scons.

>
>> What I don't like on cmake is that it uses it's own language, instead
>> of python, on the other hand, so far everything seems to just work.
>> Contrary to numscons, where it looks almost like a new python program
>> just to build numpy.
>
> Again, numscons is just a library on top of scons to support things we
> need in numpy, it is not really a new program - it is a separate
> package to avoid adding experimental code to numpy itself. Numscons is
> ~ 3000 LOC, of which 1000 is for the core, 1000 for
> blas/lapack/fortran support, and 1000 for tools which are not properly
> supported in scons (recent MSVC, python extensions).
>
> I think you would have almost as much work with cmake if not more -
> when I started numscons, cmake did not have fortran support (it now
> has, although I don't know how far - it does not seem to handle mixed
> fortran/C, for example).
>
> If you don't mind fast changing software, you should look at waf: it
> is extremely fast, based on python. It also handles a lot of
> distribution issues already (tarball generation, compiler selection,
> etc...) which scons does not.


Yes, waf is pretty cool, even though the last time I looked at it, it
wasn't able to compile my project, which was larger than just couple
files. But it seems it's development is progressing fast.

As to the other things, one nice thing about cmake is that it is
production ready right now, it is well tested (kde4) and it is in
distributions (e.g. Debian). Everything else needs nontrivial
adjustments, or is not as well tested (yet).

Ondrej


From david at ar.media.kyoto-u.ac.jp  Sun Feb  8 22:56:41 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 09 Feb 2009 12:56:41 +0900
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <85b5c3130902081933k13eda392yf892f632436b8eda@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>	<85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>	<5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>	<85b5c3130902081139n45e6b72dl770862baa87065c4@mail.gmail.com>	<5b8d13220902081658l576bc36dodfedd75a5389b23@mail.gmail.com>
	<85b5c3130902081933k13eda392yf892f632436b8eda@mail.gmail.com>
Message-ID: <498FA979.4060201@ar.media.kyoto-u.ac.jp>

Ondrej Certik wrote:
>> That's exactly what I don't like about cmake  - it means you can't
>> produce accurate builds (you need to rerun cmake everytime you change
>> the configuration or dependencies, whereas this is automatic with
>> scons/waf). It also have (used to have) very poor documentation unless
>> you buy the book - but it looks like this is changing.
>>     
>
> You can always rerun cmake if you want (and make it automatic). Imho
> that's not a problem. But maybe it is done in a better way with scons.
>   

I think it is a problem - it means you have to update it explicitly when
the configuration changes. In scons, the signature concept is quite
powerful: not only are file dependencies handled, but also command
lines, etc... For example, in numscons, if you change the blas/lapack
from atlas to say MKL, only linking and eventually configuration changes
are rebuilt. If you change the fortran compiler but not the C compiler,
only fortran code is rebuilt. All of this is 100 % automatic.

When testing software on many toolchains/platforms, this is very
valuable - it is one of the main reason why Intel, Vmware use it, at
least from the information I understood from the Mailing list. In terms
of accuracy, scons is way beyond anything cmake has to offer.

In the case of numpy, I think cmake would be difficult to integrate into
distutils

> Yes, waf is pretty cool, even though the last time I looked at it, it
> wasn't able to compile my project, which was larger than just couple
> files.
>   

I doubt this has anything to do with the project size. Waf seems more
scalable than scons, and scons can handle big projects (almost all
vmware projects are built using scons, for example), albeit slowly.

> As to the other things, one nice thing about cmake is that it is
> production ready right now, it is well tested (kde4) and it is in
> distributions (e.g. Debian).

For portability reasons, I think the build system should always
distributed with the project you are building (like autoconf does,
actually); this is another drawback of cmake. Waf's main author is
against any waf packaging - waf can be distributed as one < 100 kb
python file, which is very nice.

I don't want to sound overly critical against cmake: obviously, since it
is used by KDE4, it is definitely a good software. It supports some
features which I wish scons had (rpath and install-relinking,
distribution related features).

David


From beryl.cxy at gmail.com  Mon Feb  9 02:32:53 2009
From: beryl.cxy at gmail.com (Xiaoyu Chu)
Date: Mon, 9 Feb 2009 02:32:53 -0500
Subject: [Numpy-discussion] how to get the corresponding eigenvector for a
	specific eigen value?
Message-ID: <42099e890902082332r734667d2udf43e6f93582c15a@mail.gmail.com>

Hey all,
      I am currently working on a large matrix, and I already have a
specific eigen value that I want to use in order to find out its
corresponding eigen vector. Is there an easy way to do so?

     I have tried with linalg.solve(a, b), where I put a as the
Matrix A - eigen value* unit matrix, and b as the zero matrix. But the
solution returned is a zero matrix, which I really find disappointing.

     I have also tried with eig(A), which finds out all the eigen
vectors of matrix A, but it takes too long to run especially the order
of my matrix is like 10,000.


    So right now, I really find myself stuck. Is there anyone who can help me?

   My great gratitude.


Best,
Beryl


From hoytak at cs.ubc.ca  Mon Feb  9 02:40:25 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Sun, 8 Feb 2009 23:40:25 -0800
Subject: [Numpy-discussion] how to get the corresponding eigenvector for
	a specific eigen value?
In-Reply-To: <42099e890902082332r734667d2udf43e6f93582c15a@mail.gmail.com>
References: <42099e890902082332r734667d2udf43e6f93582c15a@mail.gmail.com>
Message-ID: <4db580fd0902082340g651c33dn26b78144bc7811d3@mail.gmail.com>

>     I have tried with linalg.solve(a, b), where I put a as the
> Matrix A - eigen value* unit matrix, and b as the zero matrix. But the
> solution returned is a zero matrix, which I really find disappointing.

So if you're trying to solve (A - \lambda I) x = b, try appending an
extra row to your  (A - \lambda I) matrix with all ones; the output of
this will be the some of the elements of your eigenvector.  Append a 1
to the end of your b.  This will exclude the case where x = 0 as valid
solution, as you'll then require that \sum_i x_i = 1.  You'll probably
want to renormalize properly afterwards.

Quick and dirty, but should work.

--Hoyt


-- 

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++


From stefan at sun.ac.za  Mon Feb  9 02:48:59 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 9 Feb 2009 09:48:59 +0200
Subject: [Numpy-discussion] NumPy buildbot
Message-ID: <9457e7c80902082348j134fba05sa0f1b677f5a35b76@mail.gmail.com>

Hi all,

Due to changes in our firewall configuration, the buildbot was offline
since 14 January.  Everything is sorted out now, so you can access the
buildbot again at

http://buildbot.scipy.org

Some of the build slaves are still offline, and I have contacted their
maintainers.

Regards
St?fan


From faltet at pytables.org  Mon Feb  9 03:34:00 2009
From: faltet at pytables.org (Francesc Alted)
Date: Mon, 9 Feb 2009 09:34:00 +0100
Subject: [Numpy-discussion] Selection of only a certain number of fields
In-Reply-To: <loom.20090208T154447-548@post.gmane.org>
References: <498B7181.5000300@enthought.com>
	<200902071917.57537.faltet@pytables.org>
	<loom.20090208T154447-548@post.gmane.org>
Message-ID: <200902090934.00885.faltet@pytables.org>

A Sunday 08 February 2009, Neil escrigu?:
> > The first one (and most important IMO), is that newarr continues to
> > be an structured array (BTW, when changed this name from the
> > original record array?), and you can use all the features of these
> > beasts with it.  Other reason (albeit a bit secondary) is that its
> > data buffer can be shared through the array interface with other
> > applications, or plain C code, in a relatively straightforward way.
> >  However, if newarr becomes a list (or dictionary), this is simply
> > not possible.
> >
> > Cheers,
>
> That's not a sample use case ;)
>
> One of the things I love about Python is that it has a small core set
> of features and tries to avoid having many ways to do the same thing.
>  This makes it extremely easy to learn.  With every new feature,
> numpy gets a little bit harder to learn, there's more to document and
> the code base gets larger and so harder to maintain.  In those
> senses, whenever you add a new function/feature to numpy, it gets a
> little bit worse.

Mmm, you have made another good point.  Actually, it is not very clear 
to me that adding too much functionality to NumPy is going to be a good 
idea for every case.  For example, lately I was thinking in that it 
would be a good idea to support column-wise structured arrays (the 
current ones are row-wise), but provided that they can be trivially 
reproduced with a combination of dictionaries and plain arrays I think 
now that implementing that in NumPy has not much sense.

Similarly, and as you said, having:

l = [rec[n] for n in ['name', 'age']]

or, if a dictionary is wanted instead:

d = dict((n,rec[n]) for n in ['name', 'age'])

would admittedly cover many of the needs of users.  In addition, one can 
get a record array easily from the above dictionary:

newrec = np.rec.fromarrays(d.values(), names=d.keys())

Having said that, I still see some value in implementing 
arr[['name', 'age']], but frankly, I'm not so sure now whether this 
idiom would be much better than:

d = dict((n,rec[n]) for n in ['name', 'age'])
newrec = np.rec.fromarrays(d.values(), names=d.keys())

or than the already implemented drop_fields() function in 
np.lib.recfunctions.

So, I'm +0 on the proposal now.

> So I think it would be nice to have some concrete examples of what
> the new feature will be useful for, just to show how it outweighs
> those negatives.  As a bonus, they'd provide nice examples to put in
> the documentation :).

Yeah, I completely agree that this would be a nice excercise to do: for 
every new asked feature, first look if it can be done easily with a 
combination of the current weaponeries of Python and NumPy together.
That would lead to a simple and powerful NumPy.

> PS.  Thanks for your work on pytables!  I've used it quite a bit,
> mostly for reading hdf5 files.

My pleasure.

-- 
Francesc Alted


From nwagner at iam.uni-stuttgart.de  Mon Feb  9 03:34:11 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 09 Feb 2009 09:34:11 +0100
Subject: [Numpy-discussion] Comparison of arrays
Message-ID: <web-116796684@uni-stuttgart.de>

Hi all,

I have two integer arrays of different shape, e.g.

   
>>> a
array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10])
>>> b
array([ 3,  4,  5,  6,  7,  8,  9, 10])

How can I extract the values that belong to the array a 
exclusively i.e. array([1,2]) ?

Nils


From faltet at pytables.org  Mon Feb  9 03:45:02 2009
From: faltet at pytables.org (Francesc Alted)
Date: Mon, 9 Feb 2009 09:45:02 +0100
Subject: [Numpy-discussion] Comparison of arrays
In-Reply-To: <web-116796684@uni-stuttgart.de>
References: <web-116796684@uni-stuttgart.de>
Message-ID: <200902090945.02962.faltet@pytables.org>

A Monday 09 February 2009, Nils Wagner escrigu?:
> Hi all,
>
> I have two integer arrays of different shape, e.g.
>
> >>> a
>
> array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10])
>
> >>> b
>
> array([ 3,  4,  5,  6,  7,  8,  9, 10])
>
> How can I extract the values that belong to the array a
> exclusively i.e. array([1,2]) ?

One possible, fast solution is using Python sets:

In [45]: np.array(list(set(a) ^ set(b)))
Out[45]: array([1, 2])

Although this is suboptimal for very large arrays as it needs temporary 
space.

Cheers,

-- 
Francesc Alted


From nwagner at iam.uni-stuttgart.de  Mon Feb  9 04:00:05 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 09 Feb 2009 10:00:05 +0100
Subject: [Numpy-discussion] Comparison of arrays
In-Reply-To: <200902090945.02962.faltet@pytables.org>
References: <web-116796684@uni-stuttgart.de>
	<200902090945.02962.faltet@pytables.org>
Message-ID: <web-116798091@uni-stuttgart.de>

On Mon, 9 Feb 2009 09:45:02 +0100
  Francesc Alted <faltet at pytables.org> wrote:
> A Monday 09 February 2009, Nils Wagner escrigu?:
>> Hi all,
>>
>> I have two integer arrays of different shape, e.g.
>>
>> >>> a
>>
>> array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10])
>>
>> >>> b
>>
>> array([ 3,  4,  5,  6,  7,  8,  9, 10])
>>
>> How can I extract the values that belong to the array a
>> exclusively i.e. array([1,2]) ?
> 
> One possible, fast solution is using Python sets:
> 
> In [45]: np.array(list(set(a) ^ set(b)))
> Out[45]: array([1, 2])
> 
> Although this is suboptimal for very large arrays as it 
>needs temporary 
> space.
> 
> Cheers,
> 
> -- 
>Francesc Alted
  
Thank you very much for your prompt response.

    Nils


From meine at informatik.uni-hamburg.de  Mon Feb  9 07:23:39 2009
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Mon, 9 Feb 2009 13:23:39 +0100
Subject: [Numpy-discussion] linalg.norm missing an 'axis' kwarg?!
In-Reply-To: <200811201154.53007.meine@informatik.uni-hamburg.de>
References: <200811201111.15930.meine@informatik.uni-hamburg.de>
	<200811201154.53007.meine@informatik.uni-hamburg.de>
Message-ID: <200902091323.40049.meine@informatik.uni-hamburg.de>

On Thursday 20 November 2008 11:54:52 Hans Meine wrote:
> On Thursday 20 November 2008 11:11:14 Hans Meine wrote:
> > I have a 2D matrix comprising a sequence of vectors, and I want to
> > compute the norm of each vector.  np.linalg.norm seems to be the best
> > bet, but it does not support axis.  Wouldn't this be a nice feature?
>
> Here's a basic implementation.  docstring + tests not updated yet, also I
> wonder whether axis should be the first argument, but that could create
> compatibility problems.

AFAICS, I never received an answer, but IMHO this should be integrated into 
NumPy.  Any objections?

Greetings,
  Hans


From stefan at sun.ac.za  Mon Feb  9 08:08:05 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 9 Feb 2009 15:08:05 +0200
Subject: [Numpy-discussion] linalg.norm missing an 'axis' kwarg?!
In-Reply-To: <200902091323.40049.meine@informatik.uni-hamburg.de>
References: <200811201111.15930.meine@informatik.uni-hamburg.de>
	<200811201154.53007.meine@informatik.uni-hamburg.de>
	<200902091323.40049.meine@informatik.uni-hamburg.de>
Message-ID: <9457e7c80902090508r5cf6e29fqb0c797eb21524b40@mail.gmail.com>

2009/2/9 Hans Meine <meine at informatik.uni-hamburg.de>:
>> Here's a basic implementation.  docstring + tests not updated yet, also I
>> wonder whether axis should be the first argument, but that could create
>> compatibility problems.
>
> AFAICS, I never received an answer, but IMHO this should be integrated into
> NumPy.  Any objections?

I often work with vectors inside an array, so I would find it useful
(even if it is just a two-line wrapper).  I'd also add an "out"
argument, so that the signature is similar to that of max, min, etc.

norm(a, ord=None, axis=None, out=None)

Cheers
St?fan


From neilcrighton at gmail.com  Mon Feb  9 09:02:12 2009
From: neilcrighton at gmail.com (Neil)
Date: Mon, 9 Feb 2009 14:02:12 +0000 (UTC)
Subject: [Numpy-discussion] Comparison of arrays
References: <web-116796684@uni-stuttgart.de>
	<200902090945.02962.faltet@pytables.org>
Message-ID: <loom.20090209T134227-444@post.gmane.org>


> > I have two integer arrays of different shape, e.g.
> >
> > >>> a
> >
> > array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10])
> >
> > >>> b
> >
> > array([ 3,  4,  5,  6,  7,  8,  9, 10])
> >
> > How can I extract the values that belong to the array a
> > exclusively i.e. array([1,2]) ?
> 

You could also use numpy.setmember1d to get a boolean mask that selects 
the values:

In [21]: a = np.array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10])
In [22]: b = np.array([ 3,  4,  5,  6,  7,  8,  9, 10])
In [23]: ismember = np.setmember1d(a,b)
In [24]: a[~ismember]
Out[24]: array([1, 2])


Neil




From faltet at pytables.org  Mon Feb  9 09:19:15 2009
From: faltet at pytables.org (Francesc Alted)
Date: Mon, 9 Feb 2009 15:19:15 +0100
Subject: [Numpy-discussion] Comparison of arrays
In-Reply-To: <loom.20090209T134227-444@post.gmane.org>
References: <web-116796684@uni-stuttgart.de>
	<200902090945.02962.faltet@pytables.org>
	<loom.20090209T134227-444@post.gmane.org>
Message-ID: <200902091519.16140.faltet@pytables.org>

A Monday 09 February 2009, Neil escrigu?:
> > > I have two integer arrays of different shape, e.g.
> > >
> > > >>> a
> > >
> > > array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10])
> > >
> > > >>> b
> > >
> > > array([ 3,  4,  5,  6,  7,  8,  9, 10])
> > >
> > > How can I extract the values that belong to the array a
> > > exclusively i.e. array([1,2]) ?
>
> You could also use numpy.setmember1d to get a boolean mask that
> selects the values:
>
> In [21]: a = np.array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10])
> In [22]: b = np.array([ 3,  4,  5,  6,  7,  8,  9, 10])
> In [23]: ismember = np.setmember1d(a,b)
> In [24]: a[~ismember]
> Out[24]: array([1, 2])

Yes, that's a nice solution.  In fact, my solution is wrong for the case 
that b is not included in a:

In [52]: a = np.array([ 1,  2,  3,  4,  5,  6,  7, 8])

In [53]: b = np.array([ 3,  4,  5,  6,  7,  8,  9, 10])

In [54]: a[~np.setmember1d(a,b)]
Out[54]: array([1, 2])

In [55]: np.array(list(set(a) ^ set(b)))
Out[55]: array([ 1,  2,  9, 10])

Cheers,

-- 
Francesc Alted


From bpederse at gmail.com  Mon Feb  9 11:22:35 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Mon, 9 Feb 2009 08:22:35 -0800
Subject: [Numpy-discussion] Comparison of arrays
In-Reply-To: <loom.20090209T134227-444@post.gmane.org>
References: <web-116796684@uni-stuttgart.de>
	<200902090945.02962.faltet@pytables.org>
	<loom.20090209T134227-444@post.gmane.org>
Message-ID: <e183a99d0902090822i2f5c06e1r8883f8c8f67aa45c@mail.gmail.com>

On Mon, Feb 9, 2009 at 6:02 AM, Neil <neilcrighton at gmail.com> wrote:
>
>> > I have two integer arrays of different shape, e.g.
>> >
>> > >>> a
>> >
>> > array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10])
>> >
>> > >>> b
>> >
>> > array([ 3,  4,  5,  6,  7,  8,  9, 10])
>> >
>> > How can I extract the values that belong to the array a
>> > exclusively i.e. array([1,2]) ?
>>
>
> You could also use numpy.setmember1d to get a boolean mask that selects
> the values:
>
> In [21]: a = np.array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10])
> In [22]: b = np.array([ 3,  4,  5,  6,  7,  8,  9, 10])
> In [23]: ismember = np.setmember1d(a,b)
> In [24]: a[~ismember]
> Out[24]: array([1, 2])
>
>
> Neil
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

there's also np.setdiff1d() which does the above in a single line.
-brent


From charlesr.harris at gmail.com  Mon Feb  9 14:05:18 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 9 Feb 2009 12:05:18 -0700
Subject: [Numpy-discussion] how to get the corresponding eigenvector for
	a specific eigen value?
In-Reply-To: <42099e890902082332r734667d2udf43e6f93582c15a@mail.gmail.com>
References: <42099e890902082332r734667d2udf43e6f93582c15a@mail.gmail.com>
Message-ID: <e06186140902091105u54591566ie57020fd0c4124f1@mail.gmail.com>

On Mon, Feb 9, 2009 at 12:32 AM, Xiaoyu Chu <beryl.cxy at gmail.com> wrote:

> Hey all,
>      I am currently working on a large matrix, and I already have a
> specific eigen value that I want to use in order to find out its
> corresponding eigen vector. Is there an easy way to do so?
>
>     I have tried with linalg.solve(a, b), where I put a as the
> Matrix A - eigen value* unit matrix, and b as the zero matrix. But the
> solution returned is a zero matrix, which I really find disappointing.
>
>     I have also tried with eig(A), which finds out all the eigen
> vectors of matrix A, but it takes too long to run especially the order
> of my matrix is like 10,000.
>
>
>    So right now, I really find myself stuck. Is there anyone who can help
> me?
>

The usual trick for this is inverse iteration,
http://en.wikipedia.org/wiki/Inverse_iteration, which involves solving the
almost singular equation A - eig*I = rhs for an appropriately chosen rhs.
The required eigenvector will dominate the solution and will grow rapidly in
amplitude over a few iterations.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090209/7e25f59b/attachment.html>

From h5py at alfven.org  Mon Feb  9 15:15:52 2009
From: h5py at alfven.org (Andrew Collette)
Date: Mon, 9 Feb 2009 12:15:52 -0800
Subject: [Numpy-discussion] ANN: HDF5 for Python 1.1
Message-ID: <f125f5e0902091215w37922edak9d61ab578515f548@mail.gmail.com>

=====================================
Announcing HDF5 for Python (h5py) 1.1
=====================================

What is h5py?
-------------

HDF5 for Python (h5py) is a general-purpose Python interface to the
Hierarchical Data Format library, version 5.  HDF5 is a versatile,
mature scientific software library designed for the fast, flexible
storage of enormous amounts of data.

>From a Python programmer's perspective, HDF5 provides a robust way to
store data, organized by name in a tree-like fashion.  You can create
datasets (arrays on disk) hundreds of gigabytes in size, and perform
random-access I/O on desired sections.  Datasets are organized in a
filesystem-like hierarchy using containers called "groups", and
accesed using the tradional POSIX /path/to/resource syntax.

In addition to providing interoperability with existing HDF5 datasets
and platforms, h5py is a convienient way to store and retrieve
arbitrary NumPy data and metadata.


New features in 1.1
-------------------

  - A new compression filter based on the LZF library, which provides
    transparent compression many times faster than the standard HDF5
    GZIP filter.

  - Efficient broadcasting using HDF5 hyperslab selections; for example,
    you can write to a (2000 x 100 x 50) selection from a (100 x 50)
    source array.

  - Now supports the NumPy boolean type

  - Auto-completion for IPython 0.9.X (contributed by Darren Dale)

  - Installable via easy_install


Standard features
-----------------

  - Supports storage of NumPy data of the following types:

    * Integer/Unsigned Integer
    * Float/Double
    * Complex/Double Complex
    * Compound ("recarray")
    * Strings
    * Boolean
    * Array (as members of a compound type only)
    * Void

  - Random access to datasets using the standard NumPy slicing syntax,
    including fancy indexing and point-based selection

  - Transparent compression of datasets using GZIP, LZF or SZIP,
    and error-detection using Fletcher32

  - "Pythonic" interface supporting dictionary and NumPy-array metaphors
    for the high-level HDF5 abstrations like groups and datasets

  - A comprehensive, object-oriented wrapping of the HDF5 low-level C API
    via Cython, in addition to the NumPy-like high-level interface.

  - Supports many new features of HDF5 1.8, including recursive iteration
    over entire files and in-library copy operations on the file tree

  - Thread-safe


Where to get it
---------------

* Main website, documentation:  http://h5py.alfven.org

* Downloads, bug tracker:       http://h5py.googlecode.com


Requires
--------

* Linux, Mac OS-X or Windows

* Python 2.5 (Windows), Python 2.5 or 2.6 (Linux/Mac OS-X)

* NumPy 1.0.3 or later

* HDF5 1.6.5 or later (including 1.8); HDF5 is included with
  the Windows version.


Thanks
------

Thanks to D. Dale, E. Lawrence and other for their continued support
and comments.  Also thanks to the Francesc Alted and the PyTables project,
for inspiration and generously providing their code to the community. Thanks
to everyone at the HDF Group for creating such a useful piece of software.


From ondrej at certik.cz  Mon Feb  9 16:06:01 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 9 Feb 2009 13:06:01 -0800
Subject: [Numpy-discussion] ANN: HDF5 for Python 1.1
In-Reply-To: <f125f5e0902091215w37922edak9d61ab578515f548@mail.gmail.com>
References: <f125f5e0902091215w37922edak9d61ab578515f548@mail.gmail.com>
Message-ID: <85b5c3130902091306m528835a9k5cff7eded4bed94d@mail.gmail.com>

On Mon, Feb 9, 2009 at 12:15 PM, Andrew Collette <h5py at alfven.org> wrote:
> =====================================
> Announcing HDF5 for Python (h5py) 1.1
> =====================================
>
> What is h5py?
> -------------
>
> HDF5 for Python (h5py) is a general-purpose Python interface to the
> Hierarchical Data Format library, version 5.  HDF5 is a versatile,
> mature scientific software library designed for the fast, flexible
> storage of enormous amounts of data.

Interesting project. My first question was how it is related to
pytables. The answer is here:

http://code.google.com/p/h5py/wiki/FAQ#What%27s_the_difference_between_h5py_and_Pytables?


Ondrej


From simon.palmer at gmail.com  Mon Feb  9 16:35:41 2009
From: simon.palmer at gmail.com (Simon Palmer)
Date: Mon, 9 Feb 2009 13:35:41 -0800 (PST)
Subject: [Numpy-discussion]  Profiling with cProfile
Message-ID: <21922525.post@talk.nabble.com>


Hi,
I am trying to profile a bit of code I have written using cProfile.  When I
run it I get the message:

TypeError: unhashable type: 'numpy.ndarray'

I am using runctx with some local variables which are ndarrays.

I am guessing that this is a lmitation of either cProfile or numpy or the
mix of the two.  What is the recommended approach for profiling my code
containing ndarrays?

-- 
View this message in context: http://www.nabble.com/Profiling-with-cProfile-tp21922525p21922525.html
Sent from the Numpy-discussion mailing list archive at Nabble.com.



From ondrej at certik.cz  Mon Feb  9 17:06:30 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 9 Feb 2009 14:06:30 -0800
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <498FA979.4060201@ar.media.kyoto-u.ac.jp>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
	<85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
	<5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>
	<85b5c3130902081139n45e6b72dl770862baa87065c4@mail.gmail.com>
	<5b8d13220902081658l576bc36dodfedd75a5389b23@mail.gmail.com>
	<85b5c3130902081933k13eda392yf892f632436b8eda@mail.gmail.com>
	<498FA979.4060201@ar.media.kyoto-u.ac.jp>
Message-ID: <85b5c3130902091406h60dd3cd0q4579da91020c3b2d@mail.gmail.com>

On Sun, Feb 8, 2009 at 7:56 PM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Ondrej Certik wrote:
>>> That's exactly what I don't like about cmake  - it means you can't
>>> produce accurate builds (you need to rerun cmake everytime you change
>>> the configuration or dependencies, whereas this is automatic with
>>> scons/waf). It also have (used to have) very poor documentation unless
>>> you buy the book - but it looks like this is changing.
>>>
>>
>> You can always rerun cmake if you want (and make it automatic). Imho
>> that's not a problem. But maybe it is done in a better way with scons.
>>
>
> I think it is a problem - it means you have to update it explicitly when
> the configuration changes. In scons, the signature concept is quite
> powerful: not only are file dependencies handled, but also command
> lines, etc... For example, in numscons, if you change the blas/lapack
> from atlas to say MKL, only linking and eventually configuration changes
> are rebuilt. If you change the fortran compiler but not the C compiler,
> only fortran code is rebuilt. All of this is 100 % automatic.


I got an email from Alexander Neundorf from kde, and with his
permission, I am forwarding it here, because he is not subscribed. As
he writes below, cmake is in fact cabable of the above.




---------- Forwarded message ----------
From: Alexander Neundorf
Date: Mon, Feb 9, 2009 at 1:28 PM
Subject: cmake information
To: Ondrej Certik


Hi Ondrej,

I hope you are the Ondrej from this thread:
http://groups.google.com/group/Numpy-discussion/browse_thread/thread/c12f96b9ac367f57

If not, please let me know and I'll try to find the right email address.
I saw this in the thread:

----------8<-------------8<----------------8<----------------

Ondrej Certik wrote:
 >> That's exactly what I don't like about cmake  - it means you can't
 >> produce accurate builds (you need to rerun cmake everytime you change
 >> the configuration or dependencies, whereas this is automatic with
 >> scons/waf). It also have (used to have) very poor documentation unless
 >> you buy the book - but it looks like this is changing.

> You can always rerun cmake if you want (and make it automatic). Imho
 > that's not a problem. But maybe it is done in a better way with scons.

I think it is a problem - it means you have to update it explicitly when
 the configuration changes. In scons, the signature concept is quite
 powerful: not only are file dependencies handled, but also command
 lines, etc... For example, in numscons, if you change the blas/lapack
 from atlas to say MKL, only linking and eventually configuration changes
 are rebuilt. If you change the fortran compiler but not the C compiler,
 only fortran code is rebuilt. All of this is 100 % automatic.

----------8<-------------8<----------------8<----------------

CMake does handle this automatically.
E.g. if include directories are changed (which you do by editing a
CMakeLists.txt or the cmake cache), all files which are affected by the are
rebuilt. If some library changes, everything linking to this library is
linked again.
If any of the files the build depends on (e.g. a CMakeLists.txt, or an
included .cmake file, or the cmake cache) is changed, cmake is automatically
rerun, and the makefiles/project files are regenerated.

I don't know what happens if you are using both C and Fortran in one project
and change one of them. I think this is actually not possible, you can (or
should) not change the compiler of an existing build tree. Basically
everything which uses this compiler is invalid then, the object files, the
results of tests etc. I'm not sure handling this separately for different
languages within one project is supported by cmake.

Alex


From mtrumpis at berkeley.edu  Mon Feb  9 17:25:54 2009
From: mtrumpis at berkeley.edu (M Trumpis)
Date: Mon, 9 Feb 2009 14:25:54 -0800
Subject: [Numpy-discussion] SVD errors
In-Reply-To: <gm908k$d55$1@ger.gmane.org>
References: <34928.128.32.52.185.1233620513.squirrel@calmail.berkeley.edu>
	<3d375d730902021627x2ef54411j5bb9504927ff831@mail.gmail.com>
	<gm908k$d55$1@ger.gmane.org>
Message-ID: <be3bc680902091425v3579660dnfef416d2006b8332@mail.gmail.com>

I played around with a C translation of that test program, and found
that dgesvd (but not dgesdd) happens to converge and return all
non-negative singular values for both operators I was having trouble
with. I'm also looking at the Octave code just now, and I think
they're using dgesvd also. Any one know why numpy uses dgesdd? speed?

Mike

On Tue, Feb 3, 2009 at 12:46 AM, Pauli Virtanen <pav at iki.fi> wrote:
> Mon, 02 Feb 2009 18:27:05 -0600, Robert Kern wrote:
>> On Mon, Feb 2, 2009 at 18:21,  <mtrumpis at berkeley.edu> wrote:
>>> Hello list.. I've run into two SVD errors over the last few days. Both
>>> errors are identical in numpy/scipy.
>>>
>>> I've submitted a ticket for the 1st problem (numpy ticket #990).
>>> Summary is: some builds of the lapack_lite module linking against
>>> system LAPACK (not the bundled dlapack_lite.o, etc) give a
>>> "LinAlgError: SVD did not converge" exception on my matrix. This error
>>> does occur using Mac's Accelerate framework LAPACK, and a coworker's
>>> Ubuntu LAPACK version. It does not seem to happen using ATLAS LAPACK
>>> (nor using Octave/Matlab on said Ubuntu)
>>
>> These are almost certainly issues with the particular implementations of
>> LAPACK that you are using. I don't think there is anything we can do
>> from numpy or scipy to change this.
>
> Yes, this is almost certainly a LAPACK problem. If in doubt, you can test
> it with the following F90 program (be sure to link it against the same
> LAPACK as Numpy). Save the matrices with 'np.savetxt("foo.txt", x.ravel
> ())' and run './test < foo.txt'.
>
> ----
> program test
>    implicit none
>    integer, parameter :: N = 128
>    double precision :: A(N*N), S(N), U(N,N), Vh(N,N)
>    double precision, allocatable :: WORK(:)
>    double precision :: tmp
>    integer :: IWORK(8*N)
>    integer :: INFO = 0, LWORK
>
>    read(*,*) A
>    A = reshape(transpose(reshape(A, (/N, N/))), (/ N*N /))
>    call dgesdd('A', N, N, A, N, S, U, N, Vh, N, tmp, -1, IWORK, INFO)
>    LWORK = tmp
>    if (info .ne. 0) stop 'lwork query failed'
>    write(*,*) 'lwork:', lwork
>    allocate(WORK(LWORK))
>    call dgesdd('A', N, N, A, N, S, U, N, Vh, N, WORK, LWORK, IWORK, INFO)
>    write(*,*) 'info:', INFO
>    write(*,*) 'min(S):', minval(S)
>    if (INFO .ne. 0) then
>        write(*,*) ' -> SVD failed to converge'
>    end if
>    if (minval(S) .lt. 0) then
>        write(*,*) ' -> negative singular value'
>    end if
> end program
> ----
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From ellisonbg.net at gmail.com  Mon Feb  9 17:35:09 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 9 Feb 2009 14:35:09 -0800
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <85b5c3130902091406h60dd3cd0q4579da91020c3b2d@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
	<85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
	<5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>
	<85b5c3130902081139n45e6b72dl770862baa87065c4@mail.gmail.com>
	<5b8d13220902081658l576bc36dodfedd75a5389b23@mail.gmail.com>
	<85b5c3130902081933k13eda392yf892f632436b8eda@mail.gmail.com>
	<498FA979.4060201@ar.media.kyoto-u.ac.jp>
	<85b5c3130902091406h60dd3cd0q4579da91020c3b2d@mail.gmail.com>
Message-ID: <6ce0ac130902091435o479568e7sf869ddcf69ef91d5@mail.gmail.com>

> CMake does handle this automatically.
> E.g. if include directories are changed (which you do by editing a
> CMakeLists.txt or the cmake cache), all files which are affected by the are
> rebuilt. If some library changes, everything linking to this library is
> linked again.
> If any of the files the build depends on (e.g. a CMakeLists.txt, or an
> included .cmake file, or the cmake cache) is changed, cmake is automatically
> rerun, and the makefiles/project files are regenerated.
>
> I don't know what happens if you are using both C and Fortran in one project
> and change one of them. I think this is actually not possible, you can (or
> should) not change the compiler of an existing build tree. Basically
> everything which uses this compiler is invalid then, the object files, the
> results of tests etc. I'm not sure handling this separately for different
> languages within one project is supported by cmake.

Is all of this handled just by calling make (after the initial cmake
call), or do you have to first recall cmake and then make?

Brian

> Alex
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From robert.kern at gmail.com  Mon Feb  9 17:40:02 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 9 Feb 2009 16:40:02 -0600
Subject: [Numpy-discussion] SVD errors
In-Reply-To: <be3bc680902091425v3579660dnfef416d2006b8332@mail.gmail.com>
References: <34928.128.32.52.185.1233620513.squirrel@calmail.berkeley.edu>
	<3d375d730902021627x2ef54411j5bb9504927ff831@mail.gmail.com>
	<gm908k$d55$1@ger.gmane.org>
	<be3bc680902091425v3579660dnfef416d2006b8332@mail.gmail.com>
Message-ID: <3d375d730902091440u23d72956le54b4894751a8dc7@mail.gmail.com>

On Mon, Feb 9, 2009 at 16:25, M Trumpis <mtrumpis at berkeley.edu> wrote:
> I played around with a C translation of that test program, and found
> that dgesvd (but not dgesdd) happens to converge and return all
> non-negative singular values for both operators I was having trouble
> with. I'm also looking at the Octave code just now, and I think
> they're using dgesvd also. Any one know why numpy uses dgesdd? speed?

Yes.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Mon Feb  9 17:44:14 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 9 Feb 2009 16:44:14 -0600
Subject: [Numpy-discussion] Profiling with cProfile
In-Reply-To: <21922525.post@talk.nabble.com>
References: <21922525.post@talk.nabble.com>
Message-ID: <3d375d730902091444o4f914aa0hf12c71b82ae9624e@mail.gmail.com>

On Mon, Feb 9, 2009 at 15:35, Simon Palmer <simon.palmer at gmail.com> wrote:
>
> Hi,
> I am trying to profile a bit of code I have written using cProfile.  When I
> run it I get the message:
>
> TypeError: unhashable type: 'numpy.ndarray'
>
> I am using runctx with some local variables which are ndarrays.
>
> I am guessing that this is a lmitation of either cProfile or numpy or the
> mix of the two.  What is the recommended approach for profiling my code
> containing ndarrays?

You might try using runcall() instead. I'm not really familiar with
what cProfile may be doing in runctx() that will make it try to use
objects in the namespace as keys in a dict.

You may want to take a look at my kernprof.py script which is a useful
driver script for profiling any Python script without your having to
bother with making your own driver script.

  http://packages.python.org/line_profiler/
  http://packages.python.org/line_profiler/kernprof.py

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From h5py at alfven.org  Mon Feb  9 17:51:57 2009
From: h5py at alfven.org (Andrew Collette)
Date: Mon, 9 Feb 2009 14:51:57 -0800
Subject: [Numpy-discussion] ANN: HDF5 for Python 1.1
In-Reply-To: <85b5c3130902091306m528835a9k5cff7eded4bed94d@mail.gmail.com>
References: <f125f5e0902091215w37922edak9d61ab578515f548@mail.gmail.com>
	<85b5c3130902091306m528835a9k5cff7eded4bed94d@mail.gmail.com>
Message-ID: <f125f5e0902091451k3d7bca13i9df01802f311973a@mail.gmail.com>

Thanks, Ondrej.  For the record, h5py is designed to provide a
"NumPy-like" interface to HDF5, along with a near-complete wrapping of
the low-level HDF5 C API.  It has none of the database-like features
of PyTables.  The FAQ entry has more info.

Andrew Collette

On Mon, Feb 9, 2009 at 1:06 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> On Mon, Feb 9, 2009 at 12:15 PM, Andrew Collette <h5py at alfven.org> wrote:
>> =====================================
>> Announcing HDF5 for Python (h5py) 1.1
>> =====================================
>>
>> What is h5py?
>> -------------
>>
>> HDF5 for Python (h5py) is a general-purpose Python interface to the
>> Hierarchical Data Format library, version 5.  HDF5 is a versatile,
>> mature scientific software library designed for the fast, flexible
>> storage of enormous amounts of data.
>
> Interesting project. My first question was how it is related to
> pytables. The answer is here:
>
> http://code.google.com/p/h5py/wiki/FAQ#What%27s_the_difference_between_h5py_and_Pytables?
>
>
> Ondrej
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From ondrej at certik.cz  Mon Feb  9 18:00:11 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 9 Feb 2009 15:00:11 -0800
Subject: [Numpy-discussion] preferred numpy build system
In-Reply-To: <6ce0ac130902091435o479568e7sf869ddcf69ef91d5@mail.gmail.com>
References: <85b5c3130902062204p6f397fa0yb5b90178bb7d9b83@mail.gmail.com>
	<5b8d13220902070642h281ce30bn2e6b6fcf1583345f@mail.gmail.com>
	<85b5c3130902071021q3f524771ra4b1f175d06f0f68@mail.gmail.com>
	<5b8d13220902080310g82f0eacyc9dfa176d8581dc5@mail.gmail.com>
	<85b5c3130902081139n45e6b72dl770862baa87065c4@mail.gmail.com>
	<5b8d13220902081658l576bc36dodfedd75a5389b23@mail.gmail.com>
	<85b5c3130902081933k13eda392yf892f632436b8eda@mail.gmail.com>
	<498FA979.4060201@ar.media.kyoto-u.ac.jp>
	<85b5c3130902091406h60dd3cd0q4579da91020c3b2d@mail.gmail.com>
	<6ce0ac130902091435o479568e7sf869ddcf69ef91d5@mail.gmail.com>
Message-ID: <85b5c3130902091500m5f72a9dftdf752c2d973880d@mail.gmail.com>

On Mon, Feb 9, 2009 at 2:35 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> CMake does handle this automatically.
>> E.g. if include directories are changed (which you do by editing a
>> CMakeLists.txt or the cmake cache), all files which are affected by the are
>> rebuilt. If some library changes, everything linking to this library is
>> linked again.
>> If any of the files the build depends on (e.g. a CMakeLists.txt, or an
>> included .cmake file, or the cmake cache) is changed, cmake is automatically
>> rerun, and the makefiles/project files are regenerated.
>>
>> I don't know what happens if you are using both C and Fortran in one project
>> and change one of them. I think this is actually not possible, you can (or
>> should) not change the compiler of an existing build tree. Basically
>> everything which uses this compiler is invalid then, the object files, the
>> results of tests etc. I'm not sure handling this separately for different
>> languages within one project is supported by cmake.
>
> Is all of this handled just by calling make (after the initial cmake
> call), or do you have to first recall cmake and then make?

I just tried that and it is handled automatically by calling "make".
It's really cool!

Ondrej


From mail at stevesimmons.com  Tue Feb 10 01:30:41 2009
From: mail at stevesimmons.com (Stephen Simmons)
Date: Tue, 10 Feb 2009 07:30:41 +0100
Subject: [Numpy-discussion] ANN: HDF5 for Python 1.1
In-Reply-To: <f125f5e0902091215w37922edak9d61ab578515f548@mail.gmail.com>
References: <f125f5e0902091215w37922edak9d61ab578515f548@mail.gmail.com>
Message-ID: <49911F11.9000109@stevesimmons.com>

Hi Andrew,

Do you have any plans to support LZO compression in h5py?
 
I have lots of LZO-compressed datasets created with PyTables.
There's a real barrier to using both h5py and PyTables if the fast
decompressor options are just LZF on h5py and LZO on PyTables.

Many thanks
Stephen


Andrew Collette wrote:
> =====================================
> Announcing HDF5 for Python (h5py) 1.1
> =====================================
>
> What is h5py?
> -------------
>
> HDF5 for Python (h5py) is a general-purpose Python interface to the
> Hierarchical Data Format library, version 5.  HDF5 is a versatile,
> mature scientific software library designed for the fast, flexible
> storage of enormous amounts of data.
>
> >From a Python programmer's perspective, HDF5 provides a robust way to
> store data, organized by name in a tree-like fashion.  You can create
> datasets (arrays on disk) hundreds of gigabytes in size, and perform
> random-access I/O on desired sections.  Datasets are organized in a
> filesystem-like hierarchy using containers called "groups", and
> accesed using the tradional POSIX /path/to/resource syntax.
>
> In addition to providing interoperability with existing HDF5 datasets
> and platforms, h5py is a convienient way to store and retrieve
> arbitrary NumPy data and metadata.
>
>
> New features in 1.1
> -------------------
>
>   - A new compression filter based on the LZF library, which provides
>     transparent compression many times faster than the standard HDF5
>     GZIP filter.
>
>   - Efficient broadcasting using HDF5 hyperslab selections; for example,
>     you can write to a (2000 x 100 x 50) selection from a (100 x 50)
>     source array.
>
>   - Now supports the NumPy boolean type
>
>   - Auto-completion for IPython 0.9.X (contributed by Darren Dale)
>
>   - Installable via easy_install
>
>
> Standard features
> -----------------
>
>   - Supports storage of NumPy data of the following types:
>
>     * Integer/Unsigned Integer
>     * Float/Double
>     * Complex/Double Complex
>     * Compound ("recarray")
>     * Strings
>     * Boolean
>     * Array (as members of a compound type only)
>     * Void
>
>   - Random access to datasets using the standard NumPy slicing syntax,
>     including fancy indexing and point-based selection
>
>   - Transparent compression of datasets using GZIP, LZF or SZIP,
>     and error-detection using Fletcher32
>
>   - "Pythonic" interface supporting dictionary and NumPy-array metaphors
>     for the high-level HDF5 abstrations like groups and datasets
>
>   - A comprehensive, object-oriented wrapping of the HDF5 low-level C API
>     via Cython, in addition to the NumPy-like high-level interface.
>
>   - Supports many new features of HDF5 1.8, including recursive iteration
>     over entire files and in-library copy operations on the file tree
>
>   - Thread-safe
>
>
> Where to get it
> ---------------
>
> * Main website, documentation:  http://h5py.alfven.org
>
> * Downloads, bug tracker:       http://h5py.googlecode.com
>
>
> Requires
> --------
>
> * Linux, Mac OS-X or Windows
>
> * Python 2.5 (Windows), Python 2.5 or 2.6 (Linux/Mac OS-X)
>
> * NumPy 1.0.3 or later
>
> * HDF5 1.6.5 or later (including 1.8); HDF5 is included with
>   the Windows version.
>
>
> Thanks
> ------
>
> Thanks to D. Dale, E. Lawrence and other for their continued support
> and comments.  Also thanks to the Francesc Alted and the PyTables project,
> for inspiration and generously providing their code to the community. Thanks
> to everyone at the HDF Group for creating such a useful piece of software.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>   



From h5py at alfven.org  Tue Feb 10 04:06:10 2009
From: h5py at alfven.org (Andrew Collette)
Date: Tue, 10 Feb 2009 01:06:10 -0800
Subject: [Numpy-discussion] ANN: HDF5 for Python 1.1
In-Reply-To: <49911F11.9000109@stevesimmons.com>
References: <f125f5e0902091215w37922edak9d61ab578515f548@mail.gmail.com>
	<49911F11.9000109@stevesimmons.com>
Message-ID: <f125f5e0902100106y61820c7cpf102b311058a871b@mail.gmail.com>

Hi Stephen,

There are no immediate plans to support LZO in h5py, and in fact I'm
starting to regret including any fast compressor at all as I'm now
responsible for maintaining it. :) The reason for the dichotomy is
that LZO is released under the GPL, which is incompatible with h5py's
license.  The LZF license lets me embed it in the distribution, which
(1) frees me from an external dependency, and (2) means you can always
open h5py-created files with h5py, regardless of the platform or build
options.

Besides which, I figured people who used PyTables LZO were happy using
PyTables. :)

Andrew Collette

On Mon, Feb 9, 2009 at 10:30 PM, Stephen Simmons <mail at stevesimmons.com> wrote:
> Hi Andrew,
>
> Do you have any plans to support LZO compression in h5py?
>
> I have lots of LZO-compressed datasets created with PyTables.
> There's a real barrier to using both h5py and PyTables if the fast
> decompressor options are just LZF on h5py and LZO on PyTables.
>
> Many thanks
> Stephen
>
>
> Andrew Collette wrote:
>> =====================================
>> Announcing HDF5 for Python (h5py) 1.1
>> =====================================
>>
>> What is h5py?
>> -------------
>>
>> HDF5 for Python (h5py) is a general-purpose Python interface to the
>> Hierarchical Data Format library, version 5.  HDF5 is a versatile,
>> mature scientific software library designed for the fast, flexible
>> storage of enormous amounts of data.
>>
>> >From a Python programmer's perspective, HDF5 provides a robust way to
>> store data, organized by name in a tree-like fashion.  You can create
>> datasets (arrays on disk) hundreds of gigabytes in size, and perform
>> random-access I/O on desired sections.  Datasets are organized in a
>> filesystem-like hierarchy using containers called "groups", and
>> accesed using the tradional POSIX /path/to/resource syntax.
>>
>> In addition to providing interoperability with existing HDF5 datasets
>> and platforms, h5py is a convienient way to store and retrieve
>> arbitrary NumPy data and metadata.
>>
>>
>> New features in 1.1
>> -------------------
>>
>>   - A new compression filter based on the LZF library, which provides
>>     transparent compression many times faster than the standard HDF5
>>     GZIP filter.
>>
>>   - Efficient broadcasting using HDF5 hyperslab selections; for example,
>>     you can write to a (2000 x 100 x 50) selection from a (100 x 50)
>>     source array.
>>
>>   - Now supports the NumPy boolean type
>>
>>   - Auto-completion for IPython 0.9.X (contributed by Darren Dale)
>>
>>   - Installable via easy_install
>>
>>
>> Standard features
>> -----------------
>>
>>   - Supports storage of NumPy data of the following types:
>>
>>     * Integer/Unsigned Integer
>>     * Float/Double
>>     * Complex/Double Complex
>>     * Compound ("recarray")
>>     * Strings
>>     * Boolean
>>     * Array (as members of a compound type only)
>>     * Void
>>
>>   - Random access to datasets using the standard NumPy slicing syntax,
>>     including fancy indexing and point-based selection
>>
>>   - Transparent compression of datasets using GZIP, LZF or SZIP,
>>     and error-detection using Fletcher32
>>
>>   - "Pythonic" interface supporting dictionary and NumPy-array metaphors
>>     for the high-level HDF5 abstrations like groups and datasets
>>
>>   - A comprehensive, object-oriented wrapping of the HDF5 low-level C API
>>     via Cython, in addition to the NumPy-like high-level interface.
>>
>>   - Supports many new features of HDF5 1.8, including recursive iteration
>>     over entire files and in-library copy operations on the file tree
>>
>>   - Thread-safe
>>
>>
>> Where to get it
>> ---------------
>>
>> * Main website, documentation:  http://h5py.alfven.org
>>
>> * Downloads, bug tracker:       http://h5py.googlecode.com
>>
>>
>> Requires
>> --------
>>
>> * Linux, Mac OS-X or Windows
>>
>> * Python 2.5 (Windows), Python 2.5 or 2.6 (Linux/Mac OS-X)
>>
>> * NumPy 1.0.3 or later
>>
>> * HDF5 1.6.5 or later (including 1.8); HDF5 is included with
>>   the Windows version.
>>
>>
>> Thanks
>> ------
>>
>> Thanks to D. Dale, E. Lawrence and other for their continued support
>> and comments.  Also thanks to the Francesc Alted and the PyTables project,
>> for inspiration and generously providing their code to the community. Thanks
>> to everyone at the HDF Group for creating such a useful piece of software.
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From faltet at pytables.org  Tue Feb 10 04:36:13 2009
From: faltet at pytables.org (Francesc Alted)
Date: Tue, 10 Feb 2009 10:36:13 +0100
Subject: [Numpy-discussion] ANN: HDF5 for Python 1.1
In-Reply-To: <85b5c3130902091306m528835a9k5cff7eded4bed94d@mail.gmail.com>
References: <f125f5e0902091215w37922edak9d61ab578515f548@mail.gmail.com>
	<85b5c3130902091306m528835a9k5cff7eded4bed94d@mail.gmail.com>
Message-ID: <200902101036.13306.faltet@pytables.org>

A Monday 09 February 2009, Ondrej Certik escrigu?:
> On Mon, Feb 9, 2009 at 12:15 PM, Andrew Collette <h5py at alfven.org> 
wrote:
> > =====================================
> > Announcing HDF5 for Python (h5py) 1.1
> > =====================================
> >
> > What is h5py?
> > -------------
> >
> > HDF5 for Python (h5py) is a general-purpose Python interface to the
> > Hierarchical Data Format library, version 5.  HDF5 is a versatile,
> > mature scientific software library designed for the fast, flexible
> > storage of enormous amounts of data.
>
> Interesting project. My first question was how it is related to
> pytables. The answer is here:
>
> http://code.google.com/p/h5py/wiki/FAQ#What%27s_the_difference_betwee
>n_h5py_and_Pytables?

And here too:

http://www.pytables.org/moin/FAQ#HowdoesPyTablescomparewiththeh5pyproject.3F

Cheers,

-- 
Francesc Alted


From markus.rosenstihl at physik.tu-darmstadt.de  Tue Feb 10 05:11:38 2009
From: markus.rosenstihl at physik.tu-darmstadt.de (Markus Rosenstihl)
Date: Tue, 10 Feb 2009 11:11:38 +0100
Subject: [Numpy-discussion] linalg.norm missing an 'axis' kwarg?!
In-Reply-To: <200811201111.15930.meine@informatik.uni-hamburg.de>
References: <200811201111.15930.meine@informatik.uni-hamburg.de>
Message-ID: <E5A50C31-1EC8-4531-B579-A4B722BE2CFA@physik.tu-darmstadt.de>


Am 20.11.2008 um 11:11 schrieb Hans Meine:

> Hi,
>
> I have a 2D matrix comprising a sequence of vectors, and I want to  
> compute the
> norm of each vector.  np.linalg.norm seems to be the best bet, but  
> it does not
> support axis.  Wouldn't this be a nice feature?


Hi,
i usually do something like this:

a = random.rand(3000)
a.resize((1000,3))
vec_norms = sqrt(sum(a**2,axis=1))


It is much faster than apply_along_axis:

%timeit apply_along_axis(linalg.norm,1,a)
10 loops, best of 3: 45.3 ms per loop

%timeit sqrt(sum(a**2,axis=1))
10000 loops, best of 3: 108 ?s per loop


The results are the same:

sum(apply_along_axis(linalg.norm,1,a)- sqrt(sum(a**2,axis=1)))
0.0

Regards,
Markus


From faltet at pytables.org  Tue Feb 10 06:29:48 2009
From: faltet at pytables.org (Francesc Alted)
Date: Tue, 10 Feb 2009 12:29:48 +0100
Subject: [Numpy-discussion] ANN: HDF5 for Python 1.1
In-Reply-To: <49911F11.9000109@stevesimmons.com>
References: <f125f5e0902091215w37922edak9d61ab578515f548@mail.gmail.com>
	<49911F11.9000109@stevesimmons.com>
Message-ID: <200902101229.49107.faltet@pytables.org>

Hi Stephen,

A Tuesday 10 February 2009, Stephen Simmons escrigu?:
> I have lots of LZO-compressed datasets created with PyTables.
> There's a real barrier to using both h5py and PyTables if the fast
> decompressor options are just LZF on h5py and LZO on PyTables.

You can always use ptrepack utility in PyTables to convert your existing 
PyTables files compressed with LZO to zlib, which is the standard 
compressor used by HDF5.  From that, you can reconvert them to LZF if 
this is your whish.

HTH,

-- 
Francesc Alted


From watson.jim at gmail.com  Tue Feb 10 07:08:16 2009
From: watson.jim at gmail.com (James Watson)
Date: Tue, 10 Feb 2009 12:08:16 +0000
Subject: [Numpy-discussion] porting NumPy to Python 3
In-Reply-To: <4989DAD7.9060507@gmail.com>
References: <f14846790902040902w57983eaav83eb2de2d7a19b86@mail.gmail.com>
	<e06186140902040944w1c71cc05ib954b6bc04a26fec@mail.gmail.com>
	<4989DAD7.9060507@gmail.com>
Message-ID: <f14846790902100408v7d155deev1774f2ad037cdca7@mail.gmail.com>

I'm taking Chuck and Bruce's suggestions and making sure that the
changes are compatible with Python 2.4 and 2.6.

I want to make sure diffs are against latest code, but keep getting
this svn error:
svn update
svn: OPTIONS of 'http://scipy.org/svn/numpy/trunk': Could not read
status line: Connection reset by peer (http://scipy.org)

Am I doing something wrong, or is the server temporarily down?

James.



>>     To get 2to3 to run without warnings, the following files require
>>     minor changes:
>>     - numpy/distutils/fcompiler/intel.py
>>     - numpy/distutils/misc_util.py
>>     - numpy/distutils/command/build_src.py
>>     - numpy/f2py/crackfortran.py
>>     - numpy/lib/function_base.py
>>     - numpy/linalg/lapack_lite/make_lite.py
>>



>>
> If you follow Guido's recommendation (at the bottom of
> http://docs.python.org/dev/3.0/whatsnew/3.0.html), then we should first
> be compatible with Python 2.6 (about the current stage) . Then
> compatible using Python 2.3 with the -3 flag (warn about Python 3.x
> incompatibilities) and fix warnings like:
>  numpy/core/__init__.py:11: DeprecationWarning: the cPickle module has
> been removed in Python 3.0
> (Of course you lose speed of cPickle if you blindly change it to use the
> old Pickle in Python 2.x)
>
> Finally use the 2to3 tool to get a Python 3 version.
>
> Bruce
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From scott.sinclair.za at gmail.com  Tue Feb 10 07:34:25 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Tue, 10 Feb 2009 14:34:25 +0200
Subject: [Numpy-discussion] porting NumPy to Python 3
In-Reply-To: <f14846790902100408v7d155deev1774f2ad037cdca7@mail.gmail.com>
References: <f14846790902040902w57983eaav83eb2de2d7a19b86@mail.gmail.com>
	<e06186140902040944w1c71cc05ib954b6bc04a26fec@mail.gmail.com>
	<4989DAD7.9060507@gmail.com>
	<f14846790902100408v7d155deev1774f2ad037cdca7@mail.gmail.com>
Message-ID: <6a17e9ee0902100434w344e8c87p963e5c029ff2b5d3@mail.gmail.com>

> 2009/2/10 James Watson <watson.jim at gmail.com>:
> I want to make sure diffs are against latest code, but keep getting
> this svn error:
> svn update
> svn: OPTIONS of 'http://scipy.org/svn/numpy/trunk': Could not read
> status line: Connection reset by peer (http://scipy.org)

There is some problem at the moment.

This seems to be quite common recently, during the very early morning
(USA time zones). I guess this is because Murphy's law states that all
server problems occur when the admin is trying to get some sleep ;-)

Cheers,
Scott


From paul at rudin.co.uk  Tue Feb 10 08:18:59 2009
From: paul at rudin.co.uk (Paul Rudin)
Date: Tue, 10 Feb 2009 13:18:59 +0000
Subject: [Numpy-discussion] numpy-izing a loop
Message-ID: <87y6we1lbw.fsf@rudin.co.uk>


I've just written this snippet of code:

result = numpy.empty((dim, dim, dim), numpy.bool)
for x in xrange(dim):
    for y in xrange(dim):
        for z in xrange(dim):
            result[x, y, z] = ((xnear[y, z] < xfar[y, z]) and 
                               (ynear[x, z] < yfar[x, z]) and
                               (znear[x, y] < zfar[x, y]))


Is there a way to get numpy to do the looping?



From meine at informatik.uni-hamburg.de  Tue Feb 10 09:02:39 2009
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Tue, 10 Feb 2009 15:02:39 +0100
Subject: [Numpy-discussion] linalg.norm missing an 'axis' kwarg?!
In-Reply-To: <E5A50C31-1EC8-4531-B579-A4B722BE2CFA@physik.tu-darmstadt.de>
References: <200811201111.15930.meine@informatik.uni-hamburg.de>
	<E5A50C31-1EC8-4531-B579-A4B722BE2CFA@physik.tu-darmstadt.de>
Message-ID: <200902101502.40982.meine@informatik.uni-hamburg.de>

On Tuesday 10 February 2009 11:11:38 Markus Rosenstihl wrote:
> i usually do something like this:
>
> a = random.rand(3000)
> a.resize((1000,3))
> vec_norms = sqrt(sum(a**2,axis=1))

If you look at the patch I posted (OK, that was some weeks ago, so I'll attach 
it again for your convenience), that's (more or less) exactly what I proposed.

Have a nice day,
  Hans
-------------- next part --------------
A non-text attachment was scrubbed...
Name: numpy_norm_axis.diff
Type: text/x-patch
Size: 1597 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090210/ae94f124/attachment.bin>

From chanley at stsci.edu  Tue Feb 10 10:01:22 2009
From: chanley at stsci.edu (Christopher Hanley)
Date: Tue, 10 Feb 2009 10:01:22 -0500
Subject: [Numpy-discussion] Solaris 8, 32 bit python issue
Message-ID: <499196C2.3060206@stsci.edu>

This problem is on a 32bit Solaris 8 system.


======================================================================
FAIL: Test find_duplicates
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/ra/pyssg/2.5.1/numpy/lib/tests/test_recfunctions.py", line 
150, in test_find_duplicates
    assert_equal(test[-1], control)
  File "/usr/stsci/pyssgdev/2.5.1/numpy/ma/testutils.py", line 121, in 
assert_equal
    return assert_array_equal(actual, desired, err_msg)
  File "/usr/stsci/pyssgdev/2.5.1/numpy/ma/testutils.py", line 193, in 
assert_array_equal
    header='Arrays are not equal')
  File "/usr/stsci/pyssgdev/2.5.1/numpy/ma/testutils.py", line 186, in 
assert_array_compare
    verbose=verbose, header=header)
  File "/usr/stsci/pyssgdev/2.5.1/numpy/testing/utils.py", line 295, in 
assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not equal

(mismatch 100.0%)
x: array([2, 0])
y: array([0, 2])

----------------------------------------------------------------------
Ran 1899 tests in 99.598s



-- 
Christopher Hanley
Senior Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338


From stefan at sun.ac.za  Tue Feb 10 10:40:10 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 10 Feb 2009 17:40:10 +0200
Subject: [Numpy-discussion] linalg.norm missing an 'axis' kwarg?!
In-Reply-To: <200902101502.40982.meine@informatik.uni-hamburg.de>
References: <200811201111.15930.meine@informatik.uni-hamburg.de>
	<E5A50C31-1EC8-4531-B579-A4B722BE2CFA@physik.tu-darmstadt.de>
	<200902101502.40982.meine@informatik.uni-hamburg.de>
Message-ID: <9457e7c80902100740r1ec14650r1c2f7883aa7326ec@mail.gmail.com>

Hi Hans

2009/2/10 Hans Meine <meine at informatik.uni-hamburg.de>:
> If you look at the patch I posted (OK, that was some weeks ago, so I'll attach
> it again for your convenience), that's (more or less) exactly what I proposed.

Would you mind adding some tests to the patch?

Cheers
St?fan


From lists_ravi at lavabit.com  Tue Feb 10 11:13:36 2009
From: lists_ravi at lavabit.com (Ravi)
Date: Tue, 10 Feb 2009 11:13:36 -0500
Subject: [Numpy-discussion] Unnecessary axes in recarrays
Message-ID: <200902101113.36678.lists_ravi@lavabit.com>

Hi,
  Recarrays seem to sprout extra axes when they are nested:

----
In [98]: np.__version__
Out[98]: '1.2.0'

In [99]: d1 = dtype( [ ('a',uint8,2), ('b',uint8,1) ] )

In [100]: d2 = dtype( [ ('c',uint8,1), ('d',d1,1) ] )

In [101]: d3 = dtype( [ ('c',uint8,1), ('d',d1,2) ] )

In [102]: a1 = zeros( (4,), dtype=uint8 ).view( d2 )

In [103]: a1['d'].shape
Out[103]: (1,)

In [104]: a2 = zeros( (7,), dtype=uint8 ).view( d3 )

In [105]: a2['d'].shape
Out[105]: (1, 2)

In [106]: d4 = dtype( [ ('c',uint8,1), ('d',d1,(1,1)) ] )

In [107]: zeros( (4,), dtype=uint8 ).view( d2 )[ 'd' ].shape
Out[107]: (1,)
----

Why does d3 field 'd' have an extra axis? And why does d4 field 'd' have only 
one axis?

Regards,
Ravi





From stefan at sun.ac.za  Tue Feb 10 11:19:54 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 10 Feb 2009 18:19:54 +0200
Subject: [Numpy-discussion] numpy-izing a loop
In-Reply-To: <87y6we1lbw.fsf@rudin.co.uk>
References: <87y6we1lbw.fsf@rudin.co.uk>
Message-ID: <9457e7c80902100819t139d6342y8885f417a4ec4443@mail.gmail.com>

Hi Paul

2009/2/10 Paul Rudin <paul at rudin.co.uk>:
>
> I've just written this snippet of code:
>
> result = numpy.empty((dim, dim, dim), numpy.bool)
> for x in xrange(dim):
>    for y in xrange(dim):
>        for z in xrange(dim):
>            result[x, y, z] = ((xnear[y, z] < xfar[y, z]) and
>                               (ynear[x, z] < yfar[x, z]) and
>                               (znear[x, y] < zfar[x, y]))
>
>
> Is there a way to get numpy to do the looping?

You can try the following:

x = np.arange(dim)
y = np.arange(dim)
z = np.arange(dim)

result = (xnear[y,z] < xfar[y, z]) & (ynear[x, z] < yfar[x, z]) &
(znear[x,y] < zfar[x,y])

It broadcasts correctly, but you'll have to verify the results to be sure.

Cheers
St?fan


From mjanikas at esri.com  Tue Feb 10 14:29:07 2009
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 10 Feb 2009 11:29:07 -0800
Subject: [Numpy-discussion] Permutations in Simulations`
Message-ID: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0BD6A@redmx1.esri.com>

Hello All,

I want to create an array that contains a column of permutations for each simulation:

import numpy as NUM
import numpy.random as RAND
x = NUM.arange(4.)
res = NUM.zeros((4,100))

for sim in range(100):
res[:,sim] = RAND.permutation(x)


Is there a way to do this without a loop?  Thanks so much ahead of time...

MJ

Mark Janikas
Product Engineer
ESRI, Geoprocessing
380 New York St.
Redlands, CA 92373
909-793-2853 (2563)
mjanikas at esri.com<mailto:mjanikas at esri.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090210/32a155fc/attachment.html>

From robert.kern at gmail.com  Tue Feb 10 14:48:59 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 10 Feb 2009 13:48:59 -0600
Subject: [Numpy-discussion] numpy-izing a loop
In-Reply-To: <9457e7c80902100819t139d6342y8885f417a4ec4443@mail.gmail.com>
References: <87y6we1lbw.fsf@rudin.co.uk>
	<9457e7c80902100819t139d6342y8885f417a4ec4443@mail.gmail.com>
Message-ID: <3d375d730902101148t10d50084k1a775e681a5e3d05@mail.gmail.com>

On Tue, Feb 10, 2009 at 10:19, St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi Paul
>
> 2009/2/10 Paul Rudin <paul at rudin.co.uk>:
>>
>> I've just written this snippet of code:
>>
>> result = numpy.empty((dim, dim, dim), numpy.bool)
>> for x in xrange(dim):
>>    for y in xrange(dim):
>>        for z in xrange(dim):
>>            result[x, y, z] = ((xnear[y, z] < xfar[y, z]) and
>>                               (ynear[x, z] < yfar[x, z]) and
>>                               (znear[x, y] < zfar[x, y]))
>>
>>
>> Is there a way to get numpy to do the looping?
>
> You can try the following:
>
> x = np.arange(dim)
> y = np.arange(dim)
> z = np.arange(dim)
>
> result = (xnear[y,z] < xfar[y, z]) & (ynear[x, z] < yfar[x, z]) &
> (znear[x,y] < zfar[x,y])
>
> It broadcasts correctly, but you'll have to verify the results to be sure.

It doesn't broadcast correctly. You will get result.shape == (dim,)
but he wants (dim,dim,dim). You can get this with the following:

  x = np.arange(dim)[:,np.newaxis,np.newaxis]
  y = np.arange(dim)[np.newaxis,:,np.newaxis]
  z = np.arange(dim)[np.newaxis,np.newaxis,:]

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From kwgoodman at gmail.com  Tue Feb 10 15:18:55 2009
From: kwgoodman at gmail.com (Keith Goodman)
Date: Tue, 10 Feb 2009 12:18:55 -0800
Subject: [Numpy-discussion] Permutations in Simulations`
In-Reply-To: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0BD6A@redmx1.esri.com>
References: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0BD6A@redmx1.esri.com>
Message-ID: <f4f93d420902101218p3bfc8a8fl280a87bd4f6e7f9d@mail.gmail.com>

On Tue, Feb 10, 2009 at 11:29 AM, Mark Janikas <mjanikas at esri.com> wrote:
> I want to create an array that contains a column of permutations for each
> simulation:
>
> import numpy as NUM
>
> import numpy.random as RAND
>
> x = NUM.arange(4.)
>
> res = NUM.zeros((4,100))
>
>
> for sim in range(100):
>
> res[:,sim] = RAND.permutation(x)
>
>
> Is there a way to do this without a loop?  Thanks so much ahead of time?

Does this work? Might not be faster but it does avoid the loop.

import numpy as np

def weirdshuffle(nx, ny):
    x = np.ones((nx,ny)).cumsum(0, dtype=np.int) - 1
    yidx = np.ones((nx,ny)).cumsum(1, dtype=np.int) - 1
    xidx = np.random.rand(nx,ny).argsort(0).argsort(0)
    return x[xidx, yidx]


From kwgoodman at gmail.com  Tue Feb 10 15:28:24 2009
From: kwgoodman at gmail.com (Keith Goodman)
Date: Tue, 10 Feb 2009 12:28:24 -0800
Subject: [Numpy-discussion] Permutations in Simulations`
In-Reply-To: <f4f93d420902101218p3bfc8a8fl280a87bd4f6e7f9d@mail.gmail.com>
References: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0BD6A@redmx1.esri.com>
	<f4f93d420902101218p3bfc8a8fl280a87bd4f6e7f9d@mail.gmail.com>
Message-ID: <f4f93d420902101228i62660221y3e31dc717f1674ee@mail.gmail.com>

On Tue, Feb 10, 2009 at 12:18 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Tue, Feb 10, 2009 at 11:29 AM, Mark Janikas <mjanikas at esri.com> wrote:
>> I want to create an array that contains a column of permutations for each
>> simulation:
>>
>> import numpy as NUM
>>
>> import numpy.random as RAND
>>
>> x = NUM.arange(4.)
>>
>> res = NUM.zeros((4,100))
>>
>>
>> for sim in range(100):
>>
>> res[:,sim] = RAND.permutation(x)
>>
>>
>> Is there a way to do this without a loop?  Thanks so much ahead of time?
>
> Does this work? Might not be faster but it does avoid the loop.
>
> import numpy as np
>
> def weirdshuffle(nx, ny):
>    x = np.ones((nx,ny)).cumsum(0, dtype=np.int) - 1
>    yidx = np.ones((nx,ny)).cumsum(1, dtype=np.int) - 1
>    xidx = np.random.rand(nx,ny).argsort(0).argsort(0)
>    return x[xidx, yidx]

Hey, it is faster for nx=4, ny=100

def baseshuffle(nx, ny):
    x = np.arange(nx)
    res = np.zeros((nx,ny))
    for sim in range(ny):
        res[:,sim] = np.random.permutation(x)
    return res

>> timeit baseshuffle(4,100)
1000 loops, best of 3: 1.11 ms per loop
>> timeit weirdshuffle(4,100)
10000 loops, best of 3: 127 ?s per loop

OK, who can cut that time in half? My first try looks clunky.


From kwgoodman at gmail.com  Tue Feb 10 15:41:45 2009
From: kwgoodman at gmail.com (Keith Goodman)
Date: Tue, 10 Feb 2009 12:41:45 -0800
Subject: [Numpy-discussion] Permutations in Simulations`
In-Reply-To: <f4f93d420902101228i62660221y3e31dc717f1674ee@mail.gmail.com>
References: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0BD6A@redmx1.esri.com>
	<f4f93d420902101218p3bfc8a8fl280a87bd4f6e7f9d@mail.gmail.com>
	<f4f93d420902101228i62660221y3e31dc717f1674ee@mail.gmail.com>
Message-ID: <f4f93d420902101241y6f2ec949kd15991bfb466f1d5@mail.gmail.com>

On Tue, Feb 10, 2009 at 12:28 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Tue, Feb 10, 2009 at 12:18 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>> On Tue, Feb 10, 2009 at 11:29 AM, Mark Janikas <mjanikas at esri.com> wrote:
>>> I want to create an array that contains a column of permutations for each
>>> simulation:
>>>
>>> import numpy as NUM
>>>
>>> import numpy.random as RAND
>>>
>>> x = NUM.arange(4.)
>>>
>>> res = NUM.zeros((4,100))
>>>
>>>
>>> for sim in range(100):
>>>
>>> res[:,sim] = RAND.permutation(x)
>>>
>>>
>>> Is there a way to do this without a loop?  Thanks so much ahead of time?
>>
>> Does this work? Might not be faster but it does avoid the loop.
>>
>> import numpy as np
>>
>> def weirdshuffle(nx, ny):
>>    x = np.ones((nx,ny)).cumsum(0, dtype=np.int) - 1
>>    yidx = np.ones((nx,ny)).cumsum(1, dtype=np.int) - 1
>>    xidx = np.random.rand(nx,ny).argsort(0).argsort(0)
>>    return x[xidx, yidx]
>
> Hey, it is faster for nx=4, ny=100
>
> def baseshuffle(nx, ny):
>    x = np.arange(nx)
>    res = np.zeros((nx,ny))
>    for sim in range(ny):
>        res[:,sim] = np.random.permutation(x)
>    return res
>
>>> timeit baseshuffle(4,100)
> 1000 loops, best of 3: 1.11 ms per loop
>>> timeit weirdshuffle(4,100)
> 10000 loops, best of 3: 127 ?s per loop
>
> OK, who can cut that time in half? My first try looks clunky.

This is a little faster:

def weirdshuffle2(nx, ny):
    one = np.ones((nx,ny), dtype=np.int)
    x = one.cumsum(0)
    x -= 1
    yidx = one.cumsum(1)
    yidx -= 1
    xidx = np.random.random_sample((nx,ny)).argsort(0).argsort(0)
    return x[xidx, yidx]

>> timeit weirdshuffle(4,100)
10000 loops, best of 3: 129 ?s per loop
>> timeit weirdshuffle2(4,100)
10000 loops, best of 3: 106 ?s per loop


From kwgoodman at gmail.com  Tue Feb 10 15:58:37 2009
From: kwgoodman at gmail.com (Keith Goodman)
Date: Tue, 10 Feb 2009 12:58:37 -0800
Subject: [Numpy-discussion] Permutations in Simulations`
In-Reply-To: <f4f93d420902101241y6f2ec949kd15991bfb466f1d5@mail.gmail.com>
References: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0BD6A@redmx1.esri.com>
	<f4f93d420902101218p3bfc8a8fl280a87bd4f6e7f9d@mail.gmail.com>
	<f4f93d420902101228i62660221y3e31dc717f1674ee@mail.gmail.com>
	<f4f93d420902101241y6f2ec949kd15991bfb466f1d5@mail.gmail.com>
Message-ID: <f4f93d420902101258h50f109fgc435fae1a70f1bc5@mail.gmail.com>

On Tue, Feb 10, 2009 at 12:41 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Tue, Feb 10, 2009 at 12:28 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>> On Tue, Feb 10, 2009 at 12:18 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>>> On Tue, Feb 10, 2009 at 11:29 AM, Mark Janikas <mjanikas at esri.com> wrote:
>>>> I want to create an array that contains a column of permutations for each
>>>> simulation:
>>>>
>>>> import numpy as NUM
>>>>
>>>> import numpy.random as RAND
>>>>
>>>> x = NUM.arange(4.)
>>>>
>>>> res = NUM.zeros((4,100))
>>>>
>>>>
>>>> for sim in range(100):
>>>>
>>>> res[:,sim] = RAND.permutation(x)
>>>>
>>>>
>>>> Is there a way to do this without a loop?  Thanks so much ahead of time?
>>>
>>> Does this work? Might not be faster but it does avoid the loop.
>>>
>>> import numpy as np
>>>
>>> def weirdshuffle(nx, ny):
>>>    x = np.ones((nx,ny)).cumsum(0, dtype=np.int) - 1
>>>    yidx = np.ones((nx,ny)).cumsum(1, dtype=np.int) - 1
>>>    xidx = np.random.rand(nx,ny).argsort(0).argsort(0)
>>>    return x[xidx, yidx]
>>
>> Hey, it is faster for nx=4, ny=100
>>
>> def baseshuffle(nx, ny):
>>    x = np.arange(nx)
>>    res = np.zeros((nx,ny))
>>    for sim in range(ny):
>>        res[:,sim] = np.random.permutation(x)
>>    return res
>>
>>>> timeit baseshuffle(4,100)
>> 1000 loops, best of 3: 1.11 ms per loop
>>>> timeit weirdshuffle(4,100)
>> 10000 loops, best of 3: 127 ?s per loop
>>
>> OK, who can cut that time in half? My first try looks clunky.
>
> This is a little faster:
>
> def weirdshuffle2(nx, ny):
>    one = np.ones((nx,ny), dtype=np.int)
>    x = one.cumsum(0)
>    x -= 1
>    yidx = one.cumsum(1)
>    yidx -= 1
>    xidx = np.random.random_sample((nx,ny)).argsort(0).argsort(0)
>    return x[xidx, yidx]
>
>>> timeit weirdshuffle(4,100)
> 10000 loops, best of 3: 129 ?s per loop
>>> timeit weirdshuffle2(4,100)
> 10000 loops, best of 3: 106 ?s per loop

Sorry for all the mail.

def weirdshuffle3(nx, ny):
    return np.random.random_sample((nx,ny)).argsort(0).argsort(0)

>> timeit weirdshuffle(4,100)
10000 loops, best of 3: 128 ?s per loop
>> timeit weirdshuffle3(4,100)
10000 loops, best of 3: 37.5 ?s per loop


From yakov.keselman at gmail.com  Tue Feb 10 16:11:36 2009
From: yakov.keselman at gmail.com (Yakov Keselman)
Date: Tue, 10 Feb 2009 13:11:36 -0800
Subject: [Numpy-discussion] from_function
In-Reply-To: <gm9d28$slc$1@ger.gmane.org>
References: <gm9d28$slc$1@ger.gmane.org>
Message-ID: <dc3116ef0902101311j7be2cd00od3aac4a83f8ea450@mail.gmail.com>

Perhaps you can do something along the following lines to get around
this limitation:

#################

# parameterizes the original function by delta, size.
def parameterized_function(delta, size, function):
    center = (size-1)/2.0
    return lambda i: function( (i-center)*delta )


# the function to which we want to apply "fromfunction".
def square(x): return x*x

# test script
from numpy import fromfunction
Delta = 0.1
Size = 9
print fromfunction( parameterized_function(Delta, Size, square), (Size,) )

###########

[ 0.16  0.09  0.04  0.01  0.    0.01  0.04  0.09  0.16]


On 2/3/09, Neal Becker <ndbecker2 at gmail.com> wrote:
> I've been using something I wrote:
>
> coef_from_function (function, delta, size)
> which does (c++ code):
>
>   double center = double(size-1)/2;
>   for (int i = 0; i < size; ++i)
>     coef[i] = call<value_t> (func, double(i - center) * delta);
>
> I thought to translate this to np.fromfunction.  It seems fromfunction is
> not as flexible, it uses only a fixed integer grid?
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


-- 
Not to laugh, not to lament, not to curse, but to understand. -- Spinoza


From stefan at sun.ac.za  Tue Feb 10 16:27:31 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 10 Feb 2009 23:27:31 +0200
Subject: [Numpy-discussion] numpy-izing a loop
In-Reply-To: <3d375d730902101148t10d50084k1a775e681a5e3d05@mail.gmail.com>
References: <87y6we1lbw.fsf@rudin.co.uk>
	<9457e7c80902100819t139d6342y8885f417a4ec4443@mail.gmail.com>
	<3d375d730902101148t10d50084k1a775e681a5e3d05@mail.gmail.com>
Message-ID: <9457e7c80902101327q5cd7cd4ax7e2c1713fcee2cd4@mail.gmail.com>

2009/2/10 Robert Kern <robert.kern at gmail.com>:
>  x = np.arange(dim)[:,np.newaxis,np.newaxis]
>  y = np.arange(dim)[np.newaxis,:,np.newaxis]
>  z = np.arange(dim)[np.newaxis,np.newaxis,:]

Yes, sorry, I should have copied from my terminal.

I think I had

x = np.arange(dim)
y = np.arange(dim)[:, None]
z = np.arange(dim)[:, None, None]

which broadcasts the same way.

Cheers
St?fan


From stefan at sun.ac.za  Tue Feb 10 16:33:35 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 10 Feb 2009 23:33:35 +0200
Subject: [Numpy-discussion] numpy-izing a loop
In-Reply-To: <9457e7c80902101327q5cd7cd4ax7e2c1713fcee2cd4@mail.gmail.com>
References: <87y6we1lbw.fsf@rudin.co.uk>
	<9457e7c80902100819t139d6342y8885f417a4ec4443@mail.gmail.com>
	<3d375d730902101148t10d50084k1a775e681a5e3d05@mail.gmail.com>
	<9457e7c80902101327q5cd7cd4ax7e2c1713fcee2cd4@mail.gmail.com>
Message-ID: <9457e7c80902101333l5883971i2b15c6290315bf8d@mail.gmail.com>

2009/2/10 St?fan van der Walt <stefan at sun.ac.za>:
> x = np.arange(dim)
> y = np.arange(dim)[:, None]
> z = np.arange(dim)[:, None, None]

Do not operate heavy machinery or attempt broadcasting while tired or
under the influence.  That order was incorrect:

> z = np.arange(dim)
> y = np.arange(dim)[:, None]
> x = np.arange(dim)[:, None, None]

Cheers
St?fan


From markperrymiller at gmail.com  Tue Feb 10 16:41:07 2009
From: markperrymiller at gmail.com (Mark Miller)
Date: Tue, 10 Feb 2009 13:41:07 -0800
Subject: [Numpy-discussion] Permutations in Simulations`
In-Reply-To: <f4f93d420902101258h50f109fgc435fae1a70f1bc5@mail.gmail.com>
References: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0BD6A@redmx1.esri.com>
	<f4f93d420902101218p3bfc8a8fl280a87bd4f6e7f9d@mail.gmail.com>
	<f4f93d420902101228i62660221y3e31dc717f1674ee@mail.gmail.com>
	<f4f93d420902101241y6f2ec949kd15991bfb466f1d5@mail.gmail.com>
	<f4f93d420902101258h50f109fgc435fae1a70f1bc5@mail.gmail.com>
Message-ID: <d1eaff10902101341w78f44361g60f6b20423c4c7f9@mail.gmail.com>

Out of curiosity, why wouldn't numpy.apply_along_axis be a reasonable
approach here.  Even more curious:  why is it slower than the original
explicit loop?

Learning,

-Mark

import numpy as np
import timeit

def baseshuffle(nx, ny):
    x = np.arange(nx)
    res = np.zeros((nx,ny),int)
    for sim in range(ny):
        res[:,sim] = np.random.permutation(x)
    return res

def axisshuffle(nx,ny):
    x=np.arange(nx).reshape(nx,1)
    res=np.zeros((nx,ny),int)
    res[:] = x
    x1 = np.apply_along_axis(np.random.permutation,0,res)
    return x1

nx = 10
ny = 100

t=timeit.Timer("baseshuffle(nx, ny)", "from __main__ import *")
print t.timeit(100)
t=timeit.Timer("axisshuffle(nx,ny)", "from __main__ import *")
print t.timeit(100)


>>>
0.111320049056
0.297792159759
>>>












On Tue, Feb 10, 2009 at 12:58 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Tue, Feb 10, 2009 at 12:41 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>> On Tue, Feb 10, 2009 at 12:28 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>>> On Tue, Feb 10, 2009 at 12:18 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>>>> On Tue, Feb 10, 2009 at 11:29 AM, Mark Janikas <mjanikas at esri.com> wrote:
>>>>> I want to create an array that contains a column of permutations for each
>>>>> simulation:
>>>>>
>>>>> import numpy as NUM
>>>>>
>>>>> import numpy.random as RAND
>>>>>
>>>>> x = NUM.arange(4.)
>>>>>
>>>>> res = NUM.zeros((4,100))
>>>>>
>>>>>
>>>>> for sim in range(100):
>>>>>
>>>>> res[:,sim] = RAND.permutation(x)
>>>>>
>>>>>
>>>>> Is there a way to do this without a loop?  Thanks so much ahead of time?
>>>>
>>>> Does this work? Might not be faster but it does avoid the loop.
>>>>
>>>> import numpy as np
>>>>
>>>> def weirdshuffle(nx, ny):
>>>>    x = np.ones((nx,ny)).cumsum(0, dtype=np.int) - 1
>>>>    yidx = np.ones((nx,ny)).cumsum(1, dtype=np.int) - 1
>>>>    xidx = np.random.rand(nx,ny).argsort(0).argsort(0)
>>>>    return x[xidx, yidx]
>>>
>>> Hey, it is faster for nx=4, ny=100
>>>
>>> def baseshuffle(nx, ny):
>>>    x = np.arange(nx)
>>>    res = np.zeros((nx,ny))
>>>    for sim in range(ny):
>>>        res[:,sim] = np.random.permutation(x)
>>>    return res
>>>
>>>>> timeit baseshuffle(4,100)
>>> 1000 loops, best of 3: 1.11 ms per loop
>>>>> timeit weirdshuffle(4,100)
>>> 10000 loops, best of 3: 127 ?s per loop
>>>
>>> OK, who can cut that time in half? My first try looks clunky.
>>
>> This is a little faster:
>>
>> def weirdshuffle2(nx, ny):
>>    one = np.ones((nx,ny), dtype=np.int)
>>    x = one.cumsum(0)
>>    x -= 1
>>    yidx = one.cumsum(1)
>>    yidx -= 1
>>    xidx = np.random.random_sample((nx,ny)).argsort(0).argsort(0)
>>    return x[xidx, yidx]
>>
>>>> timeit weirdshuffle(4,100)
>> 10000 loops, best of 3: 129 ?s per loop
>>>> timeit weirdshuffle2(4,100)
>> 10000 loops, best of 3: 106 ?s per loop
>
> Sorry for all the mail.
>
> def weirdshuffle3(nx, ny):
>    return np.random.random_sample((nx,ny)).argsort(0).argsort(0)
>
>>> timeit weirdshuffle(4,100)
> 10000 loops, best of 3: 128 ?s per loop
>>> timeit weirdshuffle3(4,100)
> 10000 loops, best of 3: 37.5 ?s per loop
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From kwgoodman at gmail.com  Tue Feb 10 16:50:15 2009
From: kwgoodman at gmail.com (Keith Goodman)
Date: Tue, 10 Feb 2009 13:50:15 -0800
Subject: [Numpy-discussion] Permutations in Simulations`
In-Reply-To: <d1eaff10902101341w78f44361g60f6b20423c4c7f9@mail.gmail.com>
References: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0BD6A@redmx1.esri.com>
	<f4f93d420902101218p3bfc8a8fl280a87bd4f6e7f9d@mail.gmail.com>
	<f4f93d420902101228i62660221y3e31dc717f1674ee@mail.gmail.com>
	<f4f93d420902101241y6f2ec949kd15991bfb466f1d5@mail.gmail.com>
	<f4f93d420902101258h50f109fgc435fae1a70f1bc5@mail.gmail.com>
	<d1eaff10902101341w78f44361g60f6b20423c4c7f9@mail.gmail.com>
Message-ID: <f4f93d420902101350s5ffddf42uec7ad4f7102bfb70@mail.gmail.com>

On Tue, Feb 10, 2009 at 1:41 PM, Mark Miller <markperrymiller at gmail.com> wrote:
> Out of curiosity, why wouldn't numpy.apply_along_axis be a reasonable
> approach here.  Even more curious:  why is it slower than the original
> explicit loop?

I took a quick look at the apply_along_axis code. It is numpy code
(not c) and it uses a while loop to loop over the axis.

In ipython just type np.apply_along_axis?? <enter>


From markperrymiller at gmail.com  Tue Feb 10 17:23:19 2009
From: markperrymiller at gmail.com (Mark Miller)
Date: Tue, 10 Feb 2009 14:23:19 -0800
Subject: [Numpy-discussion] Permutations in Simulations`
In-Reply-To: <f4f93d420902101350s5ffddf42uec7ad4f7102bfb70@mail.gmail.com>
References: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0BD6A@redmx1.esri.com>
	<f4f93d420902101218p3bfc8a8fl280a87bd4f6e7f9d@mail.gmail.com>
	<f4f93d420902101228i62660221y3e31dc717f1674ee@mail.gmail.com>
	<f4f93d420902101241y6f2ec949kd15991bfb466f1d5@mail.gmail.com>
	<f4f93d420902101258h50f109fgc435fae1a70f1bc5@mail.gmail.com>
	<d1eaff10902101341w78f44361g60f6b20423c4c7f9@mail.gmail.com>
	<f4f93d420902101350s5ffddf42uec7ad4f7102bfb70@mail.gmail.com>
Message-ID: <d1eaff10902101423g1858d0a4ka81df2c624e77be8@mail.gmail.com>

Got it.  Thanks!

On Tue, Feb 10, 2009 at 1:50 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Tue, Feb 10, 2009 at 1:41 PM, Mark Miller <markperrymiller at gmail.com> wrote:
>> Out of curiosity, why wouldn't numpy.apply_along_axis be a reasonable
>> approach here.  Even more curious:  why is it slower than the original
>> explicit loop?
>
> I took a quick look at the apply_along_axis code. It is numpy code
> (not c) and it uses a while loop to loop over the axis.
>
> In ipython just type np.apply_along_axis?? <enter>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From mjanikas at esri.com  Tue Feb 10 19:21:12 2009
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 10 Feb 2009 16:21:12 -0800
Subject: [Numpy-discussion] Permutations in Simulations`
In-Reply-To: <f4f93d420902101258h50f109fgc435fae1a70f1bc5@mail.gmail.com>
References: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0BD6A@redmx1.esri.com>
	<f4f93d420902101218p3bfc8a8fl280a87bd4f6e7f9d@mail.gmail.com>
	<f4f93d420902101228i62660221y3e31dc717f1674ee@mail.gmail.com>
	<f4f93d420902101241y6f2ec949kd15991bfb466f1d5@mail.gmail.com>
	<f4f93d420902101258h50f109fgc435fae1a70f1bc5@mail.gmail.com>
Message-ID: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0C2AE@redmx1.esri.com>

Thanks to all for your replies.  I want this to work on any vector so I was thinking this...?

import numpy as np
import timeit
x = np.array([4.,5.,10.,3.,5.,6.,7.,2.,9.,1.])
nx = 10
ny = 100

def weirdshuffle4(x, ny):
    nx = len(x)
    indices = np.random.random_sample((nx,ny)).argsort(0).argsort(0)
    return x[indices]

t=timeit.Timer("weirdshuffle4(x,ny)", "from __main__ import *") 
print t.timeit(100)

0.0148663153873


-----Original Message-----
From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Keith Goodman
Sent: Tuesday, February 10, 2009 12:59 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Permutations in Simulations`

On Tue, Feb 10, 2009 at 12:41 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Tue, Feb 10, 2009 at 12:28 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>> On Tue, Feb 10, 2009 at 12:18 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>>> On Tue, Feb 10, 2009 at 11:29 AM, Mark Janikas <mjanikas at esri.com> wrote:
>>>> I want to create an array that contains a column of permutations for each
>>>> simulation:
>>>>
>>>> import numpy as NUM
>>>>
>>>> import numpy.random as RAND
>>>>
>>>> x = NUM.arange(4.)
>>>>
>>>> res = NUM.zeros((4,100))
>>>>
>>>>
>>>> for sim in range(100):
>>>>
>>>> res[:,sim] = RAND.permutation(x)
>>>>
>>>>
>>>> Is there a way to do this without a loop?  Thanks so much ahead of time.
>>>
>>> Does this work? Might not be faster but it does avoid the loop.
>>>
>>> import numpy as np
>>>
>>> def weirdshuffle(nx, ny):
>>>    x = np.ones((nx,ny)).cumsum(0, dtype=np.int) - 1
>>>    yidx = np.ones((nx,ny)).cumsum(1, dtype=np.int) - 1
>>>    xidx = np.random.rand(nx,ny).argsort(0).argsort(0)
>>>    return x[xidx, yidx]
>>
>> Hey, it is faster for nx=4, ny=100
>>
>> def baseshuffle(nx, ny):
>>    x = np.arange(nx)
>>    res = np.zeros((nx,ny))
>>    for sim in range(ny):
>>        res[:,sim] = np.random.permutation(x)
>>    return res
>>
>>>> timeit baseshuffle(4,100)
>> 1000 loops, best of 3: 1.11 ms per loop
>>>> timeit weirdshuffle(4,100)
>> 10000 loops, best of 3: 127 ?s per loop
>>
>> OK, who can cut that time in half? My first try looks clunky.
>
> This is a little faster:
>
> def weirdshuffle2(nx, ny):
>    one = np.ones((nx,ny), dtype=np.int)
>    x = one.cumsum(0)
>    x -= 1
>    yidx = one.cumsum(1)
>    yidx -= 1
>    xidx = np.random.random_sample((nx,ny)).argsort(0).argsort(0)
>    return x[xidx, yidx]
>
>>> timeit weirdshuffle(4,100)
> 10000 loops, best of 3: 129 ?s per loop
>>> timeit weirdshuffle2(4,100)
> 10000 loops, best of 3: 106 ?s per loop

Sorry for all the mail.

def weirdshuffle3(nx, ny):
    return np.random.random_sample((nx,ny)).argsort(0).argsort(0)

>> timeit weirdshuffle(4,100)
10000 loops, best of 3: 128 ?s per loop
>> timeit weirdshuffle3(4,100)
10000 loops, best of 3: 37.5 ?s per loop
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From josef.pktd at gmail.com  Tue Feb 10 20:31:52 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 10 Feb 2009 20:31:52 -0500
Subject: [Numpy-discussion] Permutations in Simulations`
In-Reply-To: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0C2AE@redmx1.esri.com>
References: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0BD6A@redmx1.esri.com>
	<f4f93d420902101218p3bfc8a8fl280a87bd4f6e7f9d@mail.gmail.com>
	<f4f93d420902101228i62660221y3e31dc717f1674ee@mail.gmail.com>
	<f4f93d420902101241y6f2ec949kd15991bfb466f1d5@mail.gmail.com>
	<f4f93d420902101258h50f109fgc435fae1a70f1bc5@mail.gmail.com>
	<6DF3F8F869B22C4393D67CA19A35AA0E0195B0C2AE@redmx1.esri.com>
Message-ID: <1cd32cbb0902101731h21652b91o987728d521113a2d@mail.gmail.com>

very nice. What's the purpose of the second  `.argsort(0)` ? Doesn't
it also work without it, or am I missing something in how this works?>

Josef

On 2/10/09, Mark Janikas <mjanikas at esri.com> wrote:
> Thanks to all for your replies.  I want this to work on any vector so I was
> thinking this...?
>
> import numpy as np
> import timeit
> x = np.array([4.,5.,10.,3.,5.,6.,7.,2.,9.,1.])
> nx = 10
> ny = 100
>
> def weirdshuffle4(x, ny):
>     nx = len(x)
>     indices = np.random.random_sample((nx,ny)).argsort(0).argsort(0)
>     return x[indices]
>
> t=timeit.Timer("weirdshuffle4(x,ny)", "from __main__ import *")
> print t.timeit(100)
>
> 0.0148663153873
>
>
> -----Original Message-----
> From: numpy-discussion-bounces at scipy.org
> [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Keith Goodman
> Sent: Tuesday, February 10, 2009 12:59 PM
> To: Discussion of Numerical Python
> Subject: Re: [Numpy-discussion] Permutations in Simulations`
>
> On Tue, Feb 10, 2009 at 12:41 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>> On Tue, Feb 10, 2009 at 12:28 PM, Keith Goodman <kwgoodman at gmail.com>
>> wrote:
>>> On Tue, Feb 10, 2009 at 12:18 PM, Keith Goodman <kwgoodman at gmail.com>
>>> wrote:
>>>> On Tue, Feb 10, 2009 at 11:29 AM, Mark Janikas <mjanikas at esri.com>
>>>> wrote:
>>>>> I want to create an array that contains a column of permutations for
>>>>> each
>>>>> simulation:
>>>>>
>>>>> import numpy as NUM
>>>>>
>>>>> import numpy.random as RAND
>>>>>
>>>>> x = NUM.arange(4.)
>>>>>
>>>>> res = NUM.zeros((4,100))
>>>>>
>>>>>
>>>>> for sim in range(100):
>>>>>
>>>>> res[:,sim] = RAND.permutation(x)
>>>>>
>>>>>
>>>>> Is there a way to do this without a loop?  Thanks so much ahead of
>>>>> time.
>>>>
>>>> Does this work? Might not be faster but it does avoid the loop.
>>>>
>>>> import numpy as np
>>>>
>>>> def weirdshuffle(nx, ny):
>>>>    x = np.ones((nx,ny)).cumsum(0, dtype=np.int) - 1
>>>>    yidx = np.ones((nx,ny)).cumsum(1, dtype=np.int) - 1
>>>>    xidx = np.random.rand(nx,ny).argsort(0).argsort(0)
>>>>    return x[xidx, yidx]
>>>
>>> Hey, it is faster for nx=4, ny=100
>>>
>>> def baseshuffle(nx, ny):
>>>    x = np.arange(nx)
>>>    res = np.zeros((nx,ny))
>>>    for sim in range(ny):
>>>        res[:,sim] = np.random.permutation(x)
>>>    return res
>>>
>>>>> timeit baseshuffle(4,100)
>>> 1000 loops, best of 3: 1.11 ms per loop
>>>>> timeit weirdshuffle(4,100)
>>> 10000 loops, best of 3: 127 ?s per loop
>>>
>>> OK, who can cut that time in half? My first try looks clunky.
>>
>> This is a little faster:
>>
>> def weirdshuffle2(nx, ny):
>>    one = np.ones((nx,ny), dtype=np.int)
>>    x = one.cumsum(0)
>>    x -= 1
>>    yidx = one.cumsum(1)
>>    yidx -= 1
>>    xidx = np.random.random_sample((nx,ny)).argsort(0).argsort(0)
>>    return x[xidx, yidx]
>>
>>>> timeit weirdshuffle(4,100)
>> 10000 loops, best of 3: 129 ?s per loop
>>>> timeit weirdshuffle2(4,100)
>> 10000 loops, best of 3: 106 ?s per loop
>
> Sorry for all the mail.
>
> def weirdshuffle3(nx, ny):
>     return np.random.random_sample((nx,ny)).argsort(0).argsort(0)
>
>>> timeit weirdshuffle(4,100)
> 10000 loops, best of 3: 128 ?s per loop
>>> timeit weirdshuffle3(4,100)
> 10000 loops, best of 3: 37.5 ?s per loop
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From kwgoodman at gmail.com  Tue Feb 10 21:06:58 2009
From: kwgoodman at gmail.com (Keith Goodman)
Date: Tue, 10 Feb 2009 18:06:58 -0800
Subject: [Numpy-discussion] Permutations in Simulations`
In-Reply-To: <1cd32cbb0902101731h21652b91o987728d521113a2d@mail.gmail.com>
References: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0BD6A@redmx1.esri.com>
	<f4f93d420902101218p3bfc8a8fl280a87bd4f6e7f9d@mail.gmail.com>
	<f4f93d420902101228i62660221y3e31dc717f1674ee@mail.gmail.com>
	<f4f93d420902101241y6f2ec949kd15991bfb466f1d5@mail.gmail.com>
	<f4f93d420902101258h50f109fgc435fae1a70f1bc5@mail.gmail.com>
	<6DF3F8F869B22C4393D67CA19A35AA0E0195B0C2AE@redmx1.esri.com>
	<1cd32cbb0902101731h21652b91o987728d521113a2d@mail.gmail.com>
Message-ID: <f4f93d420902101806h7b8d76ecq65ab60a1ef3c77af@mail.gmail.com>

Yeah, good point. The second argsort isn't needed. That should speed things up.

The double argsort ranks the values in the array. But we don't need that here.

On Tue, Feb 10, 2009 at 5:31 PM,  <josef.pktd at gmail.com> wrote:
> very nice. What's the purpose of the second  `.argsort(0)` ? Doesn't
> it also work without it, or am I missing something in how this works?>
>
> Josef
>
> On 2/10/09, Mark Janikas <mjanikas at esri.com> wrote:
>> Thanks to all for your replies.  I want this to work on any vector so I was
>> thinking this...?
>>
>> import numpy as np
>> import timeit
>> x = np.array([4.,5.,10.,3.,5.,6.,7.,2.,9.,1.])
>> nx = 10
>> ny = 100
>>
>> def weirdshuffle4(x, ny):
>>     nx = len(x)
>>     indices = np.random.random_sample((nx,ny)).argsort(0).argsort(0)
>>     return x[indices]
>>
>> t=timeit.Timer("weirdshuffle4(x,ny)", "from __main__ import *")
>> print t.timeit(100)
>>
>> 0.0148663153873
>>
>>
>> -----Original Message-----
>> From: numpy-discussion-bounces at scipy.org
>> [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Keith Goodman
>> Sent: Tuesday, February 10, 2009 12:59 PM
>> To: Discussion of Numerical Python
>> Subject: Re: [Numpy-discussion] Permutations in Simulations`
>>
>> On Tue, Feb 10, 2009 at 12:41 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>>> On Tue, Feb 10, 2009 at 12:28 PM, Keith Goodman <kwgoodman at gmail.com>
>>> wrote:
>>>> On Tue, Feb 10, 2009 at 12:18 PM, Keith Goodman <kwgoodman at gmail.com>
>>>> wrote:
>>>>> On Tue, Feb 10, 2009 at 11:29 AM, Mark Janikas <mjanikas at esri.com>
>>>>> wrote:
>>>>>> I want to create an array that contains a column of permutations for
>>>>>> each
>>>>>> simulation:
>>>>>>
>>>>>> import numpy as NUM
>>>>>>
>>>>>> import numpy.random as RAND
>>>>>>
>>>>>> x = NUM.arange(4.)
>>>>>>
>>>>>> res = NUM.zeros((4,100))
>>>>>>
>>>>>>
>>>>>> for sim in range(100):
>>>>>>
>>>>>> res[:,sim] = RAND.permutation(x)
>>>>>>
>>>>>>
>>>>>> Is there a way to do this without a loop?  Thanks so much ahead of
>>>>>> time.
>>>>>
>>>>> Does this work? Might not be faster but it does avoid the loop.
>>>>>
>>>>> import numpy as np
>>>>>
>>>>> def weirdshuffle(nx, ny):
>>>>>    x = np.ones((nx,ny)).cumsum(0, dtype=np.int) - 1
>>>>>    yidx = np.ones((nx,ny)).cumsum(1, dtype=np.int) - 1
>>>>>    xidx = np.random.rand(nx,ny).argsort(0).argsort(0)
>>>>>    return x[xidx, yidx]
>>>>
>>>> Hey, it is faster for nx=4, ny=100
>>>>
>>>> def baseshuffle(nx, ny):
>>>>    x = np.arange(nx)
>>>>    res = np.zeros((nx,ny))
>>>>    for sim in range(ny):
>>>>        res[:,sim] = np.random.permutation(x)
>>>>    return res
>>>>
>>>>>> timeit baseshuffle(4,100)
>>>> 1000 loops, best of 3: 1.11 ms per loop
>>>>>> timeit weirdshuffle(4,100)
>>>> 10000 loops, best of 3: 127 ?s per loop
>>>>
>>>> OK, who can cut that time in half? My first try looks clunky.
>>>
>>> This is a little faster:
>>>
>>> def weirdshuffle2(nx, ny):
>>>    one = np.ones((nx,ny), dtype=np.int)
>>>    x = one.cumsum(0)
>>>    x -= 1
>>>    yidx = one.cumsum(1)
>>>    yidx -= 1
>>>    xidx = np.random.random_sample((nx,ny)).argsort(0).argsort(0)
>>>    return x[xidx, yidx]
>>>
>>>>> timeit weirdshuffle(4,100)
>>> 10000 loops, best of 3: 129 ?s per loop
>>>>> timeit weirdshuffle2(4,100)
>>> 10000 loops, best of 3: 106 ?s per loop
>>
>> Sorry for all the mail.
>>
>> def weirdshuffle3(nx, ny):
>>     return np.random.random_sample((nx,ny)).argsort(0).argsort(0)
>>
>>>> timeit weirdshuffle(4,100)
>> 10000 loops, best of 3: 128 ?s per loop
>>>> timeit weirdshuffle3(4,100)
>> 10000 loops, best of 3: 37.5 ?s per loop
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From mjanikas at esri.com  Tue Feb 10 21:21:49 2009
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 10 Feb 2009 18:21:49 -0800
Subject: [Numpy-discussion] Permutations in Simulations`
In-Reply-To: <f4f93d420902101806h7b8d76ecq65ab60a1ef3c77af@mail.gmail.com>
References: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0BD6A@redmx1.esri.com>
	<f4f93d420902101218p3bfc8a8fl280a87bd4f6e7f9d@mail.gmail.com>
	<f4f93d420902101228i62660221y3e31dc717f1674ee@mail.gmail.com>
	<f4f93d420902101241y6f2ec949kd15991bfb466f1d5@mail.gmail.com>
	<f4f93d420902101258h50f109fgc435fae1a70f1bc5@mail.gmail.com>
	<6DF3F8F869B22C4393D67CA19A35AA0E0195B0C2AE@redmx1.esri.com>
	<1cd32cbb0902101731h21652b91o987728d521113a2d@mail.gmail.com>
	<f4f93d420902101806h7b8d76ecq65ab60a1ef3c77af@mail.gmail.com>
Message-ID: <6DF3F8F869B22C4393D67CA19A35AA0E0195B0C3BD@redmx1.esri.com>

You are correct!  Thanks to all!

MJ

-----Original Message-----
From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Keith Goodman
Sent: Tuesday, February 10, 2009 6:07 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Permutations in Simulations`

Yeah, good point. The second argsort isn't needed. That should speed things up.

The double argsort ranks the values in the array. But we don't need that here.

On Tue, Feb 10, 2009 at 5:31 PM,  <josef.pktd at gmail.com> wrote:
> very nice. What's the purpose of the second  `.argsort(0)` ? Doesn't
> it also work without it, or am I missing something in how this works?>
>
> Josef
>
> On 2/10/09, Mark Janikas <mjanikas at esri.com> wrote:
>> Thanks to all for your replies.  I want this to work on any vector so I was
>> thinking this...?
>>
>> import numpy as np
>> import timeit
>> x = np.array([4.,5.,10.,3.,5.,6.,7.,2.,9.,1.])
>> nx = 10
>> ny = 100
>>
>> def weirdshuffle4(x, ny):
>>     nx = len(x)
>>     indices = np.random.random_sample((nx,ny)).argsort(0).argsort(0)
>>     return x[indices]
>>
>> t=timeit.Timer("weirdshuffle4(x,ny)", "from __main__ import *")
>> print t.timeit(100)
>>
>> 0.0148663153873
>>
>>
>> -----Original Message-----
>> From: numpy-discussion-bounces at scipy.org
>> [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Keith Goodman
>> Sent: Tuesday, February 10, 2009 12:59 PM
>> To: Discussion of Numerical Python
>> Subject: Re: [Numpy-discussion] Permutations in Simulations`
>>
>> On Tue, Feb 10, 2009 at 12:41 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>>> On Tue, Feb 10, 2009 at 12:28 PM, Keith Goodman <kwgoodman at gmail.com>
>>> wrote:
>>>> On Tue, Feb 10, 2009 at 12:18 PM, Keith Goodman <kwgoodman at gmail.com>
>>>> wrote:
>>>>> On Tue, Feb 10, 2009 at 11:29 AM, Mark Janikas <mjanikas at esri.com>
>>>>> wrote:
>>>>>> I want to create an array that contains a column of permutations for
>>>>>> each
>>>>>> simulation:
>>>>>>
>>>>>> import numpy as NUM
>>>>>>
>>>>>> import numpy.random as RAND
>>>>>>
>>>>>> x = NUM.arange(4.)
>>>>>>
>>>>>> res = NUM.zeros((4,100))
>>>>>>
>>>>>>
>>>>>> for sim in range(100):
>>>>>>
>>>>>> res[:,sim] = RAND.permutation(x)
>>>>>>
>>>>>>
>>>>>> Is there a way to do this without a loop?  Thanks so much ahead of
>>>>>> time.
>>>>>
>>>>> Does this work? Might not be faster but it does avoid the loop.
>>>>>
>>>>> import numpy as np
>>>>>
>>>>> def weirdshuffle(nx, ny):
>>>>>    x = np.ones((nx,ny)).cumsum(0, dtype=np.int) - 1
>>>>>    yidx = np.ones((nx,ny)).cumsum(1, dtype=np.int) - 1
>>>>>    xidx = np.random.rand(nx,ny).argsort(0).argsort(0)
>>>>>    return x[xidx, yidx]
>>>>
>>>> Hey, it is faster for nx=4, ny=100
>>>>
>>>> def baseshuffle(nx, ny):
>>>>    x = np.arange(nx)
>>>>    res = np.zeros((nx,ny))
>>>>    for sim in range(ny):
>>>>        res[:,sim] = np.random.permutation(x)
>>>>    return res
>>>>
>>>>>> timeit baseshuffle(4,100)
>>>> 1000 loops, best of 3: 1.11 ms per loop
>>>>>> timeit weirdshuffle(4,100)
>>>> 10000 loops, best of 3: 127 ?s per loop
>>>>
>>>> OK, who can cut that time in half? My first try looks clunky.
>>>
>>> This is a little faster:
>>>
>>> def weirdshuffle2(nx, ny):
>>>    one = np.ones((nx,ny), dtype=np.int)
>>>    x = one.cumsum(0)
>>>    x -= 1
>>>    yidx = one.cumsum(1)
>>>    yidx -= 1
>>>    xidx = np.random.random_sample((nx,ny)).argsort(0).argsort(0)
>>>    return x[xidx, yidx]
>>>
>>>>> timeit weirdshuffle(4,100)
>>> 10000 loops, best of 3: 129 ?s per loop
>>>>> timeit weirdshuffle2(4,100)
>>> 10000 loops, best of 3: 106 ?s per loop
>>
>> Sorry for all the mail.
>>
>> def weirdshuffle3(nx, ny):
>>     return np.random.random_sample((nx,ny)).argsort(0).argsort(0)
>>
>>>> timeit weirdshuffle(4,100)
>> 10000 loops, best of 3: 128 ?s per loop
>>>> timeit weirdshuffle3(4,100)
>> 10000 loops, best of 3: 37.5 ?s per loop
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From python6009 at gmail.com  Wed Feb 11 00:40:35 2009
From: python6009 at gmail.com (A B)
Date: Tue, 10 Feb 2009 21:40:35 -0800
Subject: [Numpy-discussion] loadtxt issues
Message-ID: <7cc4bc500902102140h6cc85bb3wa28c89a147ddce73@mail.gmail.com>

Hi,

How do I write a loadtxt command to read in the following file and
store each data point as the appropriate data type:

12|h|34.5|44.5
14552|bbb|34.5|42.5

Do the strings have to be read in separately from the numbers?

Why would anyone use 'S10' instead of 'string'?

dt = {'names': ('gender','age','weight','bal'), 'formats': ('i4',
'S4','f4', 'f4')}

a = loadtxt("sample_data.txt", dtype=dt)

gives

ValueError: need more than 1 value to unpack

I can do a = loadtxt("sample_data.txt", dtype="string") but can't use
'string' instead of S4 and all my data is read into strings.

Seems like all the examples on-line use either numeric or textual
input, but not both.

Thanks.


From bpederse at gmail.com  Wed Feb 11 00:52:41 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Tue, 10 Feb 2009 21:52:41 -0800
Subject: [Numpy-discussion] loadtxt issues
In-Reply-To: <7cc4bc500902102140h6cc85bb3wa28c89a147ddce73@mail.gmail.com>
References: <7cc4bc500902102140h6cc85bb3wa28c89a147ddce73@mail.gmail.com>
Message-ID: <e183a99d0902102152p2bb71821of56dc407216095c5@mail.gmail.com>

On Tue, Feb 10, 2009 at 9:40 PM, A B <python6009 at gmail.com> wrote:
> Hi,
>
> How do I write a loadtxt command to read in the following file and
> store each data point as the appropriate data type:
>
> 12|h|34.5|44.5
> 14552|bbb|34.5|42.5
>
> Do the strings have to be read in separately from the numbers?
>
> Why would anyone use 'S10' instead of 'string'?
>
> dt = {'names': ('gender','age','weight','bal'), 'formats': ('i4',
> 'S4','f4', 'f4')}
>
> a = loadtxt("sample_data.txt", dtype=dt)
>
> gives
>
> ValueError: need more than 1 value to unpack
>
> I can do a = loadtxt("sample_data.txt", dtype="string") but can't use
> 'string' instead of S4 and all my data is read into strings.
>
> Seems like all the examples on-line use either numeric or textual
> input, but not both.
>
> Thanks.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

works for me but not sure i understand the problem, did you try
setting the delimiter?


import numpy as np
from cStringIO import StringIO

txt = StringIO("""\
12|h|34.5|44.5
14552|bbb|34.5|42.5""")

dt = {'names': ('gender','age','weight','bal'), 'formats': ('i4',
'S4','f4', 'f4')}
a = np.loadtxt(txt, dtype=dt, delimiter="|")
print a.dtype


From millman at berkeley.edu  Wed Feb 11 03:25:37 2009
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 11 Feb 2009 00:25:37 -0800
Subject: [Numpy-discussion] ANN: SciPy 0.7.0
Message-ID: <c7009a550902110025h1a3331bfu12719e63815f2d59@mail.gmail.com>

I'm pleased to announce SciPy 0.7.0.  SciPy is a package of tools for
science and engineering for Python.  It includes modules for
statistics, optimization, integration, linear  algebra, Fourier
transforms, signal and image processing, ODE solvers,
and more.

This release comes sixteen months after the 0.6.0 release and contains
many new features, numerous bug-fixes, improved test coverage, and
better documentation.  Please note that SciPy 0.7.0 requires Python
2.4 or greater (but not Python 3) and NumPy 1.2.0 or greater.

For information, please see the release notes:
https://sourceforge.net/project/shownotes.php?release_id=660191&group_id=27747

You can download the release from here:
https://sourceforge.net/project/showfiles.php?group_id=27747&package_id=19531&release_id=660191

Thank you to everybody who contributed to this release.

Enjoy,

Jarrod Millman


From paul at rudin.co.uk  Wed Feb 11 03:25:40 2009
From: paul at rudin.co.uk (Paul Rudin)
Date: Wed, 11 Feb 2009 08:25:40 +0000
Subject: [Numpy-discussion] numpy-izing a loop
References: <87y6we1lbw.fsf@rudin.co.uk>
	<9457e7c80902100819t139d6342y8885f417a4ec4443@mail.gmail.com>
	<3d375d730902101148t10d50084k1a775e681a5e3d05@mail.gmail.com>
	<9457e7c80902101327q5cd7cd4ax7e2c1713fcee2cd4@mail.gmail.com>
	<9457e7c80902101333l5883971i2b15c6290315bf8d@mail.gmail.com>
Message-ID: <87skmltm63.fsf@rudin.co.uk>

St?fan van der Walt <stefan at sun.ac.za> writes:

> 2009/2/10 St?fan van der Walt <stefan at sun.ac.za>:
>> x = np.arange(dim)
>> y = np.arange(dim)[:, None]
>> z = np.arange(dim)[:, None, None]
>
> Do not operate heavy machinery or attempt broadcasting while tired or
> under the influence.  That order was incorrect:
>
>> z = np.arange(dim)
>> y = np.arange(dim)[:, None]
>> x = np.arange(dim)[:, None, None]

Thanks to you both. I can confirm that the two functions below give the
same result - as far as my comprehensive testing of one example shows :)

def compute_voxels(depth_buffers):
    assert len(depth_buffers) == 6
    dim = depth_buffers[0].shape[0]
    znear, zfar, ynear, yfar, xnear, xfar = depth_buffers
    result = numpy.empty((dim, dim, dim), numpy.bool)
    for x in xrange(dim):
        for y in xrange(dim):
            for z in xrange(dim):
                result[x, y, z] = ((xnear[y, z] < xfar[y, z]) and 
                                   (ynear[x, z] < yfar[x, z]) and
                                   (znear[x, y] < zfar[x, y]))
    return result

def compute_voxels2(depth_buffers):
    dim = depth_buffers[0].shape[0]
    znear, zfar, ynear, yfar, xnear, xfar = depth_buffers
    z = numpy.arange(dim)
    y = numpy.arange(dim)[:, None]
    x = numpy.arange(dim)[:, None, None]

    return  ((xnear[y,z] < xfar[y, z]) &
             (ynear[x, z] < yfar[x, z]) &
             (znear[x,y] < zfar[x,y]))


All that remains is for me to verify that it's actually the result I want :)

(Well - I also need to learn to spot these things for myself, but I
guess the intuition comes with practice.)



From lfriedri at imtek.de  Wed Feb 11 03:31:57 2009
From: lfriedri at imtek.de (Lars Friedrich)
Date: Wed, 11 Feb 2009 09:31:57 +0100
Subject: [Numpy-discussion] PEP: named axis
Message-ID: <49928CFD.9020000@imtek.de>

Hello list,

I am not sure, if I understood everything of the discussion on the 
named-axis-idea of numpy-arrays, since I am only a *user* of numpy. I 
never subclassed the numpy-array-class ;-)

However, I have the need to store meta-information for my arrays. I do 
this with a stand-alone class with the name 'Wave' that stores its data 
in a n-dimensional numpy-array as a member. The meta-information I store 
(using dicts and lists) is

* coordinateLabel per axis
* x0 per axis
* dx per axis

This concept is taken from the data structures in the commercial 
software IGOR, that are also called 'Waves'. An example would be an 
image I took with a microscope. The data would be 2d, say shape = (640, 
480) holding the intesity information per pixel. x0 could then be 
[-1e-6, -2e-6] and dx [100e-9, 100e-9] meaning that the image's pixel 
index [0,0] corresponds to a position of -1micrometer/-2micrometer and 
the pixels have a spacing of 100nanometers. coordinateLabels would be 
['x(m)', 'y(m)'].

If I have a movie, the data would be 3d with x0 = [-1e-6, -2e-6, 0], dx 
= [100e-9, 100e-9, 100e-3] and coordinateLabels = ['x(m)', 'y(m)', 
't(s)'] for a frame rate of 10 fps.

What I would like to say with this is the following (as a user...) :

* Meta-information is often necessary
* A string-label per axis is often not enough. Scaling is also important
* I like the idea of a most-basic-as-possible numpy-array. In my 
opinion, the meta-data-management should be done by another (sub-?) 
class. This way, numpy-arrays are simple enough for new users (as I was 
roughly two years ago...).

I would be very interested in a class that *uses* numpy-arrays to 
provide a datastructure for physical data with coordinate labels and 
scaling.

Regards,
Lars Friedrich


-- 
Dipl.-Ing. Lars Friedrich

Bio- and Nano-Photonics
Department of Microsystems Engineering -- IMTEK
University of Freiburg
Georges-K?hler-Allee 102
D-79110 Freiburg
Germany

phone: +49-761-203-7531
fax:   +49-761-203-7537
room:  01 088
email: lfriedri at imtek.de


From cournape at gmail.com  Wed Feb 11 07:46:05 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 11 Feb 2009 21:46:05 +0900
Subject: [Numpy-discussion] Numpy 1.2.1 and Scipy 0.7.0; Ubuntu packages
Message-ID: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>

Hi,

I started to set up a PPA for scipy on launchpad, which enables to
build ubuntu packages for various distributions/architectures. The
link is there:

https://edge.launchpad.net/~scipy/+archive/ppa

So you just need to add one line to your /etc/apt/sources.list, and
you will get uptodate numpy and scipy packages,

cheers,

David


From cournape at gmail.com  Wed Feb 11 07:48:55 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 11 Feb 2009 21:48:55 +0900
Subject: [Numpy-discussion] Numpy 1.2.1 and Scipy 0.7.0; Ubuntu packages
In-Reply-To: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>
References: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>
Message-ID: <5b8d13220902110448t78d20138j542d57d747663a96@mail.gmail.com>

On Wed, Feb 11, 2009 at 9:46 PM, David Cournapeau <cournape at gmail.com> wrote:
> Hi,
>
> I started to set up a PPA for scipy on launchpad, which enables to
> build ubuntu packages for various distributions/architectures. The
> link is there:
>
> https://edge.launchpad.net/~scipy/+archive/ppa
>
> So you just need to add one line to your /etc/apt/sources.list, and
> you will get uptodate numpy and scipy packages,

I forgot to mention that those packages closely follow the official
packages: once a distribution update their package, it will
automatically supercede the one above. IOW, it can be seen as a
preview of the packages to come in Ubuntu/Debian,

David


From faltet at pytables.org  Wed Feb 11 08:20:49 2009
From: faltet at pytables.org (Francesc Alted)
Date: Wed, 11 Feb 2009 14:20:49 +0100
Subject: [Numpy-discussion] ANN: Numexpr 1.2 released
Message-ID: <200902111420.49579.faltet@pytables.org>

========================
 Announcing Numexpr 1.2
========================

Numexpr is a fast numerical expression evaluator for NumPy.  With it,
expressions that operate on arrays (like "3*a+4*b") are accelerated
and use less memory than doing the same calculation in Python.

The main feature added in this version is the support of the Intel VML 
library (many thanks to Gregor Thalhammer for his nice work on this!).  
In addition, when the VML support is on, several processors can be used 
in parallel (see the new `set_vml_num_threads()` function).

When the VML support is on, the computation of transcendental functions 
(like trigonometrical, exponential, logarithmic, hyperbolic, power...) 
can be accelerated quite a few.  Typical speed-ups when using one 
single core for contiguous arrays are around 3x, with peaks of 7.5x 
(for the pow() function).  When using 2 cores the speed-ups are around 
4x and 14x respectively.

In case you want to know more in detail what has changed in this
version, have a look at the release notes:

http://code.google.com/p/numexpr/wiki/ReleaseNotes


Where I can find Numexpr?
=========================

The project is hosted at Google code in:

http://code.google.com/p/numexpr/

And you can get the packages from PyPI as well:

http://pypi.python.org/pypi


How it works?
=============

See:

http://code.google.com/p/numexpr/wiki/Overview

for a detailed description of the package.


Share your experience
=====================

Let us know of any bugs, suggestions, gripes, kudos, etc. you may
have.


Enjoy!

-- 
Francesc Alted


From wesmckinn at gmail.com  Wed Feb 11 10:10:44 2009
From: wesmckinn at gmail.com (Wes McKinney)
Date: Wed, 11 Feb 2009 10:10:44 -0500
Subject: [Numpy-discussion] PyArray_SETITEM with object arrays in Cython
Message-ID: <6c476c8a0902110710q4c9b16e4vcc3e315f64a84caf@mail.gmail.com>

Hello,

I am writing some Cython code and have noted that the buffer interface
offers very little speedup for PyObject arrays. In trying to rewrite the
same code using the C API in Cython, I find I can't get PyArray_SETITEM to
work, in a call like:

PyArray_SETITEM(result, <void *> iterresult.dataptr, obj)

where result is an ndarray of dtype object, and obj is a PyObject*.

Anyone have some experience with this can offer pointers (no pun intended!)?


Thanks,
Wes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090211/373ff15d/attachment.html>

From hanni.ali at gmail.com  Wed Feb 11 12:24:01 2009
From: hanni.ali at gmail.com (Hanni Ali)
Date: Wed, 11 Feb 2009 17:24:01 +0000
Subject: [Numpy-discussion] PyArray_SETITEM with object arrays in Cython
In-Reply-To: <6c476c8a0902110710q4c9b16e4vcc3e315f64a84caf@mail.gmail.com>
References: <6c476c8a0902110710q4c9b16e4vcc3e315f64a84caf@mail.gmail.com>
Message-ID: <789d27b10902110924j52d47552p2fb38c83e05ebc4d@mail.gmail.com>

Hi Wes,

I do not profess to be an expert, but I have been off loading a fair number
of loops to C from Python code and achieved significant improvements most
have been of the following form (which I have found to be the fastest):

size = *incomingArrayObj->dimensions;

r_dptr = PyArray_DATA(resultArray);

while(size--) {
    r_dptr = result;
    r_dptr++;
}

Where for multidimensional arrays r_dptr could be incremented by the number
of dims rather than just ++:

dims = PyArray_DIM(incomingArrayObj,1);

i have not however actually used PyArray_SETITEM so cannot comment on the
issue you are having.

Hanni



2009/2/11 Wes McKinney <wesmckinn at gmail.com>

> Hello,
>
> I am writing some Cython code and have noted that the buffer interface
> offers very little speedup for PyObject arrays. In trying to rewrite the
> same code using the C API in Cython, I find I can't get PyArray_SETITEM to
> work, in a call like:
>
> PyArray_SETITEM(result, <void *> iterresult.dataptr, obj)
>
> where result is an ndarray of dtype object, and obj is a PyObject*.
>
> Anyone have some experience with this can offer pointers (no pun
> intended!)?
>
> Thanks,
> Wes
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090211/5eb53fa0/attachment.html>

From dagss at student.matnat.uio.no  Wed Feb 11 15:25:09 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 11 Feb 2009 21:25:09 +0100 (CET)
Subject: [Numpy-discussion] PyArray_SETITEM with object arrays in Cython
In-Reply-To: <6c476c8a0902110710q4c9b16e4vcc3e315f64a84caf@mail.gmail.com>
References: <6c476c8a0902110710q4c9b16e4vcc3e315f64a84caf@mail.gmail.com>
Message-ID: <76550d7a4d21a5fbd6764825cc547ae1.squirrel@webmail.uio.no>

Wes McKinney wrote:
> I am writing some Cython code and have noted that the buffer interface
> offers very little speedup for PyObject arrays. In trying to rewrite the
> same code using the C API in Cython, I find I can't get PyArray_SETITEM to
> work, in a call like:
>
> PyArray_SETITEM(result, <void *> iterresult.dataptr, obj)
>
> where result is an ndarray of dtype object, and obj is a PyObject*.

Interesting. Whatever you end up doing, I'll make sure to integrate
whatever works faster into Cython.

I do doubt your results a bit though -- the buffer interface in Cython
increfs/decrefs the objects, but otherwise it should be completely raw
access, so using SETITEM shouldn't be faster except one INCREF/DECREF per
object (i.e. still way faster than using Python).

Could you perhaps post your Cython code?

Dag Sverre



From wesmckinn at gmail.com  Wed Feb 11 17:12:01 2009
From: wesmckinn at gmail.com (Wes McKinney)
Date: Wed, 11 Feb 2009 17:12:01 -0500
Subject: [Numpy-discussion] PyArray_SETITEM with object arrays in Cython
In-Reply-To: <76550d7a4d21a5fbd6764825cc547ae1.squirrel@webmail.uio.no>
References: <6c476c8a0902110710q4c9b16e4vcc3e315f64a84caf@mail.gmail.com>
	<76550d7a4d21a5fbd6764825cc547ae1.squirrel@webmail.uio.no>
Message-ID: <6c476c8a0902111412y2cb6985x973ff830d1a1412@mail.gmail.com>

I actually got it to work-- the function prototype in the pxi file was
wrong, needed to be:

int PyArray_SETITEM(object obj, void* itemptr, object item)

This still doesn't explain why the buffer interface was slow.

The general problem here is an indexed array (by dates or strings, for
example), that you want to conform to a new index. The arrays most of the
time contain floats but occasionally PyObjects. For some reason the access
and assignment is slow (this function can be faster by a factor of 50 with C
API macros, so clearly something is awry)-- let me know if you see anything
obviously wrong with this

def reindexObject(ndarray[object, ndim=1] index,
                  ndarray[object, ndim=1] arr,
                  dict idxMap):
    '''
    Using the provided new index, a given array, and a mapping of
index-value
    correpondences in the value array, return a new ndarray conforming to
    the new index.
    '''
    cdef object idx, value

    cdef int length  = index.shape[0]
    cdef ndarray[object, ndim = 1] result = np.empty(length, dtype=object)

    cdef int i = 0
    for i from 0 <= i < length:
        idx = index[i]
        if not PyDict_Contains(idxMap, idx):
            result[i] = None
            continue
        value = arr[idxMap[idx]]
        result[i] = value
    return result

On Wed, Feb 11, 2009 at 3:25 PM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> Wes McKinney wrote:
> > I am writing some Cython code and have noted that the buffer interface
> > offers very little speedup for PyObject arrays. In trying to rewrite the
> > same code using the C API in Cython, I find I can't get PyArray_SETITEM
> to
> > work, in a call like:
> >
> > PyArray_SETITEM(result, <void *> iterresult.dataptr, obj)
> >
> > where result is an ndarray of dtype object, and obj is a PyObject*.
>
> Interesting. Whatever you end up doing, I'll make sure to integrate
> whatever works faster into Cython.
>
> I do doubt your results a bit though -- the buffer interface in Cython
> increfs/decrefs the objects, but otherwise it should be completely raw
> access, so using SETITEM shouldn't be faster except one INCREF/DECREF per
> object (i.e. still way faster than using Python).
>
> Could you perhaps post your Cython code?
>
> Dag Sverre
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090211/5e702555/attachment.html>

From a.h.jaffe at gmail.com  Wed Feb 11 17:21:30 2009
From: a.h.jaffe at gmail.com (Andrew Jaffe)
Date: Wed, 11 Feb 2009 22:21:30 +0000
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <3d375d730902061330r29c19ce7sb44d03e8b79616fa@mail.gmail.com>
References: <498B7181.5000300@enthought.com>	<20090205231618.GA21014@phare.normalesup.org>	<498BB9F2.5080501@enthought.com>	<3d375d730902052029l21f675d1leecb0d405db5dc23@mail.gmail.com>	<9457e7c80902060122o365cbc28mf5a0bc22d5bc6f6d@mail.gmail.com>
	<3d375d730902061330r29c19ce7sb44d03e8b79616fa@mail.gmail.com>
Message-ID: <49934F6A.6070309@gmail.com>

Robert Kern wrote:
> On Fri, Feb 6, 2009 at 03:22, St?fan van der Walt <stefan at sun.ac.za> wrote:
>> Hi Robert
>>
>> 2009/2/6 Robert Kern <robert.kern at gmail.com>:
>>>> This could be implemented but would require adding information to the
>>>> NumPy array.
>>> More than that, though. Every function and method that takes an axis
>>> or reduces an axis will need to be rewritten. For that reason, I'm -1
>>> on the proposal.
>> Are you -1 on the array dictionary, or on using it to do axis mapping?
> 
> I'm -1 on rewriting every axis= argument to accept strings. 

Maybe I misunderstand the proposal, but, actually, I think this is 
completely the wrong semantics for "axis="  anyway. "axis=" in numpy 
refers to what is also a dimension,  not a  column.

More generally, then, would we restrict this to labeling only the 
column  dimension, or could it be used for any dimension?





From pav at iki.fi  Wed Feb 11 17:26:27 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 11 Feb 2009 22:26:27 +0000 (UTC)
Subject: [Numpy-discussion] PEP: named axis
References: <498B7181.5000300@enthought.com>
	<20090205231618.GA21014@phare.normalesup.org>
	<498BB9F2.5080501@enthought.com>
	<3d375d730902052029l21f675d1leecb0d405db5dc23@mail.gmail.com>
	<9457e7c80902060122o365cbc28mf5a0bc22d5bc6f6d@mail.gmail.com>
	<3d375d730902061330r29c19ce7sb44d03e8b79616fa@mail.gmail.com>
	<49934F6A.6070309@gmail.com>
Message-ID: <gmvjaj$4t0$1@ger.gmane.org>

Wed, 11 Feb 2009 22:21:30 +0000, Andrew Jaffe wrote:
[clip]
> Maybe I misunderstand the proposal, but, actually, I think this is
> completely the wrong semantics for "axis="  anyway. "axis=" in numpy
> refers to what is also a dimension,  not a  column.

I think the proposal was to add the ability to refer to dimensions with 
names instead of numbers. This is separate from referring to entries in a 
dimension. (Addressing 'columns' by name is already provided by 
structured arrays.)

-- 
Pauli Virtanen



From a.h.jaffe at gmail.com  Wed Feb 11 17:37:42 2009
From: a.h.jaffe at gmail.com (Andrew Jaffe)
Date: Wed, 11 Feb 2009 22:37:42 +0000
Subject: [Numpy-discussion] PEP: named axis
In-Reply-To: <gmvjaj$4t0$1@ger.gmane.org>
References: <498B7181.5000300@enthought.com>	<20090205231618.GA21014@phare.normalesup.org>	<498BB9F2.5080501@enthought.com>	<3d375d730902052029l21f675d1leecb0d405db5dc23@mail.gmail.com>	<9457e7c80902060122o365cbc28mf5a0bc22d5bc6f6d@mail.gmail.com>	<3d375d730902061330r29c19ce7sb44d03e8b79616fa@mail.gmail.com>	<49934F6A.6070309@gmail.com>
	<gmvjaj$4t0$1@ger.gmane.org>
Message-ID: <49935336.3080308@gmail.com>

Pauli Virtanen wrote:
> Wed, 11 Feb 2009 22:21:30 +0000, Andrew Jaffe wrote:
> [clip]
>> Maybe I misunderstand the proposal, but, actually, I think this is
>> completely the wrong semantics for "axis="  anyway. "axis=" in numpy
>> refers to what is also a dimension,  not a  column.
> 
> I think the proposal was to add the ability to refer to dimensions with 
> names instead of numbers. This is separate from referring to entries in a 
> dimension. (Addressing 'columns' by name is already provided by 
> structured arrays.)
> 

My bad -- I completely misread the proposal!

Nevermind...

Andrew


From fperez.net at gmail.com  Wed Feb 11 18:11:10 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 11 Feb 2009 15:11:10 -0800
Subject: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0;
	Ubuntu packages
In-Reply-To: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>
References: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>
Message-ID: <db6b5ecc0902111511y7d9cb105x6283d628e6473568@mail.gmail.com>

On Wed, Feb 11, 2009 at 4:46 AM, David Cournapeau <cournape at gmail.com> wrote:
> Hi,
>
> I started to set up a PPA for scipy on launchpad, which enables to
> build ubuntu packages for various distributions/architectures. The
> link is there:
>
> https://edge.launchpad.net/~scipy/+archive/ppa

Cool, thanks.  Is it easy to provide also hardy packages, or does it
require a lot of work on your part?

Cheers,

f


From cournape at gmail.com  Wed Feb 11 21:17:22 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 12 Feb 2009 11:17:22 +0900
Subject: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0;
	Ubuntu packages
In-Reply-To: <db6b5ecc0902111511y7d9cb105x6283d628e6473568@mail.gmail.com>
References: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>
	<db6b5ecc0902111511y7d9cb105x6283d628e6473568@mail.gmail.com>
Message-ID: <5b8d13220902111817q9bd42c0r8c42f6307ca1b318@mail.gmail.com>

On Thu, Feb 12, 2009 at 8:11 AM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Wed, Feb 11, 2009 at 4:46 AM, David Cournapeau <cournape at gmail.com> wrote:
>> Hi,
>>
>> I started to set up a PPA for scipy on launchpad, which enables to
>> build ubuntu packages for various distributions/architectures. The
>> link is there:
>>
>> https://edge.launchpad.net/~scipy/+archive/ppa
>
> Cool, thanks.  Is it easy to provide also hardy packages, or does it
> require a lot of work on your part?

Unfortunately, it does require some work, because hardy uses g77
instead of gfortran, so the source package has to be different (once
hardy is done, all the one below would be easy, though). I am not sure
how to do that with PPA (the doc is not great).

cheers,

David


From python6009 at gmail.com  Wed Feb 11 21:27:13 2009
From: python6009 at gmail.com (A B)
Date: Wed, 11 Feb 2009 18:27:13 -0800
Subject: [Numpy-discussion] loadtxt issues
In-Reply-To: <e183a99d0902102152p2bb71821of56dc407216095c5@mail.gmail.com>
References: <7cc4bc500902102140h6cc85bb3wa28c89a147ddce73@mail.gmail.com>
	<e183a99d0902102152p2bb71821of56dc407216095c5@mail.gmail.com>
Message-ID: <7cc4bc500902111827p3942dc40ie1216c13000fe2e0@mail.gmail.com>

On Tue, Feb 10, 2009 at 9:52 PM, Brent Pedersen <bpederse at gmail.com> wrote:
> On Tue, Feb 10, 2009 at 9:40 PM, A B <python6009 at gmail.com> wrote:
>> Hi,
>>
>> How do I write a loadtxt command to read in the following file and
>> store each data point as the appropriate data type:
>>
>> 12|h|34.5|44.5
>> 14552|bbb|34.5|42.5
>>
>> Do the strings have to be read in separately from the numbers?
>>
>> Why would anyone use 'S10' instead of 'string'?
>>
>> dt = {'names': ('gender','age','weight','bal'), 'formats': ('i4',
>> 'S4','f4', 'f4')}
>>
>> a = loadtxt("sample_data.txt", dtype=dt)
>>
>> gives
>>
>> ValueError: need more than 1 value to unpack
>>
>> I can do a = loadtxt("sample_data.txt", dtype="string") but can't use
>> 'string' instead of S4 and all my data is read into strings.
>>
>> Seems like all the examples on-line use either numeric or textual
>> input, but not both.
>>
>> Thanks.
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>
> works for me but not sure i understand the problem, did you try
> setting the delimiter?
>
>
> import numpy as np
> from cStringIO import StringIO
>
> txt = StringIO("""\
> 12|h|34.5|44.5
> 14552|bbb|34.5|42.5""")
>
> dt = {'names': ('gender','age','weight','bal'), 'formats': ('i4',
> 'S4','f4', 'f4')}
> a = np.loadtxt(txt, dtype=dt, delimiter="|")
> print a.dtype

I had tried both with and without the delimiter. In any event, it just
worked for me as well. Not sure what I was missing before. Anyway,
thank you.


From rmay31 at gmail.com  Wed Feb 11 23:38:20 2009
From: rmay31 at gmail.com (Ryan May)
Date: Wed, 11 Feb 2009 22:38:20 -0600
Subject: [Numpy-discussion] genloadtxt: dtype=None and unpack=True
Message-ID: <c7b95a020902112038w4b0fc609n844c10c9931f7c95@mail.gmail.com>

Pierre,

I noticed that using dtype=None with a heterogeneous set of data, trying to
use unpack=True to get the columns into separate arrays (instead of a
structured array) doesn't work.  I've attached a patch that, in the case of
dtype=None, unpacks the fields in the final array into a list of separate
arrays.  Does this seem like a good idea to you?

Here's a test case:

from cStringIO import StringIO
s = '2,1950-02-27,35.55\n2,1951-02-19,35.27\n'
a,b,c = np.genfromtxt(StringIO(s), delimiter=',', unpack=True, missing=' ',
dtype=None)

Ryan

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090211/88c28915/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: genloadtxt_unpack_fields.diff
Type: application/octet-stream
Size: 481 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090211/88c28915/attachment.obj>

From pgmdevlist at gmail.com  Wed Feb 11 23:47:17 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 11 Feb 2009 23:47:17 -0500
Subject: [Numpy-discussion] genloadtxt: dtype=None and unpack=True
In-Reply-To: <c7b95a020902112038w4b0fc609n844c10c9931f7c95@mail.gmail.com>
References: <c7b95a020902112038w4b0fc609n844c10c9931f7c95@mail.gmail.com>
Message-ID: <97251D70-38B3-4A41-B2EA-26A700C50452@gmail.com>


On Feb 11, 2009, at 11:38 PM, Ryan May wrote:

> Pierre,
>
> I noticed that using dtype=None with a heterogeneous set of data,  
> trying to use unpack=True to get the columns into separate arrays  
> (instead of a structured array) doesn't work.  I've attached a patch  
> that, in the case of dtype=None, unpacks the fields in the final  
> array into a list of separate arrays.  Does this seem like a good  
> idea to you?

Nope, as it breaks consistency: depending on some input parameters,  
you either get an array or a list. I think it's better to leave it as  
it is, maybe adding an extra line in the doc precising that  
unpack=True doesn't do anything for structured arrays. 


From oliphant at enthought.com  Wed Feb 11 23:53:54 2009
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 11 Feb 2009 23:53:54 -0500
Subject: [Numpy-discussion] FYI: New "select-multiple-fields" behavior
Message-ID: <4993AB62.50505@enthought.com>


Hi all,

As of r6358, I checked in the functionality to allow selection by 
multiple fields along with a couple of tests.

ary['field1', 'field3']  raises an error
ary[['field1', 'field3']] is the correct spelling and returns a copy of 
the data in those fields in a new array.


-Travis



From python6009 at gmail.com  Thu Feb 12 00:04:13 2009
From: python6009 at gmail.com (A B)
Date: Wed, 11 Feb 2009 21:04:13 -0800
Subject: [Numpy-discussion] loadtxt issues
In-Reply-To: <7cc4bc500902111827p3942dc40ie1216c13000fe2e0@mail.gmail.com>
References: <7cc4bc500902102140h6cc85bb3wa28c89a147ddce73@mail.gmail.com>
	<e183a99d0902102152p2bb71821of56dc407216095c5@mail.gmail.com>
	<7cc4bc500902111827p3942dc40ie1216c13000fe2e0@mail.gmail.com>
Message-ID: <7cc4bc500902112104w22fb9978v8b6b3e8bf25166f3@mail.gmail.com>

On Wed, Feb 11, 2009 at 6:27 PM, A B <python6009 at gmail.com> wrote:
> On Tue, Feb 10, 2009 at 9:52 PM, Brent Pedersen <bpederse at gmail.com> wrote:
>> On Tue, Feb 10, 2009 at 9:40 PM, A B <python6009 at gmail.com> wrote:
>>> Hi,
>>>
>>> How do I write a loadtxt command to read in the following file and
>>> store each data point as the appropriate data type:
>>>
>>> 12|h|34.5|44.5
>>> 14552|bbb|34.5|42.5
>>>
>>> Do the strings have to be read in separately from the numbers?
>>>
>>> Why would anyone use 'S10' instead of 'string'?
>>>
>>> dt = {'names': ('gender','age','weight','bal'), 'formats': ('i4',
>>> 'S4','f4', 'f4')}
>>>
>>> a = loadtxt("sample_data.txt", dtype=dt)
>>>
>>> gives
>>>
>>> ValueError: need more than 1 value to unpack
>>>
>>> I can do a = loadtxt("sample_data.txt", dtype="string") but can't use
>>> 'string' instead of S4 and all my data is read into strings.
>>>
>>> Seems like all the examples on-line use either numeric or textual
>>> input, but not both.
>>>
>>> Thanks.
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Numpy-discussion at scipy.org
>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>> works for me but not sure i understand the problem, did you try
>> setting the delimiter?
>>
>>
>> import numpy as np
>> from cStringIO import StringIO
>>
>> txt = StringIO("""\
>> 12|h|34.5|44.5
>> 14552|bbb|34.5|42.5""")
>>
>> dt = {'names': ('gender','age','weight','bal'), 'formats': ('i4',
>> 'S4','f4', 'f4')}
>> a = np.loadtxt(txt, dtype=dt, delimiter="|")
>> print a.dtype
>
> I had tried both with and without the delimiter. In any event, it just
> worked for me as well. Not sure what I was missing before. Anyway,
> thank you.
>

Actually, I was using two different machines and it appears that the
version of numpy available on Ubuntu is seriously out of date (1.0.4).
Wonder why ...
Version 1.2.1 on a RedHat box worked fine.


From stefan at sun.ac.za  Thu Feb 12 00:14:38 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 12 Feb 2009 07:14:38 +0200
Subject: [Numpy-discussion] FYI: New "select-multiple-fields" behavior
In-Reply-To: <4993AB62.50505@enthought.com>
References: <4993AB62.50505@enthought.com>
Message-ID: <9457e7c80902112114m14c44f79ne9cdf2db062d5fae@mail.gmail.com>

Hi Travis

2009/2/12 Travis E. Oliphant <oliphant at enthought.com>:
> ary['field1', 'field3']  raises an error
> ary[['field1', 'field3']] is the correct spelling and returns a copy of
> the data in those fields in a new array.

Is there absolutely no way of returning the result as a view?

Regards
St?fan


From python6009 at gmail.com  Thu Feb 12 00:24:39 2009
From: python6009 at gmail.com (A B)
Date: Wed, 11 Feb 2009 21:24:39 -0800
Subject: [Numpy-discussion] Outer join ?
Message-ID: <7cc4bc500902112124s6a00d59fw78ea94f71ae4bb82@mail.gmail.com>

Hi,

I have the following data structure:

col1 | col2 | col3

20080101|key1|4
20080201|key1|6
20080301|key1|5
20080301|key2|3.4
20080601|key2|5.6

For each key in the second column, I would like to create an array
where for all unique values in the first column, there will be either
a value or zero if there is no data available. Like so:

# 20080101, 20080201, 20080301, 20080601

key1 - 4, 6, 5,    0
key2 - 0, 0, 3.4, 5.6

Ideally, the results would end up in a 2d array.

What's the most efficient way to accomplish this? Currently, I am
getting a list of uniq col1 and col2 values into separate variables,
then looping through each unique value in col2

a = loadtxt(...)

dates = unique(a[:]['col1'])
keys = unique(a[:]['col2'])

for key in keys:
    b = a[where(a[:]['col2'] == key)]
    ???

Thanks in advance.


From oliphant at enthought.com  Thu Feb 12 00:28:28 2009
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 12 Feb 2009 00:28:28 -0500
Subject: [Numpy-discussion] FYI: New "select-multiple-fields" behavior
In-Reply-To: <9457e7c80902112114m14c44f79ne9cdf2db062d5fae@mail.gmail.com>
References: <4993AB62.50505@enthought.com>
	<9457e7c80902112114m14c44f79ne9cdf2db062d5fae@mail.gmail.com>
Message-ID: <4993B37C.7080701@enthought.com>

St?fan van der Walt wrote:
> Hi Travis
>
> 2009/2/12 Travis E. Oliphant <oliphant at enthought.com>:
>   
>> ary['field1', 'field3']  raises an error
>> ary[['field1', 'field3']] is the correct spelling and returns a copy of
>> the data in those fields in a new array.
>>     
>
> Is there absolutely no way of returning the result as a view?
>   
Not that I can think of --- it does match advanced indexing semantics to 
have it be a copy.

-Travis

-- 

Travis Oliphant
Enthought, Inc.
(512) 536-1057 (office)
(512) 536-1059 (fax)
http://www.enthought.com
oliphant at enthought.com



From robert.kern at gmail.com  Thu Feb 12 00:40:59 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 11 Feb 2009 23:40:59 -0600
Subject: [Numpy-discussion] Outer join ?
In-Reply-To: <7cc4bc500902112124s6a00d59fw78ea94f71ae4bb82@mail.gmail.com>
References: <7cc4bc500902112124s6a00d59fw78ea94f71ae4bb82@mail.gmail.com>
Message-ID: <3d375d730902112140j6878d379x6078ddc0ef7e4d70@mail.gmail.com>

On Wed, Feb 11, 2009 at 23:24, A B <python6009 at gmail.com> wrote:
> Hi,
>
> I have the following data structure:
>
> col1 | col2 | col3
>
> 20080101|key1|4
> 20080201|key1|6
> 20080301|key1|5
> 20080301|key2|3.4
> 20080601|key2|5.6
>
> For each key in the second column, I would like to create an array
> where for all unique values in the first column, there will be either
> a value or zero if there is no data available. Like so:
>
> # 20080101, 20080201, 20080301, 20080601
>
> key1 - 4, 6, 5,    0
> key2 - 0, 0, 3.4, 5.6
>
> Ideally, the results would end up in a 2d array.
>
> What's the most efficient way to accomplish this? Currently, I am
> getting a list of uniq col1 and col2 values into separate variables,
> then looping through each unique value in col2
>
> a = loadtxt(...)
>
> dates = unique(a[:]['col1'])
> keys = unique(a[:]['col2'])
>
> for key in keys:
>    b = a[where(a[:]['col2'] == key)]
>    ???

Take a look at setmember1d().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Thu Feb 12 00:44:36 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 12 Feb 2009 07:44:36 +0200
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <6ce0ac130902052019ne743034of817f02e31312f1f@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
	<6ce0ac130902031634ye40a829gbac28740fa281231@mail.gmail.com>
	<3d375d730902031642o65baa95ciebde9b5cc8f8b800@mail.gmail.com>
	<6ce0ac130902031653x2a304265t41b3c4f70bc257ef@mail.gmail.com>
	<3d375d730902031658l2b727ad2t3cfbaff32865e013@mail.gmail.com>
	<6ce0ac130902032122i1ba31be3y9c2d034ee0a1d2fc@mail.gmail.com>
	<3d375d730902032132l33417bedia7dacf08f4e7996e@mail.gmail.com>
	<6ce0ac130902052000x761c47cen46a3815c2b741713@mail.gmail.com>
	<3d375d730902052013t49f5fbechb89bb8e1332bfb07@mail.gmail.com>
	<6ce0ac130902052019ne743034of817f02e31312f1f@mail.gmail.com>
Message-ID: <9457e7c80902112144l32946315h8500896b6fc3b96c@mail.gmail.com>

2009/2/6 Brian Granger <ellisonbg.net at gmail.com>:
> Great, what is the best way of rolling this into numpy?

I've committed your patch.

Cheers
St?fan


From ellisonbg.net at gmail.com  Thu Feb 12 00:46:09 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 11 Feb 2009 21:46:09 -0800
Subject: [Numpy-discussion] Fast threading solution thoughts
Message-ID: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>

Hi,

This is relevant for anyone who would like to speed up array based
codes using threads.

I have a simple loop that I have implemented using Cython:

def backstep(np.ndarray opti, np.ndarray optf,
             int istart, int iend, double p, double q):
    cdef int j
    cdef double *pi
    cdef double *pf
    pi = <double *>opti.data
    pf = <double *>optf.data

    with nogil:
        for j in range(istart, iend):
            pf[j] = (p*pi[j+1] + q*pi[j])

I need to call this function *many* times and each time cannot be
performed until the previous time is completely as there are data
dependencies.  But, I still want to parallelize a single call to this
function across multiple cores (notice that I am releasing the GIL
before I do the heavy lifting).

I want to break my loop range(istart,iend) into pieces and have a
thread do each piece.  The arrays have sizes 10^3 to 10^5.

Things I have tried:

* Use pthreads and create new threads for each call to my function.
This performed horribly due to the thread creation overhead.
* Use a simple threadpool implementation in Python.  This performed
horribly as well even though I was not recreating threads each call.
The reason in this case was the time spent waiting on locks in the
thread pool implementation (which is based on Queue and threading).
This is either a problem with threading itself or a fundamental
limitation of the pthreads library itself.
* My next step is to implement a thread pool using pthreads/Cython.
Assuming I do this right, this should be as fast as I can get using
pthreads.

The only tools that I know of that are *really* designed to handle
these types of fine-grained parallel problems are:

* Intel's TBB
* OpenMP
* Cilk++
* ???

This seem like pretty heavy solutions though.

So, do people have thoughts about ways of effectively using threads
(not processes) for thin parallel loops over arrays?  This is relevant
to Numpy itself as it would be very nice if all ufuncs could release
the GIL and be run on multiple threads.

Cheers,

Brian


From ellisonbg.net at gmail.com  Thu Feb 12 00:46:23 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 11 Feb 2009 21:46:23 -0800
Subject: [Numpy-discussion] numscons/numpy.distutils bug related to
	MACOSX_DEPLOYMENT_TARGET
In-Reply-To: <9457e7c80902112144l32946315h8500896b6fc3b96c@mail.gmail.com>
References: <6ce0ac130902031612t376a1a59n431a08c899b3a517@mail.gmail.com>
	<3d375d730902031642o65baa95ciebde9b5cc8f8b800@mail.gmail.com>
	<6ce0ac130902031653x2a304265t41b3c4f70bc257ef@mail.gmail.com>
	<3d375d730902031658l2b727ad2t3cfbaff32865e013@mail.gmail.com>
	<6ce0ac130902032122i1ba31be3y9c2d034ee0a1d2fc@mail.gmail.com>
	<3d375d730902032132l33417bedia7dacf08f4e7996e@mail.gmail.com>
	<6ce0ac130902052000x761c47cen46a3815c2b741713@mail.gmail.com>
	<3d375d730902052013t49f5fbechb89bb8e1332bfb07@mail.gmail.com>
	<6ce0ac130902052019ne743034of817f02e31312f1f@mail.gmail.com>
	<9457e7c80902112144l32946315h8500896b6fc3b96c@mail.gmail.com>
Message-ID: <6ce0ac130902112146y12a6ba55kbadf3abaf17a4c10@mail.gmail.com>

Thanks much!

Brian

On Wed, Feb 11, 2009 at 9:44 PM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2009/2/6 Brian Granger <ellisonbg.net at gmail.com>:
>> Great, what is the best way of rolling this into numpy?
>
> I've committed your patch.
>
> Cheers
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From robert.kern at gmail.com  Thu Feb 12 00:52:40 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 11 Feb 2009 23:52:40 -0600
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
Message-ID: <3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>

On Wed, Feb 11, 2009 at 23:46, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Hi,
>
> This is relevant for anyone who would like to speed up array based
> codes using threads.
>
> I have a simple loop that I have implemented using Cython:
>
> def backstep(np.ndarray opti, np.ndarray optf,
>             int istart, int iend, double p, double q):
>    cdef int j
>    cdef double *pi
>    cdef double *pf
>    pi = <double *>opti.data
>    pf = <double *>optf.data
>
>    with nogil:
>        for j in range(istart, iend):
>            pf[j] = (p*pi[j+1] + q*pi[j])
>
> I need to call this function *many* times and each time cannot be
> performed until the previous time is completely as there are data
> dependencies.  But, I still want to parallelize a single call to this
> function across multiple cores (notice that I am releasing the GIL
> before I do the heavy lifting).
>
> I want to break my loop range(istart,iend) into pieces and have a
> thread do each piece.  The arrays have sizes 10^3 to 10^5.
>
> Things I have tried:
>
> * Use pthreads and create new threads for each call to my function.
> This performed horribly due to the thread creation overhead.
> * Use a simple threadpool implementation in Python.  This performed
> horribly as well even though I was not recreating threads each call.
> The reason in this case was the time spent waiting on locks in the
> thread pool implementation (which is based on Queue and threading).
> This is either a problem with threading itself or a fundamental
> limitation of the pthreads library itself.
> * My next step is to implement a thread pool using pthreads/Cython.
> Assuming I do this right, this should be as fast as I can get using
> pthreads.
>
> The only tools that I know of that are *really* designed to handle
> these types of fine-grained parallel problems are:
>
> * Intel's TBB
> * OpenMP
> * Cilk++
> * ???
>
> This seem like pretty heavy solutions though.

>From a programmer's perspective, it seems to me like OpenMP is a muck
lighter weight solution than pthreads.

> So, do people have thoughts about ways of effectively using threads
> (not processes) for thin parallel loops over arrays?  This is relevant
> to Numpy itself as it would be very nice if all ufuncs could release
> the GIL and be run on multiple threads.

Eric Jones tried to do this with pthreads in C some time ago. His work is here:

  http://svn.scipy.org/svn/numpy/branches/multicore/

The lock overhead makes it usually not worthwhile.

And there's still an open problem for determining whether two strided
arrays overlap in memory.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From scott.sinclair.za at gmail.com  Thu Feb 12 01:03:42 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Thu, 12 Feb 2009 08:03:42 +0200
Subject: [Numpy-discussion] loadtxt issues
In-Reply-To: <7cc4bc500902112104w22fb9978v8b6b3e8bf25166f3@mail.gmail.com>
References: <7cc4bc500902102140h6cc85bb3wa28c89a147ddce73@mail.gmail.com>
	<e183a99d0902102152p2bb71821of56dc407216095c5@mail.gmail.com>
	<7cc4bc500902111827p3942dc40ie1216c13000fe2e0@mail.gmail.com>
	<7cc4bc500902112104w22fb9978v8b6b3e8bf25166f3@mail.gmail.com>
Message-ID: <6a17e9ee0902112203t2f43efe1qa6a498b0e7750452@mail.gmail.com>

> 2009/2/12 A B <python6009 at gmail.com>:
> Actually, I was using two different machines and it appears that the
> version of numpy available on Ubuntu is seriously out of date (1.0.4).
> Wonder why ...

See the recent post here

http://projects.scipy.org/pipermail/numpy-discussion/2009-February/040252.html

Cheers,
Scott


From ellisonbg.net at gmail.com  Thu Feb 12 01:03:46 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 11 Feb 2009 22:03:46 -0800
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
Message-ID: <6ce0ac130902112203y3274ddd0g8196e05711361d5@mail.gmail.com>

> Eric Jones tried to do this with pthreads in C some time ago. His work is here:
>
>  http://svn.scipy.org/svn/numpy/branches/multicore/
>
> The lock overhead makes it usually not worthwhile.

I was under the impression that Eric's implementation didn't use a
thread pool.  Thus I thought the bottleneck was the thread creation
time for his implementation.  Eric can you comment?

Cheers,

Brian


From fperez.net at gmail.com  Thu Feb 12 01:13:46 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 11 Feb 2009 22:13:46 -0800
Subject: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0;
	Ubuntu packages
In-Reply-To: <5b8d13220902111817q9bd42c0r8c42f6307ca1b318@mail.gmail.com>
References: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>
	<db6b5ecc0902111511y7d9cb105x6283d628e6473568@mail.gmail.com>
	<5b8d13220902111817q9bd42c0r8c42f6307ca1b318@mail.gmail.com>
Message-ID: <db6b5ecc0902112213m6a49ba30w5faefdc0971b3d1a@mail.gmail.com>

On Wed, Feb 11, 2009 at 6:17 PM, David Cournapeau <cournape at gmail.com> wrote:

> Unfortunately, it does require some work, because hardy uses g77
> instead of gfortran, so the source package has to be different (once
> hardy is done, all the one below would be easy, though). I am not sure
> how to do that with PPA (the doc is not great).

OK, thanks for the info.  This is already very useful.

Cheers,

f


From robert.kern at gmail.com  Thu Feb 12 01:18:46 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Feb 2009 00:18:46 -0600
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <6ce0ac130902112203y3274ddd0g8196e05711361d5@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<6ce0ac130902112203y3274ddd0g8196e05711361d5@mail.gmail.com>
Message-ID: <3d375d730902112218oa83d95fo4b240e40510cf06c@mail.gmail.com>

On Thu, Feb 12, 2009 at 00:03, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> Eric Jones tried to do this with pthreads in C some time ago. His work is here:
>>
>>  http://svn.scipy.org/svn/numpy/branches/multicore/
>>
>> The lock overhead makes it usually not worthwhile.
>
> I was under the impression that Eric's implementation didn't use a
> thread pool.  Thus I thought the bottleneck was the thread creation
> time for his implementation.  Eric can you comment?

It does use a thread pool. See PyUFunc_GenericFunction:

http://svn.scipy.org/svn/numpy/branches/multicore/numpy/core/src/ufuncobject.c

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From gael.varoquaux at normalesup.org  Thu Feb 12 01:26:36 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 12 Feb 2009 07:26:36 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
Message-ID: <20090212062636.GD30330@phare.normalesup.org>

On Wed, Feb 11, 2009 at 11:52:40PM -0600, Robert Kern wrote:
> > This seem like pretty heavy solutions though.

> >From a programmer's perspective, it seems to me like OpenMP is a muck
> >lighter weight solution than pthreads.

>From a programmer's perspective, because, IMHO, openmp is implemented
using pthreads. I do have difficulties verifying this on the web, but
documents I find hint to that.

Ga?l


From david at ar.media.kyoto-u.ac.jp  Thu Feb 12 01:12:45 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Feb 2009 15:12:45 +0900
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
Message-ID: <4993BDDD.9020404@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
>
> Eric Jones tried to do this with pthreads in C some time ago. His work is here:
>
>   http://svn.scipy.org/svn/numpy/branches/multicore/
>
> The lock overhead makes it usually not worthwhile.
>   

I am curious: would you know what would be different in numpy's case
compared to matlab array model concerning locks ? Matlab, up to
recently, only spreads BLAS/LAPACK on multi-cores, but since matlab 7.3
(or 7.4), it also uses multicore for mathematical functions (cos,
etc...). So at least in matlab's model, it looks like it can be useful.
I understand that numpy's model is more flexible (I don't think strided
arrays can overlap in matlab for example, at least not from what you can
see from the public API).

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Thu Feb 12 01:15:55 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Feb 2009 15:15:55 +0900
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <20090212062636.GD30330@phare.normalesup.org>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<20090212062636.GD30330@phare.normalesup.org>
Message-ID: <4993BE9B.9060709@ar.media.kyoto-u.ac.jp>

Gael Varoquaux wrote:
> >From a programmer's perspective, because, IMHO, openmp is implemented
> using pthreads. 

Since openmp also exists on windows, I doubt that it is required that
openmp uses pthread :)

On linux, with gcc, using -fopenmp implies -pthread, so I guess it uses
pthread (can you be more low level than pthread on pthread-enabled Linux
in userland ?)

cheers,

David


From robert.kern at gmail.com  Thu Feb 12 01:42:37 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Feb 2009 00:42:37 -0600
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <4993BE9B.9060709@ar.media.kyoto-u.ac.jp>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<20090212062636.GD30330@phare.normalesup.org>
	<4993BE9B.9060709@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730902112242o27f4144dj83f835a48c48df90@mail.gmail.com>

On Thu, Feb 12, 2009 at 00:15, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Gael Varoquaux wrote:
>> >From a programmer's perspective, because, IMHO, openmp is implemented
>> using pthreads.
>
> Since openmp also exists on windows, I doubt that it is required that
> openmp uses pthread :)

It is implemented using threads, with Windows native threads on
Windows. I think Ga?l really just meant "threads" there.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ellisonbg.net at gmail.com  Thu Feb 12 01:52:57 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 11 Feb 2009 22:52:57 -0800
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
Message-ID: <6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>

> I am curious: would you know what would be different in numpy's case
> compared to matlab array model concerning locks ? Matlab, up to
> recently, only spreads BLAS/LAPACK on multi-cores, but since matlab 7.3
> (or 7.4), it also uses multicore for mathematical functions (cos,
> etc...). So at least in matlab's model, it looks like it can be useful.

Good point.  Is it possible to tell what array size it switches over
to using multiple threads?  Also, do you happen to iknow how Matlab is
doing this?

> I understand that numpy's model is more flexible (I don't think strided
> arrays can overlap in matlab for example, at least not from what you can
> see from the public API).

True, but I would be happy to just have a fast C based threadpool
implentation I could use in low level Cython based loops.  When
performance *really* matters, ufuncs have a lot of other overhead that
is typically unacceptable.  But I could imagine that all that extra
logic kill the parallel scaling through Amdahl's law (the extra logic
is serial logic).

Cheers,

Brian

> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From robert.kern at gmail.com  Thu Feb 12 01:56:31 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Feb 2009 00:56:31 -0600
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
Message-ID: <3d375d730902112256p5157010am51d5517d8c1ca060@mail.gmail.com>

On Thu, Feb 12, 2009 at 00:52, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> I am curious: would you know what would be different in numpy's case
>> compared to matlab array model concerning locks ? Matlab, up to
>> recently, only spreads BLAS/LAPACK on multi-cores, but since matlab 7.3
>> (or 7.4), it also uses multicore for mathematical functions (cos,
>> etc...). So at least in matlab's model, it looks like it can be useful.
>
> Good point.  Is it possible to tell what array size it switches over
> to using multiple threads?

Yes.

http://svn.scipy.org/svn/numpy/branches/multicore/numpy/core/threadapi.py

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ellisonbg.net at gmail.com  Thu Feb 12 01:58:46 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 11 Feb 2009 22:58:46 -0800
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <3d375d730902112256p5157010am51d5517d8c1ca060@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
	<3d375d730902112256p5157010am51d5517d8c1ca060@mail.gmail.com>
Message-ID: <6ce0ac130902112258l50c64577uf433243587c16ea4@mail.gmail.com>

>> Good point.  Is it possible to tell what array size it switches over
>> to using multiple threads?
>
> Yes.
>
> http://svn.scipy.org/svn/numpy/branches/multicore/numpy/core/threadapi.py

Sorry, I was curious about what Matlab does in this respect.  But,
this is very useful and I will look at it.

Cheers,

Brian


> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From david at ar.media.kyoto-u.ac.jp  Thu Feb 12 01:50:16 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Feb 2009 15:50:16 +0900
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
Message-ID: <4993C6A8.6030808@ar.media.kyoto-u.ac.jp>

Brian Granger wrote:
>> I am curious: would you know what would be different in numpy's case
>> compared to matlab array model concerning locks ? Matlab, up to
>> recently, only spreads BLAS/LAPACK on multi-cores, but since matlab 7.3
>> (or 7.4), it also uses multicore for mathematical functions (cos,
>> etc...). So at least in matlab's model, it looks like it can be useful.
>>     
>
> Good point.  Is it possible to tell what array size it switches over
> to using multiple threads?  Also, do you happen to iknow how Matlab is
> doing this?
>   

No - I have never seen deep explanation of the matlab model. The C api
is so small that it is hard to deduce anything from it (except that the
memory handling is not ref-counting-based, I don't know if it matters
for our discussion of speeding up ufunc). I would guess that since two
arrays cannot share data (COW-based), lock handling may be easier to
deal with ? I am not really familiar with multi-thread programming (my
only limited experience is for soft real-time programming for audio
processing, where the issues are totally different, since latency
matters as much if not more than throughput).

>
> True, but I would be happy to just have a fast C based threadpool
> implentation I could use in low level Cython based loops.

Matlab has a parallel toolbox to do this kind of things in matlab (I
don't know in C). I don't know anything about it, nor do I know if that
can be applied in any way to python/numpy's case:

http://www.mathworks.com/products/parallel-computing/

cheers,

David


From strawman at astraw.com  Thu Feb 12 03:07:02 2009
From: strawman at astraw.com (Andrew Straw)
Date: Thu, 12 Feb 2009 00:07:02 -0800
Subject: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0;
 Ubuntu packages
In-Reply-To: <db6b5ecc0902112213m6a49ba30w5faefdc0971b3d1a@mail.gmail.com>
References: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>	<db6b5ecc0902111511y7d9cb105x6283d628e6473568@mail.gmail.com>	<5b8d13220902111817q9bd42c0r8c42f6307ca1b318@mail.gmail.com>
	<db6b5ecc0902112213m6a49ba30w5faefdc0971b3d1a@mail.gmail.com>
Message-ID: <4993D8A6.5080107@astraw.com>

Fernando Perez wrote:
> On Wed, Feb 11, 2009 at 6:17 PM, David Cournapeau <cournape at gmail.com> wrote:
> 
>> Unfortunately, it does require some work, because hardy uses g77
>> instead of gfortran, so the source package has to be different (once
>> hardy is done, all the one below would be easy, though). I am not sure
>> how to do that with PPA (the doc is not great).
> 
> OK, thanks for the info.  This is already very useful.

What exactly is the expected problem and how would I verify that I'm not
getting hit by it?

The context is that I have built your packages on Hardy and got no FTBFS
errors in a clean sbuild machine and was about to upload this to my
lab's repo but saw your email. (Also, I have been using scipy
0.7.0rc2-ish apparently descended from the same Debian source package as
yours for a couple weeks and have noted no problems of late.) "nosetest
scipy" on an amd64 with lots of installed packages shows:

Ran 3211 tests in 356.572s

FAILED (SKIP=13, errors=9)

Where the errors are all known failures.


The same test on the i386 sbuild shows:

Ran 3211 tests in 312.464s

FAILED (SKIP=16, errors=204)

Most of the failures appear to be from weave/scxx and I suspect have to
do with some package not being installed -- if that's true, it's
probably a packaging bug and not a scipy bug.


From gael.varoquaux at normalesup.org  Thu Feb 12 03:12:38 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 12 Feb 2009 09:12:38 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <3d375d730902112242o27f4144dj83f835a48c48df90@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<20090212062636.GD30330@phare.normalesup.org>
	<4993BE9B.9060709@ar.media.kyoto-u.ac.jp>
	<3d375d730902112242o27f4144dj83f835a48c48df90@mail.gmail.com>
Message-ID: <20090212081238.GB20814@phare.normalesup.org>

On Thu, Feb 12, 2009 at 12:42:37AM -0600, Robert Kern wrote:
> It is implemented using threads, with Windows native threads on
> Windows. I think Ga?l really just meant "threads" there.

I guess so :). Once you reformulate my remark in proper terms, this is
indeed what comes out.

I guess all what it means is that openMP is implemented using native OS
threads, and is nothing more than an abstraction on them.

Ga?l


From david at ar.media.kyoto-u.ac.jp  Thu Feb 12 02:59:29 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Feb 2009 16:59:29 +0900
Subject: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0;
 Ubuntu packages
In-Reply-To: <4993D8A6.5080107@astraw.com>
References: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>	<db6b5ecc0902111511y7d9cb105x6283d628e6473568@mail.gmail.com>	<5b8d13220902111817q9bd42c0r8c42f6307ca1b318@mail.gmail.com>	<db6b5ecc0902112213m6a49ba30w5faefdc0971b3d1a@mail.gmail.com>
	<4993D8A6.5080107@astraw.com>
Message-ID: <4993D6E1.5080803@ar.media.kyoto-u.ac.jp>

Andrew Straw wrote:
> Fernando Perez wrote:
>   
>> On Wed, Feb 11, 2009 at 6:17 PM, David Cournapeau <cournape at gmail.com> wrote:
>>
>>     
>>> Unfortunately, it does require some work, because hardy uses g77
>>> instead of gfortran, so the source package has to be different (once
>>> hardy is done, all the one below would be easy, though). I am not sure
>>> how to do that with PPA (the doc is not great).
>>>       
>> OK, thanks for the info.  This is already very useful.
>>     
>
> What exactly is the expected problem and how would I verify that I'm not
> getting hit by it?
>   

I want to follow as closely as possible the official debian/ubuntu
packages. Ideally, any package produced on the PPA superseded by an
official package (from 1.2.1-1~ppaN to 1.2.1-1) should be identical to
the superseding package. Hardy default fortran ABI is g77, not gfortran,
so I have to use g77 for hardy - and the ppa package, limited to
intrepid, use gfortran (since intrepid ABI is gfortran's).

In your case, you built a package which uses gfortran ABI on Hardy - it
works, but is not really acceptable for an official package - and thus,
when an upgrade from ppa to official happens, you won't get the same
package. In the rpm  world, you can use conditional on distribution
version/type int the spec file (which is the control  + changelog  +
rules in one file), but AFAIK, you can't do that with debian, or at
least have not found the relevant doc.

cheers,

David


From paul at rudin.co.uk  Thu Feb 12 03:55:57 2009
From: paul at rudin.co.uk (Paul Rudin)
Date: Thu, 12 Feb 2009 08:55:57 +0000
Subject: [Numpy-discussion] numpy-izing a loop
References: <87y6we1lbw.fsf@rudin.co.uk>
	<9457e7c80902100819t139d6342y8885f417a4ec4443@mail.gmail.com>
	<3d375d730902101148t10d50084k1a775e681a5e3d05@mail.gmail.com>
	<9457e7c80902101327q5cd7cd4ax7e2c1713fcee2cd4@mail.gmail.com>
	<9457e7c80902101333l5883971i2b15c6290315bf8d@mail.gmail.com>
	<87skmltm63.fsf@rudin.co.uk>
Message-ID: <87d4doc9uq.fsf@rudin.co.uk>


Paul Rudin <paul at rudin.co.uk> writes:

> def compute_voxels2(depth_buffers):
>     dim = depth_buffers[0].shape[0]
>     znear, zfar, ynear, yfar, xnear, xfar = depth_buffers
>     z = numpy.arange(dim)
>     y = numpy.arange(dim)[:, None]
>     x = numpy.arange(dim)[:, None, None]
>
>     return  ((xnear[y,z] < xfar[y, z]) &
>              (ynear[x, z] < yfar[x, z]) &
>              (znear[x,y] < zfar[x,y]))
>

... or, more succinctly:

def compute_voxels2(znear, zfar, ynear, yfar, xnear, xfar):   
    return  ( (xnear<xfar) &
              (ynear < yfar)[:,None] &
              (znear < zfar)[:, :, None])

>
> All that remains is for me to verify that it's actually the result I want :)
>

Which it is, although getting the 6 planes passed in oriented correctly
in 3d-space makes my head hurt :/



From dagss at student.matnat.uio.no  Thu Feb 12 04:39:56 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 12 Feb 2009 10:39:56 +0100
Subject: [Numpy-discussion] PyArray_SETITEM with object arrays in Cython
In-Reply-To: <6c476c8a0902111412y2cb6985x973ff830d1a1412@mail.gmail.com>
References: <6c476c8a0902110710q4c9b16e4vcc3e315f64a84caf@mail.gmail.com>	<76550d7a4d21a5fbd6764825cc547ae1.squirrel@webmail.uio.no>
	<6c476c8a0902111412y2cb6985x973ff830d1a1412@mail.gmail.com>
Message-ID: <4993EE6C.4070507@student.matnat.uio.no>

Wes McKinney wrote:
>
> The general problem here is an indexed array (by dates or strings, for 
> example), that you want to conform to a new index. The arrays most of 
> the time contain floats but occasionally PyObjects. For some reason 
> the access and assignment is slow (this function can be faster by a 
> factor of 50 with C API macros, so clearly something is awry)-- let me 
> know if you see anything obviously wrong with this
Thanks for the example; I don't have time this week but I recorded it on 
the Cython trac and will hopefully get back to it soon.

http://trac.cython.org/cython_trac/ticket/209

Dag Sverre


From faltet at pytables.org  Thu Feb 12 05:05:54 2009
From: faltet at pytables.org (Francesc Alted)
Date: Thu, 12 Feb 2009 11:05:54 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
Message-ID: <200902121105.55893.faltet@pytables.org>

Hi Brian,

A Thursday 12 February 2009, Brian Granger escrigu?:
> Hi,
>
> This is relevant for anyone who would like to speed up array based
> codes using threads.
>
> I have a simple loop that I have implemented using Cython:
>
> def backstep(np.ndarray opti, np.ndarray optf,
>              int istart, int iend, double p, double q):
>     cdef int j
>     cdef double *pi
>     cdef double *pf
>     pi = <double *>opti.data
>     pf = <double *>optf.data
>
>     with nogil:
>         for j in range(istart, iend):
>             pf[j] = (p*pi[j+1] + q*pi[j])
>
> I need to call this function *many* times and each time cannot be
> performed until the previous time is completely as there are data
> dependencies.  But, I still want to parallelize a single call to this
> function across multiple cores (notice that I am releasing the GIL
> before I do the heavy lifting).
>
> I want to break my loop range(istart,iend) into pieces and have a
> thread do each piece.  The arrays have sizes 10^3 to 10^5.
>
> Things I have tried:
[clip]

If your problem is evaluating vector expressions just like the above 
(i.e. without using transcendental functions like sin, exp, etc...), 
usually the bottleneck is on memory access, so using several threads is 
simply not going to help you achieving better performance, but rather 
the contrary (you have to deal with the additional thread overhead).  
So, frankly, I would not waste more time trying to paralelize that.

As an example, in the recent support of VML in numexpr we have disabled 
the use of VML (as well as the OpenMP threading support that comes with 
it) in cases like yours, where only additions and multiplications are 
performed (these operations are very fast in modern processors, and the 
sole bottleneck for this case is the memory bandwidth, as I've said).  
However, in case of expressions containing operations like division or 
transcendental functions, then VML activates automatically, and you can 
make use of several cores if you want.  So, if you are in this case, 
and you have access to Intel MKL (the library that contains VML), you 
may want to give numexpr a try. 

HTH,

-- 
Francesc Alted


From dagss at student.matnat.uio.no  Thu Feb 12 05:30:57 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 12 Feb 2009 11:30:57 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
Message-ID: <4993FA61.7010000@student.matnat.uio.no>

Brian Granger wrote:
> Hi,
>
> This is relevant for anyone who would like to speed up array based
> codes using threads.
>
> I have a simple loop that I have implemented using Cython:
>
> def backstep(np.ndarray opti, np.ndarray optf,
>              int istart, int iend, double p, double q):
>     cdef int j
>     cdef double *pi
>     cdef double *pf
>     pi = <double *>opti.data
>     pf = <double *>optf.data
>
>     with nogil:
>         for j in range(istart, iend):
>             pf[j] = (p*pi[j+1] + q*pi[j])
>
> I need to call this function *many* times and each time cannot be
> performed until the previous time is completely as there are data
> dependencies.  But, I still want to parallelize a single call to this
> function across multiple cores (notice that I am releasing the GIL
> before I do the heavy lifting).
>   
A quick digression:

It would be interesting to see how a spec would look for integrating 
OpenMP natively into Cython for these kinds of purposes. Cython is still 
flexible as a language after all. Avoiding language bloat is also 
important, but it is difficult to know what kind of balance can be 
struck before some kind of spec is worked out. Has anyone managed to use 
OpenMP with Cython code in a nice way already?

I couldn't do any work on it right now but it could sit in the pipeline 
for the year to come. Also I have a strong feeling that making the spec 
and language design issues would take more time than the actual 
implementation of it anyway.

Dag Sverre


From strawman at astraw.com  Thu Feb 12 05:53:17 2009
From: strawman at astraw.com (Andrew Straw)
Date: Thu, 12 Feb 2009 02:53:17 -0800
Subject: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0;
 Ubuntu packages
In-Reply-To: <4993D6E1.5080803@ar.media.kyoto-u.ac.jp>
References: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>	<db6b5ecc0902111511y7d9cb105x6283d628e6473568@mail.gmail.com>	<5b8d13220902111817q9bd42c0r8c42f6307ca1b318@mail.gmail.com>	<db6b5ecc0902112213m6a49ba30w5faefdc0971b3d1a@mail.gmail.com>	<4993D8A6.5080107@astraw.com>
	<4993D6E1.5080803@ar.media.kyoto-u.ac.jp>
Message-ID: <4993FF9D.2010805@astraw.com>

David Cournapeau wrote:
> Andrew Straw wrote:
>   
>> Fernando Perez wrote:
>>   
>>     
>>> On Wed, Feb 11, 2009 at 6:17 PM, David Cournapeau <cournape at gmail.com> wrote:
>>>
>>>     
>>>       
>>>> Unfortunately, it does require some work, because hardy uses g77
>>>> instead of gfortran, so the source package has to be different (once
>>>> hardy is done, all the one below would be easy, though). I am not sure
>>>> how to do that with PPA (the doc is not great).
>>>>       
>>>>         
>>> OK, thanks for the info.  This is already very useful.
>>>     
>>>       
>> What exactly is the expected problem and how would I verify that I'm not
>> getting hit by it?
>>   
>>     
>
> I want to follow as closely as possible the official debian/ubuntu
> packages. Ideally, any package produced on the PPA superseded by an
> official package (from 1.2.1-1~ppaN to 1.2.1-1) should be identical to
> the superseding package. Hardy default fortran ABI is g77, not gfortran,
> so I have to use g77 for hardy - and the ppa package, limited to
> intrepid, use gfortran (since intrepid ABI is gfortran's).
>   
(Warning: this email is a little over-detailed on the packaging details
front. Believe it or not, I'm not discussing the details of Debian
packaging for fun, but rather my questions have practical importance to
me -- I don't want to break all my lab's scipy installations. :)


This doesn't make sense to me. I built the .deb on an a clean, minimal
sbuild (a chroot with only a very few basic packages installed, somewhat
mimicing Ubuntu's PPA builder). It built from your unmodified .dsc,
which auto-downloads the declared dependencies (and nothing else). It
passes the tests. To be very explicit -- I didn't specify to use g77 at
any point. (As implied by my previous statement of using your unmodified
.dsc, I used only the debian/rules and debian/control in your package.)
To understand your statement about identical, I will operationally
define "identical" for .debs to mean that they were built from the same
.dsc. Of course, in the case you describe above, it can't be _exactly_
the same .dsc because the version numbers in debian/changelog must
change and consequently so must the checksums and GPG signature in the
.dsc file, and presumably a different person will sign it. Also, there
will be timestamp differences and such for two .debs built from the
exact same .dsc, but we can ignore those. In this case, I don't see why
an "official" package, which meets this operational definition of
identical, wouldn't work on Hardy, as it would be built from nearly an
identical .dsc on nearly an identical clean build environment. (Of
course, there will never be an official package of this for Hardy, but
that's not the point.)

> In your case, you built a package which uses gfortran ABI on Hardy - it
> works, but is not really acceptable for an official package - and thus,
> when an upgrade from ppa to official happens, you won't get the same
> package.
Why is it "not really acceptable"? As long as it builds and works and
doesn't break anything, why would Ubuntu maintainers care if it uses the
gfortran ABI?

Also, in practical terms, what upgrade? A) Hardy will not upgrade
python-scipy. It's against policy for a released distribution to upgrade
software without a security reason. B) Imaging for a moment that there
would be an upgrade, why do you think it would break?
>  In the rpm  world, you can use conditional on distribution
> version/type int the spec file (which is the control  + changelog  +
> rules in one file), but AFAIK, you can't do that with debian, or at
> least have not found the relevant doc.
>   
I don't understand what you're saying. My understanding is that at the
beginning of each distribution (Hardy, Intrepid, Lenny), the maintainers
decide on a C++ (and I guess Fortran, but I'm not sure) ABI and fix the
toolchain to build this ABI. From then on, everything is built with this
ABI. And the point of a signed .dsc file and a clean sbuild/pbuilder is
that any .deb that gets built will be contingent on the files in
debian/* because that's cryptographically signed in the .dsc file. So,
if you trust the archive master and his computer (by trusting his keys
in your apt keyring), you trust that the .deb was built from the .dsc.
An the .dsc is signed by the maintainer. So there's a cryptographic
chain of trust to those control + changelog + rules files.

I'm still not sure if I'm not getting your worry, though...

Thanks,
Andrew


From gregor.thalhammer at gmail.com  Thu Feb 12 06:05:53 2009
From: gregor.thalhammer at gmail.com (Gregor Thalhammer)
Date: Thu, 12 Feb 2009 12:05:53 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
Message-ID: <49940291.4050906@googlemail.com>

Brian Granger schrieb:
>> I am curious: would you know what would be different in numpy's case
>> compared to matlab array model concerning locks ? Matlab, up to
>> recently, only spreads BLAS/LAPACK on multi-cores, but since matlab 7.3
>> (or 7.4), it also uses multicore for mathematical functions (cos,
>> etc...). So at least in matlab's model, it looks like it can be useful.
>>     
>
> Good point.  Is it possible to tell what array size it switches over
> to using multiple threads?  Also, do you happen to iknow how Matlab is
> doing this?
>
>   
Recent Matlab versions use Intels Math Kernel Library, which performs 
automatic multi-threading - also for mathematical functions like sin 
etc, but not for  addition, multiplication etc. It seems to me Matlab 
itself does not take care of multi-threading. On
http://www.intel.com/software/products/mkl/data/vml/functions/_listfunc.html
you can have a look at the performance data of the MKL vectorized math 
functions. Around a vector length between 100-1000, depending on which 
function, precision, cpu architecture, they switch to multi-threading.

Gregor


From faltet at pytables.org  Thu Feb 12 06:16:19 2009
From: faltet at pytables.org (Francesc Alted)
Date: Thu, 12 Feb 2009 12:16:19 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <4993FA61.7010000@student.matnat.uio.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<4993FA61.7010000@student.matnat.uio.no>
Message-ID: <200902121216.20222.faltet@pytables.org>

A Thursday 12 February 2009, Dag Sverre Seljebotn escrigu?:
> A quick digression:
>
> It would be interesting to see how a spec would look for integrating
> OpenMP natively into Cython for these kinds of purposes. Cython is
> still flexible as a language after all.

That would be really nice indeed.

> Avoiding language bloat is 
> also important, but it is difficult to know what kind of balance can
> be struck before some kind of spec is worked out. Has anyone managed
> to use OpenMP with Cython code in a nice way already?
>
> I couldn't do any work on it right now but it could sit in the
> pipeline for the year to come. Also I have a strong feeling that
> making the spec and language design issues would take more time than
> the actual implementation of it anyway.

I tend to aggree with you.  As a matter of fact, doing (efficient) 
parallelism is a very hairy thing and understanding all (or just most 
of them) of its issues maybe the real problem for doing the port 
(unless one can find some direct way to translate OpenMP directives 
from Cython code to the C code, which would be wonderful).

-- 
Francesc Alted


From david at ar.media.kyoto-u.ac.jp  Thu Feb 12 06:06:39 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Feb 2009 20:06:39 +0900
Subject: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0;
 Ubuntu packages
In-Reply-To: <4993FF9D.2010805@astraw.com>
References: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>	<db6b5ecc0902111511y7d9cb105x6283d628e6473568@mail.gmail.com>	<5b8d13220902111817q9bd42c0r8c42f6307ca1b318@mail.gmail.com>	<db6b5ecc0902112213m6a49ba30w5faefdc0971b3d1a@mail.gmail.com>	<4993D8A6.5080107@astraw.com>	<4993D6E1.5080803@ar.media.kyoto-u.ac.jp>
	<4993FF9D.2010805@astraw.com>
Message-ID: <499402BF.60500@ar.media.kyoto-u.ac.jp>

Andrew Straw wrote:
> (Warning: this email is a little over-detailed on the packaging details
> front. Believe it or not, I'm not discussing the details of Debian
> packaging for fun, but rather my questions have practical importance to
> me -- I don't want to break all my lab's scipy installations. :)

Well, the devil is in the details, specially for packaging :)

>
>
> This doesn't make sense to me. I built the .deb on an a clean, minimal
> sbuild (a chroot with only a very few basic packages installed, somewhat
> mimicing Ubuntu's PPA builder). It built from your unmodified .dsc,
> which auto-downloads the declared dependencies (and nothing else). It
> passes the tests. To be very explicit -- I didn't specify to use g77 at
> any point. (As implied by my previous statement of using your unmodified
> .dsc, I used only the debian/rules and debian/control in your package.)

Yes: your package will be almost the same as mine. It will use gfortran,
and not g77. There is nothing wrong with that per-se, but I don't think
it is a good practice in general. Here is an example which can break
things, to show that's not just red-herring:
    - install numpy from the PPA (gfortran ABI), pull gfortran as well
    - later, g77 is installed (as a dependency, whatever)
    - now, user build his own python extension with fortran extension,
or for example scipy -> uses g77, does not work anymore.

My main rationale to provide PPA is to avoid the never-ending queue of
emails about missing symbols, etc... because I am tired of it, because
it gives a bad mouth taste to users, because I would like to deal with
more interesting issues.

> To understand your statement about identical, I will operationally
> define "identical" for .debs to mean that they were built from the same
> .dsc.

I has an even larger definition of identical: same control/rules/patches
is identical for me.

> In this case, I don't see why
> an "official" package, which meets this operational definition of
> identical, wouldn't work on Hardy, as it would be built from nearly an
> identical .dsc on nearly an identical clean build environment. (Of
> course, there will never be an official package of this for Hardy, but
> that's not the point.)

Because a package has to be well integrated with the rest of the system
- I think my example above shows the problem of using a fortran compiler
which has a different ABI than the default one.

> Why is it "not really acceptable"? As long as it builds and works and
> doesn't break anything, why would Ubuntu maintainers care if it uses the
> gfortran ABI?

The problem is that it is impossible to avoid breaking anything without
using the g77 ABI on hardy. I don't understand "it is ok not to follow
the ABI as long as it does not break anything" : if there was no need to
follow an ABI to avoid breaking anything, then there would be no point
for an ABI in the first place :)

>
> Also, in practical terms, what upgrade? A) Hardy will not upgrade
> python-scipy. It's against policy for a released distribution to upgrade
> software without a security reason.

First, concerning the upgrade, I am just following the Ubuntu guide. To
quote it:

"
Ubuntu package names are suffixed by the version number of the package.
This allows Ubuntu to distinguish newer packages from older ones and so
remain up to date.

If you're creating an alternative version of a package already available
in Ubuntu's repositories, you should ensure that:

    * your package supersedes the official Ubuntu version
    * future Ubuntu versions will supersede your package.
"

As an example, I can think of backports.

>  B) Imaging for a moment that there
> would be an upgrade, why do you think it would break?

Well, if the upgraded package use the g77 ABI, and the old one the
gfortran ABI, this is quite likely to break some extensions built
against numpy/scipy I believe.

>>  In the rpm  world, you can use conditional on distribution
>> version/type int the spec file (which is the control  + changelog  +
>> rules in one file), but AFAIK, you can't do that with debian, or at
>> least have not found the relevant doc.
>>   
> I don't understand what you're saying.

One solution would be able to say in the control file: for hardy, do
this, for intrepid, do that. For example, with rpm packaging, I can do
the following:

%if 0%{?fedora_version} || 0%{?rhel_version} || 0%{?centos_version} 
BuildRequires: gcc-gfortran python python-devel lapack3-devel < 3.1
refblas3-devel
requires:       gcc-gfortran lapack3 < 3.1 refblas3
%endif

%if 0%{?suse_version}
BuildRequires:  gcc-fortran python python-devel lapack3-devel < 3.1
refblas3-devel
requires:       gcc-fortran lapack3 < 3.1 refblas3
%endif

Notice the different name for the dependency (not that this is a problem
with Ubuntu/Debian, but I would still need different packages).

>  And the point of a signed .dsc file and a clean sbuild/pbuilder is
> that any .deb that gets built will be contingent on the files in
> debian/* because that's cryptographically signed in the .dsc file. So,
> if you trust the archive master and his computer (by trusting his keys
> in your apt keyring), you trust that the .deb was built from the .dsc.
> An the .dsc is signed by the maintainer. So there's a cryptographic
> chain of trust to those control + changelog + rules files.

The problem has nothing to do with security - I am not sure I follow why
you mention signing issues.

cheers,

David


From dagss at student.matnat.uio.no  Thu Feb 12 06:34:42 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 12 Feb 2009 12:34:42 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <200902121216.20222.faltet@pytables.org>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<4993FA61.7010000@student.matnat.uio.no>
	<200902121216.20222.faltet@pytables.org>
Message-ID: <49940952.6060208@student.matnat.uio.no>

Francesc Alted wrote:
> A Thursday 12 February 2009, Dag Sverre Seljebotn escrigu?:
>   
>> A quick digression:
>>
>> It would be interesting to see how a spec would look for integrating
>> OpenMP natively into Cython for these kinds of purposes. Cython is
>> still flexible as a language after all.
>>     
>
> That would be really nice indeed.
>
>   
>> Avoiding language bloat is 
>> also important, but it is difficult to know what kind of balance can
>> be struck before some kind of spec is worked out. Has anyone managed
>> to use OpenMP with Cython code in a nice way already?
>>
>> I couldn't do any work on it right now but it could sit in the
>> pipeline for the year to come. Also I have a strong feeling that
>> making the spec and language design issues would take more time than
>> the actual implementation of it anyway.
>>     
>
> I tend to aggree with you.  As a matter of fact, doing (efficient) 
> parallelism is a very hairy thing and understanding all (or just most 
> of them) of its issues maybe the real problem for doing the port 
> (unless one can find some direct way to translate OpenMP directives 
> from Cython code to the C code, which would be wonderful).
>   
This is what I'm thinking. Example: This in Cython:

with cython.openmp.parallell(locals="a,b"): # implies nogil?
   for i in cython.openmp.range(0, 10):
      ...

could simply translate into emitting the right #pragmas in generated C 
at the right locations.

FYI, I am one of the core Cython developers and can make such 
modifications in Cython itself as long as there's consensus on how it 
should look on the Cython mailing list.  My problem is that I don't 
really know OpenMP and have little experience with it, so I'm not the 
best person for creating a draft for how such high-level OpenMP 
constructs should look like in Cython.

Dag Sverre


From david at ar.media.kyoto-u.ac.jp  Thu Feb 12 06:20:29 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Feb 2009 20:20:29 +0900
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <49940291.4050906@googlemail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
	<49940291.4050906@googlemail.com>
Message-ID: <499405FD.60602@ar.media.kyoto-u.ac.jp>

Gregor Thalhammer wrote:
> Recent Matlab versions use Intels Math Kernel Library, which performs 
> automatic multi-threading - also for mathematical functions like sin 
> etc, but not for  addition, multiplication etc.

It does if you have access to the parallel toolbox I mentioned earlier
in this thread (again, no experience with it, but I think it is
specially popular on clusters; in that case, though, it is not limited
to thread-based implementation).

David


From strawman at astraw.com  Thu Feb 12 06:39:59 2009
From: strawman at astraw.com (Andrew Straw)
Date: Thu, 12 Feb 2009 03:39:59 -0800
Subject: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0;
 Ubuntu packages
In-Reply-To: <499402BF.60500@ar.media.kyoto-u.ac.jp>
References: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>	<db6b5ecc0902111511y7d9cb105x6283d628e6473568@mail.gmail.com>	<5b8d13220902111817q9bd42c0r8c42f6307ca1b318@mail.gmail.com>	<db6b5ecc0902112213m6a49ba30w5faefdc0971b3d1a@mail.gmail.com>	<4993D8A6.5080107@astraw.com>	<4993D6E1.5080803@ar.media.kyoto-u.ac.jp>	<4993FF9D.2010805@astraw.com>
	<499402BF.60500@ar.media.kyoto-u.ac.jp>
Message-ID: <49940A8F.4080008@astraw.com>

OK, I think you're concerned about compatibility of Python extensions
using fortran. We don't use any (that I know of), so I'm going to stop
worrying about this and upload .debs from your .dsc (or very close) to
my repository...

...except for one last question: If Hardy uses the g77 ABI but I'm
building scipy with gfortran, shouldn't there be an ABI issue with my
ATLAS? Shouldn't I get lots of test failures with scipy? I don't.

David Cournapeau wrote:
> My main rationale to provide PPA is to avoid the never-ending queue of
> emails about missing symbols, etc... because I am tired of it, because
> it gives a bad mouth taste to users, because I would like to deal with
> more interesting issues.
>   
Well, I appreciate you doing that. Packaging is a thankless job... and
when everything works on your own computer, it's hard to work up the
motivation to make it work on others'.
>> To understand your statement about identical, I will operationally
>> define "identical" for .debs to mean that they were built from the same
>> .dsc.
>>     
>
> I has an even larger definition of identical: same control/rules/patches
> is identical for me.
>   
This is the only other point I want to make: if you're building from the
same .dsc, it means you're using the same control/rules/patches. (That's
why I brought up the checksums and the signatures.)



From david at ar.media.kyoto-u.ac.jp  Thu Feb 12 06:29:55 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Feb 2009 20:29:55 +0900
Subject: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0;
 Ubuntu packages
In-Reply-To: <49940A8F.4080008@astraw.com>
References: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>	<db6b5ecc0902111511y7d9cb105x6283d628e6473568@mail.gmail.com>	<5b8d13220902111817q9bd42c0r8c42f6307ca1b318@mail.gmail.com>	<db6b5ecc0902112213m6a49ba30w5faefdc0971b3d1a@mail.gmail.com>	<4993D8A6.5080107@astraw.com>	<4993D6E1.5080803@ar.media.kyoto-u.ac.jp>	<4993FF9D.2010805@astraw.com>	<499402BF.60500@ar.media.kyoto-u.ac.jp>
	<49940A8F.4080008@astraw.com>
Message-ID: <49940833.2060501@ar.media.kyoto-u.ac.jp>

Andrew Straw wrote:
> OK, I think you're concerned about compatibility of Python extensions
> using fortran. We don't use any (that I know of), so I'm going to stop
> worrying about this and upload .debs from your .dsc (or very close) to
> my repository...
>
> ...except for one last question: If Hardy uses the g77 ABI but I'm
> building scipy with gfortran, shouldn't there be an ABI issue with my
> ATLAS? Shouldn't I get lots of test failures with scipy? I don't.
>   

Not for hardy, because Hardy has the transitional packages (e.g. it has
both gfortran and g77 ABI packages: libatlas 3g* vs atlas3*). But this
wouldn't work for earlier distributions.

> Well, I appreciate you doing that. Packaging is a thankless job...

Reducing the number of support emails is enough for me as an incentive :)

> This is the only other point I want to make: if you're building from the
> same .dsc, it means you're using the same control/rules/patches. (That's
> why I brought up the checksums and the signatures.)
>   

Yes, the problem is not how to avoid different packages from the .dsc,
but how to conditionally defines some sections in the control file and
rule files,

David


From pav at iki.fi  Thu Feb 12 06:50:26 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 12 Feb 2009 11:50:26 +0000 (UTC)
Subject: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0;
	Ubuntu packages
References: <5b8d13220902110446x82e25a9ifb11bb563e468313@mail.gmail.com>
	<db6b5ecc0902111511y7d9cb105x6283d628e6473568@mail.gmail.com>
	<5b8d13220902111817q9bd42c0r8c42f6307ca1b318@mail.gmail.com>
	<db6b5ecc0902112213m6a49ba30w5faefdc0971b3d1a@mail.gmail.com>
	<4993D8A6.5080107@astraw.com> <4993D6E1.5080803@ar.media.kyoto-u.ac.jp>
	<4993FF9D.2010805@astraw.com> <499402BF.60500@ar.media.kyoto-u.ac.jp>
	<49940A8F.4080008@astraw.com>
Message-ID: <gn12e2$r8n$1@ger.gmane.org>

Thu, 12 Feb 2009 03:39:59 -0800, Andrew Straw wrote:
[clip]
> ...except for one last question: If Hardy uses the g77 ABI but I'm
> building scipy with gfortran, shouldn't there be an ABI issue with my
> ATLAS? Shouldn't I get lots of test failures with scipy? I don't.

On Debian Etch you get mysterious failures if you do this, see for 
example:

	http://scipy.org/scipy/scipy/ticket/815

So in general, avoiding mixing g77 and gfortran seems to be necessary for 
evading problems. Perhaps this depends on the specific g77 and gfortran 
versions, but it's easier just not to go there.

-- 
Pauli Virtanen



From gregor.thalhammer at gmail.com  Thu Feb 12 06:58:57 2009
From: gregor.thalhammer at gmail.com (Gregor Thalhammer)
Date: Thu, 12 Feb 2009 12:58:57 +0100
Subject: [Numpy-discussion] numexpr and numpy windows binaries built with MKL
Message-ID: <49940F01.4050503@googlemail.com>

Hi all,

as Francesc announced, the latest release of Numexpr 1.2 can be built 
with Intels Math Kernel Library, which gives a BIGbig increase in 
performance. Now the questions: Could somebody provide binaries for 
Windows of Numexpr, linked with Intels MKL? I know, there is the license 
problem. The same question has been discussed for numpy. Is it still 
true that Enthought is considering to provide binaries based on MKL, see 
https://svn.enthought.com/epd/ticket/216  ? 
<https://svn.enthought.com/epd/ticket/216>

Gregor



From sturla at molden.no  Thu Feb 12 07:03:53 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 12 Feb 2009 13:03:53 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <4993BE9B.9060709@ar.media.kyoto-u.ac.jp>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>	<20090212062636.GD30330@phare.normalesup.org>
	<4993BE9B.9060709@ar.media.kyoto-u.ac.jp>
Message-ID: <49941029.1030901@molden.no>

On 2/12/2009 7:15 AM, David Cournapeau wrote:

> Since openmp also exists on windows, I doubt that it is required that
> openmp uses pthread :)

On Windows, MSVC uses Win32 threads and GCC (Cygwin and MinGW) uses 
pthreads. If you use OpenMP with MinGW, the executable becomes dependent 
on pthreadGC2.dll (an LGPL library from Redhat).

S.M.





From sturla at molden.no  Thu Feb 12 07:44:25 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 12 Feb 2009 13:44:25 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <4993FA61.7010000@student.matnat.uio.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<4993FA61.7010000@student.matnat.uio.no>
Message-ID: <499419A9.4020105@molden.no>

On 2/12/2009 11:30 AM, Dag Sverre Seljebotn wrote:

> It would be interesting to see how a spec would look for integrating 
> OpenMP natively into Cython for these kinds of purposes. Cython is still 
> flexible as a language after all. Avoiding language bloat is also 
> important, but it is difficult to know what kind of balance can be 
> struck before some kind of spec is worked out. Has anyone managed to use 
> OpenMP with Cython code in a nice way already?

Here is an example of SciPy's ckdtree.pyx modified to use OpenMP. It is 
basically twice as fast on my dual-core laptop. But as Cython mangles 
the names in the C code, I had to put the OpenMP part in a separate C file.

OpenMP does not need to be a aprt of the Cython language. It can be 
special comments in the code as in Fortran. After all, "#pragma omp 
parallel" is a comment in Cython.


Sturla Molden

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: parallel_queries.c
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090212/56b9f764/attachment.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ckdtree.pyx
Type: /
Size: 22235 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090212/56b9f764/attachment.bin>

From faltet at pytables.org  Thu Feb 12 07:47:34 2009
From: faltet at pytables.org (Francesc Alted)
Date: Thu, 12 Feb 2009 13:47:34 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <49940952.6060208@student.matnat.uio.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<200902121216.20222.faltet@pytables.org>
	<49940952.6060208@student.matnat.uio.no>
Message-ID: <200902121347.35130.faltet@pytables.org>

A Thursday 12 February 2009, Dag Sverre Seljebotn escrigu?:
> FYI, I am one of the core Cython developers and can make such
> modifications in Cython itself as long as there's consensus on how it
> should look on the Cython mailing list.  My problem is that I don't
> really know OpenMP and have little experience with it, so I'm not the
> best person for creating a draft for how such high-level OpenMP
> constructs should look like in Cython.

I don't know OpenMP enough neither, but I'd say that in this list there 
could be some people that could help.

At any rate, I really like the OpenMP approach and prefer to have 
support for it in Cython much better than threading, MPI or whatever.  
But the thing is: is OpenMP stable, mature enough for allow using it in 
most of common platforms?  I think that recent GCC compilers support 
the latest incarnation (3.0) of the standard, but IIRC, MSVC 2008 only 
supports OpenMP 2.5 specification.  I'm not sure how this would affect 
a possible implementation in Cython, tough.

Cheers,

-- 
Francesc Alted


From faltet at pytables.org  Thu Feb 12 07:50:44 2009
From: faltet at pytables.org (Francesc Alted)
Date: Thu, 12 Feb 2009 13:50:44 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <499419A9.4020105@molden.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<4993FA61.7010000@student.matnat.uio.no>
	<499419A9.4020105@molden.no>
Message-ID: <200902121350.45045.faltet@pytables.org>

A Thursday 12 February 2009, Sturla Molden escrigu?:
> OpenMP does not need to be a aprt of the Cython language. It can be
> special comments in the code as in Fortran. After all, "#pragma omp
> parallel" is a comment in Cython.

Hey!  That's very nice to know.  We already have OpenMP support in 
Cython for free (or apparently it seems so :-)

-- 
Francesc Alted


From sturla at molden.no  Thu Feb 12 07:57:10 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 12 Feb 2009 13:57:10 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <499405FD.60602@ar.media.kyoto-u.ac.jp>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>	<49940291.4050906@googlemail.com>
	<499405FD.60602@ar.media.kyoto-u.ac.jp>
Message-ID: <49941CA6.3090506@molden.no>

On 2/12/2009 12:20 PM, David Cournapeau wrote:

> It does if you have access to the parallel toolbox I mentioned earlier
> in this thread (again, no experience with it, but I think it is
> specially popular on clusters; in that case, though, it is not limited
> to thread-based implementation).

As has been mentioned, Matlab is a safer language for parallel computing 
  as arrays are immutable. There is almost no need for synchronization 
of any sort, except barriers.

Maybe it is time to implement an immutable ndarray subclass?

With immutable arrays we can also avoid making temporary arrays in 
expressions like y = a*b + c. y just gets an expression and three 
immutable buffers. And then numexpr (or something like it) can take care 
of the rest.

As for Matlab, I have noticed that they are experimenting with CUDA now, 
to use nvidia's processors for hardware-acceleration. As even modest 
GPUs can yield hundreds of gigaflops, that is going to be hard to match 
(unless we make an ndarray that uses the GPU). But again, as the 
performance of GPUs comes from massive multithreading, immutability may 
be the key here as well.

Sturla Molden











From david at ar.media.kyoto-u.ac.jp  Thu Feb 12 07:48:58 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Feb 2009 21:48:58 +0900
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <49941CA6.3090506@molden.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>	<49940291.4050906@googlemail.com>	<499405FD.60602@ar.media.kyoto-u.ac.jp>
	<49941CA6.3090506@molden.no>
Message-ID: <49941ABA.1040509@ar.media.kyoto-u.ac.jp>

Sturla Molden wrote:
> On 2/12/2009 12:20 PM, David Cournapeau wrote:
>
>   
>> It does if you have access to the parallel toolbox I mentioned earlier
>> in this thread (again, no experience with it, but I think it is
>> specially popular on clusters; in that case, though, it is not limited
>> to thread-based implementation).
>>     
>
> As has been mentioned, Matlab is a safer language for parallel computing 
>   as arrays are immutable. There is almost no need for synchronization 
> of any sort, except barriers.
>
> Maybe it is time to implement an immutable ndarray subclass?
>   

Is it possible at all ? I mean immutable array as a subclass of a
mutable array ? I too would like some immutable arrays  but this sounds
like an enormous task (as much as I like numpy, I find numpy sometimes
harder to use than matlab for some problems because of the immutability
- may be my own limitation, though),

> As for Matlab, I have noticed that they are experimenting with CUDA now, 
> to use nvidia's processors for hardware-acceleration.

Labview too; there was a small boost from NI at last ICASSP, and they
were demoing some GPU based operations (mostly blas/lapack/fft from what
I understood, the easy stuff).

cheers,

David


From sturla at molden.no  Thu Feb 12 08:11:43 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 12 Feb 2009 14:11:43 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <200902121350.45045.faltet@pytables.org>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<4993FA61.7010000@student.matnat.uio.no>	<499419A9.4020105@molden.no>
	<200902121350.45045.faltet@pytables.org>
Message-ID: <4994200F.4010200@molden.no>

On 2/12/2009 1:50 PM, Francesc Alted wrote:

> Hey!  That's very nice to know.  We already have OpenMP support in 
> Cython for free (or apparently it seems so :-)

Not we don't, as variable names are different in C and Cython. But 
adding support for OpenMP would not bloat the Cython language.

Cython must exhange the variable names and leave the comment in C as a 
pragma to the C compiler.

IMHO, OpenMP is the easiest way to use parallel computing in scientific 
projects. It is much easier than manually fiddling with threads, 
processes, MPI, etc. Just write code as you normally would, debug and 
verify. Then you spend five minutes inserting pragmas to the compiler, 
and et voil? you have a parallel program. The same code will then 
compile and run correctly for parallel or sequential execution, 
depending on a compiler switch (-fopenmp). You get load balancing for 
free, as that is built into OpenMP. OpenMPs API is so small that it just 
takes 10 minutes to learn.

OpenMP currently works on SMPs (e.g. multicore CPUs), but there is work 
going on to port it to clusters as well.


S.M.


From david at ar.media.kyoto-u.ac.jp  Thu Feb 12 08:01:43 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Feb 2009 22:01:43 +0900
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <200902121347.35130.faltet@pytables.org>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<200902121216.20222.faltet@pytables.org>	<49940952.6060208@student.matnat.uio.no>
	<200902121347.35130.faltet@pytables.org>
Message-ID: <49941DB7.9020402@ar.media.kyoto-u.ac.jp>

Francesc Alted wrote:
> I don't know OpenMP enough neither, but I'd say that in this list there 
> could be some people that could help.
>
> At any rate, I really like the OpenMP approach and prefer to have 
> support for it in Cython much better than threading, MPI or whatever.  
> But the thing is: is OpenMP stable, mature enough for allow using it in 
> most of common platforms?  I think that recent GCC compilers support 
> the latest incarnation (3.0) of the standard

The yet unreleased gcc 4.4 will supports 3.0, and gcc 4.3 supports a
subset of 3.0, I believe:

http://openmp.org/wp/2008/11/openmp-30-status/

cheers,

David


From michael.abshoff at googlemail.com  Thu Feb 12 08:19:42 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Thu, 12 Feb 2009 05:19:42 -0800
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <49941CA6.3090506@molden.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>	<49940291.4050906@googlemail.com>	<499405FD.60602@ar.media.kyoto-u.ac.jp>
	<49941CA6.3090506@molden.no>
Message-ID: <499421EE.60208@gmail.com>

Sturla Molden wrote:
> On 2/12/2009 12:20 PM, David Cournapeau wrote:

Hi,

>> It does if you have access to the parallel toolbox I mentioned earlier
>> in this thread (again, no experience with it, but I think it is
>> specially popular on clusters; in that case, though, it is not limited
>> to thread-based implementation).
> 
> As has been mentioned, Matlab is a safer language for parallel computing 
>   as arrays are immutable. There is almost no need for synchronization 
> of any sort, except barriers.
> 
> Maybe it is time to implement an immutable ndarray subclass?
> 
> With immutable arrays we can also avoid making temporary arrays in 
> expressions like y = a*b + c. y just gets an expression and three 
> immutable buffers. And then numexpr (or something like it) can take care 
> of the rest.
> 
> As for Matlab, I have noticed that they are experimenting with CUDA now, 
> to use nvidia's processors for hardware-acceleration. As even modest 
> GPUs can yield hundreds of gigaflops, 

No even close. The current generation peaks at around 1.2 TFlops single 
precision, 280 GFlops double precision for ATI's hardware. The main 
problem with those numbers is that the memory on the graphics card 
cannot feed the data fast enough into the GPU to achieve theoretical 
peak. So those hundreds of GFlops are pure marketing :)

So in reality you might get anywhere from 20% to 60% (if you are lucky) 
locally before accounting for transfers from main memory to GPU memory 
and so on. Given that recent Intel CPUs give you about 7 to 11 Glops 
Double per core and libraries like ATLAS give you that performance today 
without the need to jump through hoops these number start to look a lot 
less impressive.

And Nvidia's number are lower than ATI's. NVidia's programming solution 
is much more advanced and rounded out compared to ATi's which is largely 
in closed beta. OpenCL is mostly vaporware at this point.

 > that is going to be hard to match
> (unless we make an ndarray that uses the GPU). But again, as the 
> performance of GPUs comes from massive multithreading, immutability may 
> be the key here as well.

I have a K10 system with two Tesla C1060 GPUs to play with and have 
thought about adding CUDABlas support to Numpy/Scipy, but it hasn't been 
a priority for me. My main interest here is finite field arithmetic by 
making FFPack via LinBox use CUDABlas. If anyone wants an account to 
make numpy/scipy optionally use CUDABlas feel free to ping me off list 
and I can set you up.

> Sturla Molden

Cheers,

Michael

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From dagss at student.matnat.uio.no  Thu Feb 12 08:43:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 12 Feb 2009 14:43:12 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <4994200F.4010200@molden.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<4993FA61.7010000@student.matnat.uio.no>	<499419A9.4020105@molden.no>	<200902121350.45045.faltet@pytables.org>
	<4994200F.4010200@molden.no>
Message-ID: <49942770.5080103@student.matnat.uio.no>

Sturla Molden wrote:
> On 2/12/2009 1:50 PM, Francesc Alted wrote:
>
>   
>> Hey!  That's very nice to know.  We already have OpenMP support in 
>> Cython for free (or apparently it seems so :-)
>>     
>
> Not we don't, as variable names are different in C and Cython. But 
> adding support for OpenMP would not bloat the Cython language.
>   
What I meant was "bloat the Cython compiler", i.e. Cython has to know 
about OpenMP and there must be OpenMP-specific code in the compiler. The 
grammar etc. wouldn't change.

But I think I'll be able to lobby it in, given a good spec and some time 
to implement it (about a day of work, so I think this is likely to 
happen within a year).

As Cython must understand what is going on anyway, having things like 
"with cython.openmp.parallell" is actually a lot easier to implement in 
Cython than using comments/pragmas, and looks nicer too IMO. (It means 
that only code generation and not the parser must be modified.)

Dag Sverre


From matthieu.brucher at gmail.com  Thu Feb 12 08:50:52 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 12 Feb 2009 14:50:52 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0902120550s23b9d3d4nd12e8b10d3d4e84b@mail.gmail.com>

> I am curious: would you know what would be different in numpy's case
> compared to matlab array model concerning locks ? Matlab, up to
> recently, only spreads BLAS/LAPACK on multi-cores, but since matlab 7.3
> (or 7.4), it also uses multicore for mathematical functions (cos,
> etc...). So at least in matlab's model, it looks like it can be useful.
> I understand that numpy's model is more flexible (I don't think strided
> arrays can overlap in matlab for example, at least not from what you can
> see from the public API).
>
> cheers,

And the 'more flexible' is one of the biggest drawback. It's one of
the reasons Fortran is so fast compared to C: you can't have pointer
aliases, so you will be slower. With Numpy, it is the same.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From faltet at pytables.org  Thu Feb 12 08:51:42 2009
From: faltet at pytables.org (Francesc Alted)
Date: Thu, 12 Feb 2009 14:51:42 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <4994200F.4010200@molden.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<200902121350.45045.faltet@pytables.org>
	<4994200F.4010200@molden.no>
Message-ID: <200902121451.43160.faltet@pytables.org>

A Thursday 12 February 2009, Sturla Molden escrigu?:
> On 2/12/2009 1:50 PM, Francesc Alted wrote:
> > Hey!  That's very nice to know.  We already have OpenMP support in
> > Cython for free (or apparently it seems so :-)
>
> Not we don't, as variable names are different in C and Cython. But
> adding support for OpenMP would not bloat the Cython language.
>
> Cython must exhange the variable names and leave the comment in C as
> a pragma to the C compiler.

Ah, I see.  I've just had a look at the code and seen this:

#pragma omp parallel for private(c) schedule(guided) 
num_threads(nthreads)

and I thought that the support was there.  But obviously name mangling 
is the problem.

>
> IMHO, OpenMP is the easiest way to use parallel computing in
> scientific projects. It is much easier than manually fiddling with
> threads, processes, MPI, etc. Just write code as you normally would,
> debug and verify. Then you spend five minutes inserting pragmas to
> the compiler, and et voil? you have a parallel program. The same code
> will then compile and run correctly for parallel or sequential
> execution, depending on a compiler switch (-fopenmp). You get load
> balancing for free, as that is built into OpenMP. OpenMPs API is so
> small that it just takes 10 minutes to learn.

Well, ten minutes if you are *already* used to parallelism ;-)  But I 
agree, it is, by far, the easiest path to it.

> OpenMP currently works on SMPs (e.g. multicore CPUs), but there is
> work going on to port it to clusters as well.

That's good to know.

-- 
Francesc Alted


From matthieu.brucher at gmail.com  Thu Feb 12 08:48:44 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 12 Feb 2009 14:48:44 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <20090212062636.GD30330@phare.normalesup.org>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<20090212062636.GD30330@phare.normalesup.org>
Message-ID: <e76aa17f0902120548l1859ca13x88d2bd637f6bac05@mail.gmail.com>

Yes, it is. You have to link against pthread (at least with Linux ;))

You have to write a single parallel region if you don't want this
overhead (which is not possible with Python).

Matthieu

2009/2/12 Gael Varoquaux <gael.varoquaux at normalesup.org>:
> On Wed, Feb 11, 2009 at 11:52:40PM -0600, Robert Kern wrote:
>> > This seem like pretty heavy solutions though.
>
>> >From a programmer's perspective, it seems to me like OpenMP is a muck
>> >lighter weight solution than pthreads.
>
> >From a programmer's perspective, because, IMHO, openmp is implemented
> using pthreads. I do have difficulties verifying this on the web, but
> documents I find hint to that.
>
> Ga?l
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From matthieu.brucher at gmail.com  Thu Feb 12 08:52:11 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 12 Feb 2009 14:52:11 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <4993C6A8.6030808@ar.media.kyoto-u.ac.jp>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
	<4993C6A8.6030808@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0902120552o20a26933t64989c3e884dc1e8@mail.gmail.com>

> No - I have never seen deep explanation of the matlab model. The C api
> is so small that it is hard to deduce anything from it (except that the
> memory handling is not ref-counting-based, I don't know if it matters
> for our discussion of speeding up ufunc). I would guess that since two
> arrays cannot share data (COW-based), lock handling may be easier to
> deal with ? I am not really familiar with multi-thread programming (my
> only limited experience is for soft real-time programming for audio
> processing, where the issues are totally different, since latency
> matters as much if not more than throughput).

It's not even a matter of multithread programming, in mono-core
programming, the same issue can arise.

>> True, but I would be happy to just have a fast C based threadpool
>> implentation I could use in low level Cython based loops.
>
> Matlab has a parallel toolbox to do this kind of things in matlab (I
> don't know in C). I don't know anything about it, nor do I know if that
> can be applied in any way to python/numpy's case:
>
> http://www.mathworks.com/products/parallel-computing/
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From matthieu.brucher at gmail.com  Thu Feb 12 08:58:40 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 12 Feb 2009 14:58:40 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <4994200F.4010200@molden.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<4993FA61.7010000@student.matnat.uio.no> <499419A9.4020105@molden.no>
	<200902121350.45045.faltet@pytables.org> <4994200F.4010200@molden.no>
Message-ID: <e76aa17f0902120558j29e7961am4b6f3467997a0b7a@mail.gmail.com>

2009/2/12 Sturla Molden <sturla at molden.no>:
> On 2/12/2009 1:50 PM, Francesc Alted wrote:
>
>> Hey!  That's very nice to know.  We already have OpenMP support in
>> Cython for free (or apparently it seems so :-)
>
> Not we don't, as variable names are different in C and Cython. But
> adding support for OpenMP would not bloat the Cython language.
>
> Cython must exhange the variable names and leave the comment in C as a
> pragma to the C compiler.
>
> IMHO, OpenMP is the easiest way to use parallel computing in scientific
> projects. It is much easier than manually fiddling with threads,
> processes, MPI, etc. Just write code as you normally would, debug and
> verify. Then you spend five minutes inserting pragmas to the compiler,
> and et voil? you have a parallel program. The same code will then
> compile and run correctly for parallel or sequential execution,
> depending on a compiler switch (-fopenmp). You get load balancing for
> free, as that is built into OpenMP. OpenMPs API is so small that it just
> takes 10 minutes to learn.
>
> OpenMP currently works on SMPs (e.g. multicore CPUs), but there is work
> going on to port it to clusters as well.

In this matter, I think OpenMP will never be used. Besides, OpenMP
doesn't even support an unsigned int for a parallel loop variable.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From matthieu.brucher at gmail.com  Thu Feb 12 08:55:20 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 12 Feb 2009 14:55:20 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <49940291.4050906@googlemail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
	<49940291.4050906@googlemail.com>
Message-ID: <e76aa17f0902120555y71487495hefd23d855d06e11a@mail.gmail.com>

2009/2/12 Gregor Thalhammer <gregor.thalhammer at gmail.com>:
> Brian Granger schrieb:
>>> I am curious: would you know what would be different in numpy's case
>>> compared to matlab array model concerning locks ? Matlab, up to
>>> recently, only spreads BLAS/LAPACK on multi-cores, but since matlab 7.3
>>> (or 7.4), it also uses multicore for mathematical functions (cos,
>>> etc...). So at least in matlab's model, it looks like it can be useful.
>>>
>>
>> Good point.  Is it possible to tell what array size it switches over
>> to using multiple threads?  Also, do you happen to iknow how Matlab is
>> doing this?
>>
>>
> Recent Matlab versions use Intels Math Kernel Library, which performs
> automatic multi-threading - also for mathematical functions like sin
> etc, but not for  addition, multiplication etc. It seems to me Matlab
> itself does not take care of multi-threading. On
> http://www.intel.com/software/products/mkl/data/vml/functions/_listfunc.html
> you can have a look at the performance data of the MKL vectorized math
> functions. Around a vector length between 100-1000, depending on which
> function, precision, cpu architecture, they switch to multi-threading.

For BLAS level 3, the MKL is parallelized (so matrix multiplication is).

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From david at ar.media.kyoto-u.ac.jp  Thu Feb 12 09:02:12 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Feb 2009 23:02:12 +0900
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <e76aa17f0902120552o20a26933t64989c3e884dc1e8@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>	<4993C6A8.6030808@ar.media.kyoto-u.ac.jp>
	<e76aa17f0902120552o20a26933t64989c3e884dc1e8@mail.gmail.com>
Message-ID: <49942BE4.3080706@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
>> No - I have never seen deep explanation of the matlab model. The C api
>> is so small that it is hard to deduce anything from it (except that the
>> memory handling is not ref-counting-based, I don't know if it matters
>> for our discussion of speeding up ufunc). I would guess that since two
>> arrays cannot share data (COW-based), lock handling may be easier to
>> deal with ? I am not really familiar with multi-thread programming (my
>> only limited experience is for soft real-time programming for audio
>> processing, where the issues are totally different, since latency
>> matters as much if not more than throughput).
>>     
>
> It's not even a matter of multithread programming, in mono-core
> programming, the same issue can arise.
>   

Which issue ?

David


From bernhard.voigt at gmail.com  Thu Feb 12 09:20:03 2009
From: bernhard.voigt at gmail.com (bernhard.voigt at gmail.com)
Date: Thu, 12 Feb 2009 06:20:03 -0800 (PST)
Subject: [Numpy-discussion] Outer join ?
In-Reply-To: <7cc4bc500902112124s6a00d59fw78ea94f71ae4bb82@mail.gmail.com>
References: <7cc4bc500902112124s6a00d59fw78ea94f71ae4bb82@mail.gmail.com>
Message-ID: <df18e74a-543b-49b8-bfb0-30aa81eebd9b@j1g2000yqi.googlegroups.com>

You might consider the groupby from the itertools module.

Do you have two keys only? I would prefer grouping on the first
column. For grouby you need to sort the array after the first column
then.

from itertools import groupby
a.sort(order='col1')

# target array: first col are unique dates, second col values for
key1, third col values for key2
data = numpy.zeros(len(unique(a['col1'])), dtype=dict(names=['dates',
'key1', 'key2'] , types=[long, float, float]))

for i, (date, items) in enumerate(groupby(a, lambda item: item
['col1'])):
    data[i][dates] = date
    for col1, col2, col3 in items:
        data[i][col2] = col3

Hope this works! Bernhard

On Feb 12, 6:24?am, A B <python6... at gmail.com> wrote:
> Hi,
>
> I have the following data structure:
>
> col1 | col2 | col3
>
> 20080101|key1|4
> 20080201|key1|6
> 20080301|key1|5
> 20080301|key2|3.4
> 20080601|key2|5.6
>
> For each key in the second column, I would like to create an array
> where for all unique values in the first column, there will be either
> a value or zero if there is no data available. Like so:
>
> # 20080101, 20080201, 20080301, 20080601
>
> key1 - 4, 6, 5, ? ?0
> key2 - 0, 0, 3.4, 5.6
>
> Ideally, the results would end up in a 2d array.
>
> What's the most efficient way to accomplish this? Currently, I am
> getting a list of uniq col1 and col2 values into separate variables,
> then looping through each unique value in col2
>
> a = loadtxt(...)
>
> dates = unique(a[:]['col1'])
> keys = unique(a[:]['col2'])
>
> for key in keys:
> ? ? b = a[where(a[:]['col2'] == key)]
> ? ? ???
>
> Thanks in advance.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion


From david at ar.media.kyoto-u.ac.jp  Thu Feb 12 09:04:04 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Feb 2009 23:04:04 +0900
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <e76aa17f0902120555y71487495hefd23d855d06e11a@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>	<49940291.4050906@googlemail.com>
	<e76aa17f0902120555y71487495hefd23d855d06e11a@mail.gmail.com>
Message-ID: <49942C54.7000605@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
>
> For BLAS level 3, the MKL is parallelized (so matrix multiplication is).
>   

Same for ATLAS: thread support is one focus in the 3.9 serie, currently
in development. I have never used it, I don't know how it compare to the
MKL,

David


From matthieu.brucher at gmail.com  Thu Feb 12 09:24:46 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 12 Feb 2009 15:24:46 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <49942BE4.3080706@ar.media.kyoto-u.ac.jp>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
	<4993C6A8.6030808@ar.media.kyoto-u.ac.jp>
	<e76aa17f0902120552o20a26933t64989c3e884dc1e8@mail.gmail.com>
	<49942BE4.3080706@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0902120624v21510906w958403f40473c8e2@mail.gmail.com>

2009/2/12 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
> Matthieu Brucher wrote:
>>> No - I have never seen deep explanation of the matlab model. The C api
>>> is so small that it is hard to deduce anything from it (except that the
>>> memory handling is not ref-counting-based, I don't know if it matters
>>> for our discussion of speeding up ufunc). I would guess that since two
>>> arrays cannot share data (COW-based), lock handling may be easier to
>>> deal with ? I am not really familiar with multi-thread programming (my
>>> only limited experience is for soft real-time programming for audio
>>> processing, where the issues are totally different, since latency
>>> matters as much if not more than throughput).
>>>
>>
>> It's not even a matter of multithread programming, in mono-core
>> programming, the same issue can arise.
>>
>
> Which issue ?

Sorry, I was refering to my last mail, but I sent so many in 5 minuts ;)
In C, if you have to arrays (two pointers), the compiler can't make
aggressive optimizations because they may intersect. With Fortran,
this is not possible. In this matter, Numpy behaves like C (everyone
heard about the different a[indices] += a[other_intersecting_indices]
issues), Matlab is more like Fortran.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From sturla at molden.no  Thu Feb 12 09:27:51 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 12 Feb 2009 15:27:51 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <49940952.6060208@student.matnat.uio.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<4993FA61.7010000@student.matnat.uio.no>	<200902121216.20222.faltet@pytables.org>
	<49940952.6060208@student.matnat.uio.no>
Message-ID: <499431E7.7000501@molden.no>

On 2/12/2009 12:34 PM, Dag Sverre Seljebotn wrote:

> FYI, I am one of the core Cython developers and can make such 
> modifications in Cython itself as long as there's consensus on how it 
> should look on the Cython mailing list.  My problem is that I don't 
> really know OpenMP and have little experience with it, so I'm not the 
> best person for creating a draft for how such high-level OpenMP 
> constructs should look like in Cython.

I don't know the Cython internals, but I do know OpenMP. I mostly use it 
with Fortran.

The question is: Should OpenMP be comments in the Cython code (as they 
are in C and Fortran), or should OpenMP be special objects?

As for the GIL: No I don't think nogil should be implied. But Python 
objects should only be allowed as shared variables. Synchronization will 
then be as usual for shared variables in OpenMP (#pragma omp critical).

Here is my suggestion for syntax. If you just follow a consistent 
translation scheme, you don't need to know OpenMP in details. Here is a 
suggestion:


with openmp('parallel for', argument=iterable, ...):
    --> insert pragma directly above for

with openmp(directive, argument=iterable, ...):
    --> insert pragma and brackets

with openmp('atomic'): --> insert pragma directly

openmp('barrier') --> insert pragma directly


This by the way covers all of OpenMP. This is how it should translate:


with openmp('parallel for', private=(i,), shared=(n,),
     schedule='dynamic'):

     for i in range(n):
        pass

Compiles to:

#pragma omp parallel for \
private(i) \
shared(n) \
schedule(dynamic)
for(i=0; i<n; i++) {
   /* whatever */
}


with openmp('parallel sections',
      reduction=('+',k), private=(i,j)):

     with openmp('section'):
         i = foobar()

     with openmp('section'):
         j = foobar()

     k = i + j


Compiles to:

#pragma omp parallel sections\
reduction(+:k)\
private(i,j)
{
     #pragma omp section
     {
        i = foobar();
     }

     #pragma omp section
     {
        j = foobar();
     }

     k = i+j;
}


With Python objects, the programmer must synchronize access:


with openmp('parallel for', shared=(pyobj,n), private=(i,)):
     for i in range(n):
         with openmp('critical'):
             pyobj += i


#pragma omp parallel for \
shared(pyobj,n) \
private(i)
for (i=0; i<n; i++) {
    #pragma omp critical
    {
       pyobj += i;
    }
}


Atomic and barriers:


with openmp('atomic'): i += j

#pragma omp atomic
i += j;


with openmp('parallel for', default='private', shared(n,)):
    for i in range(n):
       openmp('barrier')


#pragma omp parallel for \
default(private)\
shared(n)\
for (i=0; i<n; i++)
{
    #pragma omp barrier
}


That is my suggestion. Easy to implement as you don't need to learn 
OpenMP first (not that it is difficult).


Sturla Molden








































From david at ar.media.kyoto-u.ac.jp  Thu Feb 12 09:15:44 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Feb 2009 23:15:44 +0900
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <e76aa17f0902120624v21510906w958403f40473c8e2@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>	<4993C6A8.6030808@ar.media.kyoto-u.ac.jp>	<e76aa17f0902120552o20a26933t64989c3e884dc1e8@mail.gmail.com>	<49942BE4.3080706@ar.media.kyoto-u.ac.jp>
	<e76aa17f0902120624v21510906w958403f40473c8e2@mail.gmail.com>
Message-ID: <49942F10.9010501@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
>
> Sorry, I was refering to my last mail, but I sent so many in 5 minuts ;)
> In C, if you have to arrays (two pointers), the compiler can't make
> aggressive optimizations because they may intersect. With Fortran,
> this is not possible. In this matter, Numpy behaves like C (everyone
> heard about the different a[indices] += a[other_intersecting_indices]
> issues), Matlab is more like Fortran.
>   

I think it is hard to know exactly, because we don't know how matlab is
implemented. It is possible to handle special cases for non overlapping
arrays in numpy, once you are in C; I believe there are many codepath
which we could optimize aggressively, using openMP, SSE, etc... It is
just a lot of work to do so manually, so the real problem is how this
can be handled as generally as possible using a small core.

David


From sturla at molden.no  Thu Feb 12 09:34:53 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 12 Feb 2009 15:34:53 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <499419A9.4020105@molden.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<4993FA61.7010000@student.matnat.uio.no>
	<499419A9.4020105@molden.no>
Message-ID: <4994338D.7050003@molden.no>

On 2/12/2009 1:44 PM, Sturla Molden wrote:

> Here is an example of SciPy's ckdtree.pyx modified to use OpenMP.

It seems I managed to post an errorneous C file. :(

S.M.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: parallel_queries.c
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090212/c1635cee/attachment.c>

From michael.abshoff at googlemail.com  Thu Feb 12 09:34:58 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Thu, 12 Feb 2009 06:34:58 -0800
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <49942C54.7000605@ar.media.kyoto-u.ac.jp>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>	<49940291.4050906@googlemail.com>	<e76aa17f0902120555y71487495hefd23d855d06e11a@mail.gmail.com>
	<49942C54.7000605@ar.media.kyoto-u.ac.jp>
Message-ID: <49943392.3070505@gmail.com>

David Cournapeau wrote:
> Matthieu Brucher wrote:
>> For BLAS level 3, the MKL is parallelized (so matrix multiplication is).
>>   

Hi David,

> Same for ATLAS: thread support is one focus in the 3.9 serie, currently
> in development. 

ATLAS has had thread support for a long, long time. The 3.9 series just 
improves it substantially by using affinity when available and removes 
some long standing issues with allocation performance that one had to 
work around before by setting some defines at compile time.

 > I have never used it, I don't know how it compare to the
> MKL,

It does compare quite well and is more or less on par with the latest 
MKL releases in the 3.9 series. 3.8.2 is maybe 10% to 15% slower on i7 
as well as Core2 cores than the MKL. On big advantage of ATLAS is that 
it tends to work when using it with numpy/scipy unlike the Intel MKL 
where one has to work around a bunch of oddities and jump through hoops 
to get it to work. It seems that Intel must rename at least one library 
in each release of the MKL to keep build system maintainers occupied :)

The big disadvantage of ATLAS is that Windows support is currently 
limited to 32 bits, but 3.9.5 and higher have SFU/SUA support, so 64 bit 
support is possible. Clint told me the main issue here was lack of 
access and it isn't too high on his priority list.

> David

Cheers,

Michael

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From matthieu.brucher at gmail.com  Thu Feb 12 10:03:49 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 12 Feb 2009 16:03:49 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <49942F10.9010501@ar.media.kyoto-u.ac.jp>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
	<4993C6A8.6030808@ar.media.kyoto-u.ac.jp>
	<e76aa17f0902120552o20a26933t64989c3e884dc1e8@mail.gmail.com>
	<49942BE4.3080706@ar.media.kyoto-u.ac.jp>
	<e76aa17f0902120624v21510906w958403f40473c8e2@mail.gmail.com>
	<49942F10.9010501@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0902120703t6f92119aic98ebb49a71de8c5@mail.gmail.com>

2009/2/12 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
> Matthieu Brucher wrote:
>>
>> Sorry, I was refering to my last mail, but I sent so many in 5 minuts ;)
>> In C, if you have to arrays (two pointers), the compiler can't make
>> aggressive optimizations because they may intersect. With Fortran,
>> this is not possible. In this matter, Numpy behaves like C (everyone
>> heard about the different a[indices] += a[other_intersecting_indices]
>> issues), Matlab is more like Fortran.
>>
>
> I think it is hard to know exactly, because we don't know how matlab is
> implemented. It is possible to handle special cases for non overlapping
> arrays in numpy, once you are in C; I believe there are many codepath
> which we could optimize aggressively, using openMP, SSE, etc... It is
> just a lot of work to do so manually, so the real problem is how this
> can be handled as generally as possible using a small core.
>
> David

Indeed, Matlab may not benefit from this. At least, COW makes it
possible to optimize agressively. Then, it's a matter of implementing
the loops in C or Fortran (or to use C99 extensions and rely on the
compiler). In C89, you will have absolutely no benefit (because there
are no way you can tell the compiler that there is no aliasing), in
Fortran, it will be optimized correctly.
The problem with optimizing codepath is that you have to detect them.
On the other hand, if you implement a numpy compliant array that
explicitely forbids aliasing (you could add some additional tests to
ensure that the parents are different, or stuff like that), you can
make the compiler optimize the code better.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From wnbell at gmail.com  Thu Feb 12 10:05:36 2009
From: wnbell at gmail.com (Nathan Bell)
Date: Thu, 12 Feb 2009 10:05:36 -0500
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <499421EE.60208@gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
	<49940291.4050906@googlemail.com>
	<499405FD.60602@ar.media.kyoto-u.ac.jp> <49941CA6.3090506@molden.no>
	<499421EE.60208@gmail.com>
Message-ID: <d05265cb0902120705h14633ae5w4de34a2321b33479@mail.gmail.com>

On Thu, Feb 12, 2009 at 8:19 AM, Michael Abshoff
<michael.abshoff at googlemail.com> wrote:
>
> No even close. The current generation peaks at around 1.2 TFlops single
> precision, 280 GFlops double precision for ATI's hardware. The main
> problem with those numbers is that the memory on the graphics card
> cannot feed the data fast enough into the GPU to achieve theoretical
> peak. So those hundreds of GFlops are pure marketing :)
>

If your application is memory bandwidth limited, then yes you're not
likely to see 100s of GFlops anytime soon.  However, compute limited
application can and do achieve 100s of GFlops on GPUs.  Basic
operations like FFTs and (level 3) BLAS are compute limited, as are
the following applications:
http://www.ks.uiuc.edu/Research/gpu/
http://www.dam.brown.edu/scicomp/scg-media/report_files/BrownSC-2008-27.pdf

> So in reality you might get anywhere from 20% to 60% (if you are lucky)
> locally before accounting for transfers from main memory to GPU memory
> and so on. Given that recent Intel CPUs give you about 7 to 11 Glops
> Double per core and libraries like ATLAS give you that performance today
> without the need to jump through hoops these number start to look a lot
> less impressive.

You neglect to mention that CPUs, which have roughly 1/10th the memory
bandwidth of high-end GPUs, are memory bound on the very same
problems.  You will not see 7 to 11 GFLops on a memory bound CPU code
for the same reason you argue that GPUs don't achieve 100s of GFLops
on memory bound GPU codes.

In severely memory bound applications like sparse matrix-vector
multiplication (i.e. A*x for sparse A) the best GPU performance you
can expect is ~10 GFLops on the GPU and ~1 GFLop on the CPU (in double
precision).  We discuss this problem in the following tech report:
http://forums.nvidia.com/index.php?showtopic=83825

It's true that host<->device transfers can be a bottleneck.  In many
cases, the solution is to simply leave the data resident on the GPU.
For instance, you could imagine a variant of ndarray that held a
pointer to a device array.  Of course this requires that the other
expensive parts of your algorithm also execute on the GPU so you're
not shuttling data over the PCIe bus all the time.


Full Disclosure: I'm a researcher at NVIDIA

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From sturla at molden.no  Thu Feb 12 10:19:27 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 12 Feb 2009 16:19:27 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <e76aa17f0902120703t6f92119aic98ebb49a71de8c5@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>	<4993C6A8.6030808@ar.media.kyoto-u.ac.jp>	<e76aa17f0902120552o20a26933t64989c3e884dc1e8@mail.gmail.com>	<49942BE4.3080706@ar.media.kyoto-u.ac.jp>	<e76aa17f0902120624v21510906w958403f40473c8e2@mail.gmail.com>	<49942F10.9010501@ar.media.kyoto-u.ac.jp>
	<e76aa17f0902120703t6f92119aic98ebb49a71de8c5@mail.gmail.com>
Message-ID: <49943DFF.9070502@molden.no>

On 2/12/2009 4:03 PM, Matthieu Brucher wrote:

> In C89, you will have absolutely no benefit (because there
> are no way you can tell the compiler that there is no aliasing), in
> Fortran, it will be optimized correctly.

In ANSI C (aka C89) the effect is achieved using compiler pragmas.

In ISO C (aka C99) there is a 'restrict' qualifier for tagging pointers 
and arrays as unaliased.

Usually a C compiler will optimize better than most Fortran compilers 
when given enough information.

But the merit of Fortran is the ease by which arrays can be manipulated. 
It is almost comparable to Matlab or Python with NumPy, with performance 
close to C. That makes Fortran excellent for scientific programming.

S.M.



From python6009 at gmail.com  Thu Feb 12 10:52:36 2009
From: python6009 at gmail.com (A B)
Date: Thu, 12 Feb 2009 07:52:36 -0800
Subject: [Numpy-discussion] Outer join ?
In-Reply-To: <df18e74a-543b-49b8-bfb0-30aa81eebd9b@j1g2000yqi.googlegroups.com>
References: <7cc4bc500902112124s6a00d59fw78ea94f71ae4bb82@mail.gmail.com>
	<df18e74a-543b-49b8-bfb0-30aa81eebd9b@j1g2000yqi.googlegroups.com>
Message-ID: <7cc4bc500902120752w16b84d52h429c338c4652672d@mail.gmail.com>

This is probably more than I need but I will definitely keep it as
reference. Thank you.

On 2/12/09, bernhard.voigt at gmail.com <bernhard.voigt at gmail.com> wrote:
> You might consider the groupby from the itertools module.
>
> Do you have two keys only? I would prefer grouping on the first
> column. For grouby you need to sort the array after the first column
> then.
>
> from itertools import groupby
> a.sort(order='col1')
>
> # target array: first col are unique dates, second col values for
> key1, third col values for key2
> data = numpy.zeros(len(unique(a['col1'])), dtype=dict(names=['dates',
> 'key1', 'key2'] , types=[long, float, float]))
>
> for i, (date, items) in enumerate(groupby(a, lambda item: item
> ['col1'])):
>     data[i][dates] = date
>     for col1, col2, col3 in items:
>         data[i][col2] = col3
>
> Hope this works! Bernhard
>
> On Feb 12, 6:24 am, A B <python6... at gmail.com> wrote:
>> Hi,
>>
>> I have the following data structure:
>>
>> col1 | col2 | col3
>>
>> 20080101|key1|4
>> 20080201|key1|6
>> 20080301|key1|5
>> 20080301|key2|3.4
>> 20080601|key2|5.6
>>
>> For each key in the second column, I would like to create an array
>> where for all unique values in the first column, there will be either
>> a value or zero if there is no data available. Like so:
>>
>> # 20080101, 20080201, 20080301, 20080601
>>
>> key1 - 4, 6, 5,    0
>> key2 - 0, 0, 3.4, 5.6
>>
>> Ideally, the results would end up in a 2d array.
>>
>> What's the most efficient way to accomplish this? Currently, I am
>> getting a list of uniq col1 and col2 values into separate variables,
>> then looping through each unique value in col2
>>
>> a = loadtxt(...)
>>
>> dates = unique(a[:]['col1'])
>> keys = unique(a[:]['col2'])
>>
>> for key in keys:
>>     b = a[where(a[:]['col2'] == key)]
>>     ???
>>
>> Thanks in advance.
>> ______________________


From michael.abshoff at googlemail.com  Thu Feb 12 10:54:07 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Thu, 12 Feb 2009 07:54:07 -0800
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <d05265cb0902120705h14633ae5w4de34a2321b33479@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>	<49940291.4050906@googlemail.com>	<499405FD.60602@ar.media.kyoto-u.ac.jp>
	<49941CA6.3090506@molden.no>	<499421EE.60208@gmail.com>
	<d05265cb0902120705h14633ae5w4de34a2321b33479@mail.gmail.com>
Message-ID: <4994461F.5030302@gmail.com>

Nathan Bell wrote:
> On Thu, Feb 12, 2009 at 8:19 AM, Michael Abshoff
> <michael.abshoff at googlemail.com> wrote:

Hi,

>> No even close. The current generation peaks at around 1.2 TFlops single
>> precision, 280 GFlops double precision for ATI's hardware. The main
>> problem with those numbers is that the memory on the graphics card
>> cannot feed the data fast enough into the GPU to achieve theoretical
>> peak. So those hundreds of GFlops are pure marketing :)
>>
> 
> If your application is memory bandwidth limited, then yes you're not
> likely to see 100s of GFlops anytime soon.  However, compute limited
> application can and do achieve 100s of GFlops on GPUs.  Basic
> operations like FFTs and (level 3) BLAS are compute limited, as are
> the following applications:
> http://www.ks.uiuc.edu/Research/gpu/
> http://www.dam.brown.edu/scicomp/scg-media/report_files/BrownSC-2008-27.pdf

Yes, certainly. But Sturla implied that some "random consumer GPU" (to 
put a negative spin on it :) could do the above. There also seems to be 
a huge expectation that "porting your code to the GPU" will make it 10 
to 100 times faster. There are cases like that as mentioned above, but 
this only applies to a subset of problems.

Another problem is RAM for many datasets I work with and 512 to 1024 MB 
aren't just plainly cutting it. This means Tesla cards at $1k upward and 
all the sudden we are playing a different game.

9 months ago at the beginning when we started playing with CUDA we took 
a MacBook pro with a decent NVidia card and laughed hard after it become 
clear that its Core2 with either ATLAS or the AccelerateFramework (which 
is more or ATLAS for its BLAS bits) was faster than the build in NVidia 
card with either single or double precision. Surely, this is a consumer 
level laptop GPU, but I did expect more.

>> So in reality you might get anywhere from 20% to 60% (if you are lucky)
>> locally before accounting for transfers from main memory to GPU memory
>> and so on. Given that recent Intel CPUs give you about 7 to 11 Glops
>> Double per core and libraries like ATLAS give you that performance today
>> without the need to jump through hoops these number start to look a lot
>> less impressive.
> 
> You neglect to mention that CPUs, which have roughly 1/10th the memory
> bandwidth of high-end GPUs, are memory bound on the very same
> problems.  You will not see 7 to 11 GFLops on a memory bound CPU code
> for the same reason you argue that GPUs don't achieve 100s of GFLops
> on memory bound GPU codes.

I am seeing 7 to 11 GFLOP per core for matrix matrix multiplies on Intel 
CPUs using Strassen for matrix matrix multiplies. And we did scale out 
linear on 16 core Opterons as well as a 64 core Itanium box using ATLAS 
for BLAS level 3 matrix matrix multiplu. When you have multiple GPUs you 
do not have shared memory architectures (AFAIK the 4 GPU boxen sold by 
NVidia have fast buses between the cards, but aren't ccNUMA or anything 
like that - please correct me if I am wrong).

> In severely memory bound applications like sparse matrix-vector
> multiplication (i.e. A*x for sparse A) the best GPU performance you
> can expect is ~10 GFLops on the GPU and ~1 GFLop on the CPU (in double
> precision).  We discuss this problem in the following tech report:
> http://forums.nvidia.com/index.php?showtopic=83825

Ok, I care about dense operations primarily, but it is interesting to 
see that the GPU fares well on sparse LA.

> It's true that host<->device transfers can be a bottleneck.  In many
> cases, the solution is to simply leave the data resident on the GPU.

Well, that assumes you have enough memory locally for your working set. 
And if not you need to be clever about caching and I did not see any 
code in CUDA that takes care of that job for you. I have seen libraries 
like libflame that claim to do that for you, but I have not played with 
them yet.

> For instance, you could imagine a variant of ndarray that held a
> pointer to a device array.  Of course this requires that the other
> expensive parts of your algorithm also execute on the GPU so you're
> not shuttling data over the PCIe bus all the time.

Absolutely. I think that GPUs can fill a large niche for scientific 
computations, but it is not (yet?) the general purpose CPU it is 
sometimes made out to be.

> 
> Full Disclosure: I'm a researcher at NVIDIA

Cool. Thanks for the links by the way.

As I mentioned we have bought Tesla hardware and are working on getting 
our code to use GPUs for numerical linear algebra, exact linear algebra 
and shortly also things like monte carlo simulation. I do think that the 
GPU is extremely useful for much of the above, but there are plenty of 
programming issues to resolve and a lot of infrastructure code to be 
written before GPU computing becomes ubiquitous. After the last new 
thing I had put my hope in (the Cell CPU) basically turned out to be a 
dud I am hesitant about anything until the code I am running actually 
sees the benefit.

The thing with NVidia I am unhappy about is that CUDA is not free as in 
freedom. I am not a FSF zealot, so I will not try to convince anyone to 
make their software free. Given a choice between OpenCL and CUDA you 
have the lead at the moment because you actually have been shipping a 
working product for more than a year, but I am not so sure that in the 
long term OpenCL won't get people's mindshare. If you look at the 
history of 3D acceleration we started with numerous APIs that we all 
supplanted by OpenGL which then got pushed aside by DirectX.  Anyway, no 
point in ranting here any more ;)

Cheers,

Michael


From python6009 at gmail.com  Thu Feb 12 11:19:05 2009
From: python6009 at gmail.com (A B)
Date: Thu, 12 Feb 2009 08:19:05 -0800
Subject: [Numpy-discussion] Outer join ?
In-Reply-To: <3d375d730902112140j6878d379x6078ddc0ef7e4d70@mail.gmail.com>
References: <7cc4bc500902112124s6a00d59fw78ea94f71ae4bb82@mail.gmail.com>
	<3d375d730902112140j6878d379x6078ddc0ef7e4d70@mail.gmail.com>
Message-ID: <7cc4bc500902120819r59e1747g3aaa94ab445ca751@mail.gmail.com>

On 2/11/09, Robert Kern <robert.kern at gmail.com> wrote:
> On Wed, Feb 11, 2009 at 23:24, A B <python6009 at gmail.com> wrote:
>> Hi,
>>
>> I have the following data structure:
>>
>> col1 | col2 | col3
>>
>> 20080101|key1|4
>> 20080201|key1|6
>> 20080301|key1|5
>> 20080301|key2|3.4
>> 20080601|key2|5.6
>>
>> For each key in the second column, I would like to create an array
>> where for all unique values in the first column, there will be either
>> a value or zero if there is no data available. Like so:
>>
>> # 20080101, 20080201, 20080301, 20080601
>>
>> key1 - 4, 6, 5,    0
>> key2 - 0, 0, 3.4, 5.6
>>
>> Ideally, the results would end up in a 2d array.
>>
>> What's the most efficient way to accomplish this? Currently, I am
>> getting a list of uniq col1 and col2 values into separate variables,
>> then looping through each unique value in col2
>>
>> a = loadtxt(...)
>>
>> dates = unique(a[:]['col1'])
>> keys = unique(a[:]['col2'])
>>
>> for key in keys:
>>    b = a[where(a[:]['col2'] == key)]
>>    ???
>
> Take a look at setmember1d().
>
> --
> Robert Kern
>
Thanks. That's exactly what I need,  but I'm not sure about the next
step after I do

setmember1d(dates, b['date'])

and have the bool arr/mask ... How can I grow b to have 0 values for
the missing keys?


From gael.varoquaux at normalesup.org  Thu Feb 12 11:24:27 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 12 Feb 2009 17:24:27 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <499431E7.7000501@molden.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<4993FA61.7010000@student.matnat.uio.no>
	<200902121216.20222.faltet@pytables.org>
	<49940952.6060208@student.matnat.uio.no>
	<499431E7.7000501@molden.no>
Message-ID: <20090212162427.GC8194@phare.normalesup.org>

On Thu, Feb 12, 2009 at 03:27:51PM +0100, Sturla Molden wrote:
> The question is: Should OpenMP be comments in the Cython code (as they 
> are in C and Fortran), or should OpenMP be special objects?

My two cents: go for cython objects/statements. Not only does code in
comments looks weird and a hack, but also it means to you have to hack
the parser. Finally cython can be used in a valid Python syntax (at least
part of it) thanks to decorators giving the types. Ondrej Certik
implemented to for sympy. Having pragmas in comments would make it hard
to use the vanilla Python parser to read this code, or to do
metaprogramming on the pragmas.

The logic of Python is pretty much that everything is accessible to the
programmer, to be modified, eg for testing. Putting logics in comments
breaks this.

I like the use of the with statement for this purpose.

Ga?l


From rmay31 at gmail.com  Thu Feb 12 11:36:04 2009
From: rmay31 at gmail.com (Ryan May)
Date: Thu, 12 Feb 2009 10:36:04 -0600
Subject: [Numpy-discussion] genloadtxt: dtype=None and unpack=True
In-Reply-To: <97251D70-38B3-4A41-B2EA-26A700C50452@gmail.com>
References: <c7b95a020902112038w4b0fc609n844c10c9931f7c95@mail.gmail.com>
	<97251D70-38B3-4A41-B2EA-26A700C50452@gmail.com>
Message-ID: <c7b95a020902120836p6716d11q45efc18b0f6a0811@mail.gmail.com>

On Wed, Feb 11, 2009 at 10:47 PM, Pierre GM <pgmdevlist at gmail.com> wrote:

>
> On Feb 11, 2009, at 11:38 PM, Ryan May wrote:
>
> > Pierre,
> >
> > I noticed that using dtype=None with a heterogeneous set of data,
> > trying to use unpack=True to get the columns into separate arrays
> > (instead of a structured array) doesn't work.  I've attached a patch
> > that, in the case of dtype=None, unpacks the fields in the final
> > array into a list of separate arrays.  Does this seem like a good
> > idea to you?
>
> Nope, as it breaks consistency: depending on some input parameters,
> you either get an array or a list. I think it's better to leave it as
> it is, maybe adding an extra line in the doc precising that
> unpack=True doesn't do anything for structured arrays.
>

Ah, I hadn't thought of that.  I was only thinking in terms of the behavior
of unpacking on return, not in the actual returned object.  You're right,
it's a bad idea.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090212/c54e5225/attachment.html>

From michael.s.gilbert at gmail.com  Thu Feb 12 11:58:00 2009
From: michael.s.gilbert at gmail.com (Michael S. Gilbert)
Date: Thu, 12 Feb 2009 11:58:00 -0500
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
 mersenne twister key?
In-Reply-To: <20090206181844.9b2070ff.michael.s.gilbert@gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>
	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
	<20090206181844.9b2070ff.michael.s.gilbert@gmail.com>
Message-ID: <20090212115800.ad3049fa.michael.s.gilbert@gmail.com>

On Fri, 6 Feb 2009 18:18:44 -0500 "Michael S. Gilbert" wrote:
> BTW, there is a 64-bit version of the reference mersenne twister 
> implementation available [1].

I did some testing with this 64-bit implementation (mt19937-64). I've
found that it is actually slower than the 32-bit reference (mt19937ar)
on 64-bit systems (2.15s vs 2.25s to generate 100000000 ints). This is
likely because it generates 64-bit long long ints instead of 32-bit
long ints. However, it should be possible to break up each 64-bit int
into two 32-bit ints, then the runtime would appear to be almost twice
as fast. One other consideration to keep in mind is that the 64-bit
version is not stream-compatible with the 32-bit implementation (you
will get different sequences for the same input seed).

Would it be worth it to implement this in numpy in order to get an
almost 2x speedup on 64-bit machines?


From sturla at molden.no  Thu Feb 12 11:53:04 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 12 Feb 2009 17:53:04 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <20090212162427.GC8194@phare.normalesup.org>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<4993FA61.7010000@student.matnat.uio.no>	<200902121216.20222.faltet@pytables.org>	<49940952.6060208@student.matnat.uio.no>	<499431E7.7000501@molden.no>
	<20090212162427.GC8194@phare.normalesup.org>
Message-ID: <499453F0.7050908@molden.no>

On 2/12/2009 5:24 PM, Gael Varoquaux wrote:

> My two cents: go for cython objects/statements. Not only does code in
> comments looks weird and a hack, but also it means to you have to hack
> the parser. 

I agree with this. Particularly because Cython uses intendation as 
syntax. With comments you would have to use 'end' tags like in Fortran:

!$omp parallel do private(i) shared(n)
do i = 1,n
   !$omp parallel do private(j) shared(i,n)
   do j = i, n
      ! whatever
   end do
   !$omp end parallel do
end do
!$omp end parallel do

But this is Fortran. It is meant to look bad. :)


> Finally cython can be used in a valid Python syntax (at least
> part of it) thanks to decorators giving the types. 

I don't think OpenMP will ever be used from pure Python.

But please, add a -openmp compiler swich in Cython. If it is not there,
all openmp statements should be ignored and translate to nothing.

S.M.



From ralphkube at googlemail.com  Thu Feb 12 12:21:03 2009
From: ralphkube at googlemail.com (Ralph Kube)
Date: Thu, 12 Feb 2009 18:21:03 +0100
Subject: [Numpy-discussion] Integer cast problems
Message-ID: <49945A7F.8020706@googlemail.com>

Hi there,
I have a little problem here with array indexing, hope you see the problem.
I use the following loop to calculate some integrals

import numpy as N
from scipy.integrate import quad
T = 1
dt = 0.005
L = 3
n = 2
ints = N.zeros([T/dt])

for t in N.arange(0, T, dt):
     a = quad(lambda 	 
x:-1*(1-4*(t**4))*N.exp(-t**4)*N.exp(-x**2)*N.cos(n*N.pi*(x-L)/(2*L)), 
-L, L)[0]
     ints[int(t/dt)] = a
     print t, N.int32(t/dt), t/dt, a, ints[int(t/dt)]

The output from the print statement looks like:

0.14 28 28.0 2.52124867251e-16 2.52124867251e-16
0.145 28 29.0 2.03015199575e-16 2.03015199575e-16
0.15 30 30.0 2.40857836418e-16 2.40857836418e-16
0.155 31 31.0 2.52191011339e-16 2.52191011339e-16

The same happens on the ipython prompt:

0.145 * 0.005 = 28.999999999999996
N.int32(0.145 * 0.005) = 28

Any ideas how to deal with this?

Cheers, Ralph


From rmay31 at gmail.com  Thu Feb 12 12:35:05 2009
From: rmay31 at gmail.com (Ryan May)
Date: Thu, 12 Feb 2009 11:35:05 -0600
Subject: [Numpy-discussion] Integer cast problems
In-Reply-To: <49945A7F.8020706@googlemail.com>
References: <49945A7F.8020706@googlemail.com>
Message-ID: <c7b95a020902120935q7a43f408nc6ba8cb18714e870@mail.gmail.com>

On Thu, Feb 12, 2009 at 11:21 AM, Ralph Kube <ralphkube at googlemail.com>wrote:

> Hi there,
> I have a little problem here with array indexing, hope you see the problem.
> I use the following loop to calculate some integrals
>
> import numpy as N
> from scipy.integrate import quad
> T = 1
> dt = 0.005
> L = 3
> n = 2
> ints = N.zeros([T/dt])
>
> for t in N.arange(0, T, dt):
>     a = quad(lambda
> x:-1*(1-4*(t**4))*N.exp(-t**4)*N.exp(-x**2)*N.cos(n*N.pi*(x-L)/(2*L)),
> -L, L)[0]
>     ints[int(t/dt)] = a
>     print t, N.int32(t/dt), t/dt, a, ints[int(t/dt)]
>
> The output from the print statement looks like:
>
> 0.14 28 28.0 2.52124867251e-16 2.52124867251e-16
> 0.145 28 29.0 2.03015199575e-16 2.03015199575e-16
> 0.15 30 30.0 2.40857836418e-16 2.40857836418e-16
> 0.155 31 31.0 2.52191011339e-16 2.52191011339e-16
>
> The same happens on the ipython prompt:
>
> 0.145 * 0.005 = 28.999999999999996
> N.int32(0.145 * 0.005) = 28
>
> Any ideas how to deal with this?
>

I'm assuming you mean 0.145 / 0.005 = 28.999999999999996

When you cast to an integer, it *truncates* the fractional part, and life
with floating point says that what should be an exact result won't
necessarily be exact.  Try using N.around.

Ryan


-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090212/f1a96c3d/attachment.html>

From gregor.thalhammer at gmail.com  Thu Feb 12 12:36:04 2009
From: gregor.thalhammer at gmail.com (Gregor Thalhammer)
Date: Thu, 12 Feb 2009 18:36:04 +0100
Subject: [Numpy-discussion] Integer cast problems
In-Reply-To: <49945A7F.8020706@googlemail.com>
References: <49945A7F.8020706@googlemail.com>
Message-ID: <49945E04.1020505@googlemail.com>

Ralph Kube schrieb:
> Hi there,
> I have a little problem here with array indexing, hope you see the problem.
> I use the following loop to calculate some integrals
>
> ...
> 0.145 * 0.005 = 28.999999999999996
> N.int32(0.145 * 0.005) = 28
conversion to int truncates, it doesn't round. Try
N.int32(0.145 * 0.005  +  0.5)

Gregor


From kwgoodman at gmail.com  Thu Feb 12 12:36:14 2009
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 12 Feb 2009 09:36:14 -0800
Subject: [Numpy-discussion] Integer cast problems
In-Reply-To: <49945A7F.8020706@googlemail.com>
References: <49945A7F.8020706@googlemail.com>
Message-ID: <f4f93d420902120936w5b72e917sbd525980a0bc540e@mail.gmail.com>

On Thu, Feb 12, 2009 at 9:21 AM, Ralph Kube <ralphkube at googlemail.com> wrote:
> The same happens on the ipython prompt:
>
> 0.145 * 0.005 = 28.999999999999996
> N.int32(0.145 * 0.005) = 28
>
> Any ideas how to deal with this?

Do you want the answer to be 29? N.int32 truncates. If you want to
round instead, you could use that standard trick of adding 0.5:

>> np.int32(0.5 + 0.145 / 0.005)
   29

or

>> np.round(0.145 / 0.005)
   29.0


From robert.kern at gmail.com  Thu Feb 12 14:18:26 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Feb 2009 13:18:26 -0600
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
	mersenne twister key?
In-Reply-To: <20090212115800.ad3049fa.michael.s.gilbert@gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>
	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
	<20090206181844.9b2070ff.michael.s.gilbert@gmail.com>
	<20090212115800.ad3049fa.michael.s.gilbert@gmail.com>
Message-ID: <3d375d730902121118h529817f2y4444beb061e12e58@mail.gmail.com>

On Thu, Feb 12, 2009 at 10:58, Michael S. Gilbert
<michael.s.gilbert at gmail.com> wrote:
> On Fri, 6 Feb 2009 18:18:44 -0500 "Michael S. Gilbert" wrote:
>> BTW, there is a 64-bit version of the reference mersenne twister
>> implementation available [1].
>
> I did some testing with this 64-bit implementation (mt19937-64). I've
> found that it is actually slower than the 32-bit reference (mt19937ar)
> on 64-bit systems (2.15s vs 2.25s to generate 100000000 ints). This is
> likely because it generates 64-bit long long ints instead of 32-bit
> long ints. However, it should be possible to break up each 64-bit int
> into two 32-bit ints, then the runtime would appear to be almost twice
> as fast.

Why do you think that?

> One other consideration to keep in mind is that the 64-bit
> version is not stream-compatible with the 32-bit implementation (you
> will get different sequences for the same input seed).
>
> Would it be worth it to implement this in numpy in order to get an
> almost 2x speedup on 64-bit machines?

The incompatibility is a showstopper to replacing the PRNG on any platform.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From dagss at student.matnat.uio.no  Thu Feb 12 14:32:02 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 12 Feb 2009 20:32:02 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <499431E7.7000501@molden.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<4993FA61.7010000@student.matnat.uio.no>	<200902121216.20222.faltet@pytables.org>	<49940952.6060208@student.matnat.uio.no>
	<499431E7.7000501@molden.no>
Message-ID: <49947932.3050608@student.matnat.uio.no>

Sturla Molden wrote:
> On 2/12/2009 12:34 PM, Dag Sverre Seljebotn wrote:
> 
>> FYI, I am one of the core Cython developers and can make such 
>> modifications in Cython itself as long as there's consensus on how it 
>> should look on the Cython mailing list.  My problem is that I don't 
>> really know OpenMP and have little experience with it, so I'm not the 
>> best person for creating a draft for how such high-level OpenMP 
>> constructs should look like in Cython.
> 
> I don't know the Cython internals, but I do know OpenMP. I mostly use it 
> with Fortran.
> 
> The question is: Should OpenMP be comments in the Cython code (as they 
> are in C and Fortran), or should OpenMP be special objects?

Statements more or less like C and Fortran, but "disguised" as Python 
syntax rather than pragmas/comments.

> As for the GIL: No I don't think nogil should be implied. But Python 
> objects should only be allowed as shared variables. Synchronization will 
> then be as usual for shared variables in OpenMP (#pragma omp critical).

Good point.

> Here is my suggestion for syntax. If you just follow a consistent 
> translation scheme, you don't need to know OpenMP in details. Here is a 
> suggestion:
> 
> 
> with openmp('parallel for', argument=iterable, ...):
>     --> insert pragma directly above for
> 
> with openmp(directive, argument=iterable, ...):
>     --> insert pragma and brackets
> 
> with openmp('atomic'): --> insert pragma directly
> 
> openmp('barrier') --> insert pragma directly
> 
> 
> This by the way covers all of OpenMP. This is how it should translate:
> 
> 
> with openmp('parallel for', private=(i,), shared=(n,),

IMO there's a problem with using literal variable names here, because 
Python syntax implies that the value is passed. One shouldn't make 
syntax where private=(i,) is legal but private=(f(),) isn't. So I'd go 
for strings.

>      schedule='dynamic'):
> 
>      for i in range(n):
>         pass
> 
> Compiles to:
> 
> #pragma omp parallel for \
> private(i) \
> shared(n) \
> schedule(dynamic)
> for(i=0; i<n; i++) {
>    /* whatever */
> }

Hmm... yes. Care would need to be taken though because Cython might in 
the future very well generate a "while" loop instead for such a 
statement under some circumstances, and that won't work with OpenMP. One 
should be careful with assuming what the C result will be of Cython 
code. That's why I proposed using an alternative construct which both 
does the OpenMP stuff and contains the for loop.

> With Python objects, the programmer must synchronize access:
> 
> 
> with openmp('parallel for', shared=(pyobj,n), private=(i,)):
>      for i in range(n):
>          with openmp('critical'):
>              pyobj += i

We already have the nogil stuff, so with e.g. "with openmp.critical:" 
instead we would leave the way open for checking correctness in this area.

Anyway, thanks for the input. I've made it a ticket and might get back 
to it when I've got more time through a discussion on the Cython list.

http://trac.cython.org/cython_trac/ticket/211

-- 
Dag Sverre


From dagss at student.matnat.uio.no  Thu Feb 12 14:36:27 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 12 Feb 2009 20:36:27 +0100
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <49947932.3050608@student.matnat.uio.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>	<4993FA61.7010000@student.matnat.uio.no>	<200902121216.20222.faltet@pytables.org>	<49940952.6060208@student.matnat.uio.no>	<499431E7.7000501@molden.no>
	<49947932.3050608@student.matnat.uio.no>
Message-ID: <49947A3B.6080404@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Hmm... yes. Care would need to be taken though because Cython might in 
> the future very well generate a "while" loop instead for such a 
> statement under some circumstances, and that won't work with OpenMP. One 
> should be careful with assuming what the C result will be of Cython 
> code. That's why I proposed using an alternative construct which both 
> does the OpenMP stuff and contains the for loop.

As a matter of fact, the next Cython release might prepend all 
for-in-range-loops with an if-test under some circumstances, in order to 
better match Python looping semantics (if the loop isn't executed, the 
loop counter should never be written to -- in contrast with C). So this 
is already happening. OpenMP might need different loop semantics and so 
calls for a different construct.

-- 
Dag Sverre


From ellisonbg.net at gmail.com  Thu Feb 12 14:40:53 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 12 Feb 2009 11:40:53 -0800
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <200902121105.55893.faltet@pytables.org>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<200902121105.55893.faltet@pytables.org>
Message-ID: <6ce0ac130902121140v146e1eafme111ca59c3cadf7f@mail.gmail.com>

> If your problem is evaluating vector expressions just like the above
> (i.e. without using transcendental functions like sin, exp, etc...),
> usually the bottleneck is on memory access, so using several threads is
> simply not going to help you achieving better performance, but rather
> the contrary (you have to deal with the additional thread overhead).
> So, frankly, I would not waste more time trying to paralelize that.

I had a feeling this would be the case, I just haven't been sure about
what point this comes into play.  I really need to do some tests to
understand exactly how CPU load and memory bandwidth interplay in
these situations.  I have worked with GPUs before and often the reason
the GPU is faster than the CPU is simply the higher memory bandwidth.

> As an example, in the recent support of VML in numexpr we have disabled
> the use of VML (as well as the OpenMP threading support that comes with
> it) in cases like yours, where only additions and multiplications are
> performed (these operations are very fast in modern processors, and the
> sole bottleneck for this case is the memory bandwidth, as I've said).
> However, in case of expressions containing operations like division or
> transcendental functions, then VML activates automatically, and you can
> make use of several cores if you want.  So, if you are in this case,
> and you have access to Intel MKL (the library that contains VML), you
> may want to give numexpr a try.

OK, this is very interesting indeed.  I didn't know that numexpr has
support for VML, which has openmp support.  I will definitely have
look at this.  Thanks!

Brian

> HTH,
>
> --
> Francesc Alted
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From ellisonbg.net at gmail.com  Thu Feb 12 14:43:31 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 12 Feb 2009 11:43:31 -0800
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <49940291.4050906@googlemail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<3d375d730902112152le9c8533qf54d9c17f407f48b@mail.gmail.com>
	<4993BDDD.9020404@ar.media.kyoto-u.ac.jp>
	<6ce0ac130902112252k2e37021dledecf6d2603aec4b@mail.gmail.com>
	<49940291.4050906@googlemail.com>
Message-ID: <6ce0ac130902121143t312964a7g87f9a9bbcba64670@mail.gmail.com>

> Recent Matlab versions use Intels Math Kernel Library, which performs
> automatic multi-threading - also for mathematical functions like sin
> etc, but not for  addition, multiplication etc. It seems to me Matlab
> itself does not take care of multi-threading. On
> http://www.intel.com/software/products/mkl/data/vml/functions/_listfunc.html
> you can have a look at the performance data of the MKL vectorized math
> functions. Around a vector length between 100-1000, depending on which
> function, precision, cpu architecture, they switch to multi-threading.

Fantastic, thanks for this pointer!

Brian


From ellisonbg.net at gmail.com  Thu Feb 12 15:01:00 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 12 Feb 2009 12:01:00 -0800
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <200902121347.35130.faltet@pytables.org>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<200902121216.20222.faltet@pytables.org>
	<49940952.6060208@student.matnat.uio.no>
	<200902121347.35130.faltet@pytables.org>
Message-ID: <6ce0ac130902121201y5608b47cx740ddbf4674c4533@mail.gmail.com>

> At any rate, I really like the OpenMP approach and prefer to have
> support for it in Cython much better than threading, MPI or whatever.
> But the thing is: is OpenMP stable, mature enough for allow using it in
> most of common platforms?  I think that recent GCC compilers support
> the latest incarnation (3.0) of the standard, but IIRC, MSVC 2008 only
> supports OpenMP 2.5 specification.  I'm not sure how this would affect
> a possible implementation in Cython, tough.

OpenMP has been around for a long time.  But in other ways, I don't
consider it to be completely stable and mature - mainly because of the
fact that there haven't been free compilers (like gcc) that support
OpenMP for very long.  For example, I am running OS X Leopard, which
is back at gcc 4.0.1 - a good ways previous to the OpenMP support.
But, this will hopefully change in the next few years as everyone
moves to more recent versions of gcc.

But, Cython could simply do a test of the compiler to see if it
supports OpenMP and what version it supports.

Brian


From michael.s.gilbert at gmail.com  Thu Feb 12 15:17:17 2009
From: michael.s.gilbert at gmail.com (Michael S. Gilbert)
Date: Thu, 12 Feb 2009 15:17:17 -0500
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
 mersenne twister key?
In-Reply-To: <3d375d730902121118h529817f2y4444beb061e12e58@mail.gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>
	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
	<20090206181844.9b2070ff.michael.s.gilbert@gmail.com>
	<20090212115800.ad3049fa.michael.s.gilbert@gmail.com>
	<3d375d730902121118h529817f2y4444beb061e12e58@mail.gmail.com>
Message-ID: <20090212151717.cdacaa60.michael.s.gilbert@gmail.com>

On Thu, 12 Feb 2009 13:18:26 -0600 Robert Kern wrote:
> > I did some testing with this 64-bit implementation (mt19937-64). I've
> > found that it is actually slower than the 32-bit reference (mt19937ar)
> > on 64-bit systems (2.15s vs 2.25s to generate 100000000 ints). This is
> > likely because it generates 64-bit long long ints instead of 32-bit
> > long ints. However, it should be possible to break up each 64-bit int
> > into two 32-bit ints, then the runtime would appear to be almost twice
> > as fast.
> 
> Why do you think that?

You could also think of it the other way (in terms of generating 64-bit
ints). Instead of generating two 32-bit rints and concatenating them
for a 64-bit int, you can just directly generate the 64-bit int. Since
the 64-bit int requires only slightly more time to generate than either
of the 32-bit ints individually, an almost 2x speedup is achieved.

> > One other consideration to keep in mind is that the 64-bit
> > version is not stream-compatible with the 32-bit implementation (you
> > will get different sequences for the same input seed).
> >
> > Would it be worth it to implement this in numpy in order to get an
> > almost 2x speedup on 64-bit machines?
> 
> The incompatibility is a showstopper to replacing the PRNG on any platform.

Why is stream-compatibility such a stringent requirement?  It seems
like this contstraint majorly limits your ability to adopt new/better
technologies and reduces your flexibility to make changes to your code
as needed.  What end-use applications require stream-compatibility?  The
only thing I can think of is verification/regression testing for numpy,
but those test cases could be updated to account for the break in
compatibility (and specifically using the reference implementation's
expected output).  Wouldn't sufficient documentation of the change in
behavior be sufficient?

Regards,
Mike


From ellisonbg.net at gmail.com  Thu Feb 12 15:15:17 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 12 Feb 2009 12:15:17 -0800
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <49947932.3050608@student.matnat.uio.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<4993FA61.7010000@student.matnat.uio.no>
	<200902121216.20222.faltet@pytables.org>
	<49940952.6060208@student.matnat.uio.no> <499431E7.7000501@molden.no>
	<49947932.3050608@student.matnat.uio.no>
Message-ID: <6ce0ac130902121215h7b699631g3e2bb27d8851fc7@mail.gmail.com>

Wow, interesting thread.  Thanks everyone for the ideas.  A few more comments:

GPUs/CUDA:

* Even though there is a bottleneck between main memory and GPU
memory, as Nathan mentioned, the much larger memory bandwidth on a GPU
often makes GPUs great for memory bound computations...as long as you
can leave your data on the GPU for most of the computation.  In my
case I can do this and this is something I am pursuing as well.

OpenMP:

I don't really like OpenMP (pragma?!?), but it would be very nice if
Cython had optional support for OpenMP that didn't use comments.

Other ideas:

What I would really like is a nice, super fast *library* built on top
of pthreads that made it possible to do OpenMP-like things in Cython,
but without depending on having an OpenMP compiler.  Basically a
fancy, fast thread pool implementation in Cython.

And a question:

With the new Numpy support in Cython, does Cython release the GIL if
it can when running through through loops over numpy arrays?  Does
Cython call into the C API during these sections?

Cheers,

Brian


From dagss at student.matnat.uio.no  Thu Feb 12 15:26:17 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 12 Feb 2009 21:26:17 +0100 (CET)
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <6ce0ac130902121215h7b699631g3e2bb27d8851fc7@mail.gmail.com>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<4993FA61.7010000@student.matnat.uio.no>
	<200902121216.20222.faltet@pytables.org>
	<49940952.6060208@student.matnat.uio.no> <499431E7.7000501@molden.no>
	<49947932.3050608@student.matnat.uio.no>
	<6ce0ac130902121215h7b699631g3e2bb27d8851fc7@mail.gmail.com>
Message-ID: <2499d355612c71a44dbb74d3ef5e8dc3.squirrel@webmail.uio.no>

Brian Granger wrote:
> And a question:
>
> With the new Numpy support in Cython, does Cython release the GIL if
> it can when running through through loops over numpy arrays?  Does
> Cython call into the C API during these sections?

You know, I thought of the exact same thing when reading your post. No,
you need the GIL currently, but that's something I'd like to fix.

Ideally, it would be something like this:

cdef int i, s = 0, n = ...
cdef np.ndarray[int] arr = ... # will require the GIL
with nogil:
    for i in range(n):
        s += arr[i] # does not require GIL

The only Python operation needed on the last line is throwing an exception
if one is out of bounds; but I can always insert code to reacquire the GIL
in that exceptional case. (And in most cases like this the user will turn
off bounds-checking anyway.)

http://trac.cython.org/cython_trac/ticket/210 will contain progress on this.

Dag Sverre



From robert.kern at gmail.com  Thu Feb 12 15:31:27 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Feb 2009 14:31:27 -0600
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
	mersenne twister key?
In-Reply-To: <20090212151717.cdacaa60.michael.s.gilbert@gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>
	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
	<20090206181844.9b2070ff.michael.s.gilbert@gmail.com>
	<20090212115800.ad3049fa.michael.s.gilbert@gmail.com>
	<3d375d730902121118h529817f2y4444beb061e12e58@mail.gmail.com>
	<20090212151717.cdacaa60.michael.s.gilbert@gmail.com>
Message-ID: <3d375d730902121231v3f83bc55tdc7224f0e5e892af@mail.gmail.com>

On Thu, Feb 12, 2009 at 14:17, Michael S. Gilbert
<michael.s.gilbert at gmail.com> wrote:
> On Thu, 12 Feb 2009 13:18:26 -0600 Robert Kern wrote:
>> > I did some testing with this 64-bit implementation (mt19937-64). I've
>> > found that it is actually slower than the 32-bit reference (mt19937ar)
>> > on 64-bit systems (2.15s vs 2.25s to generate 100000000 ints). This is
>> > likely because it generates 64-bit long long ints instead of 32-bit
>> > long ints. However, it should be possible to break up each 64-bit int
>> > into two 32-bit ints, then the runtime would appear to be almost twice
>> > as fast.
>>
>> Why do you think that?
>
> You could also think of it the other way (in terms of generating 64-bit
> ints). Instead of generating two 32-bit rints and concatenating them
> for a 64-bit int, you can just directly generate the 64-bit int. Since
> the 64-bit int requires only slightly more time to generate than either
> of the 32-bit ints individually, an almost 2x speedup is achieved.

<shrug> I'll believe it when I see it.

>> > One other consideration to keep in mind is that the 64-bit
>> > version is not stream-compatible with the 32-bit implementation (you
>> > will get different sequences for the same input seed).
>> >
>> > Would it be worth it to implement this in numpy in order to get an
>> > almost 2x speedup on 64-bit machines?
>>
>> The incompatibility is a showstopper to replacing the PRNG on any platform.
>
> Why is stream-compatibility such a stringent requirement?  It seems
> like this contstraint majorly limits your ability to adopt new/better
> technologies and reduces your flexibility to make changes to your code
> as needed.  What end-use applications require stream-compatibility?  The
> only thing I can think of is verification/regression testing for numpy,
> but those test cases could be updated to account for the break in
> compatibility (and specifically using the reference implementation's
> expected output).  Wouldn't sufficient documentation of the change in
> behavior be sufficient?

Some people don't think so. People have asked for more stringent
compatibility than we can already provide (i.e. replicability even in
the face of bug fixes). People use these as inputs to their scientific
simulations. I'm not going to intentionally make their lives harder
than that.

Bruce Southey was working on exposing the innards a bit so that you
could make use the a different core PRNG while reusing the
numpy-specific stuff in RandomState. That would be the approach to
apply different technologies.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Thu Feb 12 15:32:02 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Feb 2009 14:32:02 -0600
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
	mersenne twister key?
In-Reply-To: <20090212151717.cdacaa60.michael.s.gilbert@gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>
	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
	<20090206181844.9b2070ff.michael.s.gilbert@gmail.com>
	<20090212115800.ad3049fa.michael.s.gilbert@gmail.com>
	<3d375d730902121118h529817f2y4444beb061e12e58@mail.gmail.com>
	<20090212151717.cdacaa60.michael.s.gilbert@gmail.com>
Message-ID: <3d375d730902121232r22dcb98ckab834a7771dc07fe@mail.gmail.com>

On Thu, Feb 12, 2009 at 14:17, Michael S. Gilbert
<michael.s.gilbert at gmail.com> wrote:
> On Thu, 12 Feb 2009 13:18:26 -0600 Robert Kern wrote:
>> > I did some testing with this 64-bit implementation (mt19937-64). I've
>> > found that it is actually slower than the 32-bit reference (mt19937ar)
>> > on 64-bit systems (2.15s vs 2.25s to generate 100000000 ints). This is
>> > likely because it generates 64-bit long long ints instead of 32-bit
>> > long ints. However, it should be possible to break up each 64-bit int
>> > into two 32-bit ints, then the runtime would appear to be almost twice
>> > as fast.
>>
>> Why do you think that?
>
> You could also think of it the other way (in terms of generating 64-bit
> ints). Instead of generating two 32-bit rints and concatenating them
> for a 64-bit int, you can just directly generate the 64-bit int. Since
> the 64-bit int requires only slightly more time to generate than either
> of the 32-bit ints individually, an almost 2x speedup is achieved.

<shrug> I'll believe it when I see it.

>> > One other consideration to keep in mind is that the 64-bit
>> > version is not stream-compatible with the 32-bit implementation (you
>> > will get different sequences for the same input seed).
>> >
>> > Would it be worth it to implement this in numpy in order to get an
>> > almost 2x speedup on 64-bit machines?
>>
>> The incompatibility is a showstopper to replacing the PRNG on any platform.
>
> Why is stream-compatibility such a stringent requirement?  It seems
> like this contstraint majorly limits your ability to adopt new/better
> technologies and reduces your flexibility to make changes to your code
> as needed.  What end-use applications require stream-compatibility?  The
> only thing I can think of is verification/regression testing for numpy,
> but those test cases could be updated to account for the break in
> compatibility (and specifically using the reference implementation's
> expected output).  Wouldn't sufficient documentation of the change in
> behavior be sufficient?

Some people don't think so. People have asked for more stringent
compatibility than we can already provide (i.e. replicability even in
the face of bug fixes). People use these as inputs to their scientific
simulations. I'm not going to intentionally make their lives harder
than that.

Bruce Southey was working on exposing the innards a bit so that you
could make use the a different core PRNG while reusing the
numpy-specific stuff in RandomState. That would be the approach to
apply different technologies.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From michael.s.gilbert at gmail.com  Thu Feb 12 15:46:21 2009
From: michael.s.gilbert at gmail.com (Michael S. Gilbert)
Date: Thu, 12 Feb 2009 15:46:21 -0500
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
 mersenne twister key?
In-Reply-To: <3d375d730902121232r22dcb98ckab834a7771dc07fe@mail.gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>
	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
	<20090206181844.9b2070ff.michael.s.gilbert@gmail.com>
	<20090212115800.ad3049fa.michael.s.gilbert@gmail.com>
	<3d375d730902121118h529817f2y4444beb061e12e58@mail.gmail.com>
	<20090212151717.cdacaa60.michael.s.gilbert@gmail.com>
	<3d375d730902121232r22dcb98ckab834a7771dc07fe@mail.gmail.com>
Message-ID: <20090212154621.1f03d22b.michael.s.gilbert@gmail.com>

On Thu, 12 Feb 2009 14:32:02 -0600 Robert Kern wrote:
>> You could also think of it the other way (in terms of generating 64-bit
>> ints). Instead of generating two 32-bit rints and concatenating them
>> for a 64-bit int, you can just directly generate the 64-bit int. Since
>> the 64-bit int requires only slightly more time to generate than either
>> of the 32-bit ints individually, an almost 2x speedup is achieved.
>
><shrug> I'll believe it when I see it.

I'll put together a proof of concept when I have the time.

> Some people don't think so. People have asked for more stringent
> compatibility than we can already provide (i.e. replicability even in
> the face of bug fixes). People use these as inputs to their scientific
> simulations. I'm not going to intentionally make their lives harder
> than that.
> 
> Bruce Southey was working on exposing the innards a bit so that you
> could make use the a different core PRNG while reusing the
> numpy-specific stuff in RandomState. That would be the approach to
> apply different technologies.

This would be very useful.  A "backward-compatibility=<numpy version
number>" flag could be offered to fall back to a particular
implementation so that you can push forward and still offer
compatibility as needed.

Regards,
Mike


From ellisonbg.net at gmail.com  Thu Feb 12 15:58:31 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 12 Feb 2009 12:58:31 -0800
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <2499d355612c71a44dbb74d3ef5e8dc3.squirrel@webmail.uio.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<4993FA61.7010000@student.matnat.uio.no>
	<200902121216.20222.faltet@pytables.org>
	<49940952.6060208@student.matnat.uio.no> <499431E7.7000501@molden.no>
	<49947932.3050608@student.matnat.uio.no>
	<6ce0ac130902121215h7b699631g3e2bb27d8851fc7@mail.gmail.com>
	<2499d355612c71a44dbb74d3ef5e8dc3.squirrel@webmail.uio.no>
Message-ID: <6ce0ac130902121258r26ff1d00n4ac5b77bc042c67e@mail.gmail.com>

> You know, I thought of the exact same thing when reading your post. No,
> you need the GIL currently, but that's something I'd like to fix.
>
> Ideally, it would be something like this:
>
> cdef int i, s = 0, n = ...
> cdef np.ndarray[int] arr = ... # will require the GIL
> with nogil:
>    for i in range(n):
>        s += arr[i] # does not require GIL

Yep, that would be fantastic!!!  While it doesn't make the code
multithreaded, having this sure would make it easy to get code ready
for multithreading

> The only Python operation needed on the last line is throwing an exception
> if one is out of bounds; but I can always insert code to reacquire the GIL
> in that exceptional case. (And in most cases like this the user will turn
> off bounds-checking anyway.)
>
> http://trac.cython.org/cython_trac/ticket/210 will contain progress on this.

Great, I will watch this.  I am more than willing to test this as well
as it is something I would use often.

Cheers,

Brian


From pav at iki.fi  Thu Feb 12 16:23:56 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 12 Feb 2009 21:23:56 +0000 (UTC)
Subject: [Numpy-discussion] umath build failures @ trunk on some platforms?
Message-ID: <gn241c$pkf$1@ger.gmane.org>

Hi,

The Buildbot (up once again) is showing build failures on some platforms:

	http://buildbot.scipy.org/builders/Windows_XP_x86_64_MSVC/builds/875/steps/shell/logs/stdio
	http://buildbot.scipy.org/builders/Linux_SPARC_64_Debian/builds/423/steps/shell/logs/stdio

Are these a sign of some bug (in the new math config system?), or
a glitch in the buildslaves only?

-- 
Pauli Virtanen



From charlesr.harris at gmail.com  Thu Feb 12 16:39:05 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 12 Feb 2009 14:39:05 -0700
Subject: [Numpy-discussion] umath build failures @ trunk on some
	platforms?
In-Reply-To: <gn241c$pkf$1@ger.gmane.org>
References: <gn241c$pkf$1@ger.gmane.org>
Message-ID: <e06186140902121339o7d5a206ax9e4a96916917582b@mail.gmail.com>

On Thu, Feb 12, 2009 at 2:23 PM, Pauli Virtanen <pav at iki.fi> wrote:

> Hi,
>
> The Buildbot (up once again) is showing build failures on some platforms:
>
>
> http://buildbot.scipy.org/builders/Windows_XP_x86_64_MSVC/builds/875/steps/shell/logs/stdio
>
> http://buildbot.scipy.org/builders/Linux_SPARC_64_Debian/builds/423/steps/shell/logs/stdio
>


> Are these a sign of some bug (in the new math config system?), or
> a glitch in the buildslaves only?
>

They are old problems. The Debian problem comes from missing prototypes, the
Windows problems from a missing function. I had workarounds for these but
David removed them and wants to solve them otherwise. The Debian problem
probably won't be solved at all because it is a distro problem, you might
try an upgrade.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090212/27c755b8/attachment.html>

From sturla at molden.no  Thu Feb 12 18:38:03 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 13 Feb 2009 00:38:03 +0100 (CET)
Subject: [Numpy-discussion] Fast threading solution thoughts
In-Reply-To: <49947932.3050608@student.matnat.uio.no>
References: <6ce0ac130902112146v7e858968y8a1cff3d28e5f84@mail.gmail.com>
	<4993FA61.7010000@student.matnat.uio.no>
	<200902121216.20222.faltet@pytables.org>
	<49940952.6060208@student.matnat.uio.no> <499431E7.7000501@molden.no>
	<49947932.3050608@student.matnat.uio.no>
Message-ID: <fabe04caa03ea0adcf1e1823d1faaddc.squirrel@webmail.uio.no>

> Sturla Molden wrote:

> IMO there's a problem with using literal variable names here, because
> Python syntax implies that the value is passed. One shouldn't make
> syntax where private=(i,) is legal but private=(f(),) isn't.

The latter would be illegal in OpenMP as well. OpenMP pragmas only take
variable names, not function calls.

S.M.






From cournape at gmail.com  Thu Feb 12 19:19:47 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 13 Feb 2009 09:19:47 +0900
Subject: [Numpy-discussion] umath build failures @ trunk on some
	platforms?
In-Reply-To: <e06186140902121339o7d5a206ax9e4a96916917582b@mail.gmail.com>
References: <gn241c$pkf$1@ger.gmane.org>
	<e06186140902121339o7d5a206ax9e4a96916917582b@mail.gmail.com>
Message-ID: <5b8d13220902121619t3e9a7b99r3904f5ba274a7f7b@mail.gmail.com>

On Fri, Feb 13, 2009 at 6:39 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Thu, Feb 12, 2009 at 2:23 PM, Pauli Virtanen <pav at iki.fi> wrote:
>>
>> Hi,
>>
>> The Buildbot (up once again) is showing build failures on some platforms:
>>
>>
>>  http://buildbot.scipy.org/builders/Windows_XP_x86_64_MSVC/builds/875/steps/shell/logs/stdio
>>
>>  http://buildbot.scipy.org/builders/Linux_SPARC_64_Debian/builds/423/steps/shell/logs/stdio
>>
>> Are these a sign of some bug (in the new math config system?), or
>> a glitch in the buildslaves only?
>
> They are old problems. The Debian problem comes from missing prototypes, the
> Windows problems from a missing function. I had workarounds for these but
> David removed them and wants to solve them otherwise.

> The Debian problem
> probably won't be solved at all because it is a distro problem, you might
> try an upgrade.

The Debian problem is more or less irrelevant IMHO. It is a toolchain
bug in old debian on sparc, that is a bug in an old version of a
barely used platform.

The windows problem is more serious, but I would like to avoid a
workaround which impact other platforms.

David


From python6009 at gmail.com  Thu Feb 12 20:22:58 2009
From: python6009 at gmail.com (A B)
Date: Thu, 12 Feb 2009 17:22:58 -0800
Subject: [Numpy-discussion] Filling gaps
Message-ID: <7cc4bc500902121722g6846f8d9q2b0defac524f8be3@mail.gmail.com>

Hi,
Are there any routines to fill in the gaps in an array. The simplest
would be by carrying the last known observation forward.
0,0,10,8,0,0,7,0
0,0,10,8,8,8,7,7
Or by somehow interpolating the missing values based on the previous
and next known observations (mean).
Thanks.


From pgmdevlist at gmail.com  Thu Feb 12 20:33:26 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 12 Feb 2009 20:33:26 -0500
Subject: [Numpy-discussion] Filling gaps
In-Reply-To: <7cc4bc500902121722g6846f8d9q2b0defac524f8be3@mail.gmail.com>
References: <7cc4bc500902121722g6846f8d9q2b0defac524f8be3@mail.gmail.com>
Message-ID: <7741E188-4BA5-454C-AEDE-27E685FEDB40@gmail.com>


On Feb 12, 2009, at 8:22 PM, A B wrote:

> Hi,
> Are there any routines to fill in the gaps in an array. The simplest
> would be by carrying the last known observation forward.
> 0,0,10,8,0,0,7,0
> 0,0,10,8,8,8,7,7
> Or by somehow interpolating the missing values based on the previous
> and next known observations (mean).
> Thanks.


The functions `forward_fill` and `backward_fill` in scikits.timeseries  
should do what you want. They work also on MaskedArray objects,  
meaning that you don't need to have actual series.
The catch is that you need to install scikits.timeseries, of course.  
More info here:http://pytseries.sourceforge.net/


From kwgoodman at gmail.com  Thu Feb 12 20:52:56 2009
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 12 Feb 2009 17:52:56 -0800
Subject: [Numpy-discussion] Filling gaps
In-Reply-To: <7cc4bc500902121722g6846f8d9q2b0defac524f8be3@mail.gmail.com>
References: <7cc4bc500902121722g6846f8d9q2b0defac524f8be3@mail.gmail.com>
Message-ID: <f4f93d420902121752j5f9cc9b3pc8475e2a318bc634@mail.gmail.com>

On Thu, Feb 12, 2009 at 5:22 PM, A B <python6009 at gmail.com> wrote:
> Are there any routines to fill in the gaps in an array. The simplest
> would be by carrying the last known observation forward.
> 0,0,10,8,0,0,7,0
> 0,0,10,8,8,8,7,7

Here's an obvious hack for 1d arrays:

def fill_forward(x, miss=0):
    y = x.copy()
    for i in range(x.shape[0]):
        if y[i] == miss:
            y[i] = y[i-1]
    return y

Seems to work:

>> x
   array([ 0,  0, 10,  8,  0,  0,  7,  0])
>> fill_forward(x)
   array([ 0,  0, 10,  8,  8,  8,  7,  7])


From kwgoodman at gmail.com  Thu Feb 12 21:04:32 2009
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 12 Feb 2009 18:04:32 -0800
Subject: [Numpy-discussion] Filling gaps
In-Reply-To: <f4f93d420902121752j5f9cc9b3pc8475e2a318bc634@mail.gmail.com>
References: <7cc4bc500902121722g6846f8d9q2b0defac524f8be3@mail.gmail.com>
	<f4f93d420902121752j5f9cc9b3pc8475e2a318bc634@mail.gmail.com>
Message-ID: <f4f93d420902121804s1b9b685ck50ccf6eafc5b93d2@mail.gmail.com>

On Thu, Feb 12, 2009 at 5:52 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Thu, Feb 12, 2009 at 5:22 PM, A B <python6009 at gmail.com> wrote:
>> Are there any routines to fill in the gaps in an array. The simplest
>> would be by carrying the last known observation forward.
>> 0,0,10,8,0,0,7,0
>> 0,0,10,8,8,8,7,7
>
> Here's an obvious hack for 1d arrays:
>
> def fill_forward(x, miss=0):
>    y = x.copy()
>    for i in range(x.shape[0]):
>        if y[i] == miss:
>            y[i] = y[i-1]
>    return y
>
> Seems to work:
>
>>> x
>   array([ 0,  0, 10,  8,  0,  0,  7,  0])
>>> fill_forward(x)
>   array([ 0,  0, 10,  8,  8,  8,  7,  7])

I guess that should be

    for i in range(1, x.shape[0]):

instead of

    for i in range(x.shape[0]):

to avoid replacing the first element of the array, if it is missing,
with the last.


From kwgoodman at gmail.com  Thu Feb 12 21:19:32 2009
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 12 Feb 2009 18:19:32 -0800
Subject: [Numpy-discussion] Filling gaps
In-Reply-To: <f4f93d420902121804s1b9b685ck50ccf6eafc5b93d2@mail.gmail.com>
References: <7cc4bc500902121722g6846f8d9q2b0defac524f8be3@mail.gmail.com>
	<f4f93d420902121752j5f9cc9b3pc8475e2a318bc634@mail.gmail.com>
	<f4f93d420902121804s1b9b685ck50ccf6eafc5b93d2@mail.gmail.com>
Message-ID: <f4f93d420902121819l47233105r1db050e3ea3ed386@mail.gmail.com>

On Thu, Feb 12, 2009 at 6:04 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Thu, Feb 12, 2009 at 5:52 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>> On Thu, Feb 12, 2009 at 5:22 PM, A B <python6009 at gmail.com> wrote:
>>> Are there any routines to fill in the gaps in an array. The simplest
>>> would be by carrying the last known observation forward.
>>> 0,0,10,8,0,0,7,0
>>> 0,0,10,8,8,8,7,7
>>
>> Here's an obvious hack for 1d arrays:
>>
>> def fill_forward(x, miss=0):
>>    y = x.copy()
>>    for i in range(x.shape[0]):
>>        if y[i] == miss:
>>            y[i] = y[i-1]
>>    return y
>>
>> Seems to work:
>>
>>>> x
>>   array([ 0,  0, 10,  8,  0,  0,  7,  0])
>>>> fill_forward(x)
>>   array([ 0,  0, 10,  8,  8,  8,  7,  7])
>
> I guess that should be
>
>    for i in range(1, x.shape[0]):
>
> instead of
>
>    for i in range(x.shape[0]):
>
> to avoid replacing the first element of the array, if it is missing,
> with the last.

For large 1d x arrays, this might be faster:

def fill_forward2(x, miss=0):
    y = x.copy()
    while np.any(y == miss):
        idx = np.where(y == miss)[0]
        y[idx] = y[idx-1]
    return y

But it does replace the first element of the array, if it is missing,
with the last.

We could speed it up by doing (y == miss) only once per loop. (But I
bet the np.where is the bottleneck.)


From nadavh at visionsense.com  Fri Feb 13 02:19:27 2009
From: nadavh at visionsense.com (Nadav Horesh)
Date: Fri, 13 Feb 2009 09:19:27 +0200
Subject: [Numpy-discussion] Bilateral filter:  bug corrected
Message-ID: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il>


Attached here a code for bilateral filter:

1. The code is composed of a cython back end (bilateral_base.pyx) and a python front end (bilateral.py)
2. I do not have the tools to make windows binaries (I run it on gentoo linux) .
3. It is not hard to strip the cython code to get a pure (and slow) python implementation.
4. If someone finds the licensing inadequate, I have no problem to re-license.


Any comments are (more then) welcome

  Nadav


-------------- next part --------------
A non-text attachment was scrubbed...
Name: bilateral_base.pyx
Type: application/octet-stream
Size: 4089 bytes
Desc: bilateral_base.pyx
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090213/47d94b60/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bilateral.py
Type: text/x-python
Size: 1896 bytes
Desc: bilateral.py
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090213/47d94b60/attachment.py>

From stefan at sun.ac.za  Fri Feb 13 04:50:07 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 13 Feb 2009 11:50:07 +0200
Subject: [Numpy-discussion] Bilateral filter: bug corrected
In-Reply-To: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il>
References: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il>
Message-ID: <9457e7c80902130150h4dc8755eqf6097fc0d20b6522@mail.gmail.com>

Hi Nadav

2009/2/13 Nadav Horesh <nadavh at visionsense.com>:
>
> Attached here a code for bilateral filter:
>
> 1. The code is composed of a cython back end (bilateral_base.pyx) and a python front end (bilateral.py)
> 2. I do not have the tools to make windows binaries (I run it on gentoo linux) .
> 3. It is not hard to strip the cython code to get a pure (and slow) python implementation.
> 4. If someone finds the licensing inadequate, I have no problem to re-license.
>
>
> Any comments are (more then) welcome

Thanks for the update!  I added a setup.py and fixed a small problem
with the includes.  Everything is here:

http://github.com/stefanv/bilateral/tree/master

Cheers
St?fan


From njs at pobox.com  Fri Feb 13 06:22:00 2009
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 13 Feb 2009 03:22:00 -0800
Subject: [Numpy-discussion] problem with WRITEABLE flag and array interface
Message-ID: <961fa2b40902130322p5da39205xf7887614570dc7d7@mail.gmail.com>

I'm using rpy2 to access the R statistical programming runtime from
Python, and one of rpy2's nice features is that wrappers for R arrays
support the Python array interface (via __array_struct__), so that one
can conveniently cast them to numpy arrays and edit them in place.

But this seems to expose a bug in numpy's WRITEABLE flag handling.
Such arrays start out writeable:

  >>> a = np.asarray(r)
  >>> a.flags["WRITEABLE"]
  True

And you can toggle them unwriteable:

  >>> a.flags["WRITEABLE"] = False

But then you are stuck! You cannot make the array writeable again:

  >>> a.flags["WRITEABLE"] = True
  ValueError: cannot set WRITEABLE flag to True of this array

This may seem rather trivial, but it's completely broken my API
design... the problem is that R has somewhat complex rules for when
you can write to an array and when you cannot (because of
copy-on-write stuff going on behind the scenes), and I wanted to
reflect that in my high-level API (for people who don't want to know
about R's COW minutiae) by toggling the WRITEABLE flag in a controlled
way. But I can't. Help?

Self-contained test case:

class Fraud(object):
  pass
f = Fraud()
a = np.array([1, 2, 3])
f.__array_interface__ = a.__array_interface__
f_asarray = np.asarray(f)
assert f_asarray.flags["WRITEABLE"] = True
f_asarray.flags["WRITEABLE"] = False
f_asarray.flags["WRITEABLE"] = True    # Fails

-- Nathaniel


From sturla at molden.no  Fri Feb 13 06:54:10 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 13 Feb 2009 12:54:10 +0100
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
 mersenne twister key?
In-Reply-To: <20090212154621.1f03d22b.michael.s.gilbert@gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>	<20090206181844.9b2070ff.michael.s.gilbert@gmail.com>	<20090212115800.ad3049fa.michael.s.gilbert@gmail.com>	<3d375d730902121118h529817f2y4444beb061e12e58@mail.gmail.com>	<20090212151717.cdacaa60.michael.s.gilbert@gmail.com>	<3d375d730902121232r22dcb98ckab834a7771dc07fe@mail.gmail.com>
	<20090212154621.1f03d22b.michael.s.gilbert@gmail.com>
Message-ID: <49955F62.7000203@molden.no>

On 2/12/2009 9:46 PM, Michael S. Gilbert wrote:

> I'll put together a proof of concept when I have the time.

If you are that obsessed with speed, consider to use the new SIMD 
version of the Mersenne Twister instead of Jean-Sebastien Roy's 
randomkit.c (used by NumPy). But I think randomkit.c is fast enough for 
most purposes.

http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/SFMT/index.html




Sturla Molden




From michael.s.gilbert at gmail.com  Fri Feb 13 10:07:56 2009
From: michael.s.gilbert at gmail.com (Michael S. Gilbert)
Date: Fri, 13 Feb 2009 10:07:56 -0500
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
 mersenne twister key?
In-Reply-To: <49955F62.7000203@molden.no>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>
	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
	<20090206181844.9b2070ff.michael.s.gilbert@gmail.com>
	<20090212115800.ad3049fa.michael.s.gilbert@gmail.com>
	<3d375d730902121118h529817f2y4444beb061e12e58@mail.gmail.com>
	<20090212151717.cdacaa60.michael.s.gilbert@gmail.com>
	<3d375d730902121232r22dcb98ckab834a7771dc07fe@mail.gmail.com>
	<20090212154621.1f03d22b.michael.s.gilbert@gmail.com>
	<49955F62.7000203@molden.no>
Message-ID: <20090213100756.7d2f3e5d.michael.s.gilbert@gmail.com>

On Fri, 13 Feb 2009 12:54:10 +0100 Sturla Molden wrote:
> If you are that obsessed with speed, consider to use the new SIMD 
> version of the Mersenne Twister instead of Jean-Sebastien Roy's 
> randomkit.c (used by NumPy). But I think randomkit.c is fast enough for 
> most purposes.
> 
> http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/SFMT/index.html

I had already come across this, and was going to suggest using it as
well at some point.  However, as per Robert's last comment, it sounds
like quite a bit of work needs to be done first to make drop-in
replacements for the PRNG possible.

I'm not "obsessed" with speed, but a 2x improvement is quite
significant.


From sturla at molden.no  Fri Feb 13 10:25:37 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 13 Feb 2009 16:25:37 +0100
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
 mersenne twister key?
In-Reply-To: <20090213100756.7d2f3e5d.michael.s.gilbert@gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>	<20090206181844.9b2070ff.michael.s.gilbert@gmail.com>	<20090212115800.ad3049fa.michael.s.gilbert@gmail.com>	<3d375d730902121118h529817f2y4444beb061e12e58@mail.gmail.com>	<20090212151717.cdacaa60.michael.s.gilbert@gmail.com>	<3d375d730902121232r22dcb98ckab834a7771dc07fe@mail.gmail.com>	<20090212154621.1f03d22b.michael.s.gilbert@gmail.com>	<49955F62.7000203@molden.no>
	<20090213100756.7d2f3e5d.michael.s.gilbert@gmail.com>
Message-ID: <499590F1.8060801@molden.no>

On 2/13/2009 4:07 PM, Michael S. Gilbert wrote:

> I'm not "obsessed" with speed, but a 2x improvement is quite
> significant.

Honestly, I don't care about 2x differences here. How many milliseconds 
do you save?

The PRNG in SciPy is fast enough for 99% of any use I can conceive. I 
have yet to see a program speed-limited by the PRNG. Projects that are 
speed limited by the PRNG could take the hassle to compile their own. 
And if speed is that important, perhaps one should not use the Mersenne 
Twister at all?

http://groups.google.com/group/comp.lang.c/msg/e3c4ea1169e463ae?dmode=source
http://groups.google.com/group/sci.crypt/msg/12152f657a3bb219?dmode=source



S.M.





From michael.s.gilbert at gmail.com  Fri Feb 13 10:51:53 2009
From: michael.s.gilbert at gmail.com (Michael S. Gilbert)
Date: Fri, 13 Feb 2009 10:51:53 -0500
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
 mersenne twister key?
In-Reply-To: <499590F1.8060801@molden.no>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>
	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
	<20090206181844.9b2070ff.michael.s.gilbert@gmail.com>
	<20090212115800.ad3049fa.michael.s.gilbert@gmail.com>
	<3d375d730902121118h529817f2y4444beb061e12e58@mail.gmail.com>
	<20090212151717.cdacaa60.michael.s.gilbert@gmail.com>
	<3d375d730902121232r22dcb98ckab834a7771dc07fe@mail.gmail.com>
	<20090212154621.1f03d22b.michael.s.gilbert@gmail.com>
	<49955F62.7000203@molden.no>
	<20090213100756.7d2f3e5d.michael.s.gilbert@gmail.com>
	<499590F1.8060801@molden.no>
Message-ID: <20090213105153.0fb3fe9f.michael.s.gilbert@gmail.com>

On Fri, 13 Feb 2009 16:25:37 +0100 Sturla Molden wrote:
> Honestly, I don't care about 2x differences here. How many milliseconds 
> do you save?

It's not about saving milliseconds, it's about taking half the time to
run the same simulation.  So if my runs currently take 2 hours, they
will take 1 hour instead; and if they take 2 days, they will take 1
day instead. It may not impact your application's runtime, but it does
mine.

> Projects that are speed limited by the PRNG could take the hassle to
> compile their own.

I think it would be useful for numpy to provide options and let the
user decide based on their needs (most don't want to and should not
be made to implement their own algorithms, especially since the work
has already been done).

> And if speed is that important, perhaps one should not use the Mersenne 
> Twister at all?

Just yesterday, I started looking at using /dev/urandom instead of MT,
and I had previously looked at using Marsaglia's MWC.  It's a tradeoff
between speed and random number quality.

Best Regards,
Mike


From sturla at molden.no  Fri Feb 13 11:04:48 2009
From: sturla at molden.no (Sturla Molden)
Date: Fri, 13 Feb 2009 17:04:48 +0100
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
 mersenne twister key?
In-Reply-To: <20090213105153.0fb3fe9f.michael.s.gilbert@gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>	<20090206181844.9b2070ff.michael.s.gilbert@gmail.com>	<20090212115800.ad3049fa.michael.s.gilbert@gmail.com>	<3d375d730902121118h529817f2y4444beb061e12e58@mail.gmail.com>	<20090212151717.cdacaa60.michael.s.gilbert@gmail.com>	<3d375d730902121232r22dcb98ckab834a7771dc07fe@mail.gmail.com>	<20090212154621.1f03d22b.michael.s.gilbert@gmail.com>	<49955F62.7000203@molden.no>	<20090213100756.7d2f3e5d.michael.s.gilbert@gmail.com>	<499590F1.8060801@molden.no>
	<20090213105153.0fb3fe9f.michael.s.gilbert@gmail.com>
Message-ID: <49959A20.7070701@molden.no>

On 2/13/2009 4:51 PM, Michael S. Gilbert wrote:

> It's not about saving milliseconds, it's about taking half the time to
> run the same simulation.  So if my runs currently take 2 hours, they
> will take 1 hour instead; and if they take 2 days, they will take 1
> day instead. It may not impact your application's runtime, but it does
> mine.

So you have a simulation written in *Python*, and the major bottleneck 
is the MT prng? Forgive me for not believing it.

cProfile.run('simulation()')



S.M.








From michael.s.gilbert at gmail.com  Fri Feb 13 11:17:55 2009
From: michael.s.gilbert at gmail.com (Michael S. Gilbert)
Date: Fri, 13 Feb 2009 11:17:55 -0500
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
 mersenne twister key?
In-Reply-To: <49959A20.7070701@molden.no>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>
	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
	<20090206181844.9b2070ff.michael.s.gilbert@gmail.com>
	<20090212115800.ad3049fa.michael.s.gilbert@gmail.com>
	<3d375d730902121118h529817f2y4444beb061e12e58@mail.gmail.com>
	<20090212151717.cdacaa60.michael.s.gilbert@gmail.com>
	<3d375d730902121232r22dcb98ckab834a7771dc07fe@mail.gmail.com>
	<20090212154621.1f03d22b.michael.s.gilbert@gmail.com>
	<49955F62.7000203@molden.no>
	<20090213100756.7d2f3e5d.michael.s.gilbert@gmail.com>
	<499590F1.8060801@molden.no>
	<20090213105153.0fb3fe9f.michael.s.gilbert@gmail.com>
	<49959A20.7070701@molden.no>
Message-ID: <20090213111755.000c2c71.michael.s.gilbert@gmail.com>

On Fri, 13 Feb 2009 17:04:48 +0100 Sturla Molden wrote:
> So you have a simulation written in *Python*, and the major bottleneck 
> is the MT prng? Forgive me for not believing it.

Yes, running a lot of monte carlo simulations back-to-back.  if the
PRNG were twice as fast, my code would be twice as fast.  It isn't that
unbelievable...

Honestly, I don't feel like arguing about this anymore.  Its a matter
of "show me the code," and when I have the time, I will "show you the
code."


From ralphkube at googlemail.com  Fri Feb 13 12:22:21 2009
From: ralphkube at googlemail.com (Ralph Kube)
Date: Fri, 13 Feb 2009 18:22:21 +0100
Subject: [Numpy-discussion] Integer cast problems
In-Reply-To: <f4f93d420902120936w5b72e917sbd525980a0bc540e@mail.gmail.com>
References: <49945A7F.8020706@googlemail.com>
	<f4f93d420902120936w5b72e917sbd525980a0bc540e@mail.gmail.com>
Message-ID: <4995AC4D.8090300@googlemail.com>

Thanks alot people, my problem is gone now.

Keith Goodman skrev:
> On Thu, Feb 12, 2009 at 9:21 AM, Ralph Kube <ralphkube at googlemail.com> wrote:
>> The same happens on the ipython prompt:
>>
>> 0.145 * 0.005 = 28.999999999999996
>> N.int32(0.145 * 0.005) = 28
>>
>> Any ideas how to deal with this?
> 
> Do you want the answer to be 29? N.int32 truncates. If you want to
> round instead, you could use that standard trick of adding 0.5:
> 
>>> np.int32(0.5 + 0.145 / 0.005)
>    29
> 
> or
> 
>>> np.round(0.145 / 0.005)
>    29.0
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion


From nadavh at visionsense.com  Fri Feb 13 13:50:29 2009
From: nadavh at visionsense.com (Nadav Horesh)
Date: Fri, 13 Feb 2009 20:50:29 +0200
Subject: [Numpy-discussion] Bilateral filter: bug corrected
References: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il>
	<9457e7c80902130150h4dc8755eqf6097fc0d20b6522@mail.gmail.com>
Message-ID: <710F2847B0018641891D9A216027636029C432@ex3.envision.co.il>

Hi Stefan,

  I tried the installer and it did not copy bilateral.py. I tried to improve it and the result is in the attached file.  I hope it would pass the mail filter, if not contact me directly to the email address below.

  Nadav.


-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? St?fan van der Walt
????: ? 13-??????-09 11:50
??: Discussion of Numerical Python
????: Re: [Numpy-discussion] Bilateral filter: bug corrected
 
Hi Nadav

2009/2/13 Nadav Horesh <nadavh at visionsense.com>:
>
> Attached here a code for bilateral filter:
>
> 1. The code is composed of a cython back end (bilateral_base.pyx) and a python front end (bilateral.py)
> 2. I do not have the tools to make windows binaries (I run it on gentoo linux) .
> 3. It is not hard to strip the cython code to get a pure (and slow) python implementation.
> 4. If someone finds the licensing inadequate, I have no problem to re-license.
>
>
> Any comments are (more then) welcome

Thanks for the update!  I added a setup.py and fixed a small problem
with the includes.  Everything is here:

http://github.com/stefanv/bilateral/tree/master

Cheers
St?fan
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion


-------------- next part --------------
A non-text attachment was scrubbed...
Name: bilateral.tar.gz
Type: application/x-gzip
Size: 2435 bytes
Desc: bilateral.tar.gz
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090213/276b398f/attachment.bin>

From davidh at ipac.caltech.edu  Fri Feb 13 15:04:12 2009
From: davidh at ipac.caltech.edu (David Henderson)
Date: Fri, 13 Feb 2009 12:04:12 -0800
Subject: [Numpy-discussion] improvement request to np.dot(a,
	b) - extended precision summation
Message-ID: <AE787A4B-A9E2-48FE-A988-1FD68381DE95@ipac.caltech.edu>

Hello all,

I'd like accumulate the summation in extended precision, "double"   
sum for float  inputs, "long double" sum for "double" inputs.

A way of doing this is to add an optional third argument to dot - to  
specify the summation type.

I've looked into the code, and the source file multiarraymodule.c has  
a routine:
        new_array_for_sum(PyArrayObject *ap1, PyArrayObject *ap2,

If the implementation is to pass a third argument to dot that  
specifies the resultant type, this routine would be modified to add  
that third argument or type in its priority selection logic.

I've not written any code yet, and I'm not fussy as to how it gets  
done. Is there another way to do this that I'm missing?

Thanks in advance,

David


From pav at iki.fi  Fri Feb 13 16:21:14 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 13 Feb 2009 21:21:14 +0000 (UTC)
Subject: [Numpy-discussion] improvement request to np.dot(a,
	b) - extended precision summation
References: <AE787A4B-A9E2-48FE-A988-1FD68381DE95@ipac.caltech.edu>
Message-ID: <gn4o8a$nkf$1@ger.gmane.org>

Fri, 13 Feb 2009 12:04:12 -0800, David Henderson wrote:
> I'd like accumulate the summation in extended precision, "double" sum
> for float  inputs, "long double" sum for "double" inputs.
> 
> A way of doing this is to add an optional third argument to dot - to
> specify the summation type.

`dot` does matrix-matrix, matrix-vector, or vector-vector products. These 
are usually implemented via calling the BLAS linear algebra library, and 
AFAIK BLAS only has routines for type-homogenous arguments. So I doubt 
this can be implemented for `dot` in any way.

On the other hand, `numpy.sum` already has a `dtype` argument that 
specifies the accumulator data type. Maybe you can use that?

-- 
Pauli Virtanen



From robert.kern at gmail.com  Fri Feb 13 18:14:08 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 13 Feb 2009 17:14:08 -0600
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
	mersenne twister key?
In-Reply-To: <20090213111755.000c2c71.michael.s.gilbert@gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<20090212151717.cdacaa60.michael.s.gilbert@gmail.com>
	<3d375d730902121232r22dcb98ckab834a7771dc07fe@mail.gmail.com>
	<20090212154621.1f03d22b.michael.s.gilbert@gmail.com>
	<49955F62.7000203@molden.no>
	<20090213100756.7d2f3e5d.michael.s.gilbert@gmail.com>
	<499590F1.8060801@molden.no>
	<20090213105153.0fb3fe9f.michael.s.gilbert@gmail.com>
	<49959A20.7070701@molden.no>
	<20090213111755.000c2c71.michael.s.gilbert@gmail.com>
Message-ID: <3d375d730902131514k66fda4b5p30d727aa1eb4b8c1@mail.gmail.com>

On Fri, Feb 13, 2009 at 10:17, Michael S. Gilbert
<michael.s.gilbert at gmail.com> wrote:
> On Fri, 13 Feb 2009 17:04:48 +0100 Sturla Molden wrote:
>> So you have a simulation written in *Python*, and the major bottleneck
>> is the MT prng? Forgive me for not believing it.
>
> Yes, running a lot of monte carlo simulations back-to-back.  if the
> PRNG were twice as fast, my code would be twice as fast.  It isn't that
> unbelievable...

It is somewhat. If you do even fair trivial code above getting the raw
bytes, the amount of time spent actually in the PRNG is only moderate.
Making the PRNG twice as fast only speeds up that moderate part, not
the entire program. Unless if you are *only* running the PRNG and
doing nothing else, then yes, speeding up the PRNG by a factor of 2
will speed up your program by that amount.

For example, here is a script that will either get random bytes,
standard Gaussian numbers, or random long ints. Using Instruments.app
on my Mac, I can find the amount of time actually spent in the PRNG as
opposed to the rest of Python or mtrand. For the bytes case, this is
around 96% of the time (although it will drop down to 80% or so with
1kb blocks). For the Gaussian case, this is 75% of the time. For the
long ints, this is actually 54% (I'm not sure why this is the slowest,
but a lot of time is being wasted in the method; worth looking into).
In the latter case, a double-speed PRNG will only speed up your
program by 25%. If you are doing actual computations with those random
numbers, these factors will only get worse.


import os

from numpy import random

print os.getpid()

prng = random.RandomState(1234567890)
while True:
    #x = prng.bytes(1024*1024*16)
    #x = prng.standard_normal(1024*1024*4)
    x = prng.tomaxint(1024*1024*4)


> Honestly, I don't feel like arguing about this anymore.  Its a matter
> of "show me the code," and when I have the time, I will "show you the
> code."

Before spending too much time on this, I highly recommend profiling
your code to see what is actually consuming the most time. Start with
the Python profiling tools:

  http://docs.python.org/library/profile

If it appears that the methods in numpy.random are actually a
significant bottleneck, you may need to break out a C profiler, too,
to determine how much time is actually being spent in the PRNG itself
as opposed to the non-uniform distributions and the Pyrex wrappers.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From sturla at molden.no  Fri Feb 13 19:29:01 2009
From: sturla at molden.no (Sturla Molden)
Date: Sat, 14 Feb 2009 01:29:01 +0100 (CET)
Subject: [Numpy-discussion] Purpose for bit-wise and'ing the initial
 mersenne twister key?
In-Reply-To: <20090213111755.000c2c71.michael.s.gilbert@gmail.com>
References: <20090206162418.0d8dd677.michael.s.gilbert@gmail.com>
	<3d375d730902061325h190b1de4o572f2ca63fd34d63@mail.gmail.com>
	<20090206175723.45e986fb.michael.s.gilbert@gmail.com>
	<3d375d730902061456m64fdac20uaa601f02c7ebb3e7@mail.gmail.com>
	<20090206181844.9b2070ff.michael.s.gilbert@gmail.com>
	<20090212115800.ad3049fa.michael.s.gilbert@gmail.com>
	<3d375d730902121118h529817f2y4444beb061e12e58@mail.gmail.com>
	<20090212151717.cdacaa60.michael.s.gilbert@gmail.com>
	<3d375d730902121232r22dcb98ckab834a7771dc07fe@mail.gmail.com>
	<20090212154621.1f03d22b.michael.s.gilbert@gmail.com>
	<49955F62.7000203@molden.no>
	<20090213100756.7d2f3e5d.michael.s.gilbert@gmail.com>
	<499590F1.8060801@molden.no>
	<20090213105153.0fb3fe9f.michael.s.gilbert@gmail.com>
	<49959A20.7070701@molden.no>
	<20090213111755.000c2c71.michael.s.gilbert@gmail.com>
Message-ID: <f7d02e8389a3908e414d26df4241aa50.squirrel@webmail.uio.no>

> On Fri, 13 Feb 2009 17:04:48 +0100 Sturla Molden wrote:

> Yes, running a lot of monte carlo simulations back-to-back.  if the
> PRNG were twice as fast, my code would be twice as fast.  It isn't that
> unbelievable...


Profile before you make such bold statements. You are implying that your
simulation does nothing but generate random integers for days on end.
Surely you must also do something useful with them?

If you make the PRNG twice as fast, you have med the PRNG twice as fast.
Everything else stays the same.

"Premature optimization is the root of all evil." (C.A.R. Hoare)


S.M.



From david at ar.media.kyoto-u.ac.jp  Sat Feb 14 09:38:11 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 14 Feb 2009 23:38:11 +0900
Subject: [Numpy-discussion] Float and locale formatting fixes merged in trunk
Message-ID: <4996D753.1000600@ar.media.kyoto-u.ac.jp>

Hi,

    I've just merged the branch Pauli and me have been working on
recently to fix several local and float formatting bugs. All tests pass
on Linux, but there may still be rough edges on Windows. Let me know if
this causes trouble,

cheers,

David


From pav at iki.fi  Sat Feb 14 10:41:30 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 14 Feb 2009 15:41:30 +0000 (UTC)
Subject: [Numpy-discussion] Float and locale formatting fixes merged in
	trunk
References: <4996D753.1000600@ar.media.kyoto-u.ac.jp>
Message-ID: <gn6ona$jou$1@ger.gmane.org>

Hi,

Sat, 14 Feb 2009 23:38:11 +0900, David Cournapeau wrote:
>     I've just merged the branch Pauli and me have been working on
> recently to fix several local and float formatting bugs. All tests pass
> on Linux, but there may still be rough edges on Windows. Let me know if
> this causes trouble,

Excellent.

I note that we can't test this now on Windows, since the trunk does not 
build (because of the atanhf umath build error).

-- 
Pauli Virtanen



From cournape at gmail.com  Sat Feb 14 10:57:12 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 15 Feb 2009 00:57:12 +0900
Subject: [Numpy-discussion] Float and locale formatting fixes merged in
	trunk
In-Reply-To: <gn6ona$jou$1@ger.gmane.org>
References: <4996D753.1000600@ar.media.kyoto-u.ac.jp>
	<gn6ona$jou$1@ger.gmane.org>
Message-ID: <5b8d13220902140757o54c629bfh5dd4a9b0fb8ffe29@mail.gmail.com>

On Sun, Feb 15, 2009 at 12:41 AM, Pauli Virtanen <pav at iki.fi> wrote:
> Hi,
>
> Sat, 14 Feb 2009 23:38:11 +0900, David Cournapeau wrote:
>>     I've just merged the branch Pauli and me have been working on
>> recently to fix several local and float formatting bugs. All tests pass
>> on Linux, but there may still be rough edges on Windows. Let me know if
>> this causes trouble,
>
> Excellent.
>
> I note that we can't test this now on Windows, since the trunk does not
> build (because of the atanhf umath build error).

which build problem ? I have built recent numpy fine

David
>
> --
> Pauli Virtanen
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From pav at iki.fi  Sat Feb 14 11:04:12 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 14 Feb 2009 16:04:12 +0000 (UTC)
Subject: [Numpy-discussion] Float and locale formatting fixes merged
	in	trunk
References: <4996D753.1000600@ar.media.kyoto-u.ac.jp>
	<gn6ona$jou$1@ger.gmane.org>
	<5b8d13220902140757o54c629bfh5dd4a9b0fb8ffe29@mail.gmail.com>
Message-ID: <gn6q1s$jou$2@ger.gmane.org>

Sun, 15 Feb 2009 00:57:12 +0900, David Cournapeau wrote:

> On Sun, Feb 15, 2009 at 12:41 AM, Pauli Virtanen <pav at iki.fi> wrote:
>> Hi,
>>
>> Sat, 14 Feb 2009 23:38:11 +0900, David Cournapeau wrote:
>>>     I've just merged the branch Pauli and me have been working on
>>> recently to fix several local and float formatting bugs. All tests
>>> pass on Linux, but there may still be rough edges on Windows. Let me
>>> know if this causes trouble,
>>
>> Excellent.
>>
>> I note that we can't test this now on Windows, since the trunk does not
>> build (because of the atanhf umath build error).
> 
> which build problem ? I have built recent numpy fine

This one:

http://buildbot.scipy.org/builders/Windows_XP_x86_64_MSVC/builds/875/steps/shell/logs/stdio

-- 
Pauli Virtanen



From josef.pktd at gmail.com  Sat Feb 14 11:09:01 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 14 Feb 2009 11:09:01 -0500
Subject: [Numpy-discussion] Float and locale formatting fixes merged in
	trunk
In-Reply-To: <gn6q1s$jou$2@ger.gmane.org>
References: <4996D753.1000600@ar.media.kyoto-u.ac.jp>
	<gn6ona$jou$1@ger.gmane.org>
	<5b8d13220902140757o54c629bfh5dd4a9b0fb8ffe29@mail.gmail.com>
	<gn6q1s$jou$2@ger.gmane.org>
Message-ID: <1cd32cbb0902140809p1f54c510o2a8dc2a24fa99849@mail.gmail.com>

On Sat, Feb 14, 2009 at 11:04 AM, Pauli Virtanen <pav at iki.fi> wrote:
> Sun, 15 Feb 2009 00:57:12 +0900, David Cournapeau wrote:
>
>> On Sun, Feb 15, 2009 at 12:41 AM, Pauli Virtanen <pav at iki.fi> wrote:
>>> Hi,
>>>
>>> Sat, 14 Feb 2009 23:38:11 +0900, David Cournapeau wrote:
>>>>     I've just merged the branch Pauli and me have been working on
>>>> recently to fix several local and float formatting bugs. All tests
>>>> pass on Linux, but there may still be rough edges on Windows. Let me
>>>> know if this causes trouble,
>>>
>>> Excellent.
>>>
>>> I note that we can't test this now on Windows, since the trunk does not
>>> build (because of the atanhf umath build error).
>>
>> which build problem ? I have built recent numpy fine
>
> This one:
>
> http://buildbot.scipy.org/builders/Windows_XP_x86_64_MSVC/builds/875/steps/shell/logs/stdio
>
> --
> Pauli Virtanen
>

I just build current trunk: win32, windowsXP, build with MinGW
no build problems,

Ran 1781 tests in 12.016s

FAILED (KNOWNFAIL=1, SKIP=1, errors=2, failures=3)
<nose.result.TextTestResult run=1781 errors=2 failures=3>

Josef
-------------------------------------------------------------


Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__file__
'C:\\Programs\\Python25\\lib\\site-packages\\numpy\\__init__.pyc'
>>> numpy.version.version
'1.3.0.dev6362'
>>> numpy.test()
Running unit tests for numpy
NumPy version 1.3.0.dev6362
NumPy is installed in C:\Programs\Python25\lib\site-packages\numpy
Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Int
el)]
nose version 0.10.4
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
...............................................................................K
................................................................................
...................Ignoring "Python was built with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible binaries.
Visual Studio 2003 was not found on this system. If you have Cygwin installed,
you can try compiling with MingW32, by passing "-c mingw32" to setup.py." (one s
hould fix me in fcompiler/compaq.py)
................................................................................
................................................................................
................................................................................
................................................................................
...............................................................E..F.....E.......
..................FF............................................................
................................................................................
................................................................................
................................................................................
................................................................................
.S..............................................................................
................................................................................
................................................................................
................................................................................
................................................................................
..
======================================================================
ERROR: test_mmap (test_io.TestSaveLoad)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line
 81, in test_mmap
    self.roundtrip(a, file_on_disk=True, load_kwds={'mmap_mode': 'r'})
  File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line
 89, in roundtrip
    RoundtripTest.roundtrip(self, np.save, *args, **kwargs)
  File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line
 57, in roundtrip
    arr_reloaded = np.load(load_file, **load_kwds)
  File "\Programs\Python25\Lib\site-packages\numpy\lib\io.py", line 142, in load

    fid = _file(file,"rb")
IOError: [Errno 2] No such file or directory: 'c:\\docume~1\\carrasco\\locals~1\
\temp\\tmpts5iyy'

======================================================================
ERROR: test_mmap (test_io.TestSavezLoad)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line
 81, in test_mmap
    self.roundtrip(a, file_on_disk=True, load_kwds={'mmap_mode': 'r'})
  File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line
 94, in roundtrip
    RoundtripTest.roundtrip(self, np.savez, *args, **kwargs)
  File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line
 57, in roundtrip
    arr_reloaded = np.load(load_file, **load_kwds)
  File "\Programs\Python25\Lib\site-packages\numpy\lib\io.py", line 142, in load

    fid = _file(file,"rb")
IOError: [Errno 2] No such file or directory: 'c:\\docume~1\\carrasco\\locals~1\
\temp\\tmphmdwar'

======================================================================
FAIL: test_array (test_io.TestSaveTxt)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line
 122, in test_array
    '3.000000000000000000e+00 4.000000000000000000e+00\n'])
AssertionError

======================================================================
FAIL: Test find_duplicates
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_recfunctions
.py", line 150, in test_find_duplicates
    assert_equal(test[-1], control)
  File "numpy\ma\testutils.py", line 121, in assert_equal
  File "numpy\ma\testutils.py", line 193, in assert_array_equal
  File "numpy\ma\testutils.py", line 186, in assert_array_compare
  File "\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line 295,
in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not equal

(mismatch 100.0%)
 x: array([2, 0])
 y: array([0, 2])

======================================================================
FAIL: Test the ignoremask option of find_duplicates
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_recfunctions
.py", line 181, in test_find_duplicates_ignoremask
    assert_equal(test[-1], control)
  File "numpy\ma\testutils.py", line 121, in assert_equal
  File "numpy\ma\testutils.py", line 193, in assert_array_equal
  File "numpy\ma\testutils.py", line 186, in assert_array_compare
  File "\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line 295,
in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not equal

(mismatch 50.0%)
 x: array([1, 0, 3, 4])
 y: array([0, 1, 3, 4])

----------------------------------------------------------------------
Ran 1781 tests in 12.016s

FAILED (KNOWNFAIL=1, SKIP=1, errors=2, failures=3)
<nose.result.TextTestResult run=1781 errors=2 failures=3>
>>>


From nwagner at iam.uni-stuttgart.de  Sat Feb 14 11:22:43 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 14 Feb 2009 17:22:43 +0100
Subject: [Numpy-discussion] astype
Message-ID: <web-117020614@uni-stuttgart.de>

Hi all,

How can I convert an array with string elements to
an array with float entries ?


  
>>> coord_info[:,1]
array(['0,0', '100,0', '200,0', '300,0', '400,0', '500,0', 
'600,0', '700,0', '800,0', '0.0', '100.0', '200.0', 
'300.0', '400.0', '500.0', '600.0',
        '700.0', '800.0', '0.0', '100.0', '200.0', 
'300.0', '400.0', '500.0', '600.0', '700.0', '800.0', 
'0.0', '100.0', '200.0', '300.0', '400.0',
        '500.0', '600.0', '700.0', '800.0', '0.0', 
'100.0', '200.0', '300.0', '400.0', '500.0', '600.0', 
'700.0', '800.0'],
       dtype='|S50')

>>> coord_info[:,1].astype(float)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ValueError: invalid literal for float(): 0,0


Nils


From nwagner at iam.uni-stuttgart.de  Sat Feb 14 11:27:28 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 14 Feb 2009 17:27:28 +0100
Subject: [Numpy-discussion] astype
In-Reply-To: <web-117020614@uni-stuttgart.de>
References: <web-117020614@uni-stuttgart.de>
Message-ID: <web-117020653@uni-stuttgart.de>

On Sat, 14 Feb 2009 17:22:43 +0100
  "Nils Wagner" <nwagner at iam.uni-stuttgart.de> wrote:
> Hi all,
> 
> How can I convert an array with string elements to
> an array with float entries ?
> 
> 
>  
>>>> coord_info[:,1]
> array(['0,0', '100,0', '200,0', '300,0', '400,0', 
>'500,0', 
> '600,0', '700,0', '800,0', '0.0', '100.0', '200.0', 
> '300.0', '400.0', '500.0', '600.0',
>        '700.0', '800.0', '0.0', '100.0', '200.0', 
> '300.0', '400.0', '500.0', '600.0', '700.0', '800.0', 
> '0.0', '100.0', '200.0', '300.0', '400.0',
>        '500.0', '600.0', '700.0', '800.0', '0.0', 
> '100.0', '200.0', '300.0', '400.0', '500.0', '600.0', 
> '700.0', '800.0'],
>       dtype='|S50')
> 
>>>> coord_info[:,1].astype(float)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ValueError: invalid literal for float(): 0,0
> 
> 
> Nils
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion


Sorry for the noise -  dots and commas were mixed up in 
the input file.

  


From cournape at gmail.com  Sat Feb 14 11:28:15 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 15 Feb 2009 01:28:15 +0900
Subject: [Numpy-discussion] Float and locale formatting fixes merged in
	trunk
In-Reply-To: <gn6q1s$jou$2@ger.gmane.org>
References: <4996D753.1000600@ar.media.kyoto-u.ac.jp>
	<gn6ona$jou$1@ger.gmane.org>
	<5b8d13220902140757o54c629bfh5dd4a9b0fb8ffe29@mail.gmail.com>
	<gn6q1s$jou$2@ger.gmane.org>
Message-ID: <5b8d13220902140828w15521565i8c33763b8f74b4ef@mail.gmail.com>

On Sun, Feb 15, 2009 at 1:04 AM, Pauli Virtanen <pav at iki.fi> wrote:
> Sun, 15 Feb 2009 00:57:12 +0900, David Cournapeau wrote:
>
>> On Sun, Feb 15, 2009 at 12:41 AM, Pauli Virtanen <pav at iki.fi> wrote:
>>> Hi,
>>>
>>> Sat, 14 Feb 2009 23:38:11 +0900, David Cournapeau wrote:
>>>>     I've just merged the branch Pauli and me have been working on
>>>> recently to fix several local and float formatting bugs. All tests
>>>> pass on Linux, but there may still be rough edges on Windows. Let me
>>>> know if this causes trouble,
>>>
>>> Excellent.
>>>
>>> I note that we can't test this now on Windows, since the trunk does not
>>> build (because of the atanhf umath build error).
>>
>> which build problem ? I have built recent numpy fine
>
> This one:
>
> http://buildbot.scipy.org/builders/Windows_XP_x86_64_MSVC/builds/875/steps/shell/logs/stdio
>

Note that this is windows x64, whose support has never been on part
with 32 bits ones. I will look at those.

I could build both python 2.5 and 2.6 versions with mingw, and only
two tests fail (unrelated for formatting issues - those are masked
arrays failures).

David


From pav at iki.fi  Sat Feb 14 12:21:23 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 14 Feb 2009 17:21:23 +0000 (UTC)
Subject: [Numpy-discussion] Buildbot not building?
Message-ID: <gn6uij$jou$3@ger.gmane.org>

Hi,

It seems that the buildbot.scipy.org is not picking up the changes in 
Numpy trunk.

I'd guess this could be some issue with SVNPoller. At least it doesn't 
preserve states across buildmaster restarts, so replacing it with the 
following might help:

{{{
import os
from buildbot.changes.svnpoller import SVNPoller

class PersistentSVNPoller(SVNPoller):
    persist_file = "svnpoll.rev"

    def __init__(self, *a, **kw):
        self.persist_file = kw.pop('persist_file', self.persist_file)
        SVNPoller.__init__(self, *a, **kw)

        if os.path.isfile(self.persist_file):
            f = open(self.persist_file, 'r')
            try:
                self.last_change = int(f.read())
            except ValueError:
                pass
            finally:
                f.close()

    def get_new_logentries(self, *a, **kw):
        r = SVNPoller.get_new_logentries(self, *a, **kw)
        f = open(self.persist_file, 'w')
        try:
            f.write("%d" % self.last_change)
        finally:
            f.close()
        return r
}}}

-- 
Pauli Virtanen



From cournape at gmail.com  Sat Feb 14 12:34:27 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 15 Feb 2009 02:34:27 +0900
Subject: [Numpy-discussion] Float and locale formatting fixes merged in
	trunk
In-Reply-To: <5b8d13220902140828w15521565i8c33763b8f74b4ef@mail.gmail.com>
References: <4996D753.1000600@ar.media.kyoto-u.ac.jp>
	<gn6ona$jou$1@ger.gmane.org>
	<5b8d13220902140757o54c629bfh5dd4a9b0fb8ffe29@mail.gmail.com>
	<gn6q1s$jou$2@ger.gmane.org>
	<5b8d13220902140828w15521565i8c33763b8f74b4ef@mail.gmail.com>
Message-ID: <5b8d13220902140934n478c72cfg818a63b7e4bae8df@mail.gmail.com>

On Sun, Feb 15, 2009 at 1:28 AM, David Cournapeau <cournape at gmail.com> wrote:

>
> Note that this is windows x64, whose support has never been on part
> with 32 bits ones. I will look at those.

I looked at the problem, and I have no clue about what's going on. To
make things interesting, the free 64 bits compilers are horribly buggy
(both V 14 and V 15), and actually segfault or give bogus parsing
errors when they encounter a problem. Of course, it builds fine with
the mingw-w64 compiler...

David


From stefan at sun.ac.za  Sat Feb 14 13:29:33 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sat, 14 Feb 2009 20:29:33 +0200
Subject: [Numpy-discussion] Bilateral filter: bug corrected
In-Reply-To: <710F2847B0018641891D9A216027636029C432@ex3.envision.co.il>
References: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il>
	<9457e7c80902130150h4dc8755eqf6097fc0d20b6522@mail.gmail.com>
	<710F2847B0018641891D9A216027636029C432@ex3.envision.co.il>
Message-ID: <9457e7c80902141029w451160a2te6f7e6eb57db2d1e@mail.gmail.com>

Hi Nadav

2009/2/13 Nadav Horesh <nadavh at visionsense.com>:
>  I tried the installer and it did not copy bilateral.py. I tried to improve it and the result is in the attached file.  I hope it would pass the mail filter, if not contact me directly to the email address below.

Thanks!  I applied your changes and modified the setup.py to support
in-place building.  Again available here:

http://github.com/stefanv/bilateral/tree/master

Cheers
St?fan


From charlesr.harris at gmail.com  Sat Feb 14 13:45:37 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 14 Feb 2009 11:45:37 -0700
Subject: [Numpy-discussion] Float and locale formatting fixes merged in
	trunk
In-Reply-To: <5b8d13220902140934n478c72cfg818a63b7e4bae8df@mail.gmail.com>
References: <4996D753.1000600@ar.media.kyoto-u.ac.jp>
	<gn6ona$jou$1@ger.gmane.org>
	<5b8d13220902140757o54c629bfh5dd4a9b0fb8ffe29@mail.gmail.com>
	<gn6q1s$jou$2@ger.gmane.org>
	<5b8d13220902140828w15521565i8c33763b8f74b4ef@mail.gmail.com>
	<5b8d13220902140934n478c72cfg818a63b7e4bae8df@mail.gmail.com>
Message-ID: <e06186140902141045rd16bc03w1dda9d6256f7590@mail.gmail.com>

On Sat, Feb 14, 2009 at 10:34 AM, David Cournapeau <cournape at gmail.com>wrote:

> On Sun, Feb 15, 2009 at 1:28 AM, David Cournapeau <cournape at gmail.com>
> wrote:
>
> >
> > Note that this is windows x64, whose support has never been on part
> > with 32 bits ones. I will look at those.
>
> I looked at the problem, and I have no clue about what's going on. To
> make things interesting, the free 64 bits compilers are horribly buggy
> (both V 14 and V 15), and actually segfault or give bogus parsing
> errors when they encounter a problem. Of course, it builds fine with
> the mingw-w64 compiler...
>

My work around fixed this problem. I think David's configuration test fails
for this function but it is really there, so when we define the function
there is an error. The workaround was giving the numpy function a different
name, npy_tanhf, then #define tanhf npy_tanhf.  Since all the tanhf
references are replaced before the compilation pass the problem goes away.
Ideally, the presence of tanhf should be detected, so I guess that's where
the official fix should be. All the numpy tests were passing on the
Windows_64 platform.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090214/1b0d6b92/attachment.html>

From stefan at sun.ac.za  Sat Feb 14 16:39:32 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sat, 14 Feb 2009 23:39:32 +0200
Subject: [Numpy-discussion] Buildbot not building?
In-Reply-To: <gn6uij$jou$3@ger.gmane.org>
References: <gn6uij$jou$3@ger.gmane.org>
Message-ID: <9457e7c80902141339o1475e696jf8a38967871f5cae@mail.gmail.com>

Hi Pauli

2009/2/14 Pauli Virtanen <pav at iki.fi>:
> It seems that the buildbot.scipy.org is not picking up the changes in
> Numpy trunk.

Thanks for the report.  I've let the system administrater know.

Once the server is able to poll from SVN, would you still recommend
using the persistent poller?

Regards
St?fan


From pav at iki.fi  Sat Feb 14 17:02:50 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 14 Feb 2009 22:02:50 +0000 (UTC)
Subject: [Numpy-discussion] Buildbot not building?
References: <gn6uij$jou$3@ger.gmane.org>
	<9457e7c80902141339o1475e696jf8a38967871f5cae@mail.gmail.com>
Message-ID: <gn7f2a$334$1@ger.gmane.org>

Sat, 14 Feb 2009 23:39:32 +0200, St?fan van der Walt wrote:
> 2009/2/14 Pauli Virtanen <pav at iki.fi>:
>> It seems that the buildbot.scipy.org is not picking up the changes in
>> Numpy trunk.
> 
> Thanks for the report.  I've let the system administrater know.
> 
> Once the server is able to poll from SVN, would you still recommend
> using the persistent poller?

I don't think keeping track of the last changed revision can ever be 
harmful. You may need to restart the service eventually, and if you use 
the ordinary SVNPoller, you'll AFAIK always miss the first commit after 
restart.

Hmm, this actually looks like a bug in buildbot...

-- 
Pauli Virtanen



From esyepez at sandia.gov  Sat Feb 14 17:29:08 2009
From: esyepez at sandia.gov (Yepez, Esteban)
Date: Sat, 14 Feb 2009 15:29:08 -0700
Subject: [Numpy-discussion] CROSS-COMPILING NUMPY
Message-ID: <D3D552504F106245B7696A9A5FD16A910DC32854B0@ES01SNLNT.srn.sandia.gov>

Hello,

I have having a hard time finding more information to help me, so I thought I'd ping you both.

I need to x-compile numpy for a new processor. I have a toolchain, and have x-compiled python2.5.4, boost1.37,swig1.38. I just can't figure out how to x-compile numpy. Now I had to play many tricks to get python configured, so I may not have everything I need:
For python,
- I modified the Modules/Setup to build math and other modules that my target doesn't have
- I modified configure so that it wouldn't crash when trying to run test programs
- I modified the makefile so it would not try to build sharedmods

Since Numpy uses python: python setup.py install, I figured that I could just use the host python to perform the build:
>CC=my-gcc-compiler /usr/local/bin/python setup.py install

Any help would be useful. Thanks

Steve

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090214/04ffe6d4/attachment.html>

From brennan.williams at visualreservoir.com  Sat Feb 14 17:30:24 2009
From: brennan.williams at visualreservoir.com (Brennan Williams)
Date: Sun, 15 Feb 2009 11:30:24 +1300
Subject: [Numpy-discussion] astype
In-Reply-To: <web-117020653@uni-stuttgart.de>
References: <web-117020614@uni-stuttgart.de> <web-117020653@uni-stuttgart.de>
Message-ID: <49974600.3060205@visualreservoir.com>

Nils Wagner wrote:
> On Sat, 14 Feb 2009 17:22:43 +0100
>   "Nils Wagner" <nwagner at iam.uni-stuttgart.de> wrote:
>   
>> Hi all,
>>
>> How can I convert an array with string elements to
>> an array with float entries ?
>>
>>
>>  
>>     
>>>>> coord_info[:,1]
>>>>>           
>> array(['0,0', '100,0', '200,0', '300,0', '400,0', 
>> '500,0', 
>> '600,0', '700,0', '800,0', '0.0', '100.0', '200.0', 
>> '300.0', '400.0', '500.0', '600.0',
>>        '700.0', '800.0', '0.0', '100.0', '200.0', 
>> '300.0', '400.0', '500.0', '600.0', '700.0', '800.0', 
>> '0.0', '100.0', '200.0', '300.0', '400.0',
>>        '500.0', '600.0', '700.0', '800.0', '0.0', 
>> '100.0', '200.0', '300.0', '400.0', '500.0', '600.0', 
>> '700.0', '800.0'],
>>       dtype='|S50')
>>
>>     
>>>>> coord_info[:,1].astype(float)
>>>>>           
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> ValueError: invalid literal for float(): 0,0
>>
>>
>> Nils
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>     
>
>
> Sorry for the noise -  dots and commas were mixed up in 
> the input file.
>
>   
Mind you, it raises an interesting point. What if the string was an 
amount in currency, e.g. '$1,000.00' or perhaps just '1,000.00' etc.
How would one convert a string array in that format to a float array?

>    
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   



From david at ar.media.kyoto-u.ac.jp  Sun Feb 15 02:30:41 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 15 Feb 2009 16:30:41 +0900
Subject: [Numpy-discussion] CROSS-COMPILING NUMPY
In-Reply-To: <D3D552504F106245B7696A9A5FD16A910DC32854B0@ES01SNLNT.srn.sandia.gov>
References: <D3D552504F106245B7696A9A5FD16A910DC32854B0@ES01SNLNT.srn.sandia.gov>
Message-ID: <4997C4A1.8090307@ar.media.kyoto-u.ac.jp>

Yepez, Esteban wrote:
> Hello,
>  
> I have having a hard time finding more information to help me, so I
> thought I'd ping you both.
>  
> I need to x-compile numpy for a new processor. I have a toolchain, and
> have x-compiled python2.5.4, boost1.37,swig1.38. I just can't figure
> out how to x-compile numpy. Now I had to play many tricks to get
> python configured, so I may not have everything I need:
> For python,
> - I modified the Modules/Setup to build math and other modules that my
> target doesn't have
> - I modified configure so that it wouldn't crash when trying to run
> test programs
> - I modified the makefile so it would not try to build sharedmods
>  
> Since Numpy uses python: python setup.py install, I figured that I
> could just use the host python to perform the build:
> >CC=my-gcc-compiler /usr/local/bin/python setup.py install

To put it simply, the build process in numpy does not support
cross-compilation at all, so you are mostly on your own, unfortunately,

David


From david at ar.media.kyoto-u.ac.jp  Sun Feb 15 02:48:53 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 15 Feb 2009 16:48:53 +0900
Subject: [Numpy-discussion] Core math library in numpy
Message-ID: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>

Hi,

    While trying to fix some issues on windows 64 bits related to the
float format fixes, I encountered some problems (lack of some basic math
functions, in particular isnan and co for the multiarray module). We
already have the portable replacing code for those functions
(umath_funcs_c99.inc.src), which is used for the umath module. For now,
we can only reuse this implementation by direct inclusion. I think it
would be better to have a separate library with those core math routines:
     - sharing by copying means more code, more compilation time.
     - it cannot be shared outside numpy core.
For some code, we don't have a choice, but for math functions, I don't
really see an argument for not sharing: having public, portable math
functions sounds useful to me. The only requirement is that we would
need to mangle the name, I think, to avoid clash with the C math library
on the platforms who have the maths. Would there be any other drawback ?
Would people be against this ?

cheers,

David


From robert.kern at gmail.com  Sun Feb 15 03:09:48 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 15 Feb 2009 02:09:48 -0600
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>

On Sun, Feb 15, 2009 at 01:48, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:

> Would people be against this ?

Not I.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From gael.varoquaux at normalesup.org  Sun Feb 15 09:04:00 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sun, 15 Feb 2009 15:04:00 +0100
Subject: [Numpy-discussion] Sphinx custom extension mess, and patches
Message-ID: <20090215140400.GH20403@phare.normalesup.org>

Hi all,

Sorry for the multiple posting, this concerns various groups, and I'd
rather the information not be lost.

While working on getting our in-lab library ready to be merged with NiPy,
I ran into some sort of 'sphinx extension mess' where various sphinx
extension would have side effects on each other, and most important, the
extensions did not work with sphinx trunk.

I got the side effects to be limited by cleaning up the generated code
from autosummary before each run: I added the following code in my
sphinx conf.py:

################################################################################
# Hack: run the autosummary generation script 
import shutil
if os.path.exists('generated'):
    shutil.rmtree('generated')
os.system('%s sphinxext/autosummary_generate.py -o generated *.rst' %
            sys.executable)
################################################################################

I am attaching a diff of all the modifications I made to get the various
extensions to work. I hope you can use it to get your various extensions
working on sphinx trunk quicker. For the NiPy guys, I will be committing
these changes in the fff2 tree soon, and we can go over this at the
sprint.

This does raise a problem: this extension code is all over the place, in
various repository. Some of the code cannot live in the sphinx repo, as
it introduces dependencies. However, as the extensions are not importable
from Python (I can't do 'from matplotlib.sphinxext import mathmpl'), the
different projects using them end up copying them in their repo, and thus
there are several versions floating around not updated. Some of the
extensions would do not add externa dependencies to sphinx. These should
be pushed into sphinx, with tests. That way as sphinx evolves, they do
not break.

Ga?l
-------------- next part --------------
=== modified file 'doc/sphinxext/autosummary.py'
--- doc/sphinxext/autosummary.py	2009-02-14 18:36:49 +0000
+++ doc/sphinxext/autosummary.py	2009-02-15 13:41:08 +0000
@@ -56,7 +56,7 @@
 from docutils.statemachine import ViewList
 from docutils import nodes
 
-import sphinx.addnodes, sphinx.roles, sphinx.builder
+import sphinx.addnodes, sphinx.roles
 from sphinx.util import patfilter
 
 from docscrape_sphinx import get_doc_object
@@ -160,6 +160,7 @@
         tocnode['includefiles'] = docnames
         tocnode['maxdepth'] = -1
         tocnode['glob'] = None
+        tocnode['entries'] = [] 
 
         tocnode = autosummary_toc('', '', tocnode)
         return warnings + [node] + [tocnode]

=== modified file 'doc/sphinxext/inheritance_diagram.py'
--- doc/sphinxext/inheritance_diagram.py	2009-02-14 18:36:49 +0000
+++ doc/sphinxext/inheritance_diagram.py	2009-02-15 12:53:28 +0000
@@ -40,7 +40,10 @@
 
 from docutils.nodes import Body, Element
 from docutils.writers.html4css1 import HTMLTranslator
-from sphinx.latexwriter import LaTeXTranslator
+try:
+    from sphinx.latexwriter import LaTeXTranslator
+except ImportError:
+    from sphinx.writers.latex import LaTeXTranslator
 from docutils.parsers.rst import directives
 from sphinx.roles import xfileref_role
 

=== modified file 'doc/sphinxext/mathmpl.py'
--- doc/sphinxext/mathmpl.py	2009-02-14 18:36:49 +0000
+++ doc/sphinxext/mathmpl.py	2009-02-15 12:45:33 +0000
@@ -7,7 +7,10 @@
 from docutils import nodes
 from docutils.parsers.rst import directives
 from docutils.writers.html4css1 import HTMLTranslator
-from sphinx.latexwriter import LaTeXTranslator
+try:
+    from sphinx.latexwriter import LaTeXTranslator
+except ImportError:
+    from sphinx.writers.latex import LaTeXTranslator
 import warnings
 
 # Define LaTeX math node:

=== modified file 'doc/sphinxext/only_directives.py'
--- doc/sphinxext/only_directives.py	2009-02-14 18:36:49 +0000
+++ doc/sphinxext/only_directives.py	2009-02-15 12:54:47 +0000
@@ -5,7 +5,10 @@
 
 from docutils.nodes import Body, Element
 from docutils.writers.html4css1 import HTMLTranslator
-from sphinx.latexwriter import LaTeXTranslator
+try:
+    from sphinx.latexwriter import LaTeXTranslator
+except ImportError:
+    from sphinx.writers.latex import LaTeXTranslator
 from docutils.parsers.rst import directives
 
 class html_only(Body, Element):


From josef.pktd at gmail.com  Sun Feb 15 11:09:20 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 15 Feb 2009 11:09:20 -0500
Subject: [Numpy-discussion] Sphinx custom extension mess, and patches
In-Reply-To: <20090215140400.GH20403@phare.normalesup.org>
References: <20090215140400.GH20403@phare.normalesup.org>
Message-ID: <1cd32cbb0902150809q35d28a0ej9a01d74c582ef940@mail.gmail.com>

On Sun, Feb 15, 2009 at 9:04 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> Hi all,
>
> Sorry for the multiple posting, this concerns various groups, and I'd
> rather the information not be lost.
>
> While working on getting our in-lab library ready to be merged with NiPy,
> I ran into some sort of 'sphinx extension mess' where various sphinx
> extension would have side effects on each other, and most important, the
> extensions did not work with sphinx trunk.
>
> I got the side effects to be limited by cleaning up the generated code
> from autosummary before each run: I added the following code in my
> sphinx conf.py:
>
> ################################################################################
> # Hack: run the autosummary generation script
> import shutil
> if os.path.exists('generated'):
>    shutil.rmtree('generated')
> os.system('%s sphinxext/autosummary_generate.py -o generated *.rst' %
>            sys.executable)
> ################################################################################
>
> I am attaching a diff of all the modifications I made to get the various
> extensions to work. I hope you can use it to get your various extensions
> working on sphinx trunk quicker. For the NiPy guys, I will be committing
> these changes in the fff2 tree soon, and we can go over this at the
> sprint.
>
> This does raise a problem: this extension code is all over the place, in
> various repository. Some of the code cannot live in the sphinx repo, as
> it introduces dependencies. However, as the extensions are not importable
> from Python (I can't do 'from matplotlib.sphinxext import mathmpl'), the
> different projects using them end up copying them in their repo, and thus
> there are several versions floating around not updated. Some of the
> extensions would do not add externa dependencies to sphinx. These should
> be pushed into sphinx, with tests. That way as sphinx evolves, they do
> not break.
>
> Ga?l
>


In my setup the plot directive doesn't create any graphs, it always
skips plots with the following warning

WARNING: C:\Josef\_progs\building\scipy\scipy-trunk-new-r5551\doc\source\tutoria
l\stats.rst:300: (ERROR/3) Error in "plot" directive:
no content permitted.

When debugging this, I discovered that in plot_directive.py it creates
the `class plot_directive(Directive)`
which doesn't seem to work. If I use the same function as in the other
path of the try except, then graphs are
correctly created. I don't know if this is specific to my version
combination or if this is a bug.
>>> import docutils
>>> docutils.__version__
'0.5'
>>> import sphinx
>>> sphinx.__version__
'0.5'
>>>

Josef

-------------------------------- plot_directive.py  starting  line 417
----------
try:
    from docutils.parsers.rst import Directive
except ImportError:
    from docutils.parsers.rst.directives import _directives

    def plot_directive(name, arguments, options, content, lineno,
                       content_offset, block_text, state, state_machine):
        return run(arguments, content, options, state_machine, state, lineno)
    plot_directive.__doc__ = __doc__
else:
    print 'running plot_directive class'
    #this class doesn't do anything
##    class plot_directive(Directive):
##        def run(self):
##            return run(self.arguments, self.content, self.options,
##                       self.state_machine, self.state, self.lineno)
    #copied from above
    def plot_directive(name, arguments, options, content, lineno,
                       content_offset, block_text, state, state_machine):
        return run(arguments, content, options, state_machine, state, lineno)

    plot_directive.__doc__ = __doc__


From pav at iki.fi  Sun Feb 15 12:09:24 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 15 Feb 2009 17:09:24 +0000 (UTC)
Subject: [Numpy-discussion] Sphinx custom extension mess, and patches
References: <20090215140400.GH20403@phare.normalesup.org>
	<1cd32cbb0902150809q35d28a0ej9a01d74c582ef940@mail.gmail.com>
Message-ID: <gn9i84$6r3$1@ger.gmane.org>

Sun, 15 Feb 2009 11:09:20 -0500, josef.pktd wrote:
[clip]
> In my setup the plot directive doesn't create any graphs, it always
> skips plots with the following warning
> 
> WARNING:
> C:\Josef\_progs\building\scipy\scipy-trunk-new-r5551\doc\source\tutoria
> l\stats.rst:300: (ERROR/3) Error in "plot" directive: no content
> permitted.

Docutils 0.5 incompatibility, fixed in trunk.

	Pauli



From josef.pktd at gmail.com  Sun Feb 15 13:52:07 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 15 Feb 2009 13:52:07 -0500
Subject: [Numpy-discussion] Sphinx custom extension mess, and patches
In-Reply-To: <gn9i84$6r3$1@ger.gmane.org>
References: <20090215140400.GH20403@phare.normalesup.org>
	<1cd32cbb0902150809q35d28a0ej9a01d74c582ef940@mail.gmail.com>
	<gn9i84$6r3$1@ger.gmane.org>
Message-ID: <1cd32cbb0902151052o7574ee69p7795a2e15896d9b0@mail.gmail.com>

On Sun, Feb 15, 2009 at 12:09 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Sun, 15 Feb 2009 11:09:20 -0500, josef.pktd wrote:
> [clip]
>> In my setup the plot directive doesn't create any graphs, it always
>> skips plots with the following warning
>>
>> WARNING:
>> C:\Josef\_progs\building\scipy\scipy-trunk-new-r5551\doc\source\tutoria
>> l\stats.rst:300: (ERROR/3) Error in "plot" directive: no content
>> permitted.
>
> Docutils 0.5 incompatibility, fixed in trunk.
>
>        Pauli
>

Thanks, this works.

Is there a way to prevent the following text from flowing around the
included graph (produced with the plot directive)? I have two empty
lines after the plot directive to start a new paragraph but the text
flows to the right of the graph.

Josef


From josef.pktd at gmail.com  Sun Feb 15 19:53:01 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 15 Feb 2009 19:53:01 -0500
Subject: [Numpy-discussion] Sphinx custom extension mess, and patches
In-Reply-To: <1cd32cbb0902151052o7574ee69p7795a2e15896d9b0@mail.gmail.com>
References: <20090215140400.GH20403@phare.normalesup.org>
	<1cd32cbb0902150809q35d28a0ej9a01d74c582ef940@mail.gmail.com>
	<gn9i84$6r3$1@ger.gmane.org>
	<1cd32cbb0902151052o7574ee69p7795a2e15896d9b0@mail.gmail.com>
Message-ID: <1cd32cbb0902151653r46918bbdn6508f8040ca34e94@mail.gmail.com>

I tracked down one more windows specific bug in the plot directive.
Initially the link to the source file for a graph generated by the
plot_directive didn't work (literal text instead of link). The source
was that the source link contains `\\` as separators and not `/`

The fix is in the call to jinja, replace the os.path.sep also for the
source_link  in the same way as is done for the link_dir.
After this change, the source links render correctly in the html output.


    result = jinja.from_string(TEMPLATE).render(
        link_dir=link_dir.replace(os.path.sep, '/'),
        source_link=source_link.replace(os.path.sep, '/'),       #
change os.path.sep on windows
        options=opts,
        image_names=image_names,
        source_code=source_code)

The generated or copied source files for the plots do not contain any
file extension, in my changes I also added '.py' extension back to the
filename. Is there a reason for dropping the file extension?

Josef


From mdroe at stsci.edu  Mon Feb 16 08:54:29 2009
From: mdroe at stsci.edu (Michael Droettboom)
Date: Mon, 16 Feb 2009 08:54:29 -0500
Subject: [Numpy-discussion] [matplotlib-devel] Sphinx custom extension
	mess, and patches
In-Reply-To: <20090215140400.GH20403@phare.normalesup.org>
References: <20090215140400.GH20403@phare.normalesup.org>
Message-ID: <49997015.9070108@stsci.edu>

Gael,

You raise a very good point about the duplication of code around.  As a 
case in point, the patches you provided no longer apply to the 
"canonical" (or at least original) versions of the plugins that began 
life in matplotlib.  Recent versions of Sphinx have a proper extension 
API, so that "try ... except" importing is no longer necessary.

For mathmpl.py --- I will move that into the matplotlib installed code 
tree, so, as you suggest, "from matplotlib.sphinxext import mathmpl" 
will work.  Obviously, that won't really provide traction until the next 
release of matplotlib.  And it relies on only_directive.py, so I will 
probably have to put that in matplotlib as well for now, though that is 
a reasonable candidate for inclusion in Sphinx.

I have submitted inheritance_diagram.py to Sphinx already.  There didn't 
seem to be much interest, and there were some details (particularly how 
it deals with files), that weren't "the Sphinx way", and there few 
documents and/or examples etc. at the time about how to do it right.  
Someone else ended up re-engineering it to use pygraphviz which felt 
pretty heavy weight to me.  So the thing kind of stalled.  But it's 
probably worth another push.

Mike

Gael Varoquaux wrote:
> Hi all,
>
> Sorry for the multiple posting, this concerns various groups, and I'd
> rather the information not be lost.
>
> While working on getting our in-lab library ready to be merged with NiPy,
> I ran into some sort of 'sphinx extension mess' where various sphinx
> extension would have side effects on each other, and most important, the
> extensions did not work with sphinx trunk.
>
> I got the side effects to be limited by cleaning up the generated code
> from autosummary before each run: I added the following code in my
> sphinx conf.py:
>
> ################################################################################
> # Hack: run the autosummary generation script 
> import shutil
> if os.path.exists('generated'):
>     shutil.rmtree('generated')
> os.system('%s sphinxext/autosummary_generate.py -o generated *.rst' %
>             sys.executable)
> ################################################################################
>
> I am attaching a diff of all the modifications I made to get the various
> extensions to work. I hope you can use it to get your various extensions
> working on sphinx trunk quicker. For the NiPy guys, I will be committing
> these changes in the fff2 tree soon, and we can go over this at the
> sprint.
>
> This does raise a problem: this extension code is all over the place, in
> various repository. Some of the code cannot live in the sphinx repo, as
> it introduces dependencies. However, as the extensions are not importable
> from Python (I can't do 'from matplotlib.sphinxext import mathmpl'), the
> different projects using them end up copying them in their repo, and thus
> there are several versions floating around not updated. Some of the
> extensions would do not add externa dependencies to sphinx. These should
> be pushed into sphinx, with tests. That way as sphinx evolves, they do
> not break.
>
> Ga?l
>   
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
> -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
> -Strategies to boost innovation and cut costs with open source participation
> -Receive a $600 discount off the registration fee with the source code: SFAD
> http://p.sf.net/sfu/XcvMzF8H
> ------------------------------------------------------------------------
>
> _______________________________________________
> Matplotlib-devel mailing list
> Matplotlib-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

-- 
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA



From axel.breuer at bnpparibas.com  Mon Feb 16 09:04:04 2009
From: axel.breuer at bnpparibas.com (axel.breuer at bnpparibas.com)
Date: Mon, 16 Feb 2009 15:04:04 +0100
Subject: [Numpy-discussion] numpy.linalg or scipy.linalg ?
Message-ID: <OF46D710E3.5D297F4C-ONC125755F.004CD1BC-C125755F.004D475C@bnpparibas.com>


Hi,

Which one is better numpy.linalg or scipy.linalg ?

These modules have some common functions but with different
help lines.

Furthermore, numpy.scipy.linalg.svd seems to be more stable
than numpy.linalg.svd (are they using different LAPACK methods ?)





This message and any attachments (the "message") is
intended solely for the addressees and is confidential. 
If you receive this message in error, please delete it and 
immediately notify the sender. Any use not in accord with 
its purpose, any dissemination or disclosure, either whole 
or partial, is prohibited except formal approval. The internet
can not guarantee the integrity of this message. 
BNP PARIBAS (and its subsidiaries) shall (will) not 
therefore be liable for the message if modified. 
Do not print this message unless it is necessary,
consider the environment.

                ---------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le 
"message") sont etablis a l'intention exclusive de ses 
destinataires et sont confidentiels. Si vous recevez ce 
message par erreur, merci de le detruire et d'en avertir 
immediatement l'expediteur. Toute utilisation de ce 
message non conforme a sa destination, toute diffusion 
ou toute publication, totale ou partielle, est interdite, sauf 
autorisation expresse. L'internet ne permettant pas 
d'assurer l'integrite de ce message, BNP PARIBAS (et ses
filiales) decline(nt) toute responsabilite au titre de ce 
message, dans l'hypothese ou il aurait ete modifie.
N'imprimez ce message que si necessaire,
pensez a l'environnement.



From david at ar.media.kyoto-u.ac.jp  Mon Feb 16 08:57:41 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 16 Feb 2009 22:57:41 +0900
Subject: [Numpy-discussion] numpy.linalg or scipy.linalg ?
In-Reply-To: <OF46D710E3.5D297F4C-ONC125755F.004CD1BC-C125755F.004D475C@bnpparibas.com>
References: <OF46D710E3.5D297F4C-ONC125755F.004CD1BC-C125755F.004D475C@bnpparibas.com>
Message-ID: <499970D5.1050502@ar.media.kyoto-u.ac.jp>

Hi Axel,

axel.breuer at bnpparibas.com wrote:
> Hi,
>
> Which one is better numpy.linalg or scipy.linalg ?
>   

scipy.linalg is more complete, but both numpy and scipy may use the
underlying LAPACK available on your platform.

> These modules have some common functions but with different
> help lines.
>   

Yes, there are some redundancy, unfortunately, mostly for historical
reasons I believe.

> Furthermore, numpy.scipy.linalg.svd seems to be more stable
> than numpy.linalg.svd (are they using different LAPACK methods ?)
>   

If both numpy and scipy were built on a machine with BLAS/LAPACK, they
should use the same implementation. One difference is that numpy can be
built without any blas/lapack (and then uses its own implementation).
Which platform are you on ? Did you build numpy/scipy by yourself, or
did you use a pre-built package ? There are many possible different
configurations, unfortunately.

You can also check the underlying detected implementation with show_config:

import numpy as np
import scipy as sc
print np.show_config()
print sc.show_config()

cheers,

David


From esyepez at sandia.gov  Mon Feb 16 10:23:19 2009
From: esyepez at sandia.gov (Yepez, Esteban)
Date: Mon, 16 Feb 2009 08:23:19 -0700
Subject: [Numpy-discussion] CROSS-COMPILING NUMPY
In-Reply-To: <4997C4A1.8090307@ar.media.kyoto-u.ac.jp>
Message-ID: <C5BED2F7.E71%esyepez@sandia.gov>

Are there any patches from previous versions? Can provide any other suggestions?


On 2/15/09 12:30 AM, "David Cournapeau" <david at ar.media.kyoto-u.ac.jp> wrote:

Yepez, Esteban wrote:
> Hello,
>
> I have having a hard time finding more information to help me, so I
> thought I'd ping you both.
>
> I need to x-compile numpy for a new processor. I have a toolchain, and
> have x-compiled python2.5.4, boost1.37,swig1.38. I just can't figure
> out how to x-compile numpy. Now I had to play many tricks to get
> python configured, so I may not have everything I need:
> For python,
> - I modified the Modules/Setup to build math and other modules that my
> target doesn't have
> - I modified configure so that it wouldn't crash when trying to run
> test programs
> - I modified the makefile so it would not try to build sharedmods
>
> Since Numpy uses python: python setup.py install, I figured that I
> could just use the host python to perform the build:
> >CC=my-gcc-compiler /usr/local/bin/python setup.py install

To put it simply, the build process in numpy does not support
cross-compilation at all, so you are mostly on your own, unfortunately,

David
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090216/c3cfc3ef/attachment.html>

From josef.pktd at gmail.com  Mon Feb 16 14:27:41 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 16 Feb 2009 14:27:41 -0500
Subject: [Numpy-discussion] [matplotlib-devel] Sphinx custom extension
	mess, and patches
In-Reply-To: <49997015.9070108@stsci.edu>
References: <20090215140400.GH20403@phare.normalesup.org>
	<49997015.9070108@stsci.edu>
Message-ID: <1cd32cbb0902161127y359ee483scc4e2a4967b1f134@mail.gmail.com>

another docutils 0.5 bug in the plot_directive (align doesn't work):

from docutils.parsers.rst.directives.images import Image
    align = Image.align

Image.align is not callable,
see http://svn.berlios.de/viewcvs/docutils/trunk/docutils/docutils/parsers/rst/directives/images.py?view=markup

I replaced it with:

def _option_align(arg):
    return directives.choice(arg, ("top", "middle", "bottom", "left",
"center", "right"))

from docutils.parsers.rst import directives
try:
    # docutils 0.4
    from docutils.parsers.rst.directives.images import align
except ImportError:
    # docutils 0.5
    #from docutils.parsers.rst.directives.images import Image
    align = _option_align
      # instead of Image.align

This seems to work correctly, e.g. align="center" is added to html.

The problem with the following text floating around the graph, is a
problem with Internet Explorer 6 and the created htmlhelp but not with
Firefox. As a remedy, I removed the float option in div.plot-output
.figure  in scipy.css and now the htmlhelp doesn't have a floating
text anymore.

Josef


From pav at iki.fi  Mon Feb 16 14:35:45 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 16 Feb 2009 19:35:45 +0000 (UTC)
Subject: [Numpy-discussion] [matplotlib-devel] Sphinx custom
	extension	mess, and patches
References: <20090215140400.GH20403@phare.normalesup.org>
	<49997015.9070108@stsci.edu>
	<1cd32cbb0902161127y359ee483scc4e2a4967b1f134@mail.gmail.com>
Message-ID: <gncf6h$uag$3@ger.gmane.org>

Mon, 16 Feb 2009 14:27:41 -0500, josef.pktd wrote:
> another docutils 0.5 bug in the plot_directive (align doesn't work):
>
> from docutils.parsers.rst.directives.images import Image
>     align = Image.align
> 
> Image.align is not callable,
[clip]

Strange, it didn't do that to me with docutils 0.5. Anyway, probably
better fix it anyway.

[clip]
> The problem with the following text floating around the graph, is a
> problem with Internet Explorer 6 and the created htmlhelp but not with
> Firefox. As a remedy, I removed the float option in div.plot-output
> .figure  in scipy.css and now the htmlhelp doesn't have a floating text
> anymore.

Yep, IE6 doesn't seem to understand the :after CSS attribute. But it
seems to be added in IE8, if that's any consolation :)

Ah, the joys of HTML/CSS design. I wonder if there's an easy workaround
to this...

-- 
Pauli Virtanen



From robert.kern at gmail.com  Mon Feb 16 14:36:07 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 16 Feb 2009 13:36:07 -0600
Subject: [Numpy-discussion] Sphinx custom extension mess, and patches
In-Reply-To: <20090215140400.GH20403@phare.normalesup.org>
References: <20090215140400.GH20403@phare.normalesup.org>
Message-ID: <3d375d730902161136w7ffdb5dcld8af63afe841a66c@mail.gmail.com>

On Sun, Feb 15, 2009 at 08:04, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> Hi all,
>
> Sorry for the multiple posting, this concerns various groups, and I'd
> rather the information not be lost.
>
> While working on getting our in-lab library ready to be merged with NiPy,
> I ran into some sort of 'sphinx extension mess' where various sphinx
> extension would have side effects on each other, and most important, the
> extensions did not work with sphinx trunk.

At one point, some of us had a plan to keep all of these "scientific"
extensions collected here:

  http://sphinx.googlecode.com/svn/contrib/trunk/numpyext/

SVN-using projects could use svn:externals to include these in their
projects without diverging the code. I really don't know why this plan
changed. Pauli?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From kbasye1 at jhu.edu  Mon Feb 16 18:06:46 2009
From: kbasye1 at jhu.edu (Ken Basye)
Date: Mon, 16 Feb 2009 18:06:46 -0500
Subject: [Numpy-discussion] Compute multiple outer products without a loop?
Message-ID: <4999F186.2010805@jhu.edu>

Hi List,
   I need to compute multiple outer products from 2-d data in the 
following way:
Given a and b with shape, e.g,  (10, 4), compute the 10 outer products 
of shape (4, 4) and get them into an array of shape (10, 4, 4).  
Currently I do this with a loop, but I'd really like some way to do it 
without looping.  I read that outer(a, b) is just syntactic sugar for 
a.ravel()[:, newaxis] * b.ravel()[newaxis,:] but unfortunately this 
didn't give me any bright ideas :->.
   Thanks,
       Ken
 


From charlesr.harris at gmail.com  Mon Feb 16 18:16:03 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 16 Feb 2009 16:16:03 -0700
Subject: [Numpy-discussion] Compute multiple outer products without a
	loop?
In-Reply-To: <4999F186.2010805@jhu.edu>
References: <4999F186.2010805@jhu.edu>
Message-ID: <e06186140902161516m4b2b2dd9xa2da5a021a25fa16@mail.gmail.com>

On Mon, Feb 16, 2009 at 4:06 PM, Ken Basye <kbasye1 at jhu.edu> wrote:

> Hi List,
>   I need to compute multiple outer products from 2-d data in the
> following way:
> Given a and b with shape, e.g,  (10, 4), compute the 10 outer products
> of shape (4, 4) and get them into an array of shape (10, 4, 4).
> Currently I do this with a loop, but I'd really like some way to do it
> without looping.  I read that outer(a, b) is just syntactic sugar for
> a.ravel()[:, newaxis] * b.ravel()[newaxis,:] but unfortunately this
> didn't give me any bright ideas :->.
>   Thanks,
>       Ken
>

I think you can do what you want with newaxis and broadcasting, but I'm not
sure what you want ;) Could you post your current code and a small example?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090216/79a54aff/attachment.html>

From gael.varoquaux at normalesup.org  Mon Feb 16 18:21:06 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 17 Feb 2009 00:21:06 +0100
Subject: [Numpy-discussion] Sphinx custom extension mess, and patches
In-Reply-To: <4999F3FD.4020806@python.org>
References: <20090215140400.GH20403@phare.normalesup.org>
	<4999F3FD.4020806@python.org>
Message-ID: <20090216232106.GC24472@phare.normalesup.org>

On Tue, Feb 17, 2009 at 12:17:17AM +0100, Georg Brandl wrote:
> I'm all for it.  In the case of autosummary, I'm guilty of not getting it
> in sooner.  This will change soon.  In other cases, I don't even know of
> the extension, probably because those who write it deem it as too
> project-specific to contribute it.

> I don't ask for too much if an extension is contributed, so by all means
> do at least post about your extensions!

I am not blaming anyone, just pointing out a non ideal situation. It has
already improved a lot with the matplotlib guys and the scipy guys
merging some changes in extensions and publishing the extensions in an
importable part of their source tree.

It is true that I'll be happier when I will be able to import the
only_directive, and the auto_summary from sphinx :).

Thanks for your great work,

Ga?l


From cournape at gmail.com  Mon Feb 16 22:37:02 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 17 Feb 2009 12:37:02 +0900
Subject: [Numpy-discussion] CROSS-COMPILING NUMPY
In-Reply-To: <C5BED2F7.E71%esyepez@sandia.gov>
References: <4997C4A1.8090307@ar.media.kyoto-u.ac.jp>
	<C5BED2F7.E71%esyepez@sandia.gov>
Message-ID: <5b8d13220902161937x5f2b1932i3412c0e90c8c6108@mail.gmail.com>

On Tue, Feb 17, 2009 at 12:23 AM, Yepez, Esteban <esyepez at sandia.gov> wrote:
> Are there any patches from previous versions?

Do you mean for python itself or numpy ? For numpy, not that I know
of. For old python, there are some patches in the tracker, but none of
it has been integrated AFAIK.

> Can provide any other
> suggestions?

It is difficult to provide general guidelines: distutils, the tool we
are using for compiling numpy, has never been conceived with cross
compilation in mind. One immediate problem is that distutils uses
python to get compilation flags - you would like to use the options of
the target python, but that's hard to do since you cannot execute it.
Also, some configuration tests need to be executed on the target
machine: this needs to be fixed.

You may be able to do it using many hacks, but doing it in a
maintainable and clean way (which would end up in integrated patches)
would be a lot of work, I think.

David


From jordan at math.ucsb.edu  Mon Feb 16 22:49:01 2009
From: jordan at math.ucsb.edu (jordan at math.ucsb.edu)
Date: Mon, 16 Feb 2009 19:49:01 -0800 (PST)
Subject: [Numpy-discussion] GMRES internal variables
Message-ID: <60880.189.62.54.78.1234842541.squirrel@mail.math.ucsb.edu>

Hi all,

I'm trying to run multiple instances of GMRES at the same time (one inside
another actually, used inside of the preconditioner routine) but am
running in to some problems. My guess is there is a single set of internal
variables associated with GMRES and when I initiate a new GMRES inside a
currently running GMRES, the old GMRES data gets the boot.

Is this correct, and if so any suggestions on how to fix this?

Cheers,
Jordan


From wnbell at gmail.com  Tue Feb 17 02:15:57 2009
From: wnbell at gmail.com (Nathan Bell)
Date: Tue, 17 Feb 2009 02:15:57 -0500
Subject: [Numpy-discussion] GMRES internal variables
In-Reply-To: <60880.189.62.54.78.1234842541.squirrel@mail.math.ucsb.edu>
References: <60880.189.62.54.78.1234842541.squirrel@mail.math.ucsb.edu>
Message-ID: <d05265cb0902162315s3820634bjdd74534c7977e350@mail.gmail.com>

On Mon, Feb 16, 2009 at 10:49 PM,  <jordan at math.ucsb.edu> wrote:
>
> I'm trying to run multiple instances of GMRES at the same time (one inside
> another actually, used inside of the preconditioner routine) but am
> running in to some problems. My guess is there is a single set of internal
> variables associated with GMRES and when I initiate a new GMRES inside a
> currently running GMRES, the old GMRES data gets the boot.
>
> Is this correct, and if so any suggestions on how to fix this?
>

This recently came up on SciPy-User:
http://thread.gmane.org/gmane.comp.python.scientific.user/19197/focus=19206

One solution:  PyAMG's GMRES implementation (pyamg.krylov.gmres)
should be a drop-in replacement for SciPy's gmres().  Unlike the CG
code mentioned above, you can't use gmres.py without some other
compiled components (i.e. you'll need the whole pyamg package).

We should have this resolved in the next scipy release (either 0.7.x or 0.8).

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From schut at sarvision.nl  Tue Feb 17 05:04:08 2009
From: schut at sarvision.nl (Vincent Schut)
Date: Tue, 17 Feb 2009 11:04:08 +0100
Subject: [Numpy-discussion] fancy view question
Message-ID: <gne22p$q9$1@ger.gmane.org>

Hi list,

would it be possible to create a view on an array, such that this view 
is twice as large (in some dimensions) and in fact does a nearest 
neighbour 'zoom' on the original array? E.g. using some fancy 
slicing/striding tricks?

an example:

a = [[1, 2],
      [3, 4]]

then I'd like a view on a such that this view is:

[[1, 1, 2, 2],
  [1, 1, 2, 2],
  [3, 3, 4, 4],
  [3, 3, 4, 4]]

I know several ways to create this as a copy, but as memory does play a 
role in this case (yes, I know it's cheap :-))...

If not, fancy ways to create this as a copy are apreciated. I currently 
either create an empty array and then loop-fill it (e.g. this would be 
2x2 loops), of if I' lazy I use ndimage.zoom, which actually is a bit 
overkill.

Regards,
Vincent.



From washakie at gmail.com  Tue Feb 17 05:09:12 2009
From: washakie at gmail.com (John [H2O])
Date: Tue, 17 Feb 2009 02:09:12 -0800 (PST)
Subject: [Numpy-discussion] efficient function calculation between two
 matrices
Message-ID: <22054148.post@talk.nabble.com>


hello,

I am trying to calculate the results of a function between two matrices:

>>> F.shape
(170, 2)
>>> T.shape
(170, 481, 2)

Where F contains lat/lon pairs and T contains 481 lat/lon pairs for 170
trajectories of length 481

I want a new array of shape
(170,481)

containing the results of my distance function:

gcd(x1,y1,x2,y3):
   """ returns great circle distance"""
   ...
   return dist


Is there a way to do this without the use of loops?

I'm thinking something along the lines of:

distances = [[gcd(f[0],f[1],t[0],t[1]]) for f in F] for t in T]

But obviously this doesn't work...

Thank you,
john
-- 
View this message in context: http://www.nabble.com/efficient-function-calculation-between-two-matrices-tp22054148p22054148.html
Sent from the Numpy-discussion mailing list archive at Nabble.com.



From josef.pktd at gmail.com  Tue Feb 17 09:13:42 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 17 Feb 2009 09:13:42 -0500
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <gne22p$q9$1@ger.gmane.org>
References: <gne22p$q9$1@ger.gmane.org>
Message-ID: <1cd32cbb0902170613x61952244k6b0653646b6a476d@mail.gmail.com>

On Tue, Feb 17, 2009 at 5:04 AM, Vincent Schut <schut at sarvision.nl> wrote:
> Hi list,
>
> would it be possible to create a view on an array, such that this view
> is twice as large (in some dimensions) and in fact does a nearest
> neighbour 'zoom' on the original array? E.g. using some fancy
> slicing/striding tricks?
>
> an example:
>
> a = [[1, 2],
>      [3, 4]]
>
> then I'd like a view on a such that this view is:
>
> [[1, 1, 2, 2],
>  [1, 1, 2, 2],
>  [3, 3, 4, 4],
>  [3, 3, 4, 4]]
>
> I know several ways to create this as a copy, but as memory does play a
> role in this case (yes, I know it's cheap :-))...
>
> If not, fancy ways to create this as a copy are apreciated. I currently
> either create an empty array and then loop-fill it (e.g. this would be
> 2x2 loops), of if I' lazy I use ndimage.zoom, which actually is a bit
> overkill.
>
> Regards,
> Vincent.


I don't know about the view but your array just looks like a kronecker
product to me.

>>> import numpy as np
>>> np.kron([[1, 2], [3, 4]], [[1, 1], [1, 1]])
array([[1, 1, 2, 2],
       [1, 1, 2, 2],
       [3, 3, 4, 4],
       [3, 3, 4, 4]])

Josef


From stefan at sun.ac.za  Tue Feb 17 09:31:41 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 17 Feb 2009 16:31:41 +0200
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <gne22p$q9$1@ger.gmane.org>
References: <gne22p$q9$1@ger.gmane.org>
Message-ID: <9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>

Hi Vincent

2009/2/17 Vincent Schut <schut at sarvision.nl>:
> Hi list,
>
> would it be possible to create a view on an array, such that this view
> is twice as large (in some dimensions) and in fact does a nearest
> neighbour 'zoom' on the original array? E.g. using some fancy
> slicing/striding tricks?
>
> an example:
>
> a = [[1, 2],
>      [3, 4]]
>
> then I'd like a view on a such that this view is:
>
> [[1, 1, 2, 2],
>  [1, 1, 2, 2],
>  [3, 3, 4, 4],
>  [3, 3, 4, 4]]

np.lib.stride_tricks.as_strided(x, (2, 2, 2, 2), (8, 0, 4, 0)).reshape((4, 4))

Cheers
St?fan


From stefan at sun.ac.za  Tue Feb 17 09:42:21 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 17 Feb 2009 16:42:21 +0200
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
Message-ID: <9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>

2009/2/17 St?fan van der Walt <stefan at sun.ac.za>:
> 2009/2/17 Vincent Schut <schut at sarvision.nl>:
>> Hi list,
>>
>> would it be possible to create a view on an array, such that this view
>> is twice as large (in some dimensions) and in fact does a nearest
>> neighbour 'zoom' on the original array? E.g. using some fancy
>> slicing/striding tricks?
>>
>> an example:
>>
>> a = [[1, 2],
>>      [3, 4]]
>>
>> then I'd like a view on a such that this view is:
>>
>> [[1, 1, 2, 2],
>>  [1, 1, 2, 2],
>>  [3, 3, 4, 4],
>>  [3, 3, 4, 4]]
>
> np.lib.stride_tricks.as_strided(x, (2, 2, 2, 2), (8, 0, 4, 0)).reshape((4, 4))

Or, more generally:

import numpy as np

def zoom(x, factor=2):
        rows, cols = x.shape
        row_stride, col_stride = x.strides
        view = np.lib.stride_tricks.as_strided(x,
                        (rows, factor, cols, factor),
                        (row_stride, 0, col_stride, 0))
        return view.reshape((rows*factor, cols*factor))

a = np.array([[1, 2, 3],
              [4, 5, 6]])
print zoom(a, 2)

Cheers
St?fan


From gael.varoquaux at normalesup.org  Tue Feb 17 09:44:49 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 17 Feb 2009 15:44:49 +0100
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
Message-ID: <20090217144449.GM17638@phare.normalesup.org>

On Tue, Feb 17, 2009 at 04:42:21PM +0200, St?fan van der Walt wrote:
> Or, more generally:

> import numpy as np

> def zoom(x, factor=2):
>         rows, cols = x.shape
>         row_stride, col_stride = x.strides
>         view = np.lib.stride_tricks.as_strided(x,
>                         (rows, factor, cols, factor),
>                         (row_stride, 0, col_stride, 0))
>         return view.reshape((rows*factor, cols*factor))

That's handy, you should commit this somewhere. Actually, it would be
even cooler if you could have different zoom factor in different
direction :).

Ga?l



From stefan at sun.ac.za  Tue Feb 17 09:58:57 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 17 Feb 2009 16:58:57 +0200
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <20090217144449.GM17638@phare.normalesup.org>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
Message-ID: <9457e7c80902170658n3b9834f4i63ba74474dbb90da@mail.gmail.com>

2009/2/17 Gael Varoquaux <gael.varoquaux at normalesup.org>:
> That's handy, you should commit this somewhere. Actually, it would be
> even cooler if you could have different zoom factor in different
> direction :).

Something like this:

a = np.array([[1, 2, 3],
              [4, 5, 6]])
print a
print zoom(a, x=2, y=3)

[[1 2 3]
 [4 5 6]]
[[1 1 2 2 3 3]
 [1 1 2 2 3 3]
 [1 1 2 2 3 3]
 [4 4 5 5 6 6]
 [4 4 5 5 6 6]
 [4 4 5 5 6 6]]

(Code attached)

Cheers
St?fan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zoom.py
Type: text/x-python-script
Size: 744 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090217/fdd4a0eb/attachment.bin>

From gael.varoquaux at normalesup.org  Tue Feb 17 10:01:13 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 17 Feb 2009 16:01:13 +0100
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <9457e7c80902170658n3b9834f4i63ba74474dbb90da@mail.gmail.com>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
	<9457e7c80902170658n3b9834f4i63ba74474dbb90da@mail.gmail.com>
Message-ID: <20090217150113.GO17638@phare.normalesup.org>

On Tue, Feb 17, 2009 at 04:58:57PM +0200, St?fan van der Walt wrote:
> 2009/2/17 Gael Varoquaux <gael.varoquaux at normalesup.org>:
> > That's handy, you should commit this somewhere. Actually, it would be
> > even cooler if you could have different zoom factor in different
> > direction :).

> Something like this:

> a = np.array([[1, 2, 3],
>               [4, 5, 6]])
> print a
> print zoom(a, x=2, y=3)

Exactly, but with a signature that accepts a integer, or a tuple of
integers. And commiting this in eg stride_tricks.

Ga?l


From josef.pktd at gmail.com  Tue Feb 17 10:01:21 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 17 Feb 2009 10:01:21 -0500
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <20090217144449.GM17638@phare.normalesup.org>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
Message-ID: <1cd32cbb0902170701jea399d3l8968d1b8d5970740@mail.gmail.com>

On Tue, Feb 17, 2009 at 9:44 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Tue, Feb 17, 2009 at 04:42:21PM +0200, St?fan van der Walt wrote:
>> Or, more generally:
>
>> import numpy as np
>
>> def zoom(x, factor=2):
>>         rows, cols = x.shape
>>         row_stride, col_stride = x.strides
>>         view = np.lib.stride_tricks.as_strided(x,
>>                         (rows, factor, cols, factor),
>>                         (row_stride, 0, col_stride, 0))
>>         return view.reshape((rows*factor, cols*factor))
>
> That's handy, you should commit this somewhere. Actually, it would be
> even cooler if you could have different zoom factor in different
> direction :).
>

I completely agree.

I was looking at the help file and docs, and I didn't find it.
np.lib.stride_tricks.as_strided is not in np.lib.stride_tricks.__all__
 and not in the docs. So it's still a hidden treasure.

Josef


From josef.pktd at gmail.com  Tue Feb 17 10:08:56 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 17 Feb 2009 10:08:56 -0500
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <20090217150113.GO17638@phare.normalesup.org>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
	<9457e7c80902170658n3b9834f4i63ba74474dbb90da@mail.gmail.com>
	<20090217150113.GO17638@phare.normalesup.org>
Message-ID: <1cd32cbb0902170708k6a6a43d9o433c5aa09d3f76b@mail.gmail.com>

On Tue, Feb 17, 2009 at 10:01 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Tue, Feb 17, 2009 at 04:58:57PM +0200, St?fan van der Walt wrote:
>> 2009/2/17 Gael Varoquaux <gael.varoquaux at normalesup.org>:
>> > That's handy, you should commit this somewhere. Actually, it would be
>> > even cooler if you could have different zoom factor in different
>> > direction :).
>
>> Something like this:
>
>> a = np.array([[1, 2, 3],
>>               [4, 5, 6]])
>> print a
>> print zoom(a, x=2, y=3)
>
> Exactly, but with a signature that accepts a integer, or a tuple of
> integers. And commiting this in eg stride_tricks.
>
> Ga?l

I think rows and columns are in wrong sequence (axis=0, axis=1) ?

>>> zoom(a, 2,3)
array([[1, 1, 2, 2, 3, 3],
       [1, 1, 2, 2, 3, 3],
       [1, 1, 2, 2, 3, 3],
       [4, 4, 5, 5, 6, 6],
       [4, 4, 5, 5, 6, 6],
       [4, 4, 5, 5, 6, 6]])

>>> zoom(a, 3,2)
array([[1, 1, 1, 2, 2, 2, 3, 3, 3],
       [1, 1, 1, 2, 2, 2, 3, 3, 3],
       [4, 4, 4, 5, 5, 5, 6, 6, 6],
       [4, 4, 4, 5, 5, 5, 6, 6, 6]])

Josef


From robert.kern at gmail.com  Tue Feb 17 10:09:38 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 Feb 2009 09:09:38 -0600
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <20090217144449.GM17638@phare.normalesup.org>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
Message-ID: <3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>

On Tue, Feb 17, 2009 at 08:44, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Tue, Feb 17, 2009 at 04:42:21PM +0200, St?fan van der Walt wrote:
>> Or, more generally:
>
>> import numpy as np
>
>> def zoom(x, factor=2):
>>         rows, cols = x.shape
>>         row_stride, col_stride = x.strides
>>         view = np.lib.stride_tricks.as_strided(x,
>>                         (rows, factor, cols, factor),
>>                         (row_stride, 0, col_stride, 0))
>>         return view.reshape((rows*factor, cols*factor))
>
> That's handy, you should commit this somewhere. Actually, it would be
> even cooler if you could have different zoom factor in different
> direction :).

np.repeat(np.repeat(x, 2, axis=0), 2, axis=1)

stride_tricks are fun, but this is already a solved problem in numpy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From david.huard at gmail.com  Tue Feb 17 10:10:32 2009
From: david.huard at gmail.com (David Huard)
Date: Tue, 17 Feb 2009 10:10:32 -0500
Subject: [Numpy-discussion] GMRES internal variables
In-Reply-To: <d05265cb0902162315s3820634bjdd74534c7977e350@mail.gmail.com>
References: <60880.189.62.54.78.1234842541.squirrel@mail.math.ucsb.edu>
	<d05265cb0902162315s3820634bjdd74534c7977e350@mail.gmail.com>
Message-ID: <91cf711d0902170710l1155f7bbvddbaf052e5504b79@mail.gmail.com>

Nathan,

First of all, thanks to all your work on the sparse linear algebra package,
I am starting to use it and it's much appreciated.

Just a thought: wouldn't it be more natural to write gmres as a class rather
than a function ? That way, accessing the internal work arrays for reuse
would be much easier. These are useful when using gmres in a inexact newton
loop.

Cheers,

David




On Tue, Feb 17, 2009 at 2:15 AM, Nathan Bell <wnbell at gmail.com> wrote:

> On Mon, Feb 16, 2009 at 10:49 PM,  <jordan at math.ucsb.edu> wrote:
> >
> > I'm trying to run multiple instances of GMRES at the same time (one
> inside
> > another actually, used inside of the preconditioner routine) but am
> > running in to some problems. My guess is there is a single set of
> internal
> > variables associated with GMRES and when I initiate a new GMRES inside a
> > currently running GMRES, the old GMRES data gets the boot.
> >
> > Is this correct, and if so any suggestions on how to fix this?
> >
>
> This recently came up on SciPy-User:
> http://thread.gmane.org/gmane.comp.python.scientific.user/19197/focus=19206
>
> One solution:  PyAMG's GMRES implementation (pyamg.krylov.gmres)
> should be a drop-in replacement for SciPy's gmres().  Unlike the CG
> code mentioned above, you can't use gmres.py without some other
> compiled components (i.e. you'll need the whole pyamg package).
>
> We should have this resolved in the next scipy release (either 0.7.x or
> 0.8).
>
> --
> Nathan Bell wnbell at gmail.com
> http://graphics.cs.uiuc.edu/~wnbell/<http://graphics.cs.uiuc.edu/%7Ewnbell/>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090217/4851b5bf/attachment.html>

From josef.pktd at gmail.com  Tue Feb 17 10:15:19 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 17 Feb 2009 10:15:19 -0500
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
	<3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>
Message-ID: <1cd32cbb0902170715m2077ea9pc381273fb8a89cf4@mail.gmail.com>

On Tue, Feb 17, 2009 at 10:09 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Tue, Feb 17, 2009 at 08:44, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
>> On Tue, Feb 17, 2009 at 04:42:21PM +0200, St?fan van der Walt wrote:
>>> Or, more generally:
>>
>>> import numpy as np
>>
>>> def zoom(x, factor=2):
>>>         rows, cols = x.shape
>>>         row_stride, col_stride = x.strides
>>>         view = np.lib.stride_tricks.as_strided(x,
>>>                         (rows, factor, cols, factor),
>>>                         (row_stride, 0, col_stride, 0))
>>>         return view.reshape((rows*factor, cols*factor))
>>
>> That's handy, you should commit this somewhere. Actually, it would be
>> even cooler if you could have different zoom factor in different
>> direction :).
>
> np.repeat(np.repeat(x, 2, axis=0), 2, axis=1)
>
> stride_tricks are fun, but this is already a solved problem in numpy.
>
> --
> Robert Kern
>

I'm still learning about views:

>>> b = np.repeat(np.repeat(a, 2, axis=0), 2, axis=1)
>>> b.flags
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

Does OWNDATA : True  mean it made a copy? Or is there another way to
check whether it's a view or a copy?
zoom has OWNDATA : False

>>> c = zoom(a, 3,2)
>>> c.flags
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

Josef


From pav at iki.fi  Tue Feb 17 10:19:05 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 17 Feb 2009 15:19:05 +0000 (UTC)
Subject: [Numpy-discussion] fancy view question
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
	<3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>
	<1cd32cbb0902170715m2077ea9pc381273fb8a89cf4@mail.gmail.com>
Message-ID: <gnekh9$30h$1@ger.gmane.org>

Tue, 17 Feb 2009 10:15:19 -0500, josef.pktd wrote:
[clip]
> I'm still learning about views:
[clip: c = foo(a); c.flags.owndata]
> Does OWNDATA : True  mean it made a copy? 

Yes. But owndata==False does not mean no copy was made (since the result 
could be a view to a temporary array).

> Or is there another way to check whether it's a view or a copy?

c.base is a

-- 
Pauli Virtanen



From robert.kern at gmail.com  Tue Feb 17 10:21:49 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 Feb 2009 09:21:49 -0600
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <1cd32cbb0902170715m2077ea9pc381273fb8a89cf4@mail.gmail.com>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
	<3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>
	<1cd32cbb0902170715m2077ea9pc381273fb8a89cf4@mail.gmail.com>
Message-ID: <3d375d730902170721l1e0f2dbqba20becee8020a79@mail.gmail.com>

On Tue, Feb 17, 2009 at 09:15,  <josef.pktd at gmail.com> wrote:

> I'm still learning about views:
>
>>>> b = np.repeat(np.repeat(a, 2, axis=0), 2, axis=1)
>>>> b.flags
>  C_CONTIGUOUS : True
>  F_CONTIGUOUS : False
>  OWNDATA : True
>  WRITEABLE : True
>  ALIGNED : True
>  UPDATEIFCOPY : False
>
> Does OWNDATA : True  mean it made a copy? Or is there another way to
> check whether it's a view or a copy?
> zoom has OWNDATA : False
>
>>>> c = zoom(a, 3,2)
>>>> c.flags
>  C_CONTIGUOUS : True
>  F_CONTIGUOUS : False
>  OWNDATA : False
>  WRITEABLE : True
>  ALIGNED : True
>  UPDATEIFCOPY : False

zoom() does not return a view of the input. It can't. That kind of
view *cannot* be created in the shape/strides memory model. The
.reshape() has to make a copy.

In [10]: x = array([[1,2],[3,4]])

In [11]: np.lib.stride_tricks.as_strided(x, (2, 2, 2, 2), (8, 0, 4,
0)).reshape((4, 4))
Out[11]:
array([[1, 1, 2, 2],
       [1, 1, 2, 2],
       [3, 3, 4, 4],
       [3, 3, 4, 4]])

In [12]: y = _

In [13]: x[0,0] = 10

In [14]: y
Out[14]:
array([[1, 1, 2, 2],
       [1, 1, 2, 2],
       [3, 3, 4, 4],
       [3, 3, 4, 4]])

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Tue Feb 17 10:21:51 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 17 Feb 2009 17:21:51 +0200
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
	<3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>
Message-ID: <9457e7c80902170721k78481415yca8d4f704a24ef8d@mail.gmail.com>

2009/2/17 Robert Kern <robert.kern at gmail.com>:
> stride_tricks are fun, but this is already a solved problem in numpy.

To get back to the fun part:  I see now the zoomed view is not a view
but a new array.  How do we get around that?

Cheers
St?fan


From robert.kern at gmail.com  Tue Feb 17 10:25:00 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 Feb 2009 09:25:00 -0600
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <9457e7c80902170721k78481415yca8d4f704a24ef8d@mail.gmail.com>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
	<3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>
	<9457e7c80902170721k78481415yca8d4f704a24ef8d@mail.gmail.com>
Message-ID: <3d375d730902170725g72fe2c83r2a8e2eeb7f592b20@mail.gmail.com>

On Tue, Feb 17, 2009 at 09:21, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2009/2/17 Robert Kern <robert.kern at gmail.com>:
>> stride_tricks are fun, but this is already a solved problem in numpy.
>
> To get back to the fun part:  I see now the zoomed view is not a view
> but a new array.  How do we get around that?

You can't. numpy's memory model simply cannot represent that permutation.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Tue Feb 17 10:30:14 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 17 Feb 2009 17:30:14 +0200
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <3d375d730902170725g72fe2c83r2a8e2eeb7f592b20@mail.gmail.com>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
	<3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>
	<9457e7c80902170721k78481415yca8d4f704a24ef8d@mail.gmail.com>
	<3d375d730902170725g72fe2c83r2a8e2eeb7f592b20@mail.gmail.com>
Message-ID: <9457e7c80902170730j458bdfc5k790a258cb0f47b85@mail.gmail.com>

2009/2/17 Robert Kern <robert.kern at gmail.com>:
>> To get back to the fun part:  I see now the zoomed view is not a view
>> but a new array.  How do we get around that?
>
> You can't. numpy's memory model simply cannot represent that permutation.

Right, something like the PIL's pointer-to-pointers format.

I should have realised when I had to introduce dummy dimensions to do
a reshape that this was the case.

Cheers
St?fan


From cournape at gmail.com  Tue Feb 17 11:08:17 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 18 Feb 2009 01:08:17 +0900
Subject: [Numpy-discussion] denormal test: what is it for ?
Message-ID: <5b8d13220902170808p7848b42eo2e8a47257478b79@mail.gmail.com>

Hi,

I would like to continue cleaning the setup.py from numpy/core, and
there is one test that I can't make sense of: the denormal thing
(testcode_mathlib function). The svn log has no information on it (its
presence goes back to the first revision of the file). What is it
useful for ? Denormal support ? Whether exp works ?

cheers,

David


From gael.varoquaux at normalesup.org  Tue Feb 17 11:16:10 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 17 Feb 2009 17:16:10 +0100
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
	<3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>
Message-ID: <20090217161610.GQ17638@phare.normalesup.org>

On Tue, Feb 17, 2009 at 09:09:38AM -0600, Robert Kern wrote:
> np.repeat(np.repeat(x, 2, axis=0), 2, axis=1)

> stride_tricks are fun, but this is already a solved problem in numpy.

Wow. I still have a lot to learn! How about adding a see-also in
as_strided.

Ga?l


From robert.kern at gmail.com  Tue Feb 17 11:18:11 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 Feb 2009 10:18:11 -0600
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <20090217161610.GQ17638@phare.normalesup.org>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
	<3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>
	<20090217161610.GQ17638@phare.normalesup.org>
Message-ID: <3d375d730902170818u7cd5bebaj1a9c51657ce9ca51@mail.gmail.com>

On Tue, Feb 17, 2009 at 10:16, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Tue, Feb 17, 2009 at 09:09:38AM -0600, Robert Kern wrote:
>> np.repeat(np.repeat(x, 2, axis=0), 2, axis=1)
>
>> stride_tricks are fun, but this is already a solved problem in numpy.
>
> Wow. I still have a lot to learn! How about adding a see-also in
> as_strided.

They're not really related.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From gael.varoquaux at normalesup.org  Tue Feb 17 11:20:06 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 17 Feb 2009 17:20:06 +0100
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <3d375d730902170818u7cd5bebaj1a9c51657ce9ca51@mail.gmail.com>
References: <gne22p$q9$1@ger.gmane.org>
	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>
	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>
	<20090217144449.GM17638@phare.normalesup.org>
	<3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>
	<20090217161610.GQ17638@phare.normalesup.org>
	<3d375d730902170818u7cd5bebaj1a9c51657ce9ca51@mail.gmail.com>
Message-ID: <20090217162006.GR17638@phare.normalesup.org>

On Tue, Feb 17, 2009 at 10:18:11AM -0600, Robert Kern wrote:
> On Tue, Feb 17, 2009 at 10:16, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
> > On Tue, Feb 17, 2009 at 09:09:38AM -0600, Robert Kern wrote:
> >> np.repeat(np.repeat(x, 2, axis=0), 2, axis=1)

> >> stride_tricks are fun, but this is already a solved problem in numpy.

> > Wow. I still have a lot to learn! How about adding a see-also in
> > as_strided.

> They're not really related.

Yes, I now understand that there where no views involved, on the contrary
to what I first thought, so indeed there are not related.

Ga?l


From kbasye1 at jhu.edu  Tue Feb 17 10:30:56 2009
From: kbasye1 at jhu.edu (Ken Basye)
Date: Tue, 17 Feb 2009 10:30:56 -0500
Subject: [Numpy-discussion] Compute multiple outer products without a
	loop?
In-Reply-To: <mailman.5963.1234865365.2878.numpy-discussion@scipy.org>
References: <mailman.5963.1234865365.2878.numpy-discussion@scipy.org>
Message-ID: <499AD830.80600@jhu.edu>

Hi,
   My current code looks like this:

           (k,d) = m.shape
           sq = np.zeros((k, d, d), dtype=float)
           for i in xrange(k):
               sq[i] = np.outer(m[i], m[i])
 

That is, m is treated as a sequence of k vectors of length d; the k dXd 
outer products are found and stored in sq. 

Note that the expression np.outer(m, m) will always produce an array 
with shape (p, p) where p is the product over the shape of m.  E.g. if 
m.shape == (10, 4) then outer(m, m).shape == (40, 40) whereas I want 
something with shape (10, 4, 4).   I suppose all the values I want are 
indeed somewhere in this 2d array, since it contains every pairwise 
product, but I'm not sure I want to do that much extra computation just 
to avoid the loop, plus I'd still have to figure out how to get a view 
that covered only the results I care about.

Thanks,
     Ken



From robert.kern at gmail.com  Tue Feb 17 11:36:40 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 Feb 2009 10:36:40 -0600
Subject: [Numpy-discussion] efficient function calculation between two
	matrices
In-Reply-To: <22054148.post@talk.nabble.com>
References: <22054148.post@talk.nabble.com>
Message-ID: <3d375d730902170836tdc9ec90rd301d1f12f1619d@mail.gmail.com>

On Tue, Feb 17, 2009 at 04:09, John [H2O] <washakie at gmail.com> wrote:
>
> hello,
>
> I am trying to calculate the results of a function between two matrices:
>
>>>> F.shape
> (170, 2)
>>>> T.shape
> (170, 481, 2)
>
> Where F contains lat/lon pairs and T contains 481 lat/lon pairs for 170
> trajectories of length 481
>
> I want a new array of shape
> (170,481)
>
> containing the results of my distance function:
>
> gcd(x1,y1,x2,y3):
>   """ returns great circle distance"""
>   ...
>   return dist
>
>
> Is there a way to do this without the use of loops?
>
> I'm thinking something along the lines of:
>
> distances = [[gcd(f[0],f[1],t[0],t[1]]) for f in F] for t in T]
>
> But obviously this doesn't work...

Change the shape of T to (481, 170, 2). Then it will be
broadcast-compatible with the (170, 2) array as long as your gcd()
function is written to take advantage of broadcasting.

distances = gcd(F[:,0], F[:,1], T[:,:,0], T[:,:,1])

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Tue Feb 17 13:04:56 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 17 Feb 2009 11:04:56 -0700
Subject: [Numpy-discussion] Compute multiple outer products without a
	loop?
In-Reply-To: <499AD830.80600@jhu.edu>
References: <mailman.5963.1234865365.2878.numpy-discussion@scipy.org>
	<499AD830.80600@jhu.edu>
Message-ID: <e06186140902171004g850c9aax348a3e2053bb69fa@mail.gmail.com>

On Tue, Feb 17, 2009 at 8:30 AM, Ken Basye <kbasye1 at jhu.edu> wrote:

> Hi,
>   My current code looks like this:
>
>           (k,d) = m.shape
>           sq = np.zeros((k, d, d), dtype=float)
>           for i in xrange(k):
>               sq[i] = np.outer(m[i], m[i])
>
>
> That is, m is treated as a sequence of k vectors of length d; the k dXd
> outer products are found and stored in sq.
>

Try A[:,:,newaxis]*B[:,newaxis,:] . Example

In [6]: A = array([[1,2],[3,4]])

In [7]: B = array([[1,1],[1,1]])

In [8]: A[:,:,newaxis]*B[:,newaxis,:]
Out[8]:
array([[[1, 1],
        [2, 2]],

       [[3, 3],
        [4, 4]]])

In [9]: B[:,:,newaxis]*A[:,newaxis,:]
Out[9]:
array([[[1, 2],
        [1, 2]],

       [[3, 4],
        [3, 4]]])

You can use this sort of trick along with a sum to multiply stacks of
matrices by stacks of vectors or matrices.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090217/6880612b/attachment.html>

From charlesr.harris at gmail.com  Tue Feb 17 16:16:50 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 17 Feb 2009 14:16:50 -0700
Subject: [Numpy-discussion] Warnings in current trunk
Message-ID: <e06186140902171316t6b15df96h54ea3d48b1952160@mail.gmail.com>

I see a lot of warnings like:

In file included from numpy/core/src/multiarraymodule.c:87:
numpy/core/src/umath_funcs_c99.inc.src:199:1: warning: "isnan" redefined
In file included from /usr/include/python2.5/pyport.h:204,
                 from /usr/include/python2.5/Python.h:57,
                 from numpy/core/src/multiarraymodule.c:18:

These are probably from r6363.

Also

In file included from numpy/core/src/scalartypes.inc.src:10,
                 from numpy/core/src/arrayobject.c:537,
                 from numpy/core/src/multiarraymodule.c:102:
numpy/core/src/numpyos.c: In function 'NumPyOS_ascii_strtod':
numpy/core/src/numpyos.c:431: warning: assignment discards qualifiers from
pointer target type
numpy/core/src/numpyos.c:480: warning: assignment discards qualifiers from
pointer target type

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090217/6b097ddf/attachment.html>

From beryl.cxy at gmail.com  Tue Feb 17 22:59:42 2009
From: beryl.cxy at gmail.com (Xiaoyu Chu)
Date: Tue, 17 Feb 2009 22:59:42 -0500
Subject: [Numpy-discussion] lack of memory?
Message-ID: <42099e890902171959s5e16b00j2f0dbfa059b48be1@mail.gmail.com>

Hello Everyone,

       I am trying to compute the eigenvalues of a matrix of size 200*200.
What happened is that whenever I run the program, python dies, without
showing an error message. The program works fine when the matrix size is
small, like in 10s. Does anyone know if it is because of the memory issue or
something else? By the way,  I am using the method linalg.eigvals(M)

Many thanks.
-- 
Best Regards,
Xiaoyu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090217/1289355b/attachment.html>

From mattdm at mattdm.org  Tue Feb 17 23:03:18 2009
From: mattdm at mattdm.org (Matthew Miller)
Date: Tue, 17 Feb 2009 23:03:18 -0500
Subject: [Numpy-discussion] lack of memory?
In-Reply-To: <42099e890902171959s5e16b00j2f0dbfa059b48be1@mail.gmail.com>
References: <42099e890902171959s5e16b00j2f0dbfa059b48be1@mail.gmail.com>
Message-ID: <20090218040318.GA17602@jadzia.bu.edu>

On Tue, Feb 17, 2009 at 10:59:42PM -0500, Xiaoyu Chu wrote:
>        I am trying to compute the eigenvalues of a matrix of size 200*200.
> What happened is that whenever I run the program, python dies, without
> showing an error message. The program works fine when the matrix size is
> small, like in 10s. Does anyone know if it is because of the memory issue or
> something else? By the way,  I am using the method linalg.eigvals(M)

Is this on Linux? If so, try this at a terminal prompt:

 dmesg | grep oom



-- 
Matthew Miller           mattdm at mattdm.org          <http://mattdm.org/>


From david at ar.media.kyoto-u.ac.jp  Tue Feb 17 22:48:35 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 18 Feb 2009 12:48:35 +0900
Subject: [Numpy-discussion] lack of memory?
In-Reply-To: <42099e890902171959s5e16b00j2f0dbfa059b48be1@mail.gmail.com>
References: <42099e890902171959s5e16b00j2f0dbfa059b48be1@mail.gmail.com>
Message-ID: <499B8513.7000404@ar.media.kyoto-u.ac.jp>

Xiaoyu Chu wrote:
> Hello Everyone,
>
>        I am trying to compute the eigenvalues of a matrix of size
> 200*200. What happened is that whenever I run the program, python
> dies, without showing an error message. The program works fine when
> the matrix size is small, like in 10s. Does anyone know if it is
> because of the memory issue or something else? By the way,  I am using
> the method linalg.eigvals(M)

Which platform are you on ? Which versions of numpy and scipy are you
using ? I doubt it is a memory issue,

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Tue Feb 17 22:56:59 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 18 Feb 2009 12:56:59 +0900
Subject: [Numpy-discussion] Warnings in current trunk
In-Reply-To: <e06186140902171316t6b15df96h54ea3d48b1952160@mail.gmail.com>
References: <e06186140902171316t6b15df96h54ea3d48b1952160@mail.gmail.com>
Message-ID: <499B870B.6040002@ar.media.kyoto-u.ac.jp>

Charles R Harris wrote:
> I see a lot of warnings like:
>
> In file included from numpy/core/src/multiarraymodule.c:87:
> numpy/core/src/umath_funcs_c99.inc.src:199:1: warning: "isnan" redefined
> In file included from /usr/include/python2.5/pyport.h:204,
>                  from /usr/include/python2.5/Python.h:57,
>                  from numpy/core/src/multiarraymodule.c:18:
>
> These are probably from r6363.
>
> Also
>
> In file included from numpy/core/src/scalartypes.inc.src:10,
>                  from numpy/core/src/arrayobject.c:537,
>                  from numpy/core/src/multiarraymodule.c:102:
> numpy/core/src/numpyos.c: In function 'NumPyOS_ascii_strtod':
> numpy/core/src/numpyos.c:431: warning: assignment discards qualifiers
> from pointer target type
> numpy/core/src/numpyos.c:480: warning: assignment discards qualifiers
> from pointer target type

The second one should be easy to fix, the first one not so much. The
problem is simple: for the float format fixes recently merged in the
trunk, we need some math functions - those format functions are used in
multiarray extension. Up to now, we only needed math function in ufunc,
and did so by including .c files. I first did the obvious thing,
including the same file as well in multiarray, but it does not work so
well, as you can see.

I think this "including .c" business has got to a point where it is
becoming insane :) Not only does it make the code difficult to follow
(at least to me), but it also causes various issues on MS platforms,
which are difficult to track (VS definitely does not like big files, and
often segfaults). Of course, for python API, we don't have a choice
because of C limitations (if we care about cross platform that is), but
we can make things better.

My recent question about a core math library comes exactly from this. Of
course, I then discovered that some distutils limitations makes
compiling this core math library more difficult than expected (because
the config.h generation cannot be done as it is for a library, only for
an extension).

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Tue Feb 17 23:51:19 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 18 Feb 2009 13:51:19 +0900
Subject: [Numpy-discussion] Warnings in current trunk
In-Reply-To: <499B870B.6040002@ar.media.kyoto-u.ac.jp>
References: <e06186140902171316t6b15df96h54ea3d48b1952160@mail.gmail.com>
	<499B870B.6040002@ar.media.kyoto-u.ac.jp>
Message-ID: <499B93C7.9050000@ar.media.kyoto-u.ac.jp>

David Cournapeau wrote:
> Charles R Harris wrote:
>   
>> I see a lot of warnings like:
>>
>> In file included from numpy/core/src/multiarraymodule.c:87:
>> numpy/core/src/umath_funcs_c99.inc.src:199:1: warning: "isnan" redefined
>> In file included from /usr/include/python2.5/pyport.h:204,
>>                  from /usr/include/python2.5/Python.h:57,
>>                  from numpy/core/src/multiarraymodule.c:18:
>>
>> These are probably from r6363.
>>
>> Also
>>
>> In file included from numpy/core/src/scalartypes.inc.src:10,
>>                  from numpy/core/src/arrayobject.c:537,
>>                  from numpy/core/src/multiarraymodule.c:102:
>> numpy/core/src/numpyos.c: In function 'NumPyOS_ascii_strtod':
>> numpy/core/src/numpyos.c:431: warning: assignment discards qualifiers
>> from pointer target type
>> numpy/core/src/numpyos.c:480: warning: assignment discards qualifiers
>> from pointer target type
>>     
Pauli, can you take a quick look whether the fixes for the above in
r6372 make sense ? I only tested on Linux,

cheers,

David


From charlesr.harris at gmail.com  Wed Feb 18 00:21:46 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 17 Feb 2009 22:21:46 -0700
Subject: [Numpy-discussion] Warnings in current trunk
In-Reply-To: <499B870B.6040002@ar.media.kyoto-u.ac.jp>
References: <e06186140902171316t6b15df96h54ea3d48b1952160@mail.gmail.com>
	<499B870B.6040002@ar.media.kyoto-u.ac.jp>
Message-ID: <e06186140902172121s1061121fvc1fcc0d0f410dd6d@mail.gmail.com>

On Tue, Feb 17, 2009 at 8:56 PM, David Cournapeau <
david at ar.media.kyoto-u.ac.jp> wrote:

> Charles R Harris wrote:
> > I see a lot of warnings like:
> >
> > In file included from numpy/core/src/multiarraymodule.c:87:
> > numpy/core/src/umath_funcs_c99.inc.src:199:1: warning: "isnan" redefined
> > In file included from /usr/include/python2.5/pyport.h:204,
> >                  from /usr/include/python2.5/Python.h:57,
> >                  from numpy/core/src/multiarraymodule.c:18:
> >
> > These are probably from r6363.
> >
> > Also
> >
> > In file included from numpy/core/src/scalartypes.inc.src:10,
> >                  from numpy/core/src/arrayobject.c:537,
> >                  from numpy/core/src/multiarraymodule.c:102:
> > numpy/core/src/numpyos.c: In function 'NumPyOS_ascii_strtod':
> > numpy/core/src/numpyos.c:431: warning: assignment discards qualifiers
> > from pointer target type
> > numpy/core/src/numpyos.c:480: warning: assignment discards qualifiers
> > from pointer target type
>
> The second one should be easy to fix, the first one not so much. The
> problem is simple: for the float format fixes recently merged in the
> trunk, we need some math functions - those format functions are used in
> multiarray extension. Up to now, we only needed math function in ufunc,
> and did so by including .c files. I first did the obvious thing,
> including the same file as well in multiarray, but it does not work so
> well, as you can see.
>
> I think this "including .c" business has got to a point where it is
> becoming insane :) Not only does it make the code difficult to follow
> (at least to me), but it also causes various issues on MS platforms,
> which are difficult to track (VS definitely does not like big files, and
> often segfaults). Of course, for python API, we don't have a choice
> because of C limitations (if we care about cross platform that is), but
> we can make things better.
>

The easy fix is to compile separately and link but that will expose a lot of
extraneous symbols in the extension module. Not that most would notice ;) I
believe there are linker specific commands that can be used to control that,
but we have to support a lot of platforms and no doubt life becomes messy.
It's not something I know much about in any case. I think we should also
have some honest header files, the automatically generated ones bug me.

I think your quick fix for the second problem is a bit bogus, it essentially
discards the const qualifier on purpose instead of by accident. I tried
pretty much the same quick fix and the problem was just pushed elsewhere. I
think the function needs a bit of redesign...

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090217/25483edc/attachment.html>

From charlesr.harris at gmail.com  Wed Feb 18 00:23:48 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 17 Feb 2009 22:23:48 -0700
Subject: [Numpy-discussion] lack of memory?
In-Reply-To: <42099e890902171959s5e16b00j2f0dbfa059b48be1@mail.gmail.com>
References: <42099e890902171959s5e16b00j2f0dbfa059b48be1@mail.gmail.com>
Message-ID: <e06186140902172123p20a56664y661b35e855e5751e@mail.gmail.com>

On Tue, Feb 17, 2009 at 8:59 PM, Xiaoyu Chu <beryl.cxy at gmail.com> wrote:

> Hello Everyone,
>
>        I am trying to compute the eigenvalues of a matrix of size 200*200.
> What happened is that whenever I run the program, python dies, without
> showing an error message. The program works fine when the matrix size is
> small, like in 10s. Does anyone know if it is because of the memory issue or
> something else? By the way,  I am using the method linalg.eigvals(M)
>

Smells like Atlas/Lapack to me, 200*200 isn't that big. What platform are
you on?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090217/4301c0e0/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Wed Feb 18 00:22:17 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 18 Feb 2009 14:22:17 +0900
Subject: [Numpy-discussion] Warnings in current trunk
In-Reply-To: <e06186140902172121s1061121fvc1fcc0d0f410dd6d@mail.gmail.com>
References: <e06186140902171316t6b15df96h54ea3d48b1952160@mail.gmail.com>	<499B870B.6040002@ar.media.kyoto-u.ac.jp>
	<e06186140902172121s1061121fvc1fcc0d0f410dd6d@mail.gmail.com>
Message-ID: <499B9B09.6080800@ar.media.kyoto-u.ac.jp>

Charles R Harris wrote:
>
>
> On Tue, Feb 17, 2009 at 8:56 PM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp <mailto:david at ar.media.kyoto-u.ac.jp>>
> wrote:
>
>     Charles R Harris wrote:
>     > I see a lot of warnings like:
>     >
>     > In file included from numpy/core/src/multiarraymodule.c:87:
>     > numpy/core/src/umath_funcs_c99.inc.src:199:1: warning: "isnan"
>     redefined
>     > In file included from /usr/include/python2.5/pyport.h:204,
>     >                  from /usr/include/python2.5/Python.h:57,
>     >                  from numpy/core/src/multiarraymodule.c:18:
>     >
>     > These are probably from r6363.
>     >
>     > Also
>     >
>     > In file included from numpy/core/src/scalartypes.inc.src:10,
>     >                  from numpy/core/src/arrayobject.c:537,
>     >                  from numpy/core/src/multiarraymodule.c:102:
>     > numpy/core/src/numpyos.c: In function 'NumPyOS_ascii_strtod':
>     > numpy/core/src/numpyos.c:431: warning: assignment discards
>     qualifiers
>     > from pointer target type
>     > numpy/core/src/numpyos.c:480: warning: assignment discards
>     qualifiers
>     > from pointer target type
>
>     The second one should be easy to fix, the first one not so much. The
>     problem is simple: for the float format fixes recently merged in the
>     trunk, we need some math functions - those format functions are
>     used in
>     multiarray extension. Up to now, we only needed math function in
>     ufunc,
>     and did so by including .c files. I first did the obvious thing,
>     including the same file as well in multiarray, but it does not work so
>     well, as you can see.
>
>     I think this "including .c" business has got to a point where it is
>     becoming insane :) Not only does it make the code difficult to follow
>     (at least to me), but it also causes various issues on MS platforms,
>     which are difficult to track (VS definitely does not like big
>     files, and
>     often segfaults). Of course, for python API, we don't have a choice
>     because of C limitations (if we care about cross platform that
>     is), but
>     we can make things better.
>
>
> The easy fix is to compile separately and link but that will expose a
> lot of extraneous symbols in the extension module.

Is may be easy in principle, but it is not so practically, because of
various build issues (the separate library has to be built first, but
the current code to generate config.h assumes to be built through an
extension). But I agree, that's the only portable solution, and it has
other advantages as well (being able to reuse our portable math
functions in other extensions, for example). The extraneous symbols are
OK if we decorate them (npy_log, etc...) and I think it is actually
helpful to have name to make the difference between the libc and our
function. In particular, if later we can provide several differently
optimized functions.

> Not that most would notice ;) I believe there are linker specific
> commands that can be used to control that, but we have to support a
> lot of platforms and no doubt life becomes messy.

Yes, it would be messy: I don't know any platform which does not have
this capability (Gnu linker, MS linker and Solaris linker all have
this), but to implement it would be a lot of effort for relatively
little gain.

> I think your quick fix for the second problem is a bit bogus, it
> essentially discards the const qualifier on purpose instead of by
> accident.

Yes, but that's the only solution. FWIW, I loooked a bit at how the
glibc does things, and it is pretty similar. You can't avoid casts
everywhere - and the cast I added are correct I believe.

> I tried pretty much the same quick fix and the problem was just pushed
> elsewhere.

Where ? I compiled with -W -Wall -Wextra, and there was no other warning
(in that part of the code I mean), at least on 32 bits. But before
checking more thoroughly, I want to finish cleaning numpy/core/setup.py

David



From charlesr.harris at gmail.com  Wed Feb 18 01:14:45 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 17 Feb 2009 23:14:45 -0700
Subject: [Numpy-discussion] Warnings in current trunk
In-Reply-To: <499B9B09.6080800@ar.media.kyoto-u.ac.jp>
References: <e06186140902171316t6b15df96h54ea3d48b1952160@mail.gmail.com>
	<499B870B.6040002@ar.media.kyoto-u.ac.jp>
	<e06186140902172121s1061121fvc1fcc0d0f410dd6d@mail.gmail.com>
	<499B9B09.6080800@ar.media.kyoto-u.ac.jp>
Message-ID: <e06186140902172214g50403456h63e49402de54b1db@mail.gmail.com>

On Tue, Feb 17, 2009 at 10:22 PM, David Cournapeau <
david at ar.media.kyoto-u.ac.jp> wrote:

> Charles R Harris wrote:
> >
> >
> > On Tue, Feb 17, 2009 at 8:56 PM, David Cournapeau
> > <david at ar.media.kyoto-u.ac.jp <mailto:david at ar.media.kyoto-u.ac.jp>>
> > wrote:
> >
> >     Charles R Harris wrote:
> >     > I see a lot of warnings like:
> >     >
> >     > In file included from numpy/core/src/multiarraymodule.c:87:
> >     > numpy/core/src/umath_funcs_c99.inc.src:199:1: warning: "isnan"
> >     redefined
> >     > In file included from /usr/include/python2.5/pyport.h:204,
> >     >                  from /usr/include/python2.5/Python.h:57,
> >     >                  from numpy/core/src/multiarraymodule.c:18:
> >     >
> >     > These are probably from r6363.
> >     >
> >     > Also
> >     >
> >     > In file included from numpy/core/src/scalartypes.inc.src:10,
> >     >                  from numpy/core/src/arrayobject.c:537,
> >     >                  from numpy/core/src/multiarraymodule.c:102:
> >     > numpy/core/src/numpyos.c: In function 'NumPyOS_ascii_strtod':
> >     > numpy/core/src/numpyos.c:431: warning: assignment discards
> >     qualifiers
> >     > from pointer target type
> >     > numpy/core/src/numpyos.c:480: warning: assignment discards
> >     qualifiers
> >     > from pointer target type
> >
> >     The second one should be easy to fix, the first one not so much. The
> >     problem is simple: for the float format fixes recently merged in the
> >     trunk, we need some math functions - those format functions are
> >     used in
> >     multiarray extension. Up to now, we only needed math function in
> >     ufunc,
> >     and did so by including .c files. I first did the obvious thing,
> >     including the same file as well in multiarray, but it does not work
> so
> >     well, as you can see.
> >
> >     I think this "including .c" business has got to a point where it is
> >     becoming insane :) Not only does it make the code difficult to follow
> >     (at least to me), but it also causes various issues on MS platforms,
> >     which are difficult to track (VS definitely does not like big
> >     files, and
> >     often segfaults). Of course, for python API, we don't have a choice
> >     because of C limitations (if we care about cross platform that
> >     is), but
> >     we can make things better.
> >
> >
> > The easy fix is to compile separately and link but that will expose a
> > lot of extraneous symbols in the extension module.
>
> Is may be easy in principle, but it is not so practically, because of
> various build issues (the separate library has to be built first, but
> the current code to generate config.h assumes to be built through an
> extension). But I agree, that's the only portable solution, and it has
> other advantages as well (being able to reuse our portable math
> functions in other extensions, for example). The extraneous symbols are
> OK if we decorate them (npy_log, etc...) and I think it is actually
> helpful to have name to make the difference between the libc and our
> function. In particular, if later we can provide several differently
> optimized functions.
>

I was thinking of just linking together the separately compiled object
modules. But we would normal headers to include going that route. I never
liked the automatically generated headers, it kind of obviates the whole
idea of defining the interface separately from the function code.


>
> > Not that most would notice ;) I believe there are linker specific
> > commands that can be used to control that, but we have to support a
> > lot of platforms and no doubt life becomes messy.
>
> Yes, it would be messy: I don't know any platform which does not have
> this capability (Gnu linker, MS linker and Solaris linker all have
> this), but to implement it would be a lot of effort for relatively
> little gain.
>
> > I think your quick fix for the second problem is a bit bogus, it
> > essentially discards the const qualifier on purpose instead of by
> > accident.
>
> Yes, but that's the only solution. FWIW, I loooked a bit at how the
> glibc does things, and it is pretty similar. You can't avoid casts
> everywhere - and the cast I added are correct I believe.
>
> > I tried pretty much the same quick fix and the problem was just pushed
> > elsewhere.
>
> Where ? I compiled with -W -Wall -Wextra, and there was no other warning
> (in that part of the code I mean), at least on 32 bits. But before
> checking more thoroughly, I want to finish cleaning numpy/core/setup.py
>

I just used changed the declaration to const char *p and there was a problem
with the call to PyOS_ascii_strtod. That actually seems to be a problem with
function signature (also in strtod), so you are probably right that an
explicit cast is the only way to do it.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090217/45a61ee2/attachment.html>

From charlesr.harris at gmail.com  Wed Feb 18 01:21:35 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 17 Feb 2009 23:21:35 -0700
Subject: [Numpy-discussion] Warnings in current trunk
In-Reply-To: <e06186140902172214g50403456h63e49402de54b1db@mail.gmail.com>
References: <e06186140902171316t6b15df96h54ea3d48b1952160@mail.gmail.com>
	<499B870B.6040002@ar.media.kyoto-u.ac.jp>
	<e06186140902172121s1061121fvc1fcc0d0f410dd6d@mail.gmail.com>
	<499B9B09.6080800@ar.media.kyoto-u.ac.jp>
	<e06186140902172214g50403456h63e49402de54b1db@mail.gmail.com>
Message-ID: <e06186140902172221y37881c76j837f4417d47d0dae@mail.gmail.com>

On Tue, Feb 17, 2009 at 11:14 PM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

>
>
> On Tue, Feb 17, 2009 at 10:22 PM, David Cournapeau <
> david at ar.media.kyoto-u.ac.jp> wrote:
>
>> Charles R Harris wrote:
>> >
>> >
>> > On Tue, Feb 17, 2009 at 8:56 PM, David Cournapeau
>> > <david at ar.media.kyoto-u.ac.jp <mailto:david at ar.media.kyoto-u.ac.jp>>
>> > wrote:
>> >
>> >     Charles R Harris wrote:
>> >     > I see a lot of warnings like:
>> >     >
>> >     > In file included from numpy/core/src/multiarraymodule.c:87:
>> >     > numpy/core/src/umath_funcs_c99.inc.src:199:1: warning: "isnan"
>> >     redefined
>> >     > In file included from /usr/include/python2.5/pyport.h:204,
>> >     >                  from /usr/include/python2.5/Python.h:57,
>> >     >                  from numpy/core/src/multiarraymodule.c:18:
>> >     >
>> >     > These are probably from r6363.
>> >     >
>> >     > Also
>> >     >
>> >     > In file included from numpy/core/src/scalartypes.inc.src:10,
>> >     >                  from numpy/core/src/arrayobject.c:537,
>> >     >                  from numpy/core/src/multiarraymodule.c:102:
>> >     > numpy/core/src/numpyos.c: In function 'NumPyOS_ascii_strtod':
>> >     > numpy/core/src/numpyos.c:431: warning: assignment discards
>> >     qualifiers
>> >     > from pointer target type
>> >     > numpy/core/src/numpyos.c:480: warning: assignment discards
>> >     qualifiers
>> >     > from pointer target type
>> >
>> >     The second one should be easy to fix, the first one not so much. The
>> >     problem is simple: for the float format fixes recently merged in the
>> >     trunk, we need some math functions - those format functions are
>> >     used in
>> >     multiarray extension. Up to now, we only needed math function in
>> >     ufunc,
>> >     and did so by including .c files. I first did the obvious thing,
>> >     including the same file as well in multiarray, but it does not work
>> so
>> >     well, as you can see.
>> >
>> >     I think this "including .c" business has got to a point where it is
>> >     becoming insane :) Not only does it make the code difficult to
>> follow
>> >     (at least to me), but it also causes various issues on MS platforms,
>> >     which are difficult to track (VS definitely does not like big
>> >     files, and
>> >     often segfaults). Of course, for python API, we don't have a choice
>> >     because of C limitations (if we care about cross platform that
>> >     is), but
>> >     we can make things better.
>> >
>> >
>> > The easy fix is to compile separately and link but that will expose a
>> > lot of extraneous symbols in the extension module.
>>
>> Is may be easy in principle, but it is not so practically, because of
>> various build issues (the separate library has to be built first, but
>> the current code to generate config.h assumes to be built through an
>> extension). But I agree, that's the only portable solution, and it has
>> other advantages as well (being able to reuse our portable math
>> functions in other extensions, for example). The extraneous symbols are
>> OK if we decorate them (npy_log, etc...) and I think it is actually
>> helpful to have name to make the difference between the libc and our
>> function. In particular, if later we can provide several differently
>> optimized functions.
>>
>
> I was thinking of just linking together the separately compiled object
> modules. But we would normal headers to include going that route. I never
> liked the automatically generated headers, it kind of obviates the whole
> idea of defining the interface separately from the function code.
>
>
>>
>> > Not that most would notice ;) I believe there are linker specific
>> > commands that can be used to control that, but we have to support a
>> > lot of platforms and no doubt life becomes messy.
>>
>> Yes, it would be messy: I don't know any platform which does not have
>> this capability (Gnu linker, MS linker and Solaris linker all have
>> this), but to implement it would be a lot of effort for relatively
>> little gain.
>>
>> > I think your quick fix for the second problem is a bit bogus, it
>> > essentially discards the const qualifier on purpose instead of by
>> > accident.
>>
>> Yes, but that's the only solution. FWIW, I loooked a bit at how the
>> glibc does things, and it is pretty similar. You can't avoid casts
>> everywhere - and the cast I added are correct I believe.
>>
>> > I tried pretty much the same quick fix and the problem was just pushed
>> > elsewhere.
>>
>> Where ? I compiled with -W -Wall -Wextra, and there was no other warning
>> (in that part of the code I mean), at least on 32 bits. But before
>> checking more thoroughly, I want to finish cleaning numpy/core/setup.py
>>
>
> I just used changed the declaration to const char *p and there was a
> problem with the call to PyOS_ascii_strtod. That actually seems to be a
> problem with function signature (also in strtod), so you are probably right
> that an explicit cast is the only way to do it.
>

Oh, and this should be avoided:

        if (endptr != NULL) *endptr = (char*)p;

Folks have different views about whether the single statement should be in
brackets but no one recommends putting it on the same line as the if. It's
too easy to overlook and habit draws the eye to the following line.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090217/7236bccd/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Wed Feb 18 01:37:17 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 18 Feb 2009 15:37:17 +0900
Subject: [Numpy-discussion] Warnings in current trunk
In-Reply-To: <e06186140902172221y37881c76j837f4417d47d0dae@mail.gmail.com>
References: <e06186140902171316t6b15df96h54ea3d48b1952160@mail.gmail.com>	<499B870B.6040002@ar.media.kyoto-u.ac.jp>	<e06186140902172121s1061121fvc1fcc0d0f410dd6d@mail.gmail.com>	<499B9B09.6080800@ar.media.kyoto-u.ac.jp>	<e06186140902172214g50403456h63e49402de54b1db@mail.gmail.com>
	<e06186140902172221y37881c76j837f4417d47d0dae@mail.gmail.com>
Message-ID: <499BAC9D.3010502@ar.media.kyoto-u.ac.jp>

Charles R Harris wrote:
>
> Oh, and this should be avoided:
>
>         if (endptr != NULL) *endptr = (char*)p;
>
> Folks have different views about whether the single statement should
> be in brackets but no one recommends putting it on the same line as
> the if. It's too easy to overlook and habit draws the eye to the
> following line.

That's not my code, so I did not want to change those, but I agree with
you here (I prefer explicit brackets personally, but that's not mandated
in the python code-style IIRC). There are a lot of those in numpy code,
so if one want to change them, one should change them for good.

cheers,

David


From Nicolas.Rougier at loria.fr  Wed Feb 18 02:23:44 2009
From: Nicolas.Rougier at loria.fr (Nicolas Rougier)
Date: Wed, 18 Feb 2009 08:23:44 +0100
Subject: [Numpy-discussion] Tensordot and memory consumption
Message-ID: <1234941824.6759.3.camel@sulfur.loria.fr>


Hello,

I'm using tensordot in some computation and while I've been amazed by  
the speed, I'm now trying to reduce memory consumption in some very  
particular cases:

Let S be a 2 dims array of size (s1,s2)
Let D be a 2 dims array of size (d1,d2)
Let W be a 4 dims array of size (d1,d2,s1,s2)

Currently, I'm computing D as tensordot(W,S,2) and it works really  
fine and fast.

However, in some cases, W is based solely on a single 2 dims array K and
each of the W[i,j] is a slice of K. I would like to know if there is a
way to build W such that memory footprint is reduced ? Or maybe there is
other ways to perform the same computation ? For example, I know that
when S and D are of same shape, I can use 2d convolution, but I would
need the general case.

Since I imagine my explanations are not sot clear, I join a small
example. 


Nicolas

---

import numpy

n = 3

# Source
S = numpy.random.random((n,n))

# Destination (at this stage, only D shape matters)
D = numpy.zeros((2*n,2*n))

# Kernel
K = numpy.zeros((S.shape[0]*n+1,S.shape[1]*n+1))
K[S.shape[0],S.shape[1]] = 1

# Kernel decomposition for computing tensordot
W = numpy.zeros(D.shape + S.shape)
for i in range(W.shape[0]):
     for j in range(W.shape[1]):
         x = int(i/float(W.shape[0])*float(S.shape[0]))
         y = int(j/float(W.shape[1])*float(S.shape[1]))
         W[i,j] = K[n-x:n-x+S.shape[0],n-y:n-y+S.shape[1]]

D = numpy.tensordot(W,S,2)
print S
print
print D




From schut at sarvision.nl  Wed Feb 18 03:51:47 2009
From: schut at sarvision.nl (Vincent Schut)
Date: Wed, 18 Feb 2009 09:51:47 +0100
Subject: [Numpy-discussion] fancy view question
In-Reply-To: <20090217162006.GR17638@phare.normalesup.org>
References: <gne22p$q9$1@ger.gmane.org>	<9457e7c80902170631n1c0af5d4gea5560ec0315e1ca@mail.gmail.com>	<9457e7c80902170642p6b5a6bd8u893e59283d5d066c@mail.gmail.com>	<20090217144449.GM17638@phare.normalesup.org>	<3d375d730902170709t15acfa51qbfedc5be403d6e95@mail.gmail.com>	<20090217161610.GQ17638@phare.normalesup.org>	<3d375d730902170818u7cd5bebaj1a9c51657ce9ca51@mail.gmail.com>
	<20090217162006.GR17638@phare.normalesup.org>
Message-ID: <gngi74$3s1$1@ger.gmane.org>

Gael Varoquaux wrote:
> On Tue, Feb 17, 2009 at 10:18:11AM -0600, Robert Kern wrote:
>> On Tue, Feb 17, 2009 at 10:16, Gael Varoquaux
>> <gael.varoquaux at normalesup.org> wrote:
>>> On Tue, Feb 17, 2009 at 09:09:38AM -0600, Robert Kern wrote:
>>>> np.repeat(np.repeat(x, 2, axis=0), 2, axis=1)
> 
>>>> stride_tricks are fun, but this is already a solved problem in numpy.
> 
>>> Wow. I still have a lot to learn! How about adding a see-also in
>>> as_strided.
> 
>> They're not really related.
> 
> Yes, I now understand that there where no views involved, on the contrary
> to what I first thought, so indeed there are not related.
> 
> Ga?l

Wow, did I start something! :-)
Thanks for all that shared their knowledge and ideas. I learned a lot 
beside getting answers to my questions.

Vincent.



From olivier.grisel at ensta.org  Wed Feb 18 04:06:26 2009
From: olivier.grisel at ensta.org (Olivier Grisel)
Date: Wed, 18 Feb 2009 10:06:26 +0100
Subject: [Numpy-discussion] inplace dot products
Message-ID: <db0bec8f0902180106n5483a79eg2409e0986b5b0bd4@mail.gmail.com>

Hi numpist people,

I discovered the ufunc and there ability to compute the results on
preallocated arrays:

>>> a = arange(10, dtype=float32)
>>> b = arange(10, dtype=float32) + 1
>>> c = add(a, b, a)
>>> c is a
    True
>>> a
    array([  1.,   3.,   5.,   7.,   9.,  11.,  13.,  15.,  17.,
19.], dtype=float32)

My questions is : is there a way to have an equivalent for the dot
product operation:
I want atlas to build my dot products without allocating a temporary
array and reuse
a preallocated array of results. Suppose I have:

>>> results = array((10, 3), dtype=float32)
>>> W = arange(6, dtype=float32).reshape((2, 3))
>>> x = arange(20, dtype=float32).reshape((10, 2))

What I want is the equivalent of the following without the
intermediate call to malloc:

>>> results[:] = dot(x, W)

Any idea? I tried to introspect the various docstring of numpy core
modules but I
could not get any lead.

-- 
Olivier


From ndbecker2 at gmail.com  Wed Feb 18 07:27:38 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 18 Feb 2009 12:27:38 +0000 (UTC)
Subject: [Numpy-discussion] views and object lifetime
Message-ID: <loom.20090218T122621-787@post.gmane.org>

How is it ensured, at the C api level, that when I have an array A, and a view
of it B, that the data is not destroyed until both A and B are?



From matthieu.brucher at gmail.com  Wed Feb 18 07:51:03 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 18 Feb 2009 13:51:03 +0100
Subject: [Numpy-discussion] views and object lifetime
In-Reply-To: <loom.20090218T122621-787@post.gmane.org>
References: <loom.20090218T122621-787@post.gmane.org>
Message-ID: <e76aa17f0902180451j5308caa8gb4cb9ff1a5ebe111@mail.gmail.com>

B has a reference to A.

Matthieu

2009/2/18 Neal Becker <ndbecker2 at gmail.com>:
> How is it ensured, at the C api level, that when I have an array A, and a view
> of it B, that the data is not destroyed until both A and B are?
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From ndbecker2 at gmail.com  Wed Feb 18 08:02:54 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 18 Feb 2009 13:02:54 +0000 (UTC)
Subject: [Numpy-discussion] views and object lifetime
References: <loom.20090218T122621-787@post.gmane.org>
	<e76aa17f0902180451j5308caa8gb4cb9ff1a5ebe111@mail.gmail.com>
Message-ID: <loom.20090218T130154-430@post.gmane.org>

Matthieu Brucher <matthieu.brucher <at> gmail.com> writes:

> 
> B has a reference to A.

Could you be more specific?  Where is this reference stored?  What C api
functions are used?

> Matthieu
> 
> 2009/2/18 Neal Becker <ndbecker2 <at> gmail.com>:
> > How is it ensured, at the C api level, that when I have an array A, and a
view
> > of it B, that the data is not destroyed until both A and B are?
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion <at> scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> 






From matthieu.brucher at gmail.com  Wed Feb 18 08:09:26 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 18 Feb 2009 14:09:26 +0100
Subject: [Numpy-discussion] views and object lifetime
In-Reply-To: <loom.20090218T130154-430@post.gmane.org>
References: <loom.20090218T122621-787@post.gmane.org>
	<e76aa17f0902180451j5308caa8gb4cb9ff1a5ebe111@mail.gmail.com>
	<loom.20090218T130154-430@post.gmane.org>
Message-ID: <e76aa17f0902180509r482f70e4x592d4ee841cef6bd@mail.gmail.com>

2009/2/18 Neal Becker <ndbecker2 at gmail.com>:
> Matthieu Brucher <matthieu.brucher <at> gmail.com> writes:
>
>>
>> B has a reference to A.
>
> Could you be more specific?  Where is this reference stored?  What C api
> functions are used?

I don't remember, and I don't have the Numpy book here. But if B is a
view on A, a flag indicates that B does not own the data, and another
field is a pointer to A.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From gael.varoquaux at normalesup.org  Wed Feb 18 08:15:38 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Wed, 18 Feb 2009 14:15:38 +0100
Subject: [Numpy-discussion] views and object lifetime
In-Reply-To: <loom.20090218T130154-430@post.gmane.org>
References: <loom.20090218T122621-787@post.gmane.org>
	<e76aa17f0902180451j5308caa8gb4cb9ff1a5ebe111@mail.gmail.com>
	<loom.20090218T130154-430@post.gmane.org>
Message-ID: <20090218131538.GB5600@phare.normalesup.org>

On Wed, Feb 18, 2009 at 01:02:54PM +0000, Neal Becker wrote:
> > B has a reference to A.

> Could you be more specific?  Where is this reference stored? 

In [1]: import numpy as np

In [2]: a = np.empty(10)

In [3]: b = a[::2]

In [4]: b.base is a
Out[4]: True


Ga?l


From scott.sinclair.za at gmail.com  Wed Feb 18 09:54:10 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Wed, 18 Feb 2009 16:54:10 +0200
Subject: [Numpy-discussion] views and object lifetime
In-Reply-To: <loom.20090218T130154-430@post.gmane.org>
References: <loom.20090218T122621-787@post.gmane.org>
	<e76aa17f0902180451j5308caa8gb4cb9ff1a5ebe111@mail.gmail.com>
	<loom.20090218T130154-430@post.gmane.org>
Message-ID: <6a17e9ee0902180654v4c0d985bsc65b6679c2572e47@mail.gmail.com>

> 2009/2/18 Neal Becker <ndbecker2 at gmail.com>:
> Matthieu Brucher <matthieu.brucher <at> gmail.com> writes:
>
>>
>> B has a reference to A.
>
> Could you be more specific?  Where is this reference stored?  What C api
> functions are used?

I'm probably not qualified to be much more specific, these links
should provide the necessary detail:

http://docs.scipy.org/doc/numpy/reference/c-api.html#numpy-c-api
http://docs.python.org/c-api/intro.html#objects-types-and-reference-counts
http://docs.python.org/extending/newtypes.html

The Python interpreter takes care of when to free the memory
associated with an object once it's reference count reaches zero. The
object reference counts are increased and decreased directly in C code
using the Py_INCREF and Py_DECREF macros whenever a new object is
created or a new pointer assigned to an existing object.

Cheers,
Scott


From oliphant at enthought.com  Wed Feb 18 10:40:50 2009
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 18 Feb 2009 09:40:50 -0600
Subject: [Numpy-discussion] views and object lifetime
In-Reply-To: <loom.20090218T122621-787@post.gmane.org>
References: <loom.20090218T122621-787@post.gmane.org>
Message-ID: <499C2C02.3020306@enthought.com>

Neal Becker wrote:
> How is it ensured, at the C api level, that when I have an array A, and a view
> of it B, that the data is not destroyed until both A and B are?
>   
One array, A, owns the data and will deallocate it only when its 
reference-count goes to 0.    The view, B, has a reference to A (stored 
in the base attribute) and has OWNDATA set to false so that its 
deallocator simply decreases the reference count on the array, A, that 
actually owns the data. 

In the code look at

the `array_dealloc` function  in arrayobject.c and the base and OWNDATA 
flag-bit in the array-structure for details.


-Travis



From kbasye1 at jhu.edu  Wed Feb 18 10:40:44 2009
From: kbasye1 at jhu.edu (Ken Basye)
Date: Wed, 18 Feb 2009 10:40:44 -0500
Subject: [Numpy-discussion] Compute multiple outer products,
	without a loop?
In-Reply-To: <mailman.6002.1234934517.2878.numpy-discussion@scipy.org>
References: <mailman.6002.1234934517.2878.numpy-discussion@scipy.org>
Message-ID: <499C2BFC.2030409@jhu.edu>

Thanks Chuck; that's perfect.
   Ken
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 17 Feb 2009 11:04:56 -0700
> From: Charles R Harris <charlesr.harris at gmail.com>
> Subject: Re: [Numpy-discussion] Compute multiple outer products
> 	without a	loop?
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Message-ID:
> 	<e06186140902171004g850c9aax348a3e2053bb69fa at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On Tue, Feb 17, 2009 at 8:30 AM, Ken Basye <kbasye1 at jhu.edu> wrote:
>
>   
>> Hi,
>>   My current code looks like this:
>>
>>           (k,d) = m.shape
>>           sq = np.zeros((k, d, d), dtype=float)
>>           for i in xrange(k):
>>               sq[i] = np.outer(m[i], m[i])
>>
>>
>> That is, m is treated as a sequence of k vectors of length d; the k dXd
>> outer products are found and stored in sq.
>>
>>     
>
> Try A[:,:,newaxis]*B[:,newaxis,:] . Example
>
> In [6]: A = array([[1,2],[3,4]])
>
> In [7]: B = array([[1,1],[1,1]])
>
> In [8]: A[:,:,newaxis]*B[:,newaxis,:]
> Out[8]:
> array([[[1, 1],
>         [2, 2]],
>
>        [[3, 3],
>         [4, 4]]])
>
> In [9]: B[:,:,newaxis]*A[:,newaxis,:]
> Out[9]:
> array([[[1, 2],
>         [1, 2]],
>
>        [[3, 4],
>         [3, 4]]])
>
> You can use this sort of trick along with a sum to multiply stacks of
> matrices by stacks of vectors or matrices.
>
> Chuck
>
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090218/38ea5d4f/attachment.html>

From charlesr.harris at gmail.com  Wed Feb 18 11:45:57 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 18 Feb 2009 09:45:57 -0700
Subject: [Numpy-discussion] Warnings in current trunk
In-Reply-To: <499BAC9D.3010502@ar.media.kyoto-u.ac.jp>
References: <e06186140902171316t6b15df96h54ea3d48b1952160@mail.gmail.com>
	<499B870B.6040002@ar.media.kyoto-u.ac.jp>
	<e06186140902172121s1061121fvc1fcc0d0f410dd6d@mail.gmail.com>
	<499B9B09.6080800@ar.media.kyoto-u.ac.jp>
	<e06186140902172214g50403456h63e49402de54b1db@mail.gmail.com>
	<e06186140902172221y37881c76j837f4417d47d0dae@mail.gmail.com>
	<499BAC9D.3010502@ar.media.kyoto-u.ac.jp>
Message-ID: <e06186140902180845g71e3b566t9c1681e7a71f859e@mail.gmail.com>

On Tue, Feb 17, 2009 at 11:37 PM, David Cournapeau <
david at ar.media.kyoto-u.ac.jp> wrote:

> Charles R Harris wrote:
> >
> > Oh, and this should be avoided:
> >
> >         if (endptr != NULL) *endptr = (char*)p;
> >
> > Folks have different views about whether the single statement should
> > be in brackets but no one recommends putting it on the same line as
> > the if. It's too easy to overlook and habit draws the eye to the
> > following line.
>
> That's not my code, so I did not want to change those, but I agree with
> you here (I prefer explicit brackets personally, but that's not mandated
> in the python code-style IIRC). There are a lot of those in numpy code,
> so if one want to change them, one should change them for good.
>

Yep. I'm trying to fix all of them this week which is why I complained.
Think of mopping the floor and the kid walks in with muddy shoes...

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090218/95bf40df/attachment.html>

From william at resolversystems.com  Wed Feb 18 12:03:05 2009
From: william at resolversystems.com (William Reade)
Date: Wed, 18 Feb 2009 17:03:05 +0000
Subject: [Numpy-discussion] Ironclad v0.8.1 released
Message-ID: <499C3F49.2060403@resolversystems.com>

Hi all

I'm fairly pleased to announce the release of Ironclad v0.8.1; it's not 
an enormous technical leap above v0.8, but it does now enable you to 
import and use SciPy and Matplotlib with IronPython on Win32 (with some 
restrictions; see project page) . Downloads, and more details, are 
available at http://code.google.com/p/ironclad -- please do play with 
it, and let me know if you have any problems.

Cheers
William


From davidh at ipac.caltech.edu  Wed Feb 18 12:37:54 2009
From: davidh at ipac.caltech.edu (David Henderson)
Date: Wed, 18 Feb 2009 09:37:54 -0800
Subject: [Numpy-discussion] additional dtype argument to numpy.dot() (Re:
	Numpy-discussion Digest, Vol 29, Issue 48)
In-Reply-To: <mailman.5886.1234627497.2878.numpy-discussion@scipy.org>
References: <mailman.5886.1234627497.2878.numpy-discussion@scipy.org>
Message-ID: <DD248034-4F79-478F-A8D1-1069E2996EE3@ipac.caltech.edu>

Hi Paul, list:

Thanks for the reply.

numpy.sum()  does indeed have a dtype for the accumulator for the  
sum. numpy.sum() does not implement an inner (dot) product, just a  
straight summation.

The feature I'm requesting is to add a similar accumulator type  
argument for numpy.dot().

After tracing some of the code, the there is a generic dot product  
algorithm for the various types. If a BLAS library is available, the  
BLAS routines will supplant the library, but only for the BLAS  
routines that exist.

BLAS doesn't support long double, so only a subset of the different  
calls for different types call BLAS.

The place where this feature comes into play is iterative refinement  
of a linear algebra solution. To implement iterative refinement, an  
error term is calculated as a matrix dot product with a higher  
precision than the elements of the matrix.

The floating point hardware on the i386/PPC/Sparc is ready to go for  
this application, its just that BLAS doesn't implement it.

The numpy.dot() function is missing the feature to supply the type of  
the accumulator for the sum.

I'm looking for a way to prototype some routines to perform iterative  
refinement, and python/numpy looks like an ideal platform.

numpy has _almost_ the feature set needed to implement this. Adding  
the accumulator type argument to numpy.dot() is a needed feature.

code snippet exhibiting float128 functionality that already exists:
--------------------------
Python 2.6.1 (r261:67515, Feb 12 2009, 16:56:09)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import numpy
 >>> z=numpy.longdouble((1,2,3))
 >>> z
array([1.0, 2.0, 3.0], dtype=float128)
 >>> y=numpy.double((4,5,6))
 >>> y
array([ 4.,  5.,  6.])
 >>> numpy.dot(y,y)
77.0
 >>> numpy.dot(y,z)
32.0
 >>> numpy.dot(z,z)
14.0
---------------------------

I'm prepared to make the changes to numpy. It looks like they would  
entail modifying the source file "multiarrayobject.c". The existing  
code is:

/*NUMPY_API
   Numeric.innerproduct(a,v)
*/
static PyObject *
PyArray_InnerProduct(PyObject *op1, PyObject *op2)
{

If I make changes, can I get them inserted into the numpy source base  
(after appropriate testing??)

Thanks in advance,

David Henderson


On Feb 14, 2009, at 8:04 AM, numpy-discussion-request at scipy.org wrote:

> Fri, 13 Feb 2009 12:04:12 -0800, David Henderson wrote:
>> I'd like accumulate the summation in extended precision, "double" sum
>> for float  inputs, "long double" sum for "double" inputs.
>>
>> A way of doing this is to add an optional third argument to dot - to
>> specify the summation type.
>
> `dot` does matrix-matrix, matrix-vector, or vector-vector products.  
> These
> are usually implemented via calling the BLAS linear algebra  
> library, and
> AFAIK BLAS only has routines for type-homogenous arguments. So I doubt
> this can be implemented for `dot` in any way.
>
> On the other hand, `numpy.sum` already has a `dtype` argument that
> specifies the accumulator data type. Maybe you can use that?
>
> -- 
> Pauli Virtanen
> -----------------------------
>


From cournape at gmail.com  Wed Feb 18 12:49:37 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 19 Feb 2009 02:49:37 +0900
Subject: [Numpy-discussion] additional dtype argument to numpy.dot()
	(Re: Numpy-discussion Digest, Vol 29, Issue 48)
In-Reply-To: <DD248034-4F79-478F-A8D1-1069E2996EE3@ipac.caltech.edu>
References: <mailman.5886.1234627497.2878.numpy-discussion@scipy.org>
	<DD248034-4F79-478F-A8D1-1069E2996EE3@ipac.caltech.edu>
Message-ID: <5b8d13220902180949t7ff88890q3cb0eb1f1ee811ba@mail.gmail.com>

On Thu, Feb 19, 2009 at 2:37 AM, David Henderson
<davidh at ipac.caltech.edu> wrote:
> Hi Paul, list:
>
> Thanks for the reply.
>
> numpy.sum()  does indeed have a dtype for the accumulator for the
> sum. numpy.sum() does not implement an inner (dot) product, just a
> straight summation.

You may be interested in xblas, which is a new BLAS specification
which explicitly handle mixed precision and increased accumulator
precision:

http://www.netlib.org/xblas/

For now, it is completely unsupported in numpy, but this may be a good
thing to add if you are interested in extra precision,

David


From pinto at mit.edu  Wed Feb 18 13:36:34 2009
From: pinto at mit.edu (Nicolas Pinto)
Date: Wed, 18 Feb 2009 13:36:34 -0500
Subject: [Numpy-discussion] rgb_to_hsv in scipy.misc ? (was: Optimizing
	speed for large-array inter-element algorithms (specifically,
	color space 	conversion))
Message-ID: <954ae5aa0902181036i192ad020kd152e0629974ed6c@mail.gmail.com>

Hello,

Would it be possible to include the following rgb to hsv conversion code in
scipy (probably in misc along with misc.imread, etc.) ?

What do you think?

Thanks in advance.

Best regards,

-- 
Nicolas Pinto
Ph.D. Candidate, Brain & Computer Sciences
Massachusetts Institute of Technology, USA
http://web.mit.edu/pinto


#
------------------------------------------------------------------------------
import numpy as np
def rgb_to_hsv_arr(arr):
    """ fast rgb_to_hsv using numpy array """

    # adapted from Arnar Flatberg
    # http://www.mail-archive.com/numpy-discussion at scipy.org/msg06147.html
    # it now handles NaN properly and mimics colorsys.rgb_to_hsv output

    arr = arr/255.
    out = np.empty_like(arr)

    arr_max = arr.max(-1)
    delta = arr.ptp(-1)
    s = delta / arr_max

    s[delta==0] = 0

    # red is max
    idx = (arr[:,:,0] == arr_max)
    out[idx, 0] = (arr[idx, 1] - arr[idx, 2]) / delta[idx]

    # green is max
    idx = (arr[:,:,1] == arr_max)
    out[idx, 0] = 2. + (arr[idx, 2] - arr[idx, 0] ) / delta[idx]

    # blue is max
    idx = (arr[:,:,2] == arr_max)
    out[idx, 0] = 4. + (arr[idx, 0] - arr[idx, 1] ) / delta[idx]

    out[:,:,0] = (out[:,:,0]/6.0) % 1.0
    out[:,:,1] = s
    out[:,:,2] = arr_max

    # rescale back to [0, 255]
    out *= 255.

    # remove NaN
    out[np.isnan(out)] = 0

    return out
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090218/8b9ca94c/attachment.html>

From ondrej at certik.cz  Wed Feb 18 19:14:27 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 18 Feb 2009 16:14:27 -0800
Subject: [Numpy-discussion] parallel compilation of numpy
Message-ID: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>

Hi,

I have a shiny new computer with 8 cores and numpy still takes forever
to compile --- is there a way to compile it in parallel (make -j9)?

Do distutils allow that? If not, let's move to some build system that
allows that? Just wanted to check if there is some reason for that,
apart from patches welcome. :)

Ondrej


From robert.kern at gmail.com  Wed Feb 18 19:33:13 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 Feb 2009 18:33:13 -0600
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
Message-ID: <3d375d730902181633h6e0ecbf5l1afa5918c59525ac@mail.gmail.com>

On Wed, Feb 18, 2009 at 18:14, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> I have a shiny new computer with 8 cores and numpy still takes forever
> to compile --- is there a way to compile it in parallel (make -j9)?
>
> Do distutils allow that?

No. numscons will, though.

> If not, let's move to some build system that
> allows that? Just wanted to check if there is some reason for that,
> apart from patches welcome. :)

I'm not opposed in principle to numscons becoming the default build
system for numpy. But I'm also entirely uninterested in spending my
personal time on it, for example. I'm not sure if David thinks it's
ready to fully replace the default build system, yet; he could
probably use your help. So yes, "patches welcome."  :-)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From sturla at molden.no  Wed Feb 18 20:50:01 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 19 Feb 2009 02:50:01 +0100 (CET)
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
Message-ID: <7dba8391205f88f0f9de32f7f3eb902d.squirrel@webmail.uio.no>


> I have a shiny new computer with 8 cores and numpy still takes forever
> to compile

Yes, forever/8 = forever.


Sturla Molden



From mattdm at mattdm.org  Wed Feb 18 20:55:56 2009
From: mattdm at mattdm.org (Matthew Miller)
Date: Wed, 18 Feb 2009 20:55:56 -0500
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <7dba8391205f88f0f9de32f7f3eb902d.squirrel@webmail.uio.no>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
	<7dba8391205f88f0f9de32f7f3eb902d.squirrel@webmail.uio.no>
Message-ID: <20090219015556.GA32331@jadzia.bu.edu>

On Thu, Feb 19, 2009 at 02:50:01AM +0100, Sturla Molden wrote:
> > I have a shiny new computer with 8 cores and numpy still takes forever
> > to compile
> Yes, forever/8 = forever.

Good point. nan_to_num() could be helpful here.

-- 
Matthew Miller           mattdm at mattdm.org          <http://mattdm.org/>


From cournape at gmail.com  Wed Feb 18 20:57:27 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 19 Feb 2009 10:57:27 +0900
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
Message-ID: <5b8d13220902181757x4a58ce79h7d597e432d5bf22a@mail.gmail.com>

On Thu, Feb 19, 2009 at 9:14 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> I have a shiny new computer with 8 cores and numpy still takes forever
> to compile

Forever ? It takes one minute to build :) scipy takes for ever, but it
is because of C++ more than anything else.

> --- is there a way to compile it in parallel (make -j9)?
>
> Do distutils allow that?

No, and it never will. Parallel builds requires to build with
dependency handling. Even make does not handle it well: it works most
of the time by accident, but there are numerous problems (try for
example building lapack with make -j8 on your 8 cores machine - it
will give a bogus library 90 % of the time, because it starts building
static library with ar while some object files are still built).

scons does handle it well. Now, I don't think it will make building
numpy that much faster. Here are the numbers:
 - numscons build, no parallel job: 1m08
 - distutils build: 1m17
 - numscons build, 4 jobs (on a two cores machine): 54s

Now, those numbers are on Mac OS X, and this is the worse platform to
try it on, because gcc already uses two cores even when using one
command line (I cannot confirm this, but looking at the activity
monitor, gcc always use both cores when building universal apps, which
makes sense).

There are more fundamental reasons, though:
 - for numpy, a lot of time is spent in configuration: configuration
cannot be done with multiple jobs
 - numscons has one fundamental limitation: it launched a new scons
subprocess for every subpackage, so you can't parallelize different
subpackages at the same time (says numpy.core and numpy.random).

This limitation is very hard to circumvent IMHO, and is the main
limitation of numscons ATM:
 - it means no-op builds are very slow (20s for scipy, for example),
because I have to relaunch scons for every subpackage, and scons is
relatively slow to start (and I spent quite some time to optimize this
already: but 30 subpackages in scipy, with scons taking 0.2 s only to
start means already 6 seconds to do nothing)
 - it means parallel is not as good as it can be
 - it means python setup.py sdist is very hard to to support (I have
not yet found a way - it is broken ATM if you remove the distutils
scripts - the problem is that)
 - it means numscons wastes its time doing many times the same checks
(lapack, etc...)

If I (or you :) ) solve this, I would be in favor of using numscons.
Yesterday, I wasted half a day with distutils to do something which
takes 2 minutes in numscons. Going aways from this would be a big
relief, at least as far as I am concerned.

But it is very hard - I think it would be at least one week of full
work (because it is a scons limitation).

cheers,

David


From cournape at gmail.com  Wed Feb 18 21:00:49 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 19 Feb 2009 11:00:49 +0900
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <7dba8391205f88f0f9de32f7f3eb902d.squirrel@webmail.uio.no>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
	<7dba8391205f88f0f9de32f7f3eb902d.squirrel@webmail.uio.no>
Message-ID: <5b8d13220902181800s44edb0b7w645df34172dc16ed@mail.gmail.com>

On Thu, Feb 19, 2009 at 10:50 AM, Sturla Molden <sturla at molden.no> wrote:
>
>> I have a shiny new computer with 8 cores and numpy still takes forever
>> to compile
>
> Yes, forever/8 = forever.

Not if you are a physician: my impression in undergrad was that
infinity / 8 could be anything from 0 to infinity in physics :)

More seriously, there is no chance that make -j8 will be 8 times
faster than make. 2 to 3 times, yes (IO, etc...).

David


From rmay31 at gmail.com  Wed Feb 18 21:07:45 2009
From: rmay31 at gmail.com (Ryan May)
Date: Wed, 18 Feb 2009 20:07:45 -0600
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <5b8d13220902181800s44edb0b7w645df34172dc16ed@mail.gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
	<7dba8391205f88f0f9de32f7f3eb902d.squirrel@webmail.uio.no>
	<5b8d13220902181800s44edb0b7w645df34172dc16ed@mail.gmail.com>
Message-ID: <c7b95a020902181807y6cddafc5r3412670115315241@mail.gmail.com>

On Wed, Feb 18, 2009 at 8:00 PM, David Cournapeau <cournape at gmail.com>wrote:

> On Thu, Feb 19, 2009 at 10:50 AM, Sturla Molden <sturla at molden.no> wrote:
> >
> >> I have a shiny new computer with 8 cores and numpy still takes forever
> >> to compile
> >
> > Yes, forever/8 = forever.
>
> Not if you are a physician: my impression in undergrad was that
> infinity / 8 could be anything from 0 to infinity in physics :)
>

Not to nitpick, but this is the second time I've seen this lately:

physician == medical doctor != physicist :)

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090218/606c7ea6/attachment.html>

From cournape at gmail.com  Wed Feb 18 21:19:50 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 19 Feb 2009 11:19:50 +0900
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <c7b95a020902181807y6cddafc5r3412670115315241@mail.gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
	<7dba8391205f88f0f9de32f7f3eb902d.squirrel@webmail.uio.no>
	<5b8d13220902181800s44edb0b7w645df34172dc16ed@mail.gmail.com>
	<c7b95a020902181807y6cddafc5r3412670115315241@mail.gmail.com>
Message-ID: <5b8d13220902181819n3cf19a33h97f3d21a5dd4503e@mail.gmail.com>

On Thu, Feb 19, 2009 at 11:07 AM, Ryan May <rmay31 at gmail.com> wrote:

>
> Not to nitpick, but this is the second time I've seen this lately:
>
> physician == medical doctor != physicist :)

You're right of course - the French word for physicist being
"physicien", it may be one more mistake perpetuated by the French :)

David


From lists at cheimes.de  Wed Feb 18 21:47:09 2009
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 19 Feb 2009 03:47:09 +0100
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <5b8d13220902181757x4a58ce79h7d597e432d5bf22a@mail.gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
	<5b8d13220902181757x4a58ce79h7d597e432d5bf22a@mail.gmail.com>
Message-ID: <gnih7d$u4h$1@ger.gmane.org>

David Cournapeau wrote:
> No, and it never will. Parallel builds requires to build with
> dependency handling. Even make does not handle it well: it works most
> of the time by accident, but there are numerous problems (try for
> example building lapack with make -j8 on your 8 cores machine - it
> will give a bogus library 90 % of the time, because it starts building
> static library with ar while some object files are still built).

You may call me naive and ignorant. Is it really that hard to archive
some kind of poor man's concurrency? You don't have to parallelize
everything to get a speed up on multi core machines. Usually the compile
process from C/C++ file to an object files takes up most of the time.

How about

* assemble a list of all C/C++ source files of all extensions.
* compile all source files in parallel
* do the rest (linking etc.) in serial

This should give a nice speed up without much work and without complex
dependency analysis. Do you see a possible pit fall? I don't.

Christian



From rmay31 at gmail.com  Wed Feb 18 21:48:21 2009
From: rmay31 at gmail.com (Ryan May)
Date: Wed, 18 Feb 2009 20:48:21 -0600
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <5b8d13220902181819n3cf19a33h97f3d21a5dd4503e@mail.gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
	<7dba8391205f88f0f9de32f7f3eb902d.squirrel@webmail.uio.no>
	<5b8d13220902181800s44edb0b7w645df34172dc16ed@mail.gmail.com>
	<c7b95a020902181807y6cddafc5r3412670115315241@mail.gmail.com>
	<5b8d13220902181819n3cf19a33h97f3d21a5dd4503e@mail.gmail.com>
Message-ID: <c7b95a020902181848x79c7fb5awea791c2120183dc@mail.gmail.com>

On Wed, Feb 18, 2009 at 8:19 PM, David Cournapeau <cournape at gmail.com>wrote:

> On Thu, Feb 19, 2009 at 11:07 AM, Ryan May <rmay31 at gmail.com> wrote:
>
> >
> > Not to nitpick, but this is the second time I've seen this lately:
> >
> > physician == medical doctor != physicist :)
>
> You're right of course - the French word for physicist being
> "physicien", it may be one more mistake perpetuated by the French :)


:)

Well, not nearly as bad as Jerry Lewis. :)

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
Sent from: Norman Oklahoma United States.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090218/981e4a3d/attachment.html>

From chaos.proton at gmail.com  Wed Feb 18 22:23:58 2009
From: chaos.proton at gmail.com (Grissiom)
Date: Thu, 19 Feb 2009 11:23:58 +0800
Subject: [Numpy-discussion] linalg.norm along axis?
Message-ID: <a0adeea50902181923n74995342sadc55258cd8414e@mail.gmail.com>

Hi all,

Is there any possibility to calculate norm along axis? For example:

a = np.array((
(3,4),
(6,8)))

And I want to get:
array([5.0, 10.0])

I currently use a for loop to achieve this, Is there any more elegant way to
do this?

-- 
Cheers,
Grissiom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090219/dc401fa2/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Wed Feb 18 22:08:11 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 19 Feb 2009 12:08:11 +0900
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <gnih7d$u4h$1@ger.gmane.org>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>	<5b8d13220902181757x4a58ce79h7d597e432d5bf22a@mail.gmail.com>
	<gnih7d$u4h$1@ger.gmane.org>
Message-ID: <499CCD1B.60701@ar.media.kyoto-u.ac.jp>

Christian Heimes wrote:
> David Cournapeau wrote:
>   
>> No, and it never will. Parallel builds requires to build with
>> dependency handling. Even make does not handle it well: it works most
>> of the time by accident, but there are numerous problems (try for
>> example building lapack with make -j8 on your 8 cores machine - it
>> will give a bogus library 90 % of the time, because it starts building
>> static library with ar while some object files are still built).
>>     
>
> You may call me naive and ignorant. Is it really that hard to archive
> some kind of poor man's concurrency? You don't have to parallelize
> everything to get a speed up on multi core machines. Usually the compile
> process from C/C++ file to an object files takes up most of the time.
>
> How about
>
> * assemble a list of all C/C++ source files of all extensions.
> * compile all source files in parallel
> * do the rest (linking etc.) in serial
>   

That's more or less how make works - it does not work very well IMHO.
And doing the above correctly in distutils may be harder than it seems:
both scons and waf had numerous problems with calling subtasks because
of race conditions in subprocess for example (both have their own module
for that).

More fundamentally though, I have no interest in working on distutils.
Not working on a DAG is fundamentally and hopelessly broken for a build
tool, and this is unfixable in distutils. Everything is wrong, from the
concepts to the UI through the implementation, to paraphrase a famous
saying. There is nothing to save IMHO. Of course, someone else can work
on it. I prefer working on a sane solution myself,

cheers,

David


From pinto at mit.edu  Wed Feb 18 23:12:09 2009
From: pinto at mit.edu (Nicolas Pinto)
Date: Wed, 18 Feb 2009 23:12:09 -0500
Subject: [Numpy-discussion] linalg.norm along axis?
In-Reply-To: <a0adeea50902181923n74995342sadc55258cd8414e@mail.gmail.com>
References: <a0adeea50902181923n74995342sadc55258cd8414e@mail.gmail.com>
Message-ID: <954ae5aa0902182012s17014ab7w7d07c65551ca5ef3@mail.gmail.com>

Grissiom,

Using the following doesn't require any loop:

In [9]: sqrt((a**2.).sum(1))
Out[9]: array([  5.,  10.])

Best,


On Wed, Feb 18, 2009 at 10:23 PM, Grissiom <chaos.proton at gmail.com> wrote:

> Hi all,
>
> Is there any possibility to calculate norm along axis? For example:
>
> a = np.array((
> (3,4),
> (6,8)))
>
> And I want to get:
> array([5.0, 10.0])
>
> I currently use a for loop to achieve this, Is there any more elegant way
> to do this?
>
> --
> Cheers,
> Grissiom
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Nicolas Pinto
Ph.D. Candidate, Brain & Computer Sciences
Massachusetts Institute of Technology, USA
http://web.mit.edu/pinto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090218/72a21ebb/attachment.html>

From cournape at gmail.com  Thu Feb 19 00:25:21 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 19 Feb 2009 14:25:21 +0900
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
Message-ID: <5b8d13220902182125u51f98727t40ff4fc32593a1a0@mail.gmail.com>

On Thu, Feb 19, 2009 at 9:14 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> I have a shiny new computer with 8 cores and numpy still takes forever
> to compile --- is there a way to compile it in parallel (make -j9)?
>
> Do distutils allow that? If not, let's move to some build system that
> allows that?

Since numscons was mentioned several times recently, and I have not
answered to every request, I wrote a more complete post about it -
even with somes advices for people motivated in using cmake :)

http://cournape.wordpress.com/2009/02/19/numscons-current-state-future-alternative-build-tools-for-numpy/

cheers,

David


From michael.abshoff at googlemail.com  Thu Feb 19 00:26:59 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 18 Feb 2009 21:26:59 -0800
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <499CCD1B.60701@ar.media.kyoto-u.ac.jp>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>	<5b8d13220902181757x4a58ce79h7d597e432d5bf22a@mail.gmail.com>	<gnih7d$u4h$1@ger.gmane.org>
	<499CCD1B.60701@ar.media.kyoto-u.ac.jp>
Message-ID: <499CEDA3.9020005@gmail.com>

David Cournapeau wrote:
> Christian Heimes wrote:
>> David Cournapeau wrote:

Hi,

>> You may call me naive and ignorant. Is it really that hard to archive
>> some kind of poor man's concurrency? You don't have to parallelize
>> everything to get a speed up on multi core machines. Usually the compile
>> process from C/C++ file to an object files takes up most of the time.
>>
>> How about
>>
>> * assemble a list of all C/C++ source files of all extensions.
>> * compile all source files in parallel
>> * do the rest (linking etc.) in serial
>>   

With Sage we do the cythonization in parallel and for now build 
extension serially, but we have code to do that in parallel, too. Given 
that we are building 180 extensions or so the speedup is linear. I often 
do this using 24 cores, so it seems robust since I do work on Sage daily 
and often to test builds from scratch and I never had any problems with 
that code.

We use pyprocessing to launch the jobs and the changes to disutils are 
surprisingly small, but the original version of the patch broke the 
build of numpy/scipy, but I do believe the author already has a fix for 
that, too - he is just busy finishing his PhD thesis next month and will 
then be back to work on Sage. The plan here is to definitely push things 
back into Python so that all people building extensions can benefit.

> That's more or less how make works - it does not work very well IMHO.
> And doing the above correctly in distutils may be harder than it seems:
> both scons and waf had numerous problems with calling subtasks because
> of race conditions in subprocess for example (both have their own module
> for that).
> 
> More fundamentally though, I have no interest in working on distutils.

:)

> Not working on a DAG is fundamentally and hopelessly broken for a build
> tool, and this is unfixable in distutils. Everything is wrong, from the
> concepts to the UI through the implementation, to paraphrase a famous
> saying. There is nothing to save IMHO. Of course, someone else can work
> on it. I prefer working on a sane solution myself,
> 
> cheers,
> 
> David

To taunt Ondrej: A one minute build isn't forever - numpy is tiny and I 
understand why it might seem long compared to SymPy, but just wait until 
you add Cython extensions per default and those build times will go up 
substantially ;).

Cheers,

Michael

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From chaos.proton at gmail.com  Thu Feb 19 00:31:09 2009
From: chaos.proton at gmail.com (Grissiom)
Date: Thu, 19 Feb 2009 13:31:09 +0800
Subject: [Numpy-discussion] linalg.norm along axis?
In-Reply-To: <954ae5aa0902182012s17014ab7w7d07c65551ca5ef3@mail.gmail.com>
References: <a0adeea50902181923n74995342sadc55258cd8414e@mail.gmail.com>
	<954ae5aa0902182012s17014ab7w7d07c65551ca5ef3@mail.gmail.com>
Message-ID: <a0adeea50902182131x75b58374sf74f0add3dca16c5@mail.gmail.com>

On Thu, Feb 19, 2009 at 12:12, Nicolas Pinto <pinto at mit.edu> wrote:

> Grissiom,
>
> Using the following doesn't require any loop:
>
> In [9]: sqrt((a**2.).sum(1))
> Out[9]: array([  5.,  10.])
>
> Best,
>
>
Got it~ Thanks really ;)

-- 
Cheers,
Grissiom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090219/1d50fc27/attachment.html>

From ellisonbg.net at gmail.com  Thu Feb 19 00:45:14 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 18 Feb 2009 21:45:14 -0800
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <5b8d13220902182125u51f98727t40ff4fc32593a1a0@mail.gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
	<5b8d13220902182125u51f98727t40ff4fc32593a1a0@mail.gmail.com>
Message-ID: <6ce0ac130902182145k6d5697f4u912c423ce3dc4f87@mail.gmail.com>

David,

Thank you very much for writing this summary up.  It collects a lot of
useful information about the subtle and difficult issues related to
building multi-language python packages.

The reason that Ondrej and I are asking all of these questions is that
we are currently exploring build systems for a couple of projects we
are working on (together).  In general, my sense is that distutils
won't be sufficient for our build needs.  Thus, we are looking at
Scons, cmake, numscons, waf, etc.  Honestly, compared to plain
distutils, *any* of these other solutions seem very nice, even that
each of them has its own downsides.

The big challenge as you mention is that distutils does do certain
things that plain Scons, cmake or waf doesn't:

* Install Python package in a way that is consistent with common
Python practices (site-packages, etc.)
* Integration with setuptools and eggs, which enables things like
namespace packages.
* Windows installers, source tarballs.
* Uploads to pypi.

I wonder if it wouldn't be too difficult to simply implement these
capabilities in raw cmake/Scons/waf and do away with distutils all
together.  But, I don't really know enough about distutils or
cmake/scons/waf to know if this is reasonable to dream about.

Cheers,

Brian

On Wed, Feb 18, 2009 at 9:25 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Thu, Feb 19, 2009 at 9:14 AM, Ondrej Certik <ondrej at certik.cz> wrote:
>> Hi,
>>
>> I have a shiny new computer with 8 cores and numpy still takes forever
>> to compile --- is there a way to compile it in parallel (make -j9)?
>>
>> Do distutils allow that? If not, let's move to some build system that
>> allows that?
>
> Since numscons was mentioned several times recently, and I have not
> answered to every request, I wrote a more complete post about it -
> even with somes advices for people motivated in using cmake :)
>
> http://cournape.wordpress.com/2009/02/19/numscons-current-state-future-alternative-build-tools-for-numpy/
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From david at ar.media.kyoto-u.ac.jp  Thu Feb 19 00:28:35 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 19 Feb 2009 14:28:35 +0900
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <499CEDA3.9020005@gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>	<5b8d13220902181757x4a58ce79h7d597e432d5bf22a@mail.gmail.com>	<gnih7d$u4h$1@ger.gmane.org>	<499CCD1B.60701@ar.media.kyoto-u.ac.jp>
	<499CEDA3.9020005@gmail.com>
Message-ID: <499CEE03.7080507@ar.media.kyoto-u.ac.jp>

Michael Abshoff wrote:
> David Cournapeau wrote:
>   
>> Christian Heimes wrote:
>>     
>>> David Cournapeau wrote:
>>>       
>
> Hi,
>
>   
>>> You may call me naive and ignorant. Is it really that hard to archive
>>> some kind of poor man's concurrency? You don't have to parallelize
>>> everything to get a speed up on multi core machines. Usually the compile
>>> process from C/C++ file to an object files takes up most of the time.
>>>
>>> How about
>>>
>>> * assemble a list of all C/C++ source files of all extensions.
>>> * compile all source files in parallel
>>> * do the rest (linking etc.) in serial
>>>   
>>>       
>
> With Sage we do the cythonization in parallel and for now build 
> extension serially, but we have code to do that in parallel, too. Given 
> that we are building 180 extensions or so the speedup is linear. I often 
> do this using 24 cores, so it seems robust since I do work on Sage daily 
> and often to test builds from scratch and I never had any problems with 
> that code.
>   

Note that building from scratch is the easy case, specially in the case
of parallel builds. Also, I would guess "cythonizing" is easy, at least
if it is done entirely in python. Races conditions in subprocess are a
real problem, it caused numerous issues in scons and waf, so I would be
really surprised if it did not caused any trouble in distutils.
Particularly, on windows, subprocess up to python 2.4 was problematic, I
believe (I should really check, because I was not involved in the
related discussions nor with the fixes in scons).

> To taunt Ondrej: A one minute build isn't forever - numpy is tiny and I 
> understand why it might seem long compared to SymPy, but just wait until 
> you add Cython extensions per default and those build times will go up 
> substantially
>   

Building scipy installer on windows takes 1 hour, which is already
relatively significant. But really, parallel builds is just a nice
consequence of using a sane build tool. I simply cannot stand distutils
anymore; it now feels even more painful than developing on windows.
Every time you touch something, something else, totally unrelated breaks.

cheers,

David


From michael.abshoff at googlemail.com  Thu Feb 19 01:05:45 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 18 Feb 2009 22:05:45 -0800
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <499CEE03.7080507@ar.media.kyoto-u.ac.jp>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>	<5b8d13220902181757x4a58ce79h7d597e432d5bf22a@mail.gmail.com>	<gnih7d$u4h$1@ger.gmane.org>	<499CCD1B.60701@ar.media.kyoto-u.ac.jp>	<499CEDA3.9020005@gmail.com>
	<499CEE03.7080507@ar.media.kyoto-u.ac.jp>
Message-ID: <499CF6B9.8060300@gmail.com>

David Cournapeau wrote:
> Michael Abshoff wrote:
>> David Cournapeau wrote:

Hi David,

>> With Sage we do the cythonization in parallel and for now build 
>> extension serially, but we have code to do that in parallel, too. Given 
>> that we are building 180 extensions or so the speedup is linear. I often 
>> do this using 24 cores, so it seems robust since I do work on Sage daily 
>> and often to test builds from scratch and I never had any problems with 
>> that code.
>>   
> 
> Note that building from scratch is the easy case, specially in the case
> of parallel builds.

Sure, it also works for incremental builds and I do that many, many 
times a day, i.e. for each patch I merge into the Sage library. What 
gets recompiled is decided by our own dependency tracking code which we 
want to push into Cython itself. Figuring out dependencies on the fly 
without caching takes about 1s for the whole Sage library which includes 
parsing every Cython file.

Note that we build each extension in parallel, so if you depend on a lot 
of fortran or c code to be linked into one extension this obviously 
doesn't help much. The situation with Sage's extension is that we build 
external libraries ahead of time and 99% of extensions do not have 
additional C/C++ files and those who do have usually one to three extra 
files, so for our purposes this scales linear.

> Also, I would guess "cythonizing" is easy, at least
> if it is done entirely in python. Races conditions in subprocess are a
> real problem, it caused numerous issues in scons and waf, so I would be
> really surprised if it did not caused any trouble in distutils.
> Particularly, on windows, subprocess up to python 2.4 was problematic, I
> believe (I should really check, because I was not involved in the
> related discussions nor with the fixes in scons).

We used to use threads for the "parallel stuff" and it is indeed racy, 
but that was mostly observed when running doctests since we only had one 
current directory. All those problems went away once we started to use 
Pyprocessing and while there is some overhead for the forks it is 
drowned out by the build time when using 2 cors.

>> To taunt Ondrej: A one minute build isn't forever - numpy is tiny and I 
>> understand why it might seem long compared to SymPy, but just wait until 
>> you add Cython extensions per default and those build times will go up 
>> substantially
>>   
> 
> Building scipy installer on windows takes 1 hour, which is already
> relatively significant. 

Ouch. Is that without the dependencies, i.e. ATLAS?

I was curious how you build the various version of ATLAS, i.e. no SSE, 
SSE, SSE2, etc. Do you just set the arch via -A and build them all on 
the same box? [sorry for getting slightly OT here :)]

> But really, parallel builds is just a nice
> consequence of using a sane build tool. I simply cannot stand distutils
> anymore; it now feels even more painful than developing on windows.
> Every time you touch something, something else, totally unrelated breaks.

Yeah, distutils are a pain, but numpy extending/modifying them doesn't 
make it any cleaner :(. I am looking forward to the day NumScons is part 
of Numpy though.

> cheers,
> 
> David

Cheers,

Michael

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From david at ar.media.kyoto-u.ac.jp  Thu Feb 19 00:47:48 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 19 Feb 2009 14:47:48 +0900
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <6ce0ac130902182145k6d5697f4u912c423ce3dc4f87@mail.gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>	<5b8d13220902182125u51f98727t40ff4fc32593a1a0@mail.gmail.com>
	<6ce0ac130902182145k6d5697f4u912c423ce3dc4f87@mail.gmail.com>
Message-ID: <499CF284.3060508@ar.media.kyoto-u.ac.jp>

Brian Granger wrote:
> The reason that Ondrej and I are asking all of these questions is that
> we are currently exploring build systems for a couple of projects we
> are working on (together).  In general, my sense is that distutils
> won't be sufficient for our build needs.  Thus, we are looking at
> Scons, cmake, numscons, waf, etc.  Honestly, compared to plain
> distutils, *any* of these other solutions seem very nice, even that
> each of them has its own downsides.
>   

I agree that for purely build issues, anything is better than distutils.
Heck, I would take autotools + m4 over distutils any day. To be fair to
distutils, distutils was never conceived nor intented to do complex C
handling.

> * Install Python package in a way that is consistent with common
> Python practices (site-packages, etc.)
>   

This is doable

> * Integration with setuptools and eggs, which enables things like
> namespace packages.
>   

This is not. eggs are not specified, and totally implementation defined.
I tried some time ago to  add an egg builder to scons, but I gave up.
And I don't think you can reuse the setuptools code, as everything is
coupled.

> * Uploads to pypi.
>   

Pypi has a xmlrpc-based API, so this should be doable relatively easily.

> I wonder if it wouldn't be too difficult to simply implement these
> capabilities in raw cmake/Scons/waf and do away with distutils all
> together. 
waf already has a relatively advanced python extension builder, and can
build tarballs. The problem with scons is the tight coupling of the
code: any significant change cannot be done ATM without deep knowledge
about the whole codebase. Be it for options/tarball handling or speed
issues. I don't know enough about cmake.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Thu Feb 19 01:04:13 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 19 Feb 2009 15:04:13 +0900
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <499CF6B9.8060300@gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>	<5b8d13220902181757x4a58ce79h7d597e432d5bf22a@mail.gmail.com>	<gnih7d$u4h$1@ger.gmane.org>	<499CCD1B.60701@ar.media.kyoto-u.ac.jp>	<499CEDA3.9020005@gmail.com>	<499CEE03.7080507@ar.media.kyoto-u.ac.jp>
	<499CF6B9.8060300@gmail.com>
Message-ID: <499CF65D.3000309@ar.media.kyoto-u.ac.jp>

Michael Abshoff wrote:
>
> Sure, it also works for incremental builds and I do that many, many 
> times a day, i.e. for each patch I merge into the Sage library. What 
> gets recompiled is decided by our own dependency tracking code which we 
> want to push into Cython itself. Figuring out dependencies on the fly 
> without caching takes about 1s for the whole Sage library which includes 
> parsing every Cython file.
>   

Hm, I think I would have to look at what sage is internally to really
understand the implications. But surely, if you can figure out in one
second the whole dependency for scipy, I would be more than impressed:
you would beat waf and make at their own game.

>
> We used to use threads for the "parallel stuff" and it is indeed racy, 
> but that was mostly observed when running doctests since we only had one 
> current directory. All those problems went away once we started to use 
> Pyprocessing and while there is some overhead for the forks it is 
> drowned out by the build time when using 2 cors.
>   

Does pyprocessing work well on windows as well ? I have 0 experience
with it.

>
> Ouch. Is that without the dependencies, i.e. ATLAS?
>   

Yes - but I need to build scipy three times, for each ATLAS (if I could
use numscons, it would be much better, since a library change is handled
as a dependency in scons; with distutils, the only safe way is to
rebuild from scratch for every configuration).

> I was curious how you build the various version of ATLAS, i.e. no SSE, 
> SSE, SSE2, etc. Do you just set the arch via -A and build them all on 
> the same box? [sorry for getting slightly OT here :)]
>   

This does not work: ATLAS will still use SSE if your CPU supports it,
even if you force an arch without SSE. I tried two different things:
first, using a patched qemu with options to emulate a P4 wo SSE, with
SSE2 and with SSE3, but this does not work so well (the generated
versions are too slow, and handling virtual machines in qemu is a bit of
a pain). Now, I just build on different machines, and hope I won't need
to rebuild them too often.

cheers,

David


From strawman at astraw.com  Thu Feb 19 01:24:04 2009
From: strawman at astraw.com (Andrew Straw)
Date: Wed, 18 Feb 2009 22:24:04 -0800
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <499CF284.3060508@ar.media.kyoto-u.ac.jp>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>	<5b8d13220902182125u51f98727t40ff4fc32593a1a0@mail.gmail.com>	<6ce0ac130902182145k6d5697f4u912c423ce3dc4f87@mail.gmail.com>
	<499CF284.3060508@ar.media.kyoto-u.ac.jp>
Message-ID: <499CFB04.7020305@astraw.com>

David Cournapeau wrote:
>> * Integration with setuptools and eggs, which enables things like
>> namespace packages.
>>   
> 
> This is not. eggs are not specified, and totally implementation defined.
> I tried some time ago to  add an egg builder to scons, but I gave up.
> And I don't think you can reuse the setuptools code, as everything is
> coupled.

It's an interesting idea to build Python package distributions without 
distutils. For pure Python installables, if all you seek better is 
distutils, the bar seems fairly low. For compiled stuff, it still 
doesn't seem too bad. Of course, it is easy to say this without having 
tried...

So, what do you mean by an "egg", in the context of it being hard to 
produce? An .egg zip file, an .egg directory, and/or a normal distutils 
package with an .egg-info/ sibling? Since .egg-info/  is now part of 
distutils, this should now be specified... or?

[This, though, does point out a conceptual problem with setuptools for 
me -- it does a zillion things some of which I like a lot (e.g. 
installing console scripts and gui scripts, a simple plugin 
architecture) and others I don't care about as long as they don't break 
things for me (e.g. installing multiple version of packages 
side-by-side, a problem which is much more sanely solved by setting 
PYTHONPATH, or its sophisticated cousin, virtualenv). And all of these 
things go by one name "setuptools" and sometimes even "eggs", even 
though people often use those words to discuss totally different 
features. Hence my question above.]

-Andrew


From cournape at gmail.com  Thu Feb 19 01:34:24 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 19 Feb 2009 15:34:24 +0900
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <499CFB04.7020305@astraw.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
	<5b8d13220902182125u51f98727t40ff4fc32593a1a0@mail.gmail.com>
	<6ce0ac130902182145k6d5697f4u912c423ce3dc4f87@mail.gmail.com>
	<499CF284.3060508@ar.media.kyoto-u.ac.jp>
	<499CFB04.7020305@astraw.com>
Message-ID: <5b8d13220902182234v319ebba2r22c078948a9cc800@mail.gmail.com>

On Thu, Feb 19, 2009 at 3:24 PM, Andrew Straw <strawman at astraw.com> wrote:
> David Cournapeau wrote:
>>> * Integration with setuptools and eggs, which enables things like
>>> namespace packages.
>>>
>>
>> This is not. eggs are not specified, and totally implementation defined.
>> I tried some time ago to  add an egg builder to scons, but I gave up.
>> And I don't think you can reuse the setuptools code, as everything is
>> coupled.
>
> It's an interesting idea to build Python package distributions without
> distutils. For pure Python installables, if all you seek better is
> distutils, the bar seems fairly low.

:) Being better than distutils is not difficult, indeed - that is if
you don't care about backward compatibility with distutils (which I
don't personally - since distutils is implementation defined, I don't
see any way to be backward compatible and be a significant improvement
at the same time).

> For compiled stuff, it still
> doesn't seem too bad. Of course, it is easy to say this without having
> tried...

There are some python build tools which do not have advanced
dependency handling (paver, etc...). But for compiled code, I don't
think it makes any sense not to be based on a dependency graph.
Specially for something like scipy. And this is a complex task - the
core waf code is really nice, and small (< 4000 LOC for the core).

> So, what do you mean by an "egg", in the context of it being hard to
> produce? An .egg zip file, an .egg directory, and/or a normal distutils
> package with an .egg-info/ sibling? Since .egg-info/  is now part of
> distutils, this should now be specified... or?

Producing something such as easy_install blabla.egg "works" (that is
as is intended to work in the setuptools world).

> [This, though, does point out a conceptual problem with setuptools for
> me -- it does a zillion things some of which I like a lot (e.g.
> installing console scripts and gui scripts, a simple plugin
> architecture) and others I don't care about as long as they don't break
> things for me (e.g. installing multiple version of packages
> side-by-side, a problem which is much more sanely solved by setting
> PYTHONPATH, or its sophisticated cousin, virtualenv). And all of these
> things go by one name "setuptools" and sometimes even "eggs", even
> though people often use those words to discuss totally different
> features. Hence my question above.]

I will refrain from speaking about setuptools :) But the above problem
is the same for distutils and setuptools, and exactly the fundamental
issue. If distutils was conceived as a set of lousely coupled modules
for tools, build, distribution, it would have been fixable. In the
case of setuptools, it was a conscious decision to force setuptools on
python world,

David


From michael.abshoff at googlemail.com  Thu Feb 19 01:43:40 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 18 Feb 2009 22:43:40 -0800
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <499CF65D.3000309@ar.media.kyoto-u.ac.jp>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>	<5b8d13220902181757x4a58ce79h7d597e432d5bf22a@mail.gmail.com>	<gnih7d$u4h$1@ger.gmane.org>	<499CCD1B.60701@ar.media.kyoto-u.ac.jp>	<499CEDA3.9020005@gmail.com>	<499CEE03.7080507@ar.media.kyoto-u.ac.jp>	<499CF6B9.8060300@gmail.com>
	<499CF65D.3000309@ar.media.kyoto-u.ac.jp>
Message-ID: <499CFF9C.1000901@gmail.com>

David Cournapeau wrote:
> Michael Abshoff wrote:

Hi David,

>> Sure, it also works for incremental builds and I do that many, many 
>> times a day, i.e. for each patch I merge into the Sage library. What 
>> gets recompiled is decided by our own dependency tracking code which we 
>> want to push into Cython itself. Figuring out dependencies on the fly 
>> without caching takes about 1s for the whole Sage library which includes 
>> parsing every Cython file.
>>   
> 
> Hm, I think I would have to look at what sage is internally to really
> understand the implications. But surely, if you can figure out in one
> second the whole dependency for scipy, I would be more than impressed:
> you would beat waf and make at their own game.

I didn't write the code, but thanks :)

We used to cache the dependency tree by pickling it and if no time 
stamps changed we would reuse it, so that phase would be instant. Alas, 
there was one unfixed bug in it that hit you if you removed a extension 
or file. But the main author of that code (Craig Citro - to give credit 
where credit is due) has an idea how to fix it and once his PhD thesis 
is handed in will stomp that bug out.

>> We used to use threads for the "parallel stuff" and it is indeed racy, 
>> but that was mostly observed when running doctests since we only had one 
>> current directory. All those problems went away once we started to use 
>> Pyprocessing and while there is some overhead for the forks it is 
>> drowned out by the build time when using 2 cors.
>>   
> 
> Does pyprocessing work well on windows as well ? I have 0 experience
> with it.

It should, but I haven't tested it. The last official, stand alone 
pyprocessing we have in Sage causes trouble on FreeBSD 7 by segfaulting, 
so we will likely update to the backport from Python 2.6 soon since for 
now we are stuck at Python 2.5 until numpy/scipy and a bunch of other 
Python projects like NetworkX support it officially ;).

>> Ouch. Is that without the dependencies, i.e. ATLAS?
>>   
> 
> Yes - but I need to build scipy three times, for each ATLAS (if I could
> use numscons, it would be much better, since a library change is handled
> as a dependency in scons; with distutils, the only safe way is to
> rebuild from scratch for every configuration).

In Sage if we link static libs into extension we add a dependency to the 
header. But you could do the same for libatlas.a, so dropping in a new 
version and touching it should just rebuild the extensions depending on 
atlas. I have tested this extensively and not found any problem with 
that approach.

>> I was curious how you build the various version of ATLAS, i.e. no SSE, 
>> SSE, SSE2, etc. Do you just set the arch via -A and build them all on 
>> the same box? [sorry for getting slightly OT here :)]
>>   
> 
> This does not work: ATLAS will still use SSE if your CPU supports it,
> even if you force an arch without SSE. I tried two different things:
> first, using a patched qemu with options to emulate a P4 wo SSE, with
> SSE2 and with SSE3, but this does not work so well (the generated
> versions are too slow, and handling virtual machines in qemu is a bit of
> a pain). Now, I just build on different machines, and hope I won't need
> to rebuild them too often.

Ok, I now remember what I did about this problem two days ago since I 
want to build SSE2 only binary releases of Sage. Apparently there are 
people out there who aren't using Intel/AMD CPUs with SSE3 :)

In order to make ATLAS build without say SSE3 go into the config system 
and have the SSE3 probe return "FAILURE" unconditionally. That way ATLAS 
will only pick SSE2 even if the CPU handles more. I verified by using 
objdump that the resulting lib does not contain any PNI (==SSE3) 
instructions any more, see

  http://trac.sagemath.org/sage_trac/ticket/5219

A problem here is that you get odd arches without tuning info, i.e. 
P464SSE2, so one has to build tuning info one and drop it into 
subsequent builds of ATLAS. You will also have the problem of Hammer vs. 
P4 ATLAS kernels, so one day I will measure performance.

I meant to ask Clint about adding a configure switch for a maximum SSE 
level to ATLAS itself, but since I only got the problem solved two days 
ago I hadn't gotten around to it. Given that everything else is 
configurable it seems like he would welcome it.

If you want more details on where to poke around in the config system 
let me know.

> cheers,
> 
> David

Cheers,

Michael

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From strawman at astraw.com  Thu Feb 19 02:26:52 2009
From: strawman at astraw.com (Andrew Straw)
Date: Wed, 18 Feb 2009 23:26:52 -0800
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <5b8d13220902182234v319ebba2r22c078948a9cc800@mail.gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>	<5b8d13220902182125u51f98727t40ff4fc32593a1a0@mail.gmail.com>	<6ce0ac130902182145k6d5697f4u912c423ce3dc4f87@mail.gmail.com>	<499CF284.3060508@ar.media.kyoto-u.ac.jp>	<499CFB04.7020305@astraw.com>
	<5b8d13220902182234v319ebba2r22c078948a9cc800@mail.gmail.com>
Message-ID: <499D09BC.7050806@astraw.com>

David Cournapeau wrote:
> On Thu, Feb 19, 2009 at 3:24 PM, Andrew Straw <strawman at astraw.com> wrote:
>   
>> It's an interesting idea to build Python package distributions without
>> distutils. For pure Python installables, if all you seek better is
>> distutils, the bar seems fairly low.
>>     
>
> :) Being better than distutils is not difficult, indeed - that is if
> you don't care about backward compatibility with distutils (which I
> don't personally - since distutils is implementation defined, I don't
> see any way to be backward compatible and be a significant improvement
> at the same time).
>   
Maybe if you need a level of backward compatibility, (and really, to 
gain a decent audience for this idea, I think you do need some level of 
backward compatibility) the new tool could emit setup.py files for 
consumption by distutils as a fallback plan. I could imagine 
auto-generating a setup.py is easier than emulating distutils, 
particularly if your concept starts simple. Furthermore, if you're not 
opposed to dropping in your own distutils monkeypatches, like lots of 
other packages, you probably could do anything you wanted. For example, 
bypassing the build_ext command and injecting the built products into 
the distutils install command.

This monkeypatching idea is not even so unpalatable if one remembers 
that monkeypatching, which is quite common, makes it literally 
impossible to emulate distutils without being distutils.
> I will refrain from speaking about setuptools :) But the above problem
> is the same for distutils and setuptools, and exactly the fundamental
> issue. If distutils was conceived as a set of lousely coupled modules
> for tools, build, distribution, it would have been fixable. In the
> case of setuptools, it was a conscious decision to force setuptools on
> python world,
This reminds of Linus' criticism of svn: its goal is to be a better cvs. 
Said dripping with incredulity due to his perception of the fatal flaws 
of CVS. Well, I think (parts of) setuptools are better than distutils, 
but by being distutils+, it will always share the same flawed genetic 
material...


From cournape at gmail.com  Thu Feb 19 02:47:00 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 19 Feb 2009 16:47:00 +0900
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <499D09BC.7050806@astraw.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
	<5b8d13220902182125u51f98727t40ff4fc32593a1a0@mail.gmail.com>
	<6ce0ac130902182145k6d5697f4u912c423ce3dc4f87@mail.gmail.com>
	<499CF284.3060508@ar.media.kyoto-u.ac.jp>
	<499CFB04.7020305@astraw.com>
	<5b8d13220902182234v319ebba2r22c078948a9cc800@mail.gmail.com>
	<499D09BC.7050806@astraw.com>
Message-ID: <5b8d13220902182347p3d88af5bpafa17251168464aa@mail.gmail.com>

On Thu, Feb 19, 2009 at 4:26 PM, Andrew Straw <strawman at astraw.com> wrote:

> Maybe if you need a level of backward compatibility, (and really, to
> gain a decent audience for this idea, I think you do need some level of
> backward compatibility) the new tool could emit setup.py files for
> consumption by distutils as a fallback plan.

When I say I don't care about backward compatibility, it was in the
context of numpy. If I wanted to do a better tool for python, I would
have done things differently.

> Furthermore, if you're not
> opposed to dropping in your own distutils monkeypatches, like lots of
> other packages, you probably could do anything you wanted. For example,
> bypassing the build_ext command and injecting the built products into
> the distutils install command.

you've just described the scons command in numpy.distutils :)

> This reminds of Linus' criticism of svn: its goal is to be a better cvs.
> Said dripping with incredulity due to his perception of the fatal flaws
> of CVS. Well, I think (parts of) setuptools are better than distutils,
> but by being distutils+, it will always share the same flawed genetic
> material...

We should have a Linus Torvald : very few people are capable of
building a new tool which blows away the whole field, in two weeks :)
Also, svn, for all its flaws, was at least based on concepts which
were widely shared among the people working on the problem, whereas
distutils isn't (building things from a DAG is known for decades).

David


From sturla at molden.no  Thu Feb 19 05:23:44 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 19 Feb 2009 11:23:44 +0100
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <499CF65D.3000309@ar.media.kyoto-u.ac.jp>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>	<5b8d13220902181757x4a58ce79h7d597e432d5bf22a@mail.gmail.com>	<gnih7d$u4h$1@ger.gmane.org>	<499CCD1B.60701@ar.media.kyoto-u.ac.jp>	<499CEDA3.9020005@gmail.com>	<499CEE03.7080507@ar.media.kyoto-u.ac.jp>	<499CF6B9.8060300@gmail.com>
	<499CF65D.3000309@ar.media.kyoto-u.ac.jp>
Message-ID: <499D3330.3030306@molden.no>

On 2/19/2009 7:04 AM, David Cournapeau wrote:

> Does pyprocessing work well on windows as well ? I have 0 experience
> with it.

Yes it works well on Windows, albeit process creation is a bit slower 
than on Unix (there is no os.fork in Windows, so more Python objects has 
to be pickled). From Python 2.6, pyprocessing is even in the standard 
library, renamed to 'multiprocessing'.


S.M.


From cournape at gmail.com  Thu Feb 19 10:16:42 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 20 Feb 2009 00:16:42 +0900
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
Message-ID: <5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>

On Sun, Feb 15, 2009 at 5:09 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Sun, Feb 15, 2009 at 01:48, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
>
>> Would people be against this ?
>
> Not I.

Ok, I have started this in the coremath branch - it solves the warning
issues we got since the merge of formatting stuff. I tested it on
Linux, windows (both mingw and VS - still need to test on Win64), so I
think it is good to go in the trunk. The functions exported are in a
separate header:

http://projects.scipy.org/scipy/numpy/browser/branches/coremath/numpy/core/include/numpy/npy_math.h

I did not put the complex functions there (nc_), but maybe they should
be, I am not sure.

cheers,

David


From pav at iki.fi  Thu Feb 19 10:28:42 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 19 Feb 2009 15:28:42 +0000 (UTC)
Subject: [Numpy-discussion] Core math library in numpy
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
Message-ID: <gnjtra$1bo$1@ger.gmane.org>

Fri, 20 Feb 2009 00:16:42 +0900, David Cournapeau wrote:
[clip]
> Ok, I have started this in the coremath branch - it solves the warning
> issues we got since the merge of formatting stuff. I tested it on Linux,
> windows (both mingw and VS - still need to test on Win64), so I think it
> is good to go in the trunk. The functions exported are in a separate
> header:
> 
> http://projects.scipy.org/scipy/numpy/browser/branches/coremath/numpy/
core/include/numpy/npy_math.h

One question: doesn't this add one extra function call to all umath 
functions? Could we do '#define npy_XXX XXX' in the npy_math.h header 
when the appropriate platform-specified functions are available?

> I did not put the complex functions there (nc_), but maybe they should
> be, I am not sure.

I think they should be. Then we could easily use C99 complex math 
functions on plaforms on which they are available (and so get the 
"correct" corner-case semantics for free on these platforms).

	Pauli



From cournape at gmail.com  Thu Feb 19 11:05:03 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 20 Feb 2009 01:05:03 +0900
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <gnjtra$1bo$1@ger.gmane.org>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org>
Message-ID: <5b8d13220902190805s2d12ac96va40b6062959fc53@mail.gmail.com>

On Fri, Feb 20, 2009 at 12:28 AM, Pauli Virtanen <pav at iki.fi> wrote:
> Fri, 20 Feb 2009 00:16:42 +0900, David Cournapeau wrote:
> [clip]
>> Ok, I have started this in the coremath branch - it solves the warning
>> issues we got since the merge of formatting stuff. I tested it on Linux,
>> windows (both mingw and VS - still need to test on Win64), so I think it
>> is good to go in the trunk. The functions exported are in a separate
>> header:
>>
>> http://projects.scipy.org/scipy/numpy/browser/branches/coremath/numpy/
> core/include/numpy/npy_math.h
>
> One question: doesn't this add one extra function call to all umath
> functions? Could we do '#define npy_XXX XXX' in the npy_math.h header
> when the appropriate platform-specified functions are available?

I think it does add a cost. I thought about it, but I don't know
whether this is significant or not. The problem of #define npy_XXX XXX
is that the header needs to know whether the corresponding function is
available - and we can't import HAVE_XXX symbols in npy_math.h.

Another argument against the define is that in the future, we could
load the actual implementation at runtime (SSE, etc...). If we use
define, that's not possible.

I don't know how to evaluate the cost of function call on those
functions: I tried some naive benchmark, but with cache effects and
all, I am not convinced I was measuring anything meaningful.

>
> I think they should be. Then we could easily use C99 complex math
> functions on plaforms on which they are available (and so get the
> "correct" corner-case semantics for free on these platforms).

maybe we could change the names, then ? nc is not very clear IMHO (and
since they were static up to now, we are free to change them I
believe).

cheers,

David


From cournape at gmail.com  Thu Feb 19 11:12:08 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 20 Feb 2009 01:12:08 +0900
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <5b8d13220902190805s2d12ac96va40b6062959fc53@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org>
	<5b8d13220902190805s2d12ac96va40b6062959fc53@mail.gmail.com>
Message-ID: <5b8d13220902190812g35107b5cncb2e2152661948ad@mail.gmail.com>

On Fri, Feb 20, 2009 at 1:05 AM, David Cournapeau <cournape at gmail.com> wrote:

>
> Another argument against the define is that in the future, we could
> load the actual implementation at runtime (SSE, etc...). If we use
> define, that's not possible.

Hm, that's actually a pretty stupid statement - we could certainly
have an ifdef to conditionally use define or not.

David


From cournape at gmail.com  Thu Feb 19 11:32:07 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 20 Feb 2009 01:32:07 +0900
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
Message-ID: <5b8d13220902190832q23f09867te59c5f46261642a5@mail.gmail.com>

On Fri, Feb 20, 2009 at 12:16 AM, David Cournapeau <cournape at gmail.com> wrote:
> On Sun, Feb 15, 2009 at 5:09 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Sun, Feb 15, 2009 at 01:48, David Cournapeau
>> <david at ar.media.kyoto-u.ac.jp> wrote:
>>
>>> Would people be against this ?
>>
>> Not I.
>
> Ok, I have started this in the coremath branch - it solves the warning
> issues we got since the merge of formatting stuff. I tested it on
> Linux, windows (both mingw and VS - still need to test on Win64)

Surprisingly, it worked out of the box on win64 (with python 2.6).
There are some errors, but totally unrelated to the changes. That
alone is enough of an argument for inclusion IMO - those compiler
crashes were driving me crazy,

cheers,

David


From charlesr.harris at gmail.com  Thu Feb 19 12:23:08 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 19 Feb 2009 10:23:08 -0700
Subject: [Numpy-discussion] parallel compilation of numpy
In-Reply-To: <5b8d13220902182347p3d88af5bpafa17251168464aa@mail.gmail.com>
References: <85b5c3130902181614i2e7129eau262162b343398ec4@mail.gmail.com>
	<5b8d13220902182125u51f98727t40ff4fc32593a1a0@mail.gmail.com>
	<6ce0ac130902182145k6d5697f4u912c423ce3dc4f87@mail.gmail.com>
	<499CF284.3060508@ar.media.kyoto-u.ac.jp>
	<499CFB04.7020305@astraw.com>
	<5b8d13220902182234v319ebba2r22c078948a9cc800@mail.gmail.com>
	<499D09BC.7050806@astraw.com>
	<5b8d13220902182347p3d88af5bpafa17251168464aa@mail.gmail.com>
Message-ID: <e06186140902190923q3dbf2c6ema20ae38ae3f6c732@mail.gmail.com>

On Thu, Feb 19, 2009 at 12:47 AM, David Cournapeau <cournape at gmail.com>wrote:

> On Thu, Feb 19, 2009 at 4:26 PM, Andrew Straw <strawman at astraw.com> wrote:
>
> > Maybe if you need a level of backward compatibility, (and really, to
> > gain a decent audience for this idea, I think you do need some level of
> > backward compatibility) the new tool could emit setup.py files for
> > consumption by distutils as a fallback plan.
>
> When I say I don't care about backward compatibility, it was in the
> context of numpy. If I wanted to do a better tool for python, I would
> have done things differently.
>
> > Furthermore, if you're not
> > opposed to dropping in your own distutils monkeypatches, like lots of
> > other packages, you probably could do anything you wanted. For example,
> > bypassing the build_ext command and injecting the built products into
> > the distutils install command.
>
> you've just described the scons command in numpy.distutils :)
>
> > This reminds of Linus' criticism of svn: its goal is to be a better cvs.
> > Said dripping with incredulity due to his perception of the fatal flaws
> > of CVS. Well, I think (parts of) setuptools are better than distutils,
> > but by being distutils+, it will always share the same flawed genetic
> > material...
>
> We should have a Linus Torvald : very few people are capable of
> building a new tool which blows away the whole field, in two weeks :)
> Also, svn, for all its flaws, was at least based on concepts which
> were widely shared among the people working on the problem, whereas
> distutils isn't (building things from a DAG is known for decades).
>

Linus Torvalds and git brought Larry McVoy to mind and this little piece he
has on his site. For some reason this seems like a good time to post it ;)

The Bug Count Also Rises
 by John Browne
(Imitation Hemingway Contest Winner)

In the fall of that year the rains fell as usual and washed the leaves of
the dust and dripped from the leaves onto the ground. The shuttles drove
through the rainy streets and took the people to meetings, then later
brought them back, their tires spraying the mist into the air.

Many days he stood for a long time and watched the rain and the shuttles and
drank his double-tall mochas. With the mochas he was strong.

Hernando who worked down the hall and who was large with microbrews came to
him and told him that the ship day was upon them but the bugs were not yet
out. The bugs which were always there even when you were in Cafes late at
night sipping a Redhook or a double-tall mocha and you thought you were safe
but they were there and although Enrico kept the floor swept clean and the
mochas were hot the bugs were there and they ate at you.

When Hernando told him this he asked how many bugs. "The RAID is huge with
bugs," Hernando said. "The bugs are infinite."

"Why do you ask me? You know I cannot do this thing anymore with the bugs."

"Once you were great with the bugs," Hernando said. "No one was greater," he
said again. "Even Prado."

"Prado? What of Prado? Let Prado fix the bugs."

Hernando shrugged. "Prado is finished. He was gored by three Sev 2's in
Chicago. All he does now is drink herb tea and play with his screensavers."

"Herb tea?"

"It is true, my friend." Hernando shrugged again. Later he went to his
office and sat in the dark for a long time. Then he sent e-mail to Michaels.


Michaels came to him while he was sipping a mocha. They sat silently for
awhile, then he asked Michaels, "I need you to triage for me."

Michaels looked down. "I don't do that anymore," he said.

"This is different. The bugs are enormous. There are an infinity of bugs."

"I'm finished with that," Michaels said again. "I just want to live
quietly."

"Have you heard Prado is finished? He was badly gored. Now he can only drink
herb tea."

"Herb tea?" Michaels said.

"It is true," he said sorrowfully.

Michaels stood up. "Then I will do it, my friend," he said formally. "I will
do it for Prado, who was once great with the bugs. I will do it for the time
we filled Prado's office with bouncy balls, and for the time Prado wore his
nerf weapons in the marketing hall and slew all of them with no fear and
only a great joy at the combat. I will do it for all the pizza we ate and
the bottles of Coke we drank."

Together they walked slowly back, knowing it would be good. As they walked
the rain dripped softly from the leaves, and the shuttles carried the bodies
back from the meetings.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090219/04784bb8/attachment.html>

From ondrej at certik.cz  Thu Feb 19 12:36:42 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 19 Feb 2009 09:36:42 -0800
Subject: [Numpy-discussion] numpy 1.2.1 uploaded to unstable
Message-ID: <85b5c3130902190936keb8a821gc424c1ea72bc3948@mail.gmail.com>

Hi,

I finally found time to upload the numpy 1.2.1 to Debian unstable
(currently it's in incoming: http://incoming.debian.org/). The package
is lintian clean, but there is one test that failed for me in chroot.
I'll wait until it gets to mirrors and then try it on my laptop and
report a bug (I uploaded from a ubuntu machine, but of course I
compiled it in pbuilder with sid and tested in chroot).

Ondrej


From charlesr.harris at gmail.com  Thu Feb 19 13:41:58 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 19 Feb 2009 11:41:58 -0700
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <5b8d13220902190832q23f09867te59c5f46261642a5@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<5b8d13220902190832q23f09867te59c5f46261642a5@mail.gmail.com>
Message-ID: <e06186140902191041m1fbf74fap3384a058800d33a1@mail.gmail.com>

On Thu, Feb 19, 2009 at 9:32 AM, David Cournapeau <cournape at gmail.com>wrote:

> On Fri, Feb 20, 2009 at 12:16 AM, David Cournapeau <cournape at gmail.com>
> wrote:
> > On Sun, Feb 15, 2009 at 5:09 PM, Robert Kern <robert.kern at gmail.com>
> wrote:
> >> On Sun, Feb 15, 2009 at 01:48, David Cournapeau
> >> <david at ar.media.kyoto-u.ac.jp> wrote:
> >>
> >>> Would people be against this ?
> >>
> >> Not I.
> >
> > Ok, I have started this in the coremath branch - it solves the warning
> > issues we got since the merge of formatting stuff. I tested it on
> > Linux, windows (both mingw and VS - still need to test on Win64)
>
> Surprisingly, it worked out of the box on win64 (with python 2.6).
> There are some errors, but totally unrelated to the changes. That
> alone is enough of an argument for inclusion IMO - those compiler
> crashes were driving me crazy,
>

I think this is a bit obfuscated:

--- branches/coremath/numpy/core/code_generators/generate_umath.py
 2009-02-19 11:45:30 UTC (rev 6417)
+++ branches/coremath/numpy/core/code_generators/generate_umath.py
 2009-02-19 11:46:09 UTC (rev 6418)
@@ -37,7 +37,7 @@
            self.out = self.type * nout
        assert len(self.out) == nout

-_fdata_map = dict(f='%sf', d='%s', g='%sl',
+_fdata_map = dict(f='npy_%sf', d='npy_%s', g='npy_%sl',
                  F='nc_%sf', D='nc_%s', G='nc_%sl')
 def build_func_data(types, f):
    func_data = []---
branches/coremath/numpy/core/code_generators/generate_umath.py
     2009-02-19 11:45:30 UTC (rev 6417)
+++ branches/coremath/numpy/core/code_generators/generate_umath.py
 2009-02-19 11:46:09 UTC (rev 6418)
@@ -37,7 +37,7 @@
            self.out = self.type * nout
        assert len(self.out) == nout

-_fdata_map = dict(f='%sf', d='%s', g='%sl',
+_fdata_map = dict(f='npy_%sf', d='npy_%s', g='npy_%sl',
                  F='nc_%sf', D='nc_%s', G='nc_%sl')
 def build_func_data(types, f):
    func_data = []


I think it better to make the npy_* explicit in the TD constructor calls.
That way folks don't have to dig through the code to discover where the
magic npy came from.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090219/6da4bb6d/attachment.html>

From ndbecker2 at gmail.com  Thu Feb 19 14:19:18 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 19 Feb 2009 19:19:18 +0000 (UTC)
Subject: [Numpy-discussion] Build with MKL on linux
Message-ID: <loom.20090219T191632-54@post.gmane.org>

Trying to build numpy-1.2.1 with intel mkl 10.1.1.019 on linux F10 x86_64.

echo $LD_LIBRARY_PATH
/opt/intel/mkl/10.1.1.019/lib/em64t


strace -e trace=file python -c 'import numpy; numpy.test()' 2>stuff
Running unit tests for numpy
NumPy version 1.2.1
NumPy is installed in /usr/lib64/python2.5/site-packages/numpy
Python version 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) [GCC 4.3.2 20080917
(Red Hat 4.3.2-4)]
nose version 0.10.3

MKL FATAL ERROR: Cannot load neither libmkl_mc.so nor libmkl_def.so

strace shows:
...
open("/usr/lib64/python2.5/site-packages/numpy/core/tests/test_blasdot.py",
O_RDONLY) = 3
open("/usr/lib64/python2.5/site-packages/numpy/core/tests/test_blasdot.pyc",
O_RDONLY) = 5
open("/opt/intel/mkl/10.1.1.019/lib/em64t/libmkl_mc.so", O_RDONLY) = 3
open("/opt/intel/mkl/10.1.1.019/lib/em64t/libmkl_mc.so", O_RDONLY) = 3
open("/opt/intel/mkl/10.1.1.019/lib/em64t/libmkl_def.so", O_RDONLY) = 3
open("/opt/intel/mkl/10.1.1.019/lib/em64t/libmkl_def.so", O_RDONLY) = 3

The files are there and found, what could be wrong?



From nwagner at iam.uni-stuttgart.de  Thu Feb 19 15:51:35 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Thu, 19 Feb 2009 21:51:35 +0100
Subject: [Numpy-discussion] Summary of ticket 937
Message-ID: <web-117210804@uni-stuttgart.de>

Hi all,

The summary of ticket 937 is incomplete.
It should be "Complex matrices and lstsq".

http://projects.scipy.org/scipy/numpy/ticket/937

Nils


From robert.kern at gmail.com  Thu Feb 19 16:21:26 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 19 Feb 2009 15:21:26 -0600
Subject: [Numpy-discussion] ANN: line_profiler 1.0b2 released
Message-ID: <3d375d730902191321m396f7ddeu8b84d9fdc562c952@mail.gmail.com>

http://pypi.python.org/pypi/line_profiler/
http://packages.python.org/line_profiler/

This release fixes the "negative timings" issue on Windows.

Future announcements will occur on python-announce. I just wanted to
make sure my earliest users here who ran into this bug are aware of
the fix.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From bsouthey at gmail.com  Thu Feb 19 16:24:24 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Thu, 19 Feb 2009 15:24:24 -0600
Subject: [Numpy-discussion] Update webpage for python requirements for
	Numpy/SciPy
Message-ID: <499DCE08.903@gmail.com>

Hi,
Could someone please update the website to clearly state that numpy 1.2 
requires Python 2.4 or later?
I know it is in the release notes but that assumes people read them :-)

It would be great to state this on the download and installation pages:
http://www.scipy.org/Download
http://www.scipy.org/Installing_SciPy

Also there should be a mention that nose is required at for testing.

A couple of dated material that I noticed are:

http://www.scipy.org/Installing_SciPy/BuildingGeneral
'To build SciPy, Python version 2.3 or newer is required.'

The FAQ page (http://www.scipy.org/FAQ)
NumPy/SciPy installation
'Prerequisities
NumPy requires the following software installed:
1. Python 2.3.x or 2.4.x or 2.5.x '

Thanks
Bruce


From mattdm at mattdm.org  Thu Feb 19 16:26:19 2009
From: mattdm at mattdm.org (Matthew Miller)
Date: Thu, 19 Feb 2009 16:26:19 -0500
Subject: [Numpy-discussion] ANN: line_profiler 1.0b2 released
In-Reply-To: <3d375d730902191321m396f7ddeu8b84d9fdc562c952@mail.gmail.com>
References: <3d375d730902191321m396f7ddeu8b84d9fdc562c952@mail.gmail.com>
Message-ID: <20090219212619.GA10229@jadzia.bu.edu>

On Thu, Feb 19, 2009 at 03:21:26PM -0600, Robert Kern wrote:
> http://pypi.python.org/pypi/line_profiler/
> http://packages.python.org/line_profiler/
> This release fixes the "negative timings" issue on Windows.

Cool. I'm still interested in making Fedora packages for this once I get a
few spare cycles.

-- 
Matthew Miller           mattdm at mattdm.org          <http://mattdm.org/>


From millman at berkeley.edu  Thu Feb 19 16:52:54 2009
From: millman at berkeley.edu (Jarrod Millman)
Date: Thu, 19 Feb 2009 13:52:54 -0800
Subject: [Numpy-discussion] Update webpage for python requirements for
	Numpy/SciPy
In-Reply-To: <499DCE08.903@gmail.com>
References: <499DCE08.903@gmail.com>
Message-ID: <c7009a550902191352u53345cf4pfebd3e39194b6ec3@mail.gmail.com>

On Thu, Feb 19, 2009 at 1:24 PM, Bruce Southey <bsouthey at gmail.com> wrote:
> Hi,
> Could someone please update the website to clearly state that numpy 1.2
> requires Python 2.4 or later?
> I know it is in the release notes but that assumes people read them :-)
>
> It would be great to state this on the download and installation pages:
> http://www.scipy.org/Download
> http://www.scipy.org/Installing_SciPy
>
> Also there should be a mention that nose is required at for testing.
>
> A couple of dated material that I noticed are:
>
> http://www.scipy.org/Installing_SciPy/BuildingGeneral
> 'To build SciPy, Python version 2.3 or newer is required.'
>
> The FAQ page (http://www.scipy.org/FAQ)
> NumPy/SciPy installation
> 'Prerequisities
> NumPy requires the following software installed:
> 1. Python 2.3.x or 2.4.x or 2.5.x '

It is extremely difficult to keep track of the numerous pages that
explain what the requirements are and how to build and install
everything.  I would love it if someone would volunteer to add this
information to the user documentation for numpy:
  http://docs.scipy.org/doc/numpy/user/
and scipy:
  http://docs.scipy.org/doc/scipy/reference/    (maybe in the tutorial)?

Then we can just link to the relevant authoritative site on as many
pages as we want.  Since the docs are checked into the source code, it
will be much easier--and more likely--that developers will update this
information will they are working on the code.

Unfortunately I don't have the time to do this myself, but I would be
extremely appreciative if some else was able to take the time to do
this.  It would be very useful.

You could start here:
  http://projects.scipy.org/scipy/scipy/browser/trunk/INSTALL.txt
  http://projects.scipy.org/scipy/numpy/browser/trunk/INSTALL.txt

And the integrate the various information from the other sites listed
above.  As soon as the information is moved from the wiki to the
official docs, you could replace the source pages with links to the
official, authoritative site.

I am thinking the final version would look something like this:
  http://neuroimaging.scipy.org/site/doc/manual/html/users/installation.html
  http://neuroimaging.scipy.org/site/doc/manual/html/devel/install/index.html

Of course, the information might not always be correct; but if there
is one authoritative site where everyone can point out things that are
broken or don't work, it hopefully won't take too long to get
everything in good shape.  So if you are up to the challenge, I would
recommend just merging everything to start and not worrying to much
about testing and verifying that everything works correctly before
starting.

Thanks,
Jarrod


From ellisonbg.net at gmail.com  Thu Feb 19 17:21:40 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 19 Feb 2009 14:21:40 -0800
Subject: [Numpy-discussion] ANN: line_profiler 1.0b2 released
In-Reply-To: <3d375d730902191321m396f7ddeu8b84d9fdc562c952@mail.gmail.com>
References: <3d375d730902191321m396f7ddeu8b84d9fdc562c952@mail.gmail.com>
Message-ID: <6ce0ac130902191421r4d93a762o35863ae254e62b9d@mail.gmail.com>

Robert,

Thanks for the announcement.  I have recently started to use
line_profiler to profile Twisted using servers and clients.  I quickly
found that line_profiler needed some modifications to properly handle
timing functions that return Deferred's.  I have written some small
extensions to line_profiler (basically a subclass of LineProfiler
called DeferredLineProfiler) to handle these things.  I need to do
some further testing before I contribute this, but how would you like
to handle contributions (assuming you are open to contributions :)?

For me, it is probably easiest to simply throw a branch up on github.

Cheers and thanks for a fantastic package!

Brian

On Thu, Feb 19, 2009 at 1:21 PM, Robert Kern <robert.kern at gmail.com> wrote:
> http://pypi.python.org/pypi/line_profiler/
> http://packages.python.org/line_profiler/
>
> This release fixes the "negative timings" issue on Windows.
>
> Future announcements will occur on python-announce. I just wanted to
> make sure my earliest users here who ran into this bug are aware of
> the fix.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From robert.kern at gmail.com  Thu Feb 19 17:33:21 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 19 Feb 2009 16:33:21 -0600
Subject: [Numpy-discussion] ANN: line_profiler 1.0b2 released
In-Reply-To: <6ce0ac130902191421r4d93a762o35863ae254e62b9d@mail.gmail.com>
References: <3d375d730902191321m396f7ddeu8b84d9fdc562c952@mail.gmail.com>
	<6ce0ac130902191421r4d93a762o35863ae254e62b9d@mail.gmail.com>
Message-ID: <3d375d730902191433t4eeb6effg5d949ee7eadab3a9@mail.gmail.com>

On Thu, Feb 19, 2009 at 16:21, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Robert,
>
> Thanks for the announcement.  I have recently started to use
> line_profiler to profile Twisted using servers and clients.  I quickly
> found that line_profiler needed some modifications to properly handle
> timing functions that return Deferred's.  I have written some small
> extensions to line_profiler (basically a subclass of LineProfiler
> called DeferredLineProfiler) to handle these things.  I need to do
> some further testing before I contribute this, but how would you like
> to handle contributions (assuming you are open to contributions :)?
>
> For me, it is probably easiest to simply throw a branch up on github.

github is fine, but bitbucket would be a better impedance match.
Either way, I'm looking forward to it. Thanks!

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ellisonbg.net at gmail.com  Thu Feb 19 17:35:24 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 19 Feb 2009 14:35:24 -0800
Subject: [Numpy-discussion] ANN: line_profiler 1.0b2 released
In-Reply-To: <3d375d730902191433t4eeb6effg5d949ee7eadab3a9@mail.gmail.com>
References: <3d375d730902191321m396f7ddeu8b84d9fdc562c952@mail.gmail.com>
	<6ce0ac130902191421r4d93a762o35863ae254e62b9d@mail.gmail.com>
	<3d375d730902191433t4eeb6effg5d949ee7eadab3a9@mail.gmail.com>
Message-ID: <6ce0ac130902191435h1b8572cbw87fcbf39040a4f27@mail.gmail.com>

> github is fine, but bitbucket would be a better impedance match.
> Either way, I'm looking forward to it. Thanks!

I will give bitbucket a shot and let you know when I have something
for you to look at.

Cheers,

Brian

> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From frank at gis4weather.com  Thu Feb 19 18:03:58 2009
From: frank at gis4weather.com (Frank Peacock)
Date: Thu, 19 Feb 2009 23:03:58 -0000
Subject: [Numpy-discussion] Initialize numpy array with other numpy arrays
Message-ID: <00a901c992e6$59103ea0$0b30bbe0$@com>

Hello

 

I would like to know whether I can do the following in some other way as
this fails with setting an array with a sequence on each of the colour
arrays:

 

h,w=720,410

img = ones((h,w,3), uint8)*255

img[ngridn,ngride]=(ncolour[0],ncolour[1],ncolour[2])

pilImage = Image.fromarray(img, 'RGB')

 

where ngridn,ngride,ncolour[m] are all 1-D with the same dimension
(effectively ngridn and ngride values map within the bounds of the image)

 

The following works fine:

h,w=720,410

img = ones((h,w,3), uint8)*255

img[ngridn,ngride]=(255,0,0)

pilImage = Image.fromarray(img, 'RGB')

 

I would prefer not to use indices to solve the problem like the following:
(as it is a lot slower)

 

h,w=720,410

img = ones((h,w,3), uint8)*255

for n in range(len(gride)):

img[ngridn[n],ngride[n]]=(ncolour[0][n],ncolour[1][n],ncolour[n][2])

pilImage = Image.fromarray(img, 'RGB')

 

It is possible to not use indices and use numpy functions instead.

 

Thanks


Frank

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090219/5f1aa510/attachment.html>

From robert.kern at gmail.com  Thu Feb 19 19:24:54 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 19 Feb 2009 18:24:54 -0600
Subject: [Numpy-discussion] Initialize numpy array with other numpy
	arrays
In-Reply-To: <00a901c992e6$59103ea0$0b30bbe0$@com>
References: <00a901c992e6$59103ea0$0b30bbe0$@com>
Message-ID: <3d375d730902191624x3f7f5d41k3256b1e1f1d8e015@mail.gmail.com>

On Thu, Feb 19, 2009 at 17:03, Frank Peacock <frank at gis4weather.com> wrote:
> Hello
>
>
>
> I would like to know whether I can do the following in some other way as
> this fails with setting an array with a sequence on each of the colour
> arrays:
>
>
>
> h,w=720,410
>
> img = ones((h,w,3), uint8)*255
>
> img[ngridn,ngride]=(ncolour[0],ncolour[1],ncolour[2])
>
> pilImage = Image.fromarray(img, 'RGB')

You probably want to do

for i in range(3):
    img[ngridn,ngride,i] = ncolour[i]

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From cournape at gmail.com  Thu Feb 19 23:37:21 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 20 Feb 2009 13:37:21 +0900
Subject: [Numpy-discussion] Trac is overloaded
Message-ID: <5b8d13220902192037h6107cebcv8986fd4422464721@mail.gmail.com>

Hi,

Just to mention that It looks like numpy trac is overloaded (error 500
and locked database)

cheers,

David


From scott.sinclair.za at gmail.com  Fri Feb 20 01:34:06 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Fri, 20 Feb 2009 08:34:06 +0200
Subject: [Numpy-discussion] Update webpage for python requirements for
	Numpy/SciPy
In-Reply-To: <c7009a550902191352u53345cf4pfebd3e39194b6ec3@mail.gmail.com>
References: <499DCE08.903@gmail.com>
	<c7009a550902191352u53345cf4pfebd3e39194b6ec3@mail.gmail.com>
Message-ID: <6a17e9ee0902192234j4fbfdec1r8887fa321fd5d782@mail.gmail.com>

> 2009/2/19 Jarrod Millman <millman at berkeley.edu>:
> On Thu, Feb 19, 2009 at 1:24 PM, Bruce Southey <bsouthey at gmail.com> wrote:
>> Hi,
>> Could someone please update the website to clearly state that numpy 1.2
>> requires Python 2.4 or later?
>> I know it is in the release notes but that assumes people read them :-)
>>
>
> It is extremely difficult to keep track of the numerous pages that
> explain what the requirements are and how to build and install
> everything.  I would love it if someone would volunteer to add this
> information to the user documentation for numpy:
>  http://docs.scipy.org/doc/numpy/user/
> and scipy:
>  http://docs.scipy.org/doc/scipy/reference/    (maybe in the tutorial)?

If someone can add a stub to the docs in SVN (patch attached for
Numpy), I'm prepared to work on this. I can't see how to add pages in
the doc-wiki...

Cheers,
Scott
-------------- next part --------------
A non-text attachment was scrubbed...
Name: numpy_install.patch
Type: text/x-diff
Size: 685 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090220/2237ee60/attachment.patch>

From charlesr.harris at gmail.com  Fri Feb 20 02:09:08 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 20 Feb 2009 00:09:08 -0700
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
Message-ID: <e06186140902192309n64a08d66g17a9089c602508ae@mail.gmail.com>

On Thu, Feb 19, 2009 at 8:16 AM, David Cournapeau <cournape at gmail.com>wrote:

> On Sun, Feb 15, 2009 at 5:09 PM, Robert Kern <robert.kern at gmail.com>
> wrote:
> > On Sun, Feb 15, 2009 at 01:48, David Cournapeau
> > <david at ar.media.kyoto-u.ac.jp> wrote:
> >
> >> Would people be against this ?
> >
> > Not I.
>
> Ok, I have started this in the coremath branch - it solves the warning
> issues we got since the merge of formatting stuff. I tested it on
> Linux, windows (both mingw and VS - still need to test on Win64), so I
> think it is good to go in the trunk. The functions exported are in a
> separate header:
>
>
> http://projects.scipy.org/scipy/numpy/browser/branches/coremath/numpy/core/include/numpy/npy_math.h
>
> I did not put the complex functions there (nc_), but maybe they should
> be, I am not sure.
>

I'm thinking yes, we might want them in scalarmathmodule.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090220/aa9072d3/attachment.html>

From frank at gis4weather.com  Fri Feb 20 02:33:52 2009
From: frank at gis4weather.com (Frank Peacock)
Date: Fri, 20 Feb 2009 07:33:52 -0000
Subject: [Numpy-discussion] Initialize numpy array with other numpy	arrays
Message-ID: <00d901c9932d$9460c0f0$bd2242d0$@com>

Many thanks Robert.

 

Frank

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090220/7737b6eb/attachment.html>

From faltet at pytables.org  Fri Feb 20 02:44:11 2009
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 20 Feb 2009 08:44:11 +0100
Subject: [Numpy-discussion] Trac is overloaded
In-Reply-To: <5b8d13220902192037h6107cebcv8986fd4422464721@mail.gmail.com>
References: <5b8d13220902192037h6107cebcv8986fd4422464721@mail.gmail.com>
Message-ID: <200902200844.12451.faltet@pytables.org>

A Friday 20 February 2009, David Cournapeau escrigu?:
> Hi,
>
> Just to mention that It looks like numpy trac is overloaded (error
> 500 and locked database)

FWIW, in my experience Trac hogs (possibly leaks) vast amounts of 
memory, and that makes it extremely slow and prone to fail.  My fix has 
been to restart the Trac process everyday and fortunately everything 
goes smoothly since then.

Cheers,

-- 
Francesc Alted


From pav at iki.fi  Fri Feb 20 03:26:19 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 20 Feb 2009 08:26:19 +0000 (UTC)
Subject: [Numpy-discussion] Core math library in numpy
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org>
	<5b8d13220902190805s2d12ac96va40b6062959fc53@mail.gmail.com>
Message-ID: <gnlpfb$hih$1@ger.gmane.org>

Fri, 20 Feb 2009 01:05:03 +0900, David Cournapeau wrote:
[clip]
>> I think they should be. Then we could easily use C99 complex math
>> functions on plaforms on which they are available (and so get the
>> "correct" corner-case semantics for free on these platforms).
> 
> maybe we could change the names, then ? nc is not very clear IMHO (and
> since they were static up to now, we are free to change them I believe).

I think it would make sense to change them to follow C99 function names, 
with a npy_ prefix.

-- 
Pauli Virtanen



From pav at iki.fi  Fri Feb 20 03:27:50 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 20 Feb 2009 08:27:50 +0000 (UTC)
Subject: [Numpy-discussion] Update webpage for python requirements
	for	Numpy/SciPy
References: <499DCE08.903@gmail.com>
	<c7009a550902191352u53345cf4pfebd3e39194b6ec3@mail.gmail.com>
	<6a17e9ee0902192234j4fbfdec1r8887fa321fd5d782@mail.gmail.com>
Message-ID: <gnlpi6$hih$2@ger.gmane.org>

Fri, 20 Feb 2009 08:34:06 +0200, Scott Sinclair wrote:
[clip]
> If someone can add a stub to the docs in SVN (patch attached for Numpy),
> I'm prepared to work on this. I can't see how to add pages in the
> doc-wiki...

You can do also this in the doc-wiki, just use the "New item" form:

	http://docs.scipy.org/numpy/docs/numpy-docs/user/

No need to add stubs to SVN.

-- 
Pauli Virtanen



From silva at lma.cnrs-mrs.fr  Fri Feb 20 03:22:08 2009
From: silva at lma.cnrs-mrs.fr (Fabrice Silva)
Date: Fri, 20 Feb 2009 09:22:08 +0100
Subject: [Numpy-discussion] Initialize numpy array with other
	numpy	arrays
In-Reply-To: <3d375d730902191624x3f7f5d41k3256b1e1f1d8e015@mail.gmail.com>
References: <00a901c992e6$59103ea0$0b30bbe0$@com>
	<3d375d730902191624x3f7f5d41k3256b1e1f1d8e015@mail.gmail.com>
Message-ID: <1235118128.2889.1.camel@localhost>

> On Thu, Feb 19, 2009 at 17:03, Frank Peacock <frank at gis4weather.com> wrote:
> > img[ngridn,ngride]=(ncolour[0],ncolour[1],ncolour[2])

Le jeudi 19 f?vrier 2009 ? 18:24 -0600, Robert Kern a ?crit :
> for i in range(3):
>     img[ngridn,ngride,i] = ncolour[i]

Is it not possible to simply use
        img[ngridn, ngride, :] = ncolour[:]

-- 
Fabricio




From scott.sinclair.za at gmail.com  Fri Feb 20 04:24:36 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Fri, 20 Feb 2009 11:24:36 +0200
Subject: [Numpy-discussion] Update webpage for python requirements for
	Numpy/SciPy
In-Reply-To: <gnlpi6$hih$2@ger.gmane.org>
References: <499DCE08.903@gmail.com>
	<c7009a550902191352u53345cf4pfebd3e39194b6ec3@mail.gmail.com>
	<6a17e9ee0902192234j4fbfdec1r8887fa321fd5d782@mail.gmail.com>
	<gnlpi6$hih$2@ger.gmane.org>
Message-ID: <6a17e9ee0902200124p297e63a5u2eda3d9c853be7f2@mail.gmail.com>

> 2009/2/20 Pauli Virtanen <pav at iki.fi>:
> Fri, 20 Feb 2009 08:34:06 +0200, Scott Sinclair wrote:
> [clip]
>> If someone can add a stub to the docs in SVN (patch attached for Numpy),
>> I'm prepared to work on this. I can't see how to add pages in the
>> doc-wiki...
>
> You can do also this in the doc-wiki, just use the "New item" form:
>
>        http://docs.scipy.org/numpy/docs/numpy-docs/user/
>
> No need to add stubs to SVN.

That's useful!

It's a bit hard to navigate to unless you're already aware of the feature.

Cheers,
Scott


From dwf at cs.toronto.edu  Fri Feb 20 06:18:23 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Fri, 20 Feb 2009 06:18:23 -0500
Subject: [Numpy-discussion] inplace dot products
In-Reply-To: <db0bec8f0902180106n5483a79eg2409e0986b5b0bd4@mail.gmail.com>
References: <db0bec8f0902180106n5483a79eg2409e0986b5b0bd4@mail.gmail.com>
Message-ID: <B70159E4-A555-456D-9B23-18C46F82AF74@cs.toronto.edu>

Hi Olivier,

There was this idea posted on the Scipy-user list a while back:

	http://projects.scipy.org/pipermail/scipy-user/2008-August/017954.html

but it doesn't look like he got anywhere with it, or even got a  
response.

I just tried it and I observe the same behaviour. A quick look at the  
SciPy sources tells me there is something fishy.

subroutine  
< 
tchar=s,d,c,z>gemm(m,n,k,alpha,a,b,beta,c,trans_a,trans_b,lda,ka,ldb,kb)
   ! c = gemm(alpha,a,b,beta=0,c=0,trans_a=0,trans_b=0,overwrite_c=0)
   ! Calculate C <- alpha * op(A) * op(B) + beta * C

I don't read Fortran very well, but it seems to me as though the  
Fortran prototype doesn't match the python prototype.

I'll poke around a little more, but in summary: there's no numpy- 
sanctioned way to specify an output array for a dot(), AFAIK. This is  
a bit of an annoyance, I agree, though I seem to remember Robert Kern  
offering a fairly compelling argument why it's hard. I just don't know  
what that argument is :)

David


On 18-Feb-09, at 4:06 AM, Olivier Grisel wrote:

> Hi numpist people,
>
> I discovered the ufunc and there ability to compute the results on
> preallocated arrays:
>
>>>> a = arange(10, dtype=float32)
>>>> b = arange(10, dtype=float32) + 1
>>>> c = add(a, b, a)
>>>> c is a
>    True
>>>> a
>    array([  1.,   3.,   5.,   7.,   9.,  11.,  13.,  15.,  17.,
> 19.], dtype=float32)
>
> My questions is : is there a way to have an equivalent for the dot
> product operation:
> I want atlas to build my dot products without allocating a temporary
> array and reuse
> a preallocated array of results. Suppose I have:
>
>>>> results = array((10, 3), dtype=float32)
>>>> W = arange(6, dtype=float32).reshape((2, 3))
>>>> x = arange(20, dtype=float32).reshape((10, 2))
>
> What I want is the equivalent of the following without the
> intermediate call to malloc:
>
>>>> results[:] = dot(x, W)
>
> Any idea? I tried to introspect the various docstring of numpy core
> modules but I
> could not get any lead.
>
> -- 
> Olivier
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From olivier.grisel at ensta.org  Fri Feb 20 06:41:12 2009
From: olivier.grisel at ensta.org (Olivier Grisel)
Date: Fri, 20 Feb 2009 12:41:12 +0100
Subject: [Numpy-discussion] inplace dot products
In-Reply-To: <B70159E4-A555-456D-9B23-18C46F82AF74@cs.toronto.edu>
References: <db0bec8f0902180106n5483a79eg2409e0986b5b0bd4@mail.gmail.com>
	<B70159E4-A555-456D-9B23-18C46F82AF74@cs.toronto.edu>
Message-ID: <db0bec8f0902200341n643830od566c8ec5dbe3388@mail.gmail.com>

2009/2/20 David Warde-Farley <dwf at cs.toronto.edu>:
> Hi Olivier,
>
> There was this idea posted on the Scipy-user list a while back:
>
> ? ? ? ?http://projects.scipy.org/pipermail/scipy-user/2008-August/017954.html
>
> but it doesn't look like he got anywhere with it, or even got a
> response.
>
> I just tried it and I observe the same behaviour. A quick look at the
> SciPy sources tells me there is something fishy.
>
> subroutine
> <
> tchar=s,d,c,z>gemm(m,n,k,alpha,a,b,beta,c,trans_a,trans_b,lda,ka,ldb,kb)
> ? ! c = gemm(alpha,a,b,beta=0,c=0,trans_a=0,trans_b=0,overwrite_c=0)
> ? ! Calculate C <- alpha * op(A) * op(B) + beta * C
>
> I don't read Fortran very well, but it seems to me as though the
> Fortran prototype doesn't match the python prototype.
>
> I'll poke around a little more, but in summary: there's no numpy-
> sanctioned way to specify an output array for a dot(), AFAIK. This is
> a bit of an annoyance, I agree, though I seem to remember Robert Kern
> offering a fairly compelling argument why it's hard. I just don't know
> what that argument is :)
>

Alright, thanks for the reply.

Is there a canonical way /sample code to gain low level access to blas / lapack
atlas routines using ctypes from numpy / scipy code?

I don't mind fixing the dimensions and the ndtype of my array if it can
decrease the memory overhead.

BTW, Robert if your insight on this topic would be very much appreciated.

-- 
Olivier


From david at ar.media.kyoto-u.ac.jp  Fri Feb 20 06:39:49 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 20 Feb 2009 20:39:49 +0900
Subject: [Numpy-discussion] inplace dot products
In-Reply-To: <db0bec8f0902200341n643830od566c8ec5dbe3388@mail.gmail.com>
References: <db0bec8f0902180106n5483a79eg2409e0986b5b0bd4@mail.gmail.com>	<B70159E4-A555-456D-9B23-18C46F82AF74@cs.toronto.edu>
	<db0bec8f0902200341n643830od566c8ec5dbe3388@mail.gmail.com>
Message-ID: <499E9685.7050107@ar.media.kyoto-u.ac.jp>

Olivier Grisel wrote:
> 2009/2/20 David Warde-Farley <dwf at cs.toronto.edu>:
>   
>> Hi Olivier,
>>
>> There was this idea posted on the Scipy-user list a while back:
>>
>>        http://projects.scipy.org/pipermail/scipy-user/2008-August/017954.html
>>
>> but it doesn't look like he got anywhere with it, or even got a
>> response.
>>
>> I just tried it and I observe the same behaviour. A quick look at the
>> SciPy sources tells me there is something fishy.
>>
>> subroutine
>> <
>> tchar=s,d,c,z>gemm(m,n,k,alpha,a,b,beta,c,trans_a,trans_b,lda,ka,ldb,kb)
>>   ! c = gemm(alpha,a,b,beta=0,c=0,trans_a=0,trans_b=0,overwrite_c=0)
>>   ! Calculate C <- alpha * op(A) * op(B) + beta * C
>>
>> I don't read Fortran very well, but it seems to me as though the
>> Fortran prototype doesn't match the python prototype.
>>
>> I'll poke around a little more, but in summary: there's no numpy-
>> sanctioned way to specify an output array for a dot(), AFAIK. This is
>> a bit of an annoyance, I agree, though I seem to remember Robert Kern
>> offering a fairly compelling argument why it's hard. I just don't know
>> what that argument is :)
>>
>>     
>
> Alright, thanks for the reply.
>
> Is there a canonical way /sample code to gain low level access to blas / lapack
> atlas routines using ctypes from numpy / scipy code?
>   

You can just use ctypes to access ATLAS, as you would do for any
library. Or do you mean something else ?

cheers,

David


From gregor.thalhammer at gmail.com  Fri Feb 20 07:15:18 2009
From: gregor.thalhammer at gmail.com (Gregor Thalhammer)
Date: Fri, 20 Feb 2009 13:15:18 +0100
Subject: [Numpy-discussion] Build with MKL on linux
In-Reply-To: <loom.20090219T191632-54@post.gmane.org>
References: <loom.20090219T191632-54@post.gmane.org>
Message-ID: <499E9ED6.3090504@googlemail.com>

Neal Becker schrieb:
> Trying to build numpy-1.2.1 with intel mkl 10.1.1.019 on linux F10 x86_64.
>
> echo $LD_LIBRARY_PATH
> /opt/intel/mkl/10.1.1.019/lib/em64t
>
>
> strace -e trace=file python -c 'import numpy; numpy.test()' 2>stuff
> Running unit tests for numpy
> NumPy version 1.2.1
> NumPy is installed in /usr/lib64/python2.5/site-packages/numpy
> Python version 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) [GCC 4.3.2 20080917
> (Red Hat 4.3.2-4)]
> nose version 0.10.3
>
> MKL FATAL ERROR: Cannot load neither libmkl_mc.so nor libmkl_def.so
>
> strace shows:
> ...
> open("/usr/lib64/python2.5/site-packages/numpy/core/tests/test_blasdot.py",
> O_RDONLY) = 3
> open("/usr/lib64/python2.5/site-packages/numpy/core/tests/test_blasdot.pyc",
> O_RDONLY) = 5
> open("/opt/intel/mkl/10.1.1.019/lib/em64t/libmkl_mc.so", O_RDONLY) = 3
> open("/opt/intel/mkl/10.1.1.019/lib/em64t/libmkl_mc.so", O_RDONLY) = 3
> open("/opt/intel/mkl/10.1.1.019/lib/em64t/libmkl_def.so", O_RDONLY) = 3
> open("/opt/intel/mkl/10.1.1.019/lib/em64t/libmkl_def.so", O_RDONLY) = 3
>
> The files are there and found, what could be wrong?
>
>   
Try MKL version 10.0.2.018. Some more information about this problem you 
find on 
http://software.intel.com/en-us/forums/intel-math-kernel-library/topic/60460/

Gregor


From dwf at cs.toronto.edu  Fri Feb 20 07:25:39 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Fri, 20 Feb 2009 07:25:39 -0500
Subject: [Numpy-discussion] inplace dot products
In-Reply-To: <499E9685.7050107@ar.media.kyoto-u.ac.jp>
References: <db0bec8f0902180106n5483a79eg2409e0986b5b0bd4@mail.gmail.com>	<B70159E4-A555-456D-9B23-18C46F82AF74@cs.toronto.edu>
	<db0bec8f0902200341n643830od566c8ec5dbe3388@mail.gmail.com>
	<499E9685.7050107@ar.media.kyoto-u.ac.jp>
Message-ID: <7BF4A860-B3BB-4473-8219-47DCBEED5ABE@cs.toronto.edu>


On 20-Feb-09, at 6:39 AM, David Cournapeau wrote:

> You can just use ctypes to access ATLAS, as you would do for any
> library. Or do you mean something else ?

Say, David... :)

Do you have any idea why the pyf wrapper for fblas3 completely ignores  
the overwrite_c argument? Fiddling around I've found other blas/lapack  
functions where the same arg is offered but the choice actually does  
something.

Regards,

(Other) David


From robert.kern at gmail.com  Fri Feb 20 10:39:53 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Feb 2009 09:39:53 -0600
Subject: [Numpy-discussion] inplace dot products
In-Reply-To: <7BF4A860-B3BB-4473-8219-47DCBEED5ABE@cs.toronto.edu>
References: <db0bec8f0902180106n5483a79eg2409e0986b5b0bd4@mail.gmail.com>
	<B70159E4-A555-456D-9B23-18C46F82AF74@cs.toronto.edu>
	<db0bec8f0902200341n643830od566c8ec5dbe3388@mail.gmail.com>
	<499E9685.7050107@ar.media.kyoto-u.ac.jp>
	<7BF4A860-B3BB-4473-8219-47DCBEED5ABE@cs.toronto.edu>
Message-ID: <3d375d730902200739h418d7bd2h7b8dcc216fa9e30a@mail.gmail.com>

On Fri, Feb 20, 2009 at 06:25, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>
> On 20-Feb-09, at 6:39 AM, David Cournapeau wrote:
>
>> You can just use ctypes to access ATLAS, as you would do for any
>> library. Or do you mean something else ?
>
> Say, David... :)
>
> Do you have any idea why the pyf wrapper for fblas3 completely ignores
> the overwrite_c argument?

I believe the "copy" is the culprit.

  double precision dimension(m,n),intent(in,out,copy),depend(m,n),optional :: c

> Fiddling around I've found other blas/lapack
> functions where the same arg is offered but the choice actually does
> something.

Examples?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Fri Feb 20 10:45:38 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Feb 2009 09:45:38 -0600
Subject: [Numpy-discussion] Initialize numpy array with other numpy
	arrays
In-Reply-To: <1235118128.2889.1.camel@localhost>
References: <00a901c992e6$59103ea0$0b30bbe0$@com>
	<3d375d730902191624x3f7f5d41k3256b1e1f1d8e015@mail.gmail.com>
	<1235118128.2889.1.camel@localhost>
Message-ID: <3d375d730902200745r1b080b00p684e573ca3522b0a@mail.gmail.com>

On Fri, Feb 20, 2009 at 02:22, Fabrice Silva <silva at lma.cnrs-mrs.fr> wrote:
>> On Thu, Feb 19, 2009 at 17:03, Frank Peacock <frank at gis4weather.com> wrote:
>> > img[ngridn,ngride]=(ncolour[0],ncolour[1],ncolour[2])
>
> Le jeudi 19 f?vrier 2009 ? 18:24 -0600, Robert Kern a ?crit :
>> for i in range(3):
>>     img[ngridn,ngride,i] = ncolour[i]
>
> Is it not possible to simply use
>        img[ngridn, ngride, :] = ncolour[:]

No. This would, though:

  img[ngridn, ngride, :] = ncolour.T

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Fri Feb 20 10:52:03 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Feb 2009 09:52:03 -0600
Subject: [Numpy-discussion] inplace dot products
In-Reply-To: <B70159E4-A555-456D-9B23-18C46F82AF74@cs.toronto.edu>
References: <db0bec8f0902180106n5483a79eg2409e0986b5b0bd4@mail.gmail.com>
	<B70159E4-A555-456D-9B23-18C46F82AF74@cs.toronto.edu>
Message-ID: <3d375d730902200752k6b71efc9gb1e0ac1d260d65d@mail.gmail.com>

On Fri, Feb 20, 2009 at 05:18, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> Hi Olivier,
>
> There was this idea posted on the Scipy-user list a while back:
>
>        http://projects.scipy.org/pipermail/scipy-user/2008-August/017954.html
>
> but it doesn't look like he got anywhere with it, or even got a
> response.
>
> I just tried it and I observe the same behaviour. A quick look at the
> SciPy sources tells me there is something fishy.
>
> subroutine
> <
> tchar=s,d,c,z>gemm(m,n,k,alpha,a,b,beta,c,trans_a,trans_b,lda,ka,ldb,kb)
>   ! c = gemm(alpha,a,b,beta=0,c=0,trans_a=0,trans_b=0,overwrite_c=0)
>   ! Calculate C <- alpha * op(A) * op(B) + beta * C
>
> I don't read Fortran very well, but it seems to me as though the
> Fortran prototype doesn't match the python prototype.

What do you mean? Based on the rest of the argument information in
that block, f2py creates the Python prototype. For example, all of the
m,n,k,lda,ka,ldb,kb dimensions are found from the input arrays
themselves, optional arguments are given defaults, etc.

> I'll poke around a little more, but in summary: there's no numpy-
> sanctioned way to specify an output array for a dot(), AFAIK. This is
> a bit of an annoyance, I agree, though I seem to remember Robert Kern
> offering a fairly compelling argument why it's hard. I just don't know

I believe I was only talking about why it would be hard to use a
higher-precision accumulator.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From davidh at ipac.caltech.edu  Fri Feb 20 14:20:03 2009
From: davidh at ipac.caltech.edu (David Henderson)
Date: Fri, 20 Feb 2009 11:20:03 -0800
Subject: [Numpy-discussion] inplace dot products (Robert Kern) (Re:
	Numpy-discussion Digest, Vol 29, Issue 69)
In-Reply-To: <mailman.9.1235152845.11388.numpy-discussion@scipy.org>
References: <mailman.9.1235152845.11388.numpy-discussion@scipy.org>
Message-ID: <0ADDF139-18FA-4B76-836D-2DC82CDBD90D@ipac.caltech.edu>

Hello all,

I've been toying with the idea of an extended precision accumulator  
for the dot product written in numpy/core/src/multiarraymodule.c

Once the modification is being performed, there is no reason not to  
allow the specification of an output array.

The functions that exist now:

The routine cumsum already exists with an accumulator of a specific  
type. The output location is an optional argument.
numpy.cumsum(a, axis=None, dtype=None, out=None)

Various forms of inner product (dot product) also exist:

numpy.tensordot(a, b, axes=2)
Returns the tensor dot product for (ndim >= 1) arrays along an axes.
The first element of the sequence determines the axis or axes
in `a` to sum over, and the second element in `axes` argument sequence
determines the axis or axes in `b` to sum over.

numpy.vdot(a,b)
Returns the dot product of a and b for scalars and vectors
of floating point and complex types.  The first argument, a, is  
conjugated.

numpy.innerproduct(a,b)
Returns the inner product of a and b for arrays of floating point types.
Like the generic NumPy equivalent the product sum is over
the last dimension of a and b.
NB: The first argument is not conjugated.

The proposed extensions:

numpy.tensordot(a, b, axes=2, dtype=None, out=None)
numpy.vdot(a,b, dtype=None, out=None)
numpy.innerproduct(a,b, dtype=None, out=None)   (found in numpy/core/ 
src/multiarraymodule.c)

Is this so difficult an extension to implement? Is there a better  
functional specification to be made?

Granted, these routines do not exist in blas. Therefore, they  
wouldn't be speedy, at least without blas extensions. But... the key  
routines to make the generic extension already exist in numpy without  
blas.

from numeric.py:
         When Numpy is built with an accelerated BLAS like ATLAS,  
these functions
         are replaced to make use of the faster implementations.  The  
faster
         implementations only affect float32, float64, complex64, and  
complex128
         arrays. Furthermore, the BLAS API only includes matrix-matrix,
         matrix-vector, and vector-vector products. Products of  
arrays with larger
         dimensionalities use the built in functions and are not  
accelerated.


Looking at numpy/core/src/multiarraymodule.c:

The existing routine that allows an accumulator specification and  
output array:

static PyObject *
PyArray_Sum(PyArrayObject *self, int axis, int rtype, PyArrayObject  
*out)

The routine that would be modified and its new function prototypes:

static PyObject *
PyArray_InnerProduct(PyObject *op1, PyObject *op2, int rtype,  
PyArrayObject *out)

Other C code that might be modified to support the new functionality  
is in arraytypes.inc.src. Routines here perform a dot product along  
one dimension for various argument dtypes and accumulator dtypes.

It looks like the accumulator type is encoded in the #out signature.  
As I read it, there is no change needed to this source as all the  
different accumulator types are present in the output signature.

/**begin repeat
#name=BYTE, UBYTE, SHORT, USHORT, INT, UINT, LONG, ULONG, LONGLONG,  
ULONGLONG, FLOAT, DOUBLE, LONGDOUBLE#
#type= byte, ubyte, short, ushort, int, uint, long, ulong, longlong,  
ulonglong, float, double, longdouble#
#out= long, ulong, long, ulong, long, ulong, long, ulong, longlong,  
ulonglong, float, double, longdouble#
*/
static void
@name at _dot(char *ip1, intp is1, char *ip2, intp is2, char *op, intp n,


So far, the changes dont seem to be too messy. Am I missing anything  
here?

David



On Feb 20, 2009, at 10:00 AM, numpy-discussion-request at scipy.org wrote:
>
> Message: 2
> Date: Fri, 20 Feb 2009 09:52:03 -0600
> From: Robert Kern <robert.kern at gmail.com>
> Subject: Re: [Numpy-discussion] inplace dot products
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Message-ID:
> 	<3d375d730902200752k6b71efc9gb1e0ac1d260d65d at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> On Fri, Feb 20, 2009 at 05:18, David Warde-Farley  
> <dwf at cs.toronto.edu> wrote:
>> Hi Olivier,
>>
>> There was this idea posted on the Scipy-user list a while back:
>>
>>        http://projects.scipy.org/pipermail/scipy-user/2008-August/ 
>> 017954.html
>>
>> but it doesn't look like he got anywhere with it, or even got a
>> response.
>>
>> I just tried it and I observe the same behaviour. A quick look at the
>> SciPy sources tells me there is something fishy.
>>
>> subroutine
>> <
>> tchar=s,d,c,z>gemm 
>> (m,n,k,alpha,a,b,beta,c,trans_a,trans_b,lda,ka,ldb,kb)
>>   ! c = gemm(alpha,a,b,beta=0,c=0,trans_a=0,trans_b=0,overwrite_c=0)
>>   ! Calculate C <- alpha * op(A) * op(B) + beta * C
>>
>> I don't read Fortran very well, but it seems to me as though the
>> Fortran prototype doesn't match the python prototype.
>
> What do you mean? Based on the rest of the argument information in
> that block, f2py creates the Python prototype. For example, all of the
> m,n,k,lda,ka,ldb,kb dimensions are found from the input arrays
> themselves, optional arguments are given defaults, etc.
>
>> I'll poke around a little more, but in summary: there's no numpy-
>> sanctioned way to specify an output array for a dot(), AFAIK. This is
>> a bit of an annoyance, I agree, though I seem to remember Robert Kern
>> offering a fairly compelling argument why it's hard. I just don't  
>> know
>
> I believe I was only talking about why it would be hard to use a
> higher-precision accumulator.
>
> -- 
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>   -- Umberto Eco
>
>
> ------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
> End of Numpy-discussion Digest, Vol 29, Issue 69
> ************************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090220/56d7e2b2/attachment.html>

From soemraws at xs4all.nl  Fri Feb 20 15:19:24 2009
From: soemraws at xs4all.nl (Sumant S.R. Oemrawsingh)
Date: Fri, 20 Feb 2009 12:19:24 -0800 (PST)
Subject: [Numpy-discussion] Problem looping over numpy array in C
Message-ID: <35905.128.111.8.119.1235161164.squirrel@webmail.xs4all.nl>

Hi guys,

I have a problem with looping over numpy arrays in C. I modify the array
in-place (and return None), but after modification, the array doesn't seem
to play nice any more.

Below, I have the C code for a function do_something (a stripped version
of my original function), which has as arguments a 1D numpy array (float
or complex) and either an int or a sequence of ints.

In python, I do the following using only the integer:

>>> a = array([0., 1., 2., 3., 2., 1., 0.])
>>> do_something(a,3)
>>> savetxt('bla',a)
>>>

Which works fine. However, when I do the same, but with a list of any
length larger than 0:

>>> a = array([0., 1., 2., 3., 2., 1., 0.])
>>> do_something(a,[3])
>>> savetxt('bla',a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/site-packages/numpy/core/numeric.py", line 767,
in savetxt
    X = asarray(X)
  File "/usr/lib/python2.5/site-packages/numpy/core/numeric.py", line 132,
in asarray
    return array(a, dtype, copy=False, order=order)
TypeError: an integer is required
>>> savetxt('bla',a)
>>>

For some reason, the first time it doesn't work; the asarray fails but
then succeeds on a second try. The resulting file is identical to when I
would have looped over the list in python, and called the do_something
function with each integer separately. For instance:

for i in [3,1,0]:
  do_something(a, i)

works fine as well. So apparently looping in C temporarily leaves the
array in a weird state but manages to automatically be restored after one
exception. I've checked that type(a), a.dtype, a.shape and a.ndim remain
the same before and after calling do_something with a sequence or with an
integer as second argument. That doesn't seem to be the problem.

The reason that I want do the loop in C is that I need some
precalculations, before being able to do the actual loop. But they might
not be time-consuming enough to warrant writing it in C, so I could just
do the loop in python and not have this problem any more. However, if the
loop in C manages to somehow (temporarily) corrupt the array, how can I be
sure that the single-shot call doesn't succeed by accident?

If anyone can help, suggest something to try or spot my mistake, I'd
appreciate it.

Thanks,
Sumant


static PyObject*
do_something(PyObject *self, PyObject *args, PyObject *kwargs)
{
  PyObject *input, *s, *item;
  PyArrayObject *array = NULL;
  npy_intp i,n;
  static char *kwlist[] = {"a", "index", NULL};

  if (!PyArg_ParseTupleAndKeywords(args, kwargs, "Oi", kwlist,
        &input, &i))
    goto sequence;

  if ((array = (PyArrayObject *) PyArray_FromAny(input, NULL, 1, 1,
          NPY_CARRAY | NPY_UPDATEIFCOPY, NULL)) == NULL)
    goto error;

  n = PyArray_DIM(array,0);

  if (PyArray_ISCOMPLEX(array))
    do_something_complex((complex*)(PyArray_DATA(array)), n, i);
  else if (PyArray_ISFLOAT(array))
    do_something_double((double*)(PyArray_DATA(array)), n, i);
  else
    goto error;

  Py_DECREF(array);
  Py_RETURN_NONE;

sequence:
  if (!PyArg_ParseTupleAndKeywords(args, kwargs, "OO", kwlist,
        &input, &s))
    goto error;

  if (PySequence_Check(s) == 0)
    goto error;

  if ((array = (PyArrayObject *) PyArray_FromAny(input, NULL, 1, 1,
          NPY_CARRAY | NPY_UPDATEIFCOPY, NULL)) == NULL)
    goto error;

  n = PyArray_DIM(array,0);

  if (PyArray_ISCOMPLEX(array))
  {
    complex* data = (complex*)PyArray_DATA(array);
    for (i = 0; i < PySequence_Size(s); i++)
    {
      item = PySequence_GetItem(s, i);
      do_something_complex(data, n, PyInt_AsSsize_t(item));
      Py_DECREF(item);
    }
  }
  else if (PyArray_ISFLOAT(array))
  {
    double* data = (double*)PyArray_DATA(array);
    for (i = 0; i < PySequence_Size(s); i++)
    {
      item = PySequence_GetItem(s, i);
      do_something_double(data, n, PyInt_AsSsize_t(item));
      Py_DECREF(item);
    }
  }
  else
    goto error;

  Py_DECREF(array);
  Py_RETURN_NONE;

error:
  Py_XDECREF(array);
  return NULL;
}



From dwf at cs.toronto.edu  Fri Feb 20 16:06:48 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Fri, 20 Feb 2009 16:06:48 -0500
Subject: [Numpy-discussion] inplace dot products
In-Reply-To: <3d375d730902200739h418d7bd2h7b8dcc216fa9e30a@mail.gmail.com>
References: <db0bec8f0902180106n5483a79eg2409e0986b5b0bd4@mail.gmail.com>
	<B70159E4-A555-456D-9B23-18C46F82AF74@cs.toronto.edu>
	<db0bec8f0902200341n643830od566c8ec5dbe3388@mail.gmail.com>
	<499E9685.7050107@ar.media.kyoto-u.ac.jp>
	<7BF4A860-B3BB-4473-8219-47DCBEED5ABE@cs.toronto.edu>
	<3d375d730902200739h418d7bd2h7b8dcc216fa9e30a@mail.gmail.com>
Message-ID: <2B07D98B-9CD6-4FA5-B5ED-9C1430E6F1AF@cs.toronto.edu>

On 20-Feb-09, at 10:39 AM, Robert Kern wrote:

>> Fiddling around I've found other blas/lapack
>> functions where the same arg is offered but the choice actually does
>> something.
>
> Examples?

scipy.lib.lapack.flapack.dpotri, for example. I'm not sure of the  
proper usage, but when I pass it an identity matrix, depending on  
whether overwrite_c is True or not, the memory pointed to by the  
variable gets overwritten.

David


From frank at gis4weather.com  Fri Feb 20 16:21:17 2009
From: frank at gis4weather.com (Frank Peacock)
Date: Fri, 20 Feb 2009 21:21:17 -0000
Subject: [Numpy-discussion] Multiple draw text or paste image using numpy
Message-ID: <014d01c993a1$2ae18410$80a48c30$@com>

Hello

 

I am familiar with the fromarray function in the pil library Image module to
plot pixels. Is it possible to do the same for drawtext for text or paste
for images without looping with lists?

 

Thanks


Frank 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090220/7a6269d6/attachment.html>

From robert.kern at gmail.com  Fri Feb 20 16:47:36 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Feb 2009 15:47:36 -0600
Subject: [Numpy-discussion] Multiple draw text or paste image using numpy
In-Reply-To: <014d01c993a1$2ae18410$80a48c30$@com>
References: <014d01c993a1$2ae18410$80a48c30$@com>
Message-ID: <3d375d730902201347o59778b86r18134cd8cd949e7c@mail.gmail.com>

On Fri, Feb 20, 2009 at 15:21, Frank Peacock <frank at gis4weather.com> wrote:
> Hello
>
> I am familiar with the fromarray function in the pil library Image module to
> plot pixels. Is it possible to do the same for drawtext for text or paste
> for images without looping with lists?

No.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From dwf at cs.toronto.edu  Fri Feb 20 19:41:25 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Fri, 20 Feb 2009 19:41:25 -0500
Subject: [Numpy-discussion] inplace dot products
In-Reply-To: <3d375d730902200739h418d7bd2h7b8dcc216fa9e30a@mail.gmail.com>
References: <db0bec8f0902180106n5483a79eg2409e0986b5b0bd4@mail.gmail.com>
	<B70159E4-A555-456D-9B23-18C46F82AF74@cs.toronto.edu>
	<db0bec8f0902200341n643830od566c8ec5dbe3388@mail.gmail.com>
	<499E9685.7050107@ar.media.kyoto-u.ac.jp>
	<7BF4A860-B3BB-4473-8219-47DCBEED5ABE@cs.toronto.edu>
	<3d375d730902200739h418d7bd2h7b8dcc216fa9e30a@mail.gmail.com>
Message-ID: <84370AAA-0E28-4498-94D7-2546406BCC57@cs.toronto.edu>

On 20-Feb-09, at 10:39 AM, Robert Kern wrote:

>> Fiddling around I've found other blas/lapack
>> functions where the same arg is offered but the choice actually does
>> something.
>
> Examples?

An even better example is scipy.linalg.fblas.dgemv, the matrix-vector  
equivalent of dgemm. overwrite_y behaves correctly there.

David


From fperez.net at gmail.com  Fri Feb 20 20:02:45 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 20 Feb 2009 17:02:45 -0800
Subject: [Numpy-discussion] [Nipy-devel] Sphinx custom extension mess,
	and patches
In-Reply-To: <20090216232106.GC24472@phare.normalesup.org>
References: <20090215140400.GH20403@phare.normalesup.org>
	<4999F3FD.4020806@python.org>
	<20090216232106.GC24472@phare.normalesup.org>
Message-ID: <db6b5ecc0902201702u3c843ae3ga7f24ad81ec7aeb0@mail.gmail.com>

On Mon, Feb 16, 2009 at 3:21 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:

> I am not blaming anyone, just pointing out a non ideal situation. It has
> already improved a lot with the matplotlib guys and the scipy guys
> merging some changes in extensions and publishing the extensions in an
> importable part of their source tree.

In keeping with the spirit of trying to get all of these extension
changes upstream so that we can all eventually stop carrying our own
copies, below is a tiny change I just made to the inheritance diagram
one.  This is needed to ensure that the figure is separated from any
surrounding text, since otherwise you get hideous off-screen diagrams
in the rendered PDF.

This has been committed to the nipy trunk already.

Similarly (for the pymvpa crowd), the api autogen code is now a
module, and it also contains a few small fixes, in particular
regarding chapter titles.  Feel free to grab and update your copy:

http://bazaar.launchpad.net/~nipy-developers/nipy/trunk/annotate/head%3A/tools/apigen.py

I've been told the gods of numpy/sphinx don't like auto-generated
docs, but I think there's a valid use case for these tools, so
hopefully in the future it will be possible to include them upstream
for us lesser mortals to use.  If not, I guess we'll just continue to
carry our copies around :)

Cheers,

f

# diff, inline because it's so trivial:

=== modified file 'doc/sphinxext/inheritance_diagram.py'
--- doc/sphinxext/inheritance_diagram.py	2009-01-30 02:00:57 +0000
+++ doc/sphinxext/inheritance_diagram.py	2009-02-20 21:11:38 +0000
@@ -370,7 +370,7 @@

     graph.run_dot(['-Tpdf', '-o%s' % pdf_path],
                   name, parts, graph_options={'size': '"6.0,6.0"'})
-    return '\\includegraphics{%s}' % pdf_path
+    return '\n\\includegraphics{%s}\n\n' % pdf_path

 def visit_inheritance_diagram(inner_func):
     """


From pinto at mit.edu  Fri Feb 20 23:46:09 2009
From: pinto at mit.edu (Nicolas Pinto)
Date: Fri, 20 Feb 2009 23:46:09 -0500
Subject: [Numpy-discussion] Efficient numpy slicing for a "sliding window
	approach".
Message-ID: <954ae5aa0902202046k5d89a67cn5d260c2844866dab@mail.gmail.com>

Dear all,

I'm trying to optimize the code below and I was wondering if there is an
efficient method that could reduce the numpy slicing overheard without going
with cython. Is there anyway I could use mgrid to get a matrix with all my
"windows" and then do a large matrix multiply instead?

Any idea?

Thanks in advance.


Best regards,

-- 
Nicolas Pinto
Ph.D. Candidate, Brain & Computer Sciences
Massachusetts Institute of Technology, USA
http://web.mit.edu/pinto


========================================

import numpy as np
from numpy import dot

arrh, arrw, arrd = 480,640,96
arr = np.random.randn(arrh, arrw, arrd).astype("float32")

stride = 16
winh, winw, wind = 128,64,96

limit = 100

clas_w = np.random.randn(8,4,96).astype("float32").ravel()

@profile
def func():
    nh, nw = arrh-winh+1, arrw-winw+1
    rngh = np.arange(nh)
    rngw = np.arange(nw)

    np.random.shuffle(rngh)
    np.random.shuffle(rngw)

    rngh = rngh[:limit]
    rngw = rngw[:limit]

    rngh = np.sort(rngh)
    rngw = np.sort(rngw)
    resps = np.empty((nh,nw), dtype="float32")
    for j in rngh:
        for i in rngw:
            win = arr[j:j+winh:stride, i:i+winw:stride].ravel()
            resp = dot(win, clas_w)
            resps[j,i] = resp

    resps = resps.ravel()

    # ...

func()


========================================

% python kernprof.py -l -v sliding_win.py
Wrote profile results to sliding_win.py.lprof
Timer unit: 1e-06 s

File: sliding_win.py
Function: func at line 14
Total time: 0.224315 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    14                                           @profile
    15                                           def func():
    16         1            7      7.0      0.0      nh, nw = arrh-winh+1,
arrw-winw+1
    17         1           14     14.0      0.0      rngh = np.arange(nh)
    18         1            4      4.0      0.0      rngw = np.arange(nw)
    19
    20         1           89     89.0      0.0      np.random.shuffle(rngh)
    21         1          124    124.0      0.1      np.random.shuffle(rngw)
    22
    23         1            3      3.0      0.0      rngh = rngh[:limit]
    24         1            2      2.0      0.0      rngw = rngw[:limit]
    25
    26         1           39     39.0      0.0      rngh = np.sort(rngh)
    27         1           19     19.0      0.0      rngw = np.sort(rngw)
    28         1           24     24.0      0.0      resps =
np.empty((nh,nw), dtype="float32")
    29       101          127      1.3      0.1      for j in rngh:
    30     10100        12636      1.3      5.6          for i in rngw:
    31     10000       118190     11.8     52.7              win =
arr[j:j+winh:stride, i:i+winw:stride].ravel()
    32     10000        73854      7.4     32.9              resp = dot(win,
clas_w)
    33     10000        19180      1.9      8.6              resps[j,i] =
resp
    34
    35         1            3      3.0      0.0      resps = resps.ravel()
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090220/19d6815b/attachment.html>

From cournape at gmail.com  Fri Feb 20 23:55:43 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 21 Feb 2009 13:55:43 +0900
Subject: [Numpy-discussion] Efficient numpy slicing for a "sliding
	window approach".
In-Reply-To: <954ae5aa0902202046k5d89a67cn5d260c2844866dab@mail.gmail.com>
References: <954ae5aa0902202046k5d89a67cn5d260c2844866dab@mail.gmail.com>
Message-ID: <5b8d13220902202055k59e06fd9hd2f553c5411a3efd@mail.gmail.com>

On Sat, Feb 21, 2009 at 1:46 PM, Nicolas Pinto <pinto at mit.edu> wrote:
> Dear all,
>
> I'm trying to optimize the code below and I was wondering if there is an
> efficient method that could reduce the numpy slicing overheard without going
> with cython. Is there anyway I could use mgrid to get a matrix with all my
> "windows" and then do a large matrix multiply instead?

If you only care about removing the two loops for the per-window
processing, Anne Archibald and Robert Kern wrote a very useful
function, segment_axis, which is like the matlab buffer function on
steroids, using numpy stride tricks (to avoid copies in many cases). I
think that would do everything you want, right ? I use it a lot in my
own code, it may be worth being included in numpy or scipy proper.

http://projects.scipy.org/scipy/scikits/browser/trunk/talkbox/scikits/talkbox/tools/segmentaxis.py

David


From pinto at mit.edu  Sat Feb 21 00:36:15 2009
From: pinto at mit.edu (Nicolas Pinto)
Date: Sat, 21 Feb 2009 00:36:15 -0500
Subject: [Numpy-discussion] Efficient numpy slicing for a "sliding
	window approach".
In-Reply-To: <5b8d13220902202055k59e06fd9hd2f553c5411a3efd@mail.gmail.com>
References: <954ae5aa0902202046k5d89a67cn5d260c2844866dab@mail.gmail.com>
	<5b8d13220902202055k59e06fd9hd2f553c5411a3efd@mail.gmail.com>
Message-ID: <954ae5aa0902202136x513770a7o12409270436b87bd@mail.gmail.com>

Thanks a lot for the pointer to segmentaxis. I'm trying to use it "as is"
and it seems that I need to a big reshape before the matrix multiplication.
Am I missing something ?


========================================

import numpy as np
from numpy import dot, transpose

arrh, arrw, arrd = 480,640,96
arr = np.random.randn(arrh, arrw, arrd).astype("float32")

stride = 16
winh, winw, wind = 128,64,96

limit = 100

clas_w = np.random.randn(8,4,96).astype("float32").ravel()

from segmentaxis import segment_axis

nh, nw = arrh-winh+1, arrw-winw+1

@profile
def func_loop(arr):

    resps = np.empty((nh,nw), dtype="float32")
    for j in xrange(nh):
         for i in xrange(nw):
            win = arr[j:j+winh:stride, i:i+winw:stride]
            win = win.ravel()
            resp = dot(win, clas_w)
            resps[j,i] = resp

    resps = resps.ravel()
    print resps.mean()


@profile
def func_segment(arr):

    arr = segment_axis(arr, winh, winh-1, axis=0)
    arr = arr[:, ::stride]

    arr = segment_axis(arr, winw, winw-1, axis=2)
    arr = arr[:, :, :, ::stride]

    arr = transpose(arr, [0, 2, 1, 3, 4])
    arr = arr.reshape(-1, clas_w.size)

    resps2 = dot(arr, clas_w)

    resps2 = resps2.ravel()
    print resps2.mean()

    # ...

func_loop(arr)

func_segment(arr)


========================================


% python kernprof.py -l -v sliding_win_all.py
w-0.0500485824034
segmentaxis.py:94: UserWarning: Problem with ndarray creation forces copy.
  warnings.warn("Problem with ndarray creation forces copy.")
-0.0500485824034
Wrote profile results to sliding_win_all.py.lprof
Timer unit: 1e-06 s

File: sliding_win_all.py
Function: func_loop at line 18
Total time: 3.9785 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    18                                           @profile
    19                                           def func_loop(arr):
    20
    21         1           28     28.0      0.0      resps =
np.empty((nh,nw), dtype="float32")
    22       354          308      0.9      0.0      for j in xrange(nh):
    23    204034       199753      1.0      5.0           for i in
xrange(nw):
    24    203681       691915      3.4     17.4              win =
arr[j:j+winh:stride, i:i+winw:stride]
    25    203681      1341174      6.6     33.7              win =
win.ravel()
    26    203681      1417998      7.0     35.6              resp = dot(win,
clas_w)
    27    203681       326520      1.6      8.2              resps[j,i] =
resp
    28
    29         1            2      2.0      0.0      resps = resps.ravel()
    30         1          805    805.0      0.0      print resps.mean()

File: sliding_win_all.py
Function: func_segment at line 33
Total time: 3.82026 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    33                                           @profile
    34                                           def func_segment(arr):
    35
    36         1           43     43.0      0.0      arr = segment_axis(arr,
winh, winh-1, axis=0)
    37         1            4      4.0      0.0      arr = arr[:, ::stride]
    38
    39         1       546505 546505.0     14.3      arr = segment_axis(arr,
winw, winw-1, axis=2)
    40         1           12     12.0      0.0      arr = arr[:, :, :,
::stride]
    41
    42         1           12     12.0      0.0      arr = transpose(arr,
[0, 2, 1, 3, 4])
    43         1      2435047 2435047.0     63.7      arr = arr.reshape(-1,
clas_w.size)
    44
    45         1       837700 837700.0     21.9      resps2 = dot(arr,
clas_w)
    46
    47         1           41     41.0      0.0      resps2 = resps2.ravel()
    48         1          892    892.0      0.0      print resps2.mean()


On Fri, Feb 20, 2009 at 11:55 PM, David Cournapeau <cournape at gmail.com>wrote:

> On Sat, Feb 21, 2009 at 1:46 PM, Nicolas Pinto <pinto at mit.edu> wrote:
> > Dear all,
> >
> > I'm trying to optimize the code below and I was wondering if there is an
> > efficient method that could reduce the numpy slicing overheard without
> going
> > with cython. Is there anyway I could use mgrid to get a matrix with all
> my
> > "windows" and then do a large matrix multiply instead?
>
> If you only care about removing the two loops for the per-window
> processing, Anne Archibald and Robert Kern wrote a very useful
> function, segment_axis, which is like the matlab buffer function on
> steroids, using numpy stride tricks (to avoid copies in many cases). I
> think that would do everything you want, right ? I use it a lot in my
> own code, it may be worth being included in numpy or scipy proper.
>
>
> http://projects.scipy.org/scipy/scikits/browser/trunk/talkbox/scikits/talkbox/tools/segmentaxis.py
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


Thanks again!

-- 
Nicolas Pinto
Ph.D. Candidate, Brain & Computer Sciences
Massachusetts Institute of Technology, USA
http://web.mit.edu/pinto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090221/c03623b8/attachment.html>

From josef.pktd at gmail.com  Sat Feb 21 00:56:48 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 21 Feb 2009 00:56:48 -0500
Subject: [Numpy-discussion] Efficient numpy slicing for a "sliding
	window approach".
In-Reply-To: <954ae5aa0902202136x513770a7o12409270436b87bd@mail.gmail.com>
References: <954ae5aa0902202046k5d89a67cn5d260c2844866dab@mail.gmail.com>
	<5b8d13220902202055k59e06fd9hd2f553c5411a3efd@mail.gmail.com>
	<954ae5aa0902202136x513770a7o12409270436b87bd@mail.gmail.com>
Message-ID: <1cd32cbb0902202156g4380b9d6je4ced629bc290ecd@mail.gmail.com>

On Sat, Feb 21, 2009 at 12:36 AM, Nicolas Pinto <pinto at mit.edu> wrote:
> Thanks a lot for the pointer to segmentaxis. I'm trying to use it "as is"
> and it seems that I need to a big reshape before the matrix multiplication.
> Am I missing something ?
>
>
> ========================================
>
> import numpy as np
> from numpy import dot, transpose
>
> arrh, arrw, arrd = 480,640,96
> arr = np.random.randn(arrh, arrw, arrd).astype("float32")
>
> stride = 16
> winh, winw, wind = 128,64,96
>
> limit = 100
>
> clas_w = np.random.randn(8,4,96).astype("float32").ravel()
>
> from segmentaxis import segment_axis
>
> nh, nw = arrh-winh+1, arrw-winw+1
>
> @profile
> def func_loop(arr):
>
>     resps = np.empty((nh,nw), dtype="float32")
>     for j in xrange(nh):
>          for i in xrange(nw):
>             win = arr[j:j+winh:stride, i:i+winw:stride]
>             win = win.ravel()
>             resp = dot(win, clas_w)
>             resps[j,i] = resp
>
>     resps = resps.ravel()
>     print resps.mean()
>
>
> @profile
> def func_segment(arr):
>
>     arr = segment_axis(arr, winh, winh-1, axis=0)
>     arr = arr[:, ::stride]
>
>     arr = segment_axis(arr, winw, winw-1, axis=2)
>     arr = arr[:, :, :, ::stride]
>
>     arr = transpose(arr, [0, 2, 1, 3, 4])
>     arr = arr.reshape(-1, clas_w.size)
>
>     resps2 = dot(arr, clas_w)
>
>     resps2 = resps2.ravel()
>     print resps2.mean()
>
>     # ...
>
> func_loop(arr)
>
> func_segment(arr)
>
>
> ========================================
>
>
> % python kernprof.py -l -v sliding_win_all.py
> w-0.0500485824034
> segmentaxis.py:94: UserWarning: Problem with ndarray creation forces copy.
>   warnings.warn("Problem with ndarray creation forces copy.")
> -0.0500485824034
> Wrote profile results to sliding_win_all.py.lprof
> Timer unit: 1e-06 s
>
> File: sliding_win_all.py
> Function: func_loop at line 18
> Total time: 3.9785 s
>
> Line #      Hits         Time  Per Hit   % Time  Line Contents
> ==============================================================
>     18                                           @profile
>     19                                           def func_loop(arr):
>     20
>     21         1           28     28.0      0.0      resps =
> np.empty((nh,nw), dtype="float32")
>     22       354          308      0.9      0.0      for j in xrange(nh):
>     23    204034       199753      1.0      5.0           for i in
> xrange(nw):
>     24    203681       691915      3.4     17.4              win =
> arr[j:j+winh:stride, i:i+winw:stride]
>     25    203681      1341174      6.6     33.7              win =
> win.ravel()
>     26    203681      1417998      7.0     35.6              resp = dot(win,
> clas_w)
>     27    203681       326520      1.6      8.2              resps[j,i] =
> resp
>     28
>     29         1            2      2.0      0.0      resps = resps.ravel()
>     30         1          805    805.0      0.0      print resps.mean()
>
> File: sliding_win_all.py
> Function: func_segment at line 33
> Total time: 3.82026 s
>
> Line #      Hits         Time  Per Hit   % Time  Line Contents
> ==============================================================
>     33                                           @profile
>     34                                           def func_segment(arr):
>     35
>     36         1           43     43.0      0.0      arr = segment_axis(arr,
> winh, winh-1, axis=0)
>     37         1            4      4.0      0.0      arr = arr[:, ::stride]
>     38
>     39         1       546505 546505.0     14.3      arr = segment_axis(arr,
> winw, winw-1, axis=2)
>     40         1           12     12.0      0.0      arr = arr[:, :, :,
> ::stride]
>     41
>     42         1           12     12.0      0.0      arr = transpose(arr,
> [0, 2, 1, 3, 4])
>     43         1      2435047 2435047.0     63.7      arr = arr.reshape(-1,
> clas_w.size)
>     44
>     45         1       837700 837700.0     21.9      resps2 = dot(arr,
> clas_w)
>     46
>     47         1           41     41.0      0.0      resps2 = resps2.ravel()
>     48         1          892    892.0      0.0      print resps2.mean()
>
>
> On Fri, Feb 20, 2009 at 11:55 PM, David Cournapeau <cournape at gmail.com>
> wrote:
>>
>> On Sat, Feb 21, 2009 at 1:46 PM, Nicolas Pinto <pinto at mit.edu> wrote:
>> > Dear all,
>> >
>> > I'm trying to optimize the code below and I was wondering if there is an
>> > efficient method that could reduce the numpy slicing overheard without
>> > going
>> > with cython. Is there anyway I could use mgrid to get a matrix with all
>> > my
>> > "windows" and then do a large matrix multiply instead?
>>
>> If you only care about removing the two loops for the per-window
>> processing, Anne Archibald and Robert Kern wrote a very useful
>> function, segment_axis, which is like the matlab buffer function on
>> steroids, using numpy stride tricks (to avoid copies in many cases). I
>> think that would do everything you want, right ? I use it a lot in my
>> own code, it may be worth being included in numpy or scipy proper.
>>
>>
>> http://projects.scipy.org/scipy/scikits/browser/trunk/talkbox/scikits/talkbox/tools/segmentaxis.py
>>
>> David
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
> Thanks again!
>
> --
> Nicolas Pinto
> Ph.D. Candidate, Brain & Computer Sciences
> Massachusetts Institute of Technology, USA
> http://web.mit.edu/pinto
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>

Hi,

I don't really understand your array dimension, but to me it seems
that you could use ndimage.correlate2d for this.

Josef


From pinto at mit.edu  Sat Feb 21 01:08:15 2009
From: pinto at mit.edu (Nicolas Pinto)
Date: Sat, 21 Feb 2009 01:08:15 -0500
Subject: [Numpy-discussion] Efficient numpy slicing for a "sliding
	window approach".
In-Reply-To: <1cd32cbb0902202156g4380b9d6je4ced629bc290ecd@mail.gmail.com>
References: <954ae5aa0902202046k5d89a67cn5d260c2844866dab@mail.gmail.com>
	<5b8d13220902202055k59e06fd9hd2f553c5411a3efd@mail.gmail.com>
	<954ae5aa0902202136x513770a7o12409270436b87bd@mail.gmail.com>
	<1cd32cbb0902202156g4380b9d6je4ced629bc290ecd@mail.gmail.com>
Message-ID: <954ae5aa0902202208x54f5a93dmdb3af46d2a280856@mail.gmail.com>

Thanks Josef.

I'm not sure how I could use correlate2d because of the 'stride' parameter
on the y and x axes, but I may be able to do something on the z axis.

On Sat, Feb 21, 2009 at 12:56 AM, <josef.pktd at gmail.com> wrote:

> On Sat, Feb 21, 2009 at 12:36 AM, Nicolas Pinto <pinto at mit.edu> wrote:
> > Thanks a lot for the pointer to segmentaxis. I'm trying to use it "as is"
> > and it seems that I need to a big reshape before the matrix
> multiplication.
> > Am I missing something ?
> >
> >
> > ========================================
> >
> > import numpy as np
> > from numpy import dot, transpose
> >
> > arrh, arrw, arrd = 480,640,96
> > arr = np.random.randn(arrh, arrw, arrd).astype("float32")
> >
> > stride = 16
> > winh, winw, wind = 128,64,96
> >
> > limit = 100
> >
> > clas_w = np.random.randn(8,4,96).astype("float32").ravel()
> >
> > from segmentaxis import segment_axis
> >
> > nh, nw = arrh-winh+1, arrw-winw+1
> >
> > @profile
> > def func_loop(arr):
> >
> >     resps = np.empty((nh,nw), dtype="float32")
> >     for j in xrange(nh):
> >          for i in xrange(nw):
> >             win = arr[j:j+winh:stride, i:i+winw:stride]
> >             win = win.ravel()
> >             resp = dot(win, clas_w)
> >             resps[j,i] = resp
> >
> >     resps = resps.ravel()
> >     print resps.mean()
> >
> >
> > @profile
> > def func_segment(arr):
> >
> >     arr = segment_axis(arr, winh, winh-1, axis=0)
> >     arr = arr[:, ::stride]
> >
> >     arr = segment_axis(arr, winw, winw-1, axis=2)
> >     arr = arr[:, :, :, ::stride]
> >
> >     arr = transpose(arr, [0, 2, 1, 3, 4])
> >     arr = arr.reshape(-1, clas_w.size)
> >
> >     resps2 = dot(arr, clas_w)
> >
> >     resps2 = resps2.ravel()
> >     print resps2.mean()
> >
> >     # ...
> >
> > func_loop(arr)
> >
> > func_segment(arr)
> >
> >
> > ========================================
> >
> >
> > % python kernprof.py -l -v sliding_win_all.py
> > w-0.0500485824034
> > segmentaxis.py:94: UserWarning: Problem with ndarray creation forces
> copy.
> >   warnings.warn("Problem with ndarray creation forces copy.")
> > -0.0500485824034
> > Wrote profile results to sliding_win_all.py.lprof
> > Timer unit: 1e-06 s
> >
> > File: sliding_win_all.py
> > Function: func_loop at line 18
> > Total time: 3.9785 s
> >
> > Line #      Hits         Time  Per Hit   % Time  Line Contents
> > ==============================================================
> >     18                                           @profile
> >     19                                           def func_loop(arr):
> >     20
> >     21         1           28     28.0      0.0      resps =
> > np.empty((nh,nw), dtype="float32")
> >     22       354          308      0.9      0.0      for j in xrange(nh):
> >     23    204034       199753      1.0      5.0           for i in
> > xrange(nw):
> >     24    203681       691915      3.4     17.4              win =
> > arr[j:j+winh:stride, i:i+winw:stride]
> >     25    203681      1341174      6.6     33.7              win =
> > win.ravel()
> >     26    203681      1417998      7.0     35.6              resp =
> dot(win,
> > clas_w)
> >     27    203681       326520      1.6      8.2              resps[j,i] =
> > resp
> >     28
> >     29         1            2      2.0      0.0      resps =
> resps.ravel()
> >     30         1          805    805.0      0.0      print resps.mean()
> >
> > File: sliding_win_all.py
> > Function: func_segment at line 33
> > Total time: 3.82026 s
> >
> > Line #      Hits         Time  Per Hit   % Time  Line Contents
> > ==============================================================
> >     33                                           @profile
> >     34                                           def func_segment(arr):
> >     35
> >     36         1           43     43.0      0.0      arr =
> segment_axis(arr,
> > winh, winh-1, axis=0)
> >     37         1            4      4.0      0.0      arr = arr[:,
> ::stride]
> >     38
> >     39         1       546505 546505.0     14.3      arr =
> segment_axis(arr,
> > winw, winw-1, axis=2)
> >     40         1           12     12.0      0.0      arr = arr[:, :, :,
> > ::stride]
> >     41
> >     42         1           12     12.0      0.0      arr = transpose(arr,
> > [0, 2, 1, 3, 4])
> >     43         1      2435047 2435047.0     63.7      arr =
> arr.reshape(-1,
> > clas_w.size)
> >     44
> >     45         1       837700 837700.0     21.9      resps2 = dot(arr,
> > clas_w)
> >     46
> >     47         1           41     41.0      0.0      resps2 =
> resps2.ravel()
> >     48         1          892    892.0      0.0      print resps2.mean()
> >
> >
> > On Fri, Feb 20, 2009 at 11:55 PM, David Cournapeau <cournape at gmail.com>
> > wrote:
> >>
> >> On Sat, Feb 21, 2009 at 1:46 PM, Nicolas Pinto <pinto at mit.edu> wrote:
> >> > Dear all,
> >> >
> >> > I'm trying to optimize the code below and I was wondering if there is
> an
> >> > efficient method that could reduce the numpy slicing overheard without
> >> > going
> >> > with cython. Is there anyway I could use mgrid to get a matrix with
> all
> >> > my
> >> > "windows" and then do a large matrix multiply instead?
> >>
> >> If you only care about removing the two loops for the per-window
> >> processing, Anne Archibald and Robert Kern wrote a very useful
> >> function, segment_axis, which is like the matlab buffer function on
> >> steroids, using numpy stride tricks (to avoid copies in many cases). I
> >> think that would do everything you want, right ? I use it a lot in my
> >> own code, it may be worth being included in numpy or scipy proper.
> >>
> >>
> >>
> http://projects.scipy.org/scipy/scikits/browser/trunk/talkbox/scikits/talkbox/tools/segmentaxis.py
> >>
> >> David
> >> _______________________________________________
> >> Numpy-discussion mailing list
> >> Numpy-discussion at scipy.org
> >> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> > Thanks again!
> >
> > --
> > Nicolas Pinto
> > Ph.D. Candidate, Brain & Computer Sciences
> > Massachusetts Institute of Technology, USA
> > http://web.mit.edu/pinto
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
>
> Hi,
>
> I don't really understand your array dimension, but to me it seems
> that you could use ndimage.correlate2d for this.
>
> Josef
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Nicolas Pinto
Ph.D. Candidate, Brain & Computer Sciences
Massachusetts Institute of Technology, USA
http://web.mit.edu/pinto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090221/025da060/attachment.html>

From stefan at sun.ac.za  Sat Feb 21 05:58:14 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sat, 21 Feb 2009 12:58:14 +0200
Subject: [Numpy-discussion] RFR: 995 - numpy.load can't handle gzip file
	handles
Message-ID: <9457e7c80902210258h3622f473h54e7776046cb195b@mail.gmail.com>

Hi,

Based on an example on Effbot, I implemented a workaround for reverse
seeking in gzip files.  I need someone with Python 2.4 to review:

http://www.scipy.org/scipy/numpy/ticket/995

Thanks!
St?fan


From nwagner at iam.uni-stuttgart.de  Sat Feb 21 14:34:53 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 21 Feb 2009 20:34:53 +0100
Subject: [Numpy-discussion] RFR: 995 - numpy.load can't handle gzip
 file	handles
In-Reply-To: <9457e7c80902210258h3622f473h54e7776046cb195b@mail.gmail.com>
References: <9457e7c80902210258h3622f473h54e7776046cb195b@mail.gmail.com>
Message-ID: <web-117266524@uni-stuttgart.de>

On Sat, 21 Feb 2009 12:58:14 +0200
  St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi,
> 
> Based on an example on Effbot, I implemented a 
>workaround for reverse
> seeking in gzip files.  I need someone with Python 2.4 
>to review:
> 
> http://www.scipy.org/scipy/numpy/ticket/995
> 
> Thanks!
> St?fan
> _________________

Hi Stefan,

I would like to help but I failed to install numpy
(python2.4 Suse Linux 9.3)

In file included from 
numpy/core/src/multiarraymodule.c:96:
numpy/core/src/umath_funcs_c99.inc.src:269: warning: 
conflicting types for built-in function `sinl'
numpy/core/src/umath_funcs_c99.inc.src:269: warning: 
conflicting types for built-in function `cosl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `tanl'
/usr/include/bits/mathcalls.h:68: error: previous 
declaration of `tanl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `sinhl'
/usr/include/bits/mathcalls.h:75: error: previous 
declaration of `sinhl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `coshl'
/usr/include/bits/mathcalls.h:73: error: previous 
declaration of `coshl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `tanhl'
/usr/include/bits/mathcalls.h:77: error: previous 
declaration of `tanhl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `fabsl'
/usr/include/bits/mathinline.h:476: error: previous 
declaration of `fabsl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `floorl'
/usr/include/bits/mathinline.h:530: error: previous 
declaration of `floorl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `ceill'
/usr/include/bits/mathinline.h:541: error: previous 
declaration of `ceill'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `rintl'
/usr/include/bits/mathcalls.h:280: error: previous 
declaration of `rintl'
numpy/core/src/umath_funcs_c99.inc.src:269: warning: 
conflicting types for built-in function `truncl'
numpy/core/src/umath_funcs_c99.inc.src:269: warning: 
conflicting types for built-in function `sqrtl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `log10l'
/usr/include/bits/mathcalls.h:113: error: previous 
declaration of `log10l'
numpy/core/src/umath_funcs_c99.inc.src:269: warning: 
conflicting types for built-in function `logl'
numpy/core/src/umath_funcs_c99.inc.src:269: warning: 
conflicting types for built-in function `expl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `expm1l'
/usr/include/bits/mathcalls.h:129: error: previous 
declaration of `expm1l'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `asinl'
/usr/include/bits/mathcalls.h:57: error: previous 
declaration of `asinl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `acosl'
/usr/include/bits/mathcalls.h:55: error: previous 
declaration of `acosl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `atanl'
/usr/include/bits/mathcalls.h:59: error: previous 
declaration of `atanl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `asinhl'
/usr/include/bits/mathcalls.h:91: error: previous 
declaration of `asinhl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `acoshl'
/usr/include/bits/mathcalls.h:89: error: previous 
declaration of `acoshl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `atanhl'
/usr/include/bits/mathcalls.h:93: error: previous 
declaration of `atanhl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `log1pl'
/usr/include/bits/mathcalls.h:132: error: previous 
declaration of `log1pl'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `exp2l'
/usr/include/bits/mathcalls.h:142: error: previous 
declaration of `exp2l'
numpy/core/src/umath_funcs_c99.inc.src:269: error: 
conflicting types for `log2l'
/usr/include/bits/mathcalls.h:145: error: previous 
declaration of `log2l'
numpy/core/src/umath_funcs_c99.inc.src:285: error: 
conflicting types for `atan2l'
/usr/include/bits/mathcalls.h:61: error: previous 
declaration of `atan2l'
numpy/core/src/umath_funcs_c99.inc.src:285: error: 
conflicting types for `hypotl'
/usr/include/bits/mathcalls.h:163: error: previous 
declaration of `hypotl'
numpy/core/src/umath_funcs_c99.inc.src:285: error: 
conflicting types for `powl'
/usr/include/bits/mathcalls.h:154: error: previous 
declaration of `powl'
numpy/core/src/umath_funcs_c99.inc.src:285: error: 
conflicting types for `fmodl'
/usr/include/bits/mathcalls.h:188: error: previous 
declaration of `fmodl'
numpy/core/src/umath_funcs_c99.inc.src:296: error: 
conflicting types for `modfl'
/usr/include/bits/mathcalls.h:116: error: previous 
declaration of `modfl'
In file included from 
numpy/core/src/scalartypes.inc.src:8,
                  from numpy/core/src/arrayobject.c:545,
                  from 
numpy/core/src/multiarraymodule.c:111:
build/src.linux-i686-2.4/numpy/core/include/numpy/config.h:10:1: 
warning: "SIZEOF_LONG_DOUBLE" redefined
build/src.linux-i686-2.4/numpy/core/include/numpy/config.h:6:1: 
warning: this is the location of the previous definition
In file included from numpy/core/src/arraytypes.inc.src:2,
                  from numpy/core/src/arrayobject.c:546,
                  from 
numpy/core/src/multiarraymodule.c:111:
build/src.linux-i686-2.4/numpy/core/include/numpy/config.h:6:1: 
warning: "SIZEOF_LONG_DOUBLE" redefined
In file included from 
numpy/core/src/scalartypes.inc.src:8,
                  from numpy/core/src/arrayobject.c:545,
                  from 
numpy/core/src/multiarraymodule.c:111:
build/src.linux-i686-2.4/numpy/core/include/numpy/config.h:10:1: 
warning: this is the location of the previous definition
In file included from numpy/core/src/arraytypes.inc.src:2,
                  from numpy/core/src/arrayobject.c:546,
                  from 
numpy/core/src/multiarraymodule.c:111:
build/src.linux-i686-2.4/numpy/core/include/numpy/config.h:10:1: 
warning: "SIZEOF_LONG_DOUBLE" redefined
build/src.linux-i686-2.4/numpy/core/include/numpy/config.h:6:1: 
warning: this is the location of the previous definition
error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG 
-O2 -march=i586 -mcpu=i686 -fmessage-length=0 -Wall -g 
-fPIC -Ibuild/src.linux-i686-2.4/numpy/core/src 
-Inumpy/core/include 
-Ibuild/src.linux-i686-2.4/numpy/core/include/numpy 
-Inumpy/core/src -Inumpy/core/include 
-I/usr/include/python2.4 -c 
numpy/core/src/multiarraymodule.c -o 
build/temp.linux-i686-2.4/numpy/core/src/multiarraymodule.o" 
failed with exit status 1

Cheers,

       Nils

  


From stefan at sun.ac.za  Sat Feb 21 18:46:28 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 22 Feb 2009 01:46:28 +0200
Subject: [Numpy-discussion] RFR: 995 - numpy.load can't handle gzip file
	handles
In-Reply-To: <web-117266524@uni-stuttgart.de>
References: <9457e7c80902210258h3622f473h54e7776046cb195b@mail.gmail.com>
	<web-117266524@uni-stuttgart.de>
Message-ID: <9457e7c80902211546s67f74ef0gcdf8d802ee8f71ad@mail.gmail.com>

2009/2/21 Nils Wagner <nwagner at iam.uni-stuttgart.de>:
> Hi Stefan,
>
> I would like to help but I failed to install numpy
> (python2.4 Suse Linux 9.3)
>
> In file included from
> numpy/core/src/multiarraymodule.c:96:
> numpy/core/src/umath_funcs_c99.inc.src:269: warning:
> conflicting types for built-in function `sinl'

[...]

Thanks, Nils.  I've seen these errors on the buildbot too.  David
should have this sorted out pretty soon.

Regards
St?fan


From cournape at gmail.com  Sat Feb 21 22:55:23 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 22 Feb 2009 12:55:23 +0900
Subject: [Numpy-discussion] RFR: 995 - numpy.load can't handle gzip file
	handles
In-Reply-To: <9457e7c80902211546s67f74ef0gcdf8d802ee8f71ad@mail.gmail.com>
References: <9457e7c80902210258h3622f473h54e7776046cb195b@mail.gmail.com>
	<web-117266524@uni-stuttgart.de>
	<9457e7c80902211546s67f74ef0gcdf8d802ee8f71ad@mail.gmail.com>
Message-ID: <5b8d13220902211955j4383ffe5x47decf87ce9fab9b@mail.gmail.com>

On Sun, Feb 22, 2009 at 8:46 AM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2009/2/21 Nils Wagner <nwagner at iam.uni-stuttgart.de>:
>> Hi Stefan,
>>
>> I would like to help but I failed to install numpy
>> (python2.4 Suse Linux 9.3)
>>
>> In file included from
>> numpy/core/src/multiarraymodule.c:96:
>> numpy/core/src/umath_funcs_c99.inc.src:269: warning:
>> conflicting types for built-in function `sinl'
>
> [...]
>
> Thanks, Nils.  I've seen these errors on the buildbot too.  David
> should have this sorted out pretty soon.

Sorry about that, I screwed up while merging back into the trunk the
branch about windows x64 support - this should be fixed now (it fixes
the problem on mac os X at least),

cheers,

David


From nwagner at iam.uni-stuttgart.de  Sun Feb 22 05:01:03 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sun, 22 Feb 2009 11:01:03 +0100
Subject: [Numpy-discussion] RFR: 995 - numpy.load can't handle gzip
 file	handles
In-Reply-To: <9457e7c80902210258h3622f473h54e7776046cb195b@mail.gmail.com>
References: <9457e7c80902210258h3622f473h54e7776046cb195b@mail.gmail.com>
Message-ID: <web-117274771@uni-stuttgart.de>

On Sat, 21 Feb 2009 12:58:14 +0200
  St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi,
> 
> Based on an example on Effbot, I implemented a 
>workaround for reverse
> seeking in gzip files.  I need someone with Python 2.4 
>to review:
> 
> http://www.scipy.org/scipy/numpy/ticket/995
> 
> Thanks!
> St?fan


Done. See http://www.scipy.org/scipy/numpy/ticket/995
for details.

Cheers,

     Nils
  


From stefan at sun.ac.za  Sun Feb 22 06:03:03 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 22 Feb 2009 13:03:03 +0200
Subject: [Numpy-discussion] RFR: 995 - numpy.load can't handle gzip file
	handles
In-Reply-To: <web-117274771@uni-stuttgart.de>
References: <9457e7c80902210258h3622f473h54e7776046cb195b@mail.gmail.com>
	<web-117274771@uni-stuttgart.de>
Message-ID: <9457e7c80902220303w39730e4et38fcf9937ffed6b3@mail.gmail.com>

Hi Nils,

2009/2/22 Nils Wagner <nwagner at iam.uni-stuttgart.de>:
> Done. See http://www.scipy.org/scipy/numpy/ticket/995
> for details.

Thanks.  Did you have a NumPy array stored with numpy.save in test.gz?
 I finally got access to a 2.4 machine and the patch works there.

Cheers
St?fan


From nwagner at iam.uni-stuttgart.de  Sun Feb 22 06:17:04 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sun, 22 Feb 2009 12:17:04 +0100
Subject: [Numpy-discussion] RFR: 995 - numpy.load can't handle gzip
 file	handles
In-Reply-To: <9457e7c80902220303w39730e4et38fcf9937ffed6b3@mail.gmail.com>
References: <9457e7c80902210258h3622f473h54e7776046cb195b@mail.gmail.com>
	<web-117274771@uni-stuttgart.de>
	<9457e7c80902220303w39730e4et38fcf9937ffed6b3@mail.gmail.com>
Message-ID: <web-117275771@uni-stuttgart.de>

On Sun, 22 Feb 2009 13:03:03 +0200
  St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi Nils,
> 
> 2009/2/22 Nils Wagner <nwagner at iam.uni-stuttgart.de>:
>> Done. See http://www.scipy.org/scipy/numpy/ticket/995
>> for details.
> 
> Thanks.  Did you have a NumPy array stored with 
>numpy.save in test.gz?
> I finally got access to a 2.4 machine and the patch 
>works there.
> 
> Cheers
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

The attachment is missing.

Nils
  


From stefan at sun.ac.za  Sun Feb 22 11:37:05 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 22 Feb 2009 18:37:05 +0200
Subject: [Numpy-discussion] RFR: 991 - Make savez able to write ZIP64 files
Message-ID: <9457e7c80902220837m7144bb53qbe9ee14caca8dd4f@mail.gmail.com>

Hi all,

Please review the patch attached to

http://scipy.org/scipy/numpy/ticket/991

which enables ZIP64 extensions when saving and loading zipped data
under Python >= 2.5

Thanks,
St?fan


From nwagner at iam.uni-stuttgart.de  Sun Feb 22 11:57:37 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sun, 22 Feb 2009 17:57:37 +0100
Subject: [Numpy-discussion] RFR: 991 - Make savez able to write ZIP64
 files
In-Reply-To: <9457e7c80902220837m7144bb53qbe9ee14caca8dd4f@mail.gmail.com>
References: <9457e7c80902220837m7144bb53qbe9ee14caca8dd4f@mail.gmail.com>
Message-ID: <web-117280659@uni-stuttgart.de>

On Sun, 22 Feb 2009 18:37:05 +0200
  St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi all,
> 
> Please review the patch attached to
> 
> http://scipy.org/scipy/numpy/ticket/991
> 
> which enables ZIP64 extensions when saving and loading 
>zipped data
> under Python >= 2.5
> 
> Thanks,
> St?fan

Hi Stefan,

Please can you provide a short test ?

Thanks in advance

                     Nils
  


From dsdale24 at gmail.com  Sun Feb 22 15:17:36 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 22 Feb 2009 15:17:36 -0500
Subject: [Numpy-discussion] small suggestion for numpy.testing utils
Message-ID: <a08e5f80902221217r50e76bcbk2586612e7c97e9a1@mail.gmail.com>

Hello,

I am using numpy's assert_array_equal and assert_array_almost_equal to unit
test my physical quantities package. I made a single minor change to
assert_array_compare that I think might make these functions more useful to
ndarray subclasses, and thought maybe they could be useful to numpy itself.
I tried applying this diff to numpy and running the test suite, and instead
of 9 known failures I got 1 known failure, 11 skips, 2 errors and 2
failures. Perhaps it is possible that by not forcing the input arrays to be
ndarray instances, some additional numpy features are exposed.

Thanks,
Darren

$ svn diff
Index: numpy/testing/utils.py
===================================================================
--- numpy/testing/utils.py      (revision 6370)
+++ numpy/testing/utils.py      (working copy)
@@ -240,9 +240,9 @@

 def assert_array_compare(comparison, x, y, err_msg='', verbose=True,
                          header=''):
-    from numpy.core import asarray, isnan, any
-    x = asarray(x)
-    y = asarray(y)
+    from numpy.core import array, isnan, any
+    x = array(x, copy=False, subok=True)
+    y = array(y, copy=False, subok=True)

     def isnumber(x):
         return x.dtype.char in '?bhilqpBHILQPfdgFDG'
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/641d41fc/attachment.html>

From stefan at sun.ac.za  Sun Feb 22 15:18:32 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 22 Feb 2009 22:18:32 +0200
Subject: [Numpy-discussion] RFR: 991 - Make savez able to write ZIP64
	files
In-Reply-To: <web-117280659@uni-stuttgart.de>
References: <9457e7c80902220837m7144bb53qbe9ee14caca8dd4f@mail.gmail.com>
	<web-117280659@uni-stuttgart.de>
Message-ID: <9457e7c80902221218r3137c708v6d3eed1999a4a9a9@mail.gmail.com>

Hi Nils

2009/2/22 Nils Wagner <nwagner at iam.uni-stuttgart.de>:
>> http://scipy.org/scipy/numpy/ticket/991
>>
>> which enables ZIP64 extensions when saving and loading
>>zipped data
>> under Python >= 2.5

You can just run "nosetests numpy.lib" on both a 2.4 and a 2.5
installation to see if it works.

Thanks!
St?fan


From dsdale24 at gmail.com  Sun Feb 22 15:22:15 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 22 Feb 2009 15:22:15 -0500
Subject: [Numpy-discussion] small suggestion for numpy.testing utils
In-Reply-To: <a08e5f80902221217r50e76bcbk2586612e7c97e9a1@mail.gmail.com>
References: <a08e5f80902221217r50e76bcbk2586612e7c97e9a1@mail.gmail.com>
Message-ID: <a08e5f80902221222j63a3ce21s74122a9d6932c0be@mail.gmail.com>

On Sun, Feb 22, 2009 at 3:17 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> Hello,
>
> I am using numpy's assert_array_equal and assert_array_almost_equal to unit
> test my physical quantities package. I made a single minor change to
> assert_array_compare that I think might make these functions more useful to
> ndarray subclasses, and thought maybe they could be useful to numpy itself.
> I tried applying this diff to numpy and running the test suite, and instead
> of 9 known failures I got 1 known failure, 11 skips, 2 errors and 2
> failures. Perhaps it is possible that by not forcing the input arrays to be
> ndarray instances, some additional numpy features are exposed.
>
> Thanks,
> Darren
>
> $ svn diff
> Index: numpy/testing/utils.py
> ===================================================================
> --- numpy/testing/utils.py      (revision 6370)
> +++ numpy/testing/utils.py      (working copy)
> @@ -240,9 +240,9 @@
>
>  def assert_array_compare(comparison, x, y, err_msg='', verbose=True,
>                           header=''):
> -    from numpy.core import asarray, isnan, any
> -    x = asarray(x)
> -    y = asarray(y)
> +    from numpy.core import array, isnan, any
> +    x = array(x, copy=False, subok=True)
> +    y = array(y, copy=False, subok=True)
>
>      def isnumber(x):
>          return x.dtype.char in '?bhilqpBHILQPfdgFDG'
>

Actually, my svn checkout was not up to date. With this patch applied, I get
1 known failure and 11 skips.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/3fc7bec3/attachment.html>

From dsdale24 at gmail.com  Sun Feb 22 15:38:21 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 22 Feb 2009 15:38:21 -0500
Subject: [Numpy-discussion] small suggestion for numpy.testing utils
In-Reply-To: <a08e5f80902221222j63a3ce21s74122a9d6932c0be@mail.gmail.com>
References: <a08e5f80902221217r50e76bcbk2586612e7c97e9a1@mail.gmail.com>
	<a08e5f80902221222j63a3ce21s74122a9d6932c0be@mail.gmail.com>
Message-ID: <a08e5f80902221238o257b5995x126c6588114fd7e0@mail.gmail.com>

On Sun, Feb 22, 2009 at 3:22 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> On Sun, Feb 22, 2009 at 3:17 PM, Darren Dale <dsdale24 at gmail.com> wrote:
>
>> Hello,
>>
>> I am using numpy's assert_array_equal and assert_array_almost_equal to
>> unit test my physical quantities package. I made a single minor change to
>> assert_array_compare that I think might make these functions more useful to
>> ndarray subclasses, and thought maybe they could be useful to numpy itself.
>> I tried applying this diff to numpy and running the test suite, and instead
>> of 9 known failures I got 1 known failure, 11 skips, 2 errors and 2
>> failures. Perhaps it is possible that by not forcing the input arrays to be
>> ndarray instances, some additional numpy features are exposed.
>>
>> Thanks,
>> Darren
>>
>> $ svn diff
>> Index: numpy/testing/utils.py
>> ===================================================================
>> --- numpy/testing/utils.py      (revision 6370)
>> +++ numpy/testing/utils.py      (working copy)
>> @@ -240,9 +240,9 @@
>>
>>  def assert_array_compare(comparison, x, y, err_msg='', verbose=True,
>>                           header=''):
>> -    from numpy.core import asarray, isnan, any
>> -    x = asarray(x)
>> -    y = asarray(y)
>> +    from numpy.core import array, isnan, any
>> +    x = array(x, copy=False, subok=True)
>> +    y = array(y, copy=False, subok=True)
>>
>>      def isnumber(x):
>>          return x.dtype.char in '?bhilqpBHILQPfdgFDG'
>>
>
> Actually, my svn checkout was not up to date. With this patch applied, I
> get 1 known failure and 11 skips.
>
>
I just double checked and I think I get the same results running the svn
6456 test suite with and without this patch applied. I tried posting an
enhancement request at the trac website, but I cant file the ticket because
I get "500 Internal Server Error", so I'm posting it here.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/a98f381b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_subclasses.diff
Type: text/x-diff
Size: 609 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/a98f381b/attachment.diff>

From nwagner at iam.uni-stuttgart.de  Sun Feb 22 15:50:22 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sun, 22 Feb 2009 21:50:22 +0100
Subject: [Numpy-discussion] RFR: 991 - Make savez able to write
 ZIP64	files
In-Reply-To: <9457e7c80902221218r3137c708v6d3eed1999a4a9a9@mail.gmail.com>
References: <9457e7c80902220837m7144bb53qbe9ee14caca8dd4f@mail.gmail.com>
	<web-117280659@uni-stuttgart.de>
	<9457e7c80902221218r3137c708v6d3eed1999a4a9a9@mail.gmail.com>
Message-ID: <web-117283978@uni-stuttgart.de>

On Sun, 22 Feb 2009 22:18:32 +0200
  St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi Nils
> 
> 2009/2/22 Nils Wagner <nwagner at iam.uni-stuttgart.de>:
>>> http://scipy.org/scipy/numpy/ticket/991
>>>
>>> which enables ZIP64 extensions when saving and loading
>>>zipped data
>>> under Python >= 2.5
> 
> You can just run "nosetests numpy.lib" on both a 2.4 and 
>a 2.5
> installation to see if it works.
> 
> Thanks!
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion


Done. I am using python2.6 now.

patch io.py < 0001-Add-ZIP64-support.patch


nosetests numpy.lib
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 938 tests in 8.616s

OK


Cheers,
             Nils


From stefan at sun.ac.za  Sun Feb 22 16:07:35 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 22 Feb 2009 23:07:35 +0200
Subject: [Numpy-discussion] RFR: 991 - Make savez able to write ZIP64
	files
In-Reply-To: <web-117283978@uni-stuttgart.de>
References: <9457e7c80902220837m7144bb53qbe9ee14caca8dd4f@mail.gmail.com>
	<web-117280659@uni-stuttgart.de>
	<9457e7c80902221218r3137c708v6d3eed1999a4a9a9@mail.gmail.com>
	<web-117283978@uni-stuttgart.de>
Message-ID: <9457e7c80902221307m7af667e4icda518fa4848f54f@mail.gmail.com>

2009/2/22 Nils Wagner <nwagner at iam.uni-stuttgart.de>:
>> You can just run "nosetests numpy.lib" on both a 2.4 and
>>a 2.5
>> installation to see if it works.
>
> Done. I am using python2.6 now.
>
> patch io.py < 0001-Add-ZIP64-support.patch

Thanks for testing!

Cheers
St?fan


From stefan at sun.ac.za  Sun Feb 22 17:12:21 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 23 Feb 2009 00:12:21 +0200
Subject: [Numpy-discussion] small suggestion for numpy.testing utils
In-Reply-To: <a08e5f80902221217r50e76bcbk2586612e7c97e9a1@mail.gmail.com>
References: <a08e5f80902221217r50e76bcbk2586612e7c97e9a1@mail.gmail.com>
Message-ID: <9457e7c80902221412t10e9a3f3pb5a20f1a80023e81@mail.gmail.com>

Hi Darren

2009/2/22 Darren Dale <dsdale24 at gmail.com>:
> I am using numpy's assert_array_equal and assert_array_almost_equal to unit
> test my physical quantities package. I made a single minor change to
> assert_array_compare that I think might make these functions more useful to
> ndarray subclasses, and thought maybe they could be useful to numpy itself.

Your patch makes good sense.  I applied it in r6457.  I'll keep an eye
on the thread to see if anyone else has further comments.

Thanks!
St?fan


From dsdale24 at gmail.com  Sun Feb 22 17:33:52 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 22 Feb 2009 17:33:52 -0500
Subject: [Numpy-discussion] small suggestion for numpy.testing utils
In-Reply-To: <9457e7c80902221412t10e9a3f3pb5a20f1a80023e81@mail.gmail.com>
References: <a08e5f80902221217r50e76bcbk2586612e7c97e9a1@mail.gmail.com>
	<9457e7c80902221412t10e9a3f3pb5a20f1a80023e81@mail.gmail.com>
Message-ID: <a08e5f80902221433j189209d0o2b4dfcaaaa61d20@mail.gmail.com>

On Sun, Feb 22, 2009 at 5:12 PM, St?fan van der Walt <stefan at sun.ac.za>wrote:

> Hi Darren
>
> 2009/2/22 Darren Dale <dsdale24 at gmail.com>:
> > I am using numpy's assert_array_equal and assert_array_almost_equal to
> unit
> > test my physical quantities package. I made a single minor change to
> > assert_array_compare that I think might make these functions more useful
> to
> > ndarray subclasses, and thought maybe they could be useful to numpy
> itself.
>
> Your patch makes good sense.  I applied it in r6457.  I'll keep an eye
> on the thread to see if anyone else has further comments.
>

That's great, thank you. I finally got the trac website to take the ticket,
by the way.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/6c8ce810/attachment.html>

From stefan at sun.ac.za  Sun Feb 22 17:39:03 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 23 Feb 2009 00:39:03 +0200
Subject: [Numpy-discussion] small suggestion for numpy.testing utils
In-Reply-To: <a08e5f80902221433j189209d0o2b4dfcaaaa61d20@mail.gmail.com>
References: <a08e5f80902221217r50e76bcbk2586612e7c97e9a1@mail.gmail.com>
	<9457e7c80902221412t10e9a3f3pb5a20f1a80023e81@mail.gmail.com>
	<a08e5f80902221433j189209d0o2b4dfcaaaa61d20@mail.gmail.com>
Message-ID: <9457e7c80902221439m67c2e980l8f3c02b744d7a09a@mail.gmail.com>

2009/2/23 Darren Dale <dsdale24 at gmail.com>:
> That's great, thank you. I finally got the trac website to take the ticket,
> by the way.

Actually, it's been taking all your tickets... from 1016 through 1021 :)

Cheers
St?fan


From dsdale24 at gmail.com  Sun Feb 22 17:51:33 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 22 Feb 2009 17:51:33 -0500
Subject: [Numpy-discussion] possible bug: __array_wrap__ is not called
	during arithmetic operations in some cases
Message-ID: <a08e5f80902221451q7aaa220cua3c13072f1993398@mail.gmail.com>

Does anyone know why __array_wrap__ is not called for subclasses during
arithmetic operations where an iterable like a list or tuple appears to the
right of the subclass? When I do "mine*[1,2,3]", array_wrap is not called
and I get an ndarray instead of a MyArray. "[1,2,3]*mine" is fine, as is
"mine*array([1,2,3])". I see the same issue with division, addition, etc.
Here is a demonstration, observed with svn 6456:

import numpy as np

class MyArray(np.ndarray):

    __array_priority__ = 20

    def __new__(cls):
        return np.asarray(1).view(cls).copy()

    def __array_wrap__(self, obj, context=None):
        print 'array wrap:', self, obj, context
        return obj.view(type(self))

    def __str__(self):
        return 'MyArray(%s)'%super(MyArray,self).__str__()

mine = MyArray()


print 3*mine
print mine*3
print [1,2,3]*mine
print mine*[1,2,3]
print
print 3/mine
print mine/3
print [1,2,3]*mine
print mine*[1,2,3]
print
print 3+mine
print mine+3
print [1,2,3]+mine
print mine+[1,2,3]
print
print 3/mine
print mine/3
print [1,2,3]/mine
print mine/[1,2,3]
print
print 3**mine
print mine**3
print [1,2,3]**mine
print mine**[1,2,3]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/176c2158/attachment.html>

From dsdale24 at gmail.com  Sun Feb 22 17:54:16 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 22 Feb 2009 17:54:16 -0500
Subject: [Numpy-discussion] small suggestion for numpy.testing utils
In-Reply-To: <9457e7c80902221439m67c2e980l8f3c02b744d7a09a@mail.gmail.com>
References: <a08e5f80902221217r50e76bcbk2586612e7c97e9a1@mail.gmail.com>
	<9457e7c80902221412t10e9a3f3pb5a20f1a80023e81@mail.gmail.com>
	<a08e5f80902221433j189209d0o2b4dfcaaaa61d20@mail.gmail.com>
	<9457e7c80902221439m67c2e980l8f3c02b744d7a09a@mail.gmail.com>
Message-ID: <a08e5f80902221454xee6b038gc48fd62225b7e7ab@mail.gmail.com>

On Sun, Feb 22, 2009 at 5:39 PM, St?fan van der Walt <stefan at sun.ac.za>wrote:

> 2009/2/23 Darren Dale <dsdale24 at gmail.com>:
> > That's great, thank you. I finally got the trac website to take the
> ticket,
> > by the way.
>
> Actually, it's been taking all your tickets... from 1016 through 1021 :)
>

Oops, sorry about that. (I was taking a drink of water when I read this and
it nearly came out my nose)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/07f2bd0c/attachment.html>

From efiring at hawaii.edu  Sun Feb 22 18:21:59 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Sun, 22 Feb 2009 13:21:59 -1000
Subject: [Numpy-discussion] possible bug: __array_wrap__ is not called
 during arithmetic operations in some cases
In-Reply-To: <a08e5f80902221451q7aaa220cua3c13072f1993398@mail.gmail.com>
References: <a08e5f80902221451q7aaa220cua3c13072f1993398@mail.gmail.com>
Message-ID: <49A1DE17.4070001@hawaii.edu>

Darren Dale wrote:
> Does anyone know why __array_wrap__ is not called for subclasses during 
> arithmetic operations where an iterable like a list or tuple appears to 
> the right of the subclass? When I do "mine*[1,2,3]", array_wrap is not 
> called and I get an ndarray instead of a MyArray. "[1,2,3]*mine" is 
> fine, as is "mine*array([1,2,3])". I see the same issue with division, 

The masked array subclass does not show this behavior:


In [3]:np.ma.arange(3) * [1,2,3]
Out[3]:
masked_array(data = [0 2 6],
              mask = False,
        fill_value = 999999)


In [4]:[1,2,3] * np.ma.arange(3)
Out[4]:
masked_array(data = [0 2 6],
              mask = False,
        fill_value = 999999)


Eric

> addition, etc. Here is a demonstration, observed with svn 6456:
> 
> import numpy as np
> 
> class MyArray(np.ndarray):
> 
>     __array_priority__ = 20
> 
>     def __new__(cls):
>         return np.asarray(1).view(cls).copy()
> 
>     def __array_wrap__(self, obj, context=None):
>         print 'array wrap:', self, obj, context
>         return obj.view(type(self))
> 
>     def __str__(self):
>         return 'MyArray(%s)'%super(MyArray,self).__str__()
> 
> mine = MyArray()
> 
> 
> print 3*mine
> print mine*3
> print [1,2,3]*mine
> print mine*[1,2,3]
> print
> print 3/mine
> print mine/3
> print [1,2,3]*mine
> print mine*[1,2,3]
> print
> print 3+mine
> print mine+3
> print [1,2,3]+mine
> print mine+[1,2,3]
> print
> print 3/mine
> print mine/3
> print [1,2,3]/mine
> print mine/[1,2,3]
> print
> print 3**mine
> print mine**3
> print [1,2,3]**mine
> print mine**[1,2,3]
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From timlee126 at yahoo.com  Sun Feb 22 18:19:10 2009
From: timlee126 at yahoo.com (Tim)
Date: Sun, 22 Feb 2009 15:19:10 -0800 (PST)
Subject: [Numpy-discussion] Import NumPy in Self-defined function script
Message-ID: <69639.66983.qm@web63101.mail.re1.yahoo.com>

Hi,
I am defining a function in file "trainer.py". It requires a module called numpy as following:

[code]
from numpy import *
import string

def trainer(train_seqs):

??? nul_dict = {"A":0,"C":1,"G":2,"T":3}# nucleotide to index of array
??? init_prob = zeros(4,double)# initial prob at? 1st col
??? tran_prob = zeros([5,4,4],double)# transition prob indexed by ( col, nul in previous col, nul in current col)
??? for line in train_seqs: # counts
??????? if line != '\n':
??????????? z = line.split()
??????????? z = z[0]
??????????? init_prob[nul_dict[z[0]]] += 1
??????????? for i in range(0,5):
??????????????? tran_prob[i][nul_dict[z[i]]][nul_dict[z[i+1]]] += 1
?? ?

??? init_prob /= init_prob.sum() # normalize transition initial prob
[/code]

Another script "controller.py" calls this function as:

[code]#! /usr/bin/env python
# from numpy import *
import string
import trainer

import pdb; pdb.set_trace()

ESE_file = file("./ESE.txt",'r')
ESE_seqs = ESE_file.readlines()
ESE_file.close()

ESE_model=trainer.trainer(ESE_seqs)????????? ?
[/code]

When I try to tun "controller.py", I get the error:
[quote]
Traceback (most recent call last):
? File "<stdin>", line 1, in <module>
? File "controller.py", line 12, in <module>
??? ESE_model=trainer.trainer(ESE_seqs)
? File "trainer.py", line 4, in trainer
?? ?
NameError: global name 'zeros' is not defined
[/quote]

I think zeros belongs to Module numpy, and I import it in my function file "trainer.py". So I was wonering what's the mistake I am making?? Thanks for help!

-Tim



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/89422a28/attachment.html>

From stefan at sun.ac.za  Sun Feb 22 18:28:20 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 23 Feb 2009 01:28:20 +0200
Subject: [Numpy-discussion] possible bug: __array_wrap__ is not called
	during arithmetic operations in some cases
In-Reply-To: <49A1DE17.4070001@hawaii.edu>
References: <a08e5f80902221451q7aaa220cua3c13072f1993398@mail.gmail.com>
	<49A1DE17.4070001@hawaii.edu>
Message-ID: <9457e7c80902221528q5fe34442jbbc7f25238943ec6@mail.gmail.com>

2009/2/23 Eric Firing <efiring at hawaii.edu>:
> Darren Dale wrote:
>> Does anyone know why __array_wrap__ is not called for subclasses during
>> arithmetic operations where an iterable like a list or tuple appears to
>> the right of the subclass? When I do "mine*[1,2,3]", array_wrap is not
>> called and I get an ndarray instead of a MyArray. "[1,2,3]*mine" is
>> fine, as is "mine*array([1,2,3])". I see the same issue with division,
>
> The masked array subclass does not show this behavior:

Maybe because it defines __mul__.  If I change Darren's class to include

    def __mul__(self, y):
        return y * self

It works as expected.

St?fan


From pgmdevlist at gmail.com  Sun Feb 22 18:28:56 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 22 Feb 2009 18:28:56 -0500
Subject: [Numpy-discussion] possible bug: __array_wrap__ is not called
	during arithmetic operations in some cases
In-Reply-To: <49A1DE17.4070001@hawaii.edu>
References: <a08e5f80902221451q7aaa220cua3c13072f1993398@mail.gmail.com>
	<49A1DE17.4070001@hawaii.edu>
Message-ID: <444F6AEF-9255-4350-AE1B-B9494A039D43@gmail.com>


On Feb 22, 2009, at 6:21 PM, Eric Firing wrote:

> Darren Dale wrote:
>> Does anyone know why __array_wrap__ is not called for subclasses  
>> during
>> arithmetic operations where an iterable like a list or tuple  
>> appears to
>> the right of the subclass? When I do "mine*[1,2,3]", array_wrap is  
>> not
>> called and I get an ndarray instead of a MyArray. "[1,2,3]*mine" is
>> fine, as is "mine*array([1,2,3])". I see the same issue with  
>> division,
>
> The masked array subclass does not show this behavior:

Because MaskedArray.__mul__ and others are redefined.

Darren, you can fix your problem by redefining MyArray.__mul__ as:

     def __mul__(self, other):
         return np.ndarray.__mul__(self, np.asanyarray(other))

forcing the second term to be a ndarray (or a subclass of). You can do  
the same thing for the other functions (__add__, __radd__, ...)


From dsdale24 at gmail.com  Sun Feb 22 18:35:47 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 22 Feb 2009 18:35:47 -0500
Subject: [Numpy-discussion] possible bug: __array_wrap__ is not called
	during arithmetic operations in some cases
In-Reply-To: <444F6AEF-9255-4350-AE1B-B9494A039D43@gmail.com>
References: <a08e5f80902221451q7aaa220cua3c13072f1993398@mail.gmail.com>
	<49A1DE17.4070001@hawaii.edu>
	<444F6AEF-9255-4350-AE1B-B9494A039D43@gmail.com>
Message-ID: <a08e5f80902221535l3db78a66j81967dd6cca8d3e5@mail.gmail.com>

On Sun, Feb 22, 2009 at 6:28 PM, Pierre GM <pgmdevlist at gmail.com> wrote:

>
> On Feb 22, 2009, at 6:21 PM, Eric Firing wrote:
>
> > Darren Dale wrote:
> >> Does anyone know why __array_wrap__ is not called for subclasses
> >> during
> >> arithmetic operations where an iterable like a list or tuple
> >> appears to
> >> the right of the subclass? When I do "mine*[1,2,3]", array_wrap is
> >> not
> >> called and I get an ndarray instead of a MyArray. "[1,2,3]*mine" is
> >> fine, as is "mine*array([1,2,3])". I see the same issue with
> >> division,
> >
> > The masked array subclass does not show this behavior:
>
> Because MaskedArray.__mul__ and others are redefined.
>
> Darren, you can fix your problem by redefining MyArray.__mul__ as:
>
>     def __mul__(self, other):
>         return np.ndarray.__mul__(self, np.asanyarray(other))
>
> forcing the second term to be a ndarray (or a subclass of). You can do
> the same thing for the other functions (__add__, __radd__, ...)


Thanks for the suggestion. I know this can be done, but ufuncs like
np.multiply(mine,[1,2,3]) will still not work. Plus, if I reimplement these
methods, I take some small performance hit. I've been putting a lot of work
in lately to get quantities to work with numpy's stock ufuncs.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/cf8fec54/attachment.html>

From stefan at sun.ac.za  Sun Feb 22 18:45:11 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 23 Feb 2009 01:45:11 +0200
Subject: [Numpy-discussion] Import NumPy in Self-defined function script
In-Reply-To: <69639.66983.qm@web63101.mail.re1.yahoo.com>
References: <69639.66983.qm@web63101.mail.re1.yahoo.com>
Message-ID: <9457e7c80902221545w72ae315dp3c575f7cc0319ba6@mail.gmail.com>

Hi Tim

2009/2/23 Tim <timlee126 at yahoo.com>:
> I think zeros belongs to Module numpy, and I import it in my function file
> "trainer.py". So I was wonering what's the mistake I am making?  Thanks for
> help!

What is the output of the following (please execute this inside Python
inside trainer.py's directory):

import numpy
print numpy
print numpy.__version__

Cheers
St?fan


From markus.rosenstihl at physik.tu-darmstadt.de  Sun Feb 22 18:47:41 2009
From: markus.rosenstihl at physik.tu-darmstadt.de (Markus Rosenstihl)
Date: Mon, 23 Feb 2009 00:47:41 +0100
Subject: [Numpy-discussion] Import NumPy in Self-defined function script
In-Reply-To: <69639.66983.qm@web63101.mail.re1.yahoo.com>
References: <69639.66983.qm@web63101.mail.re1.yahoo.com>
Message-ID: <30376402-EB69-4DF5-82C2-F2E2572E55C0@physik.tu-darmstadt.de>


Am 23.02.2009 um 00:19 schrieb Tim:

> Hi,
> I am defining a function in file "trainer.py". It requires a module  
> called numpy as following:
>
>
> Another script "controller.py" calls this function as:
>
> [code]#! /usr/bin/env python
> # from numpy import *
     ^ This is a comment, you are not importing numpy!

I think it is generally better to not do

from ... import *

because it messes up the namespace; it can happen that a second module  
overwrites a function from the first module.
This happens for example with:

from numpy import *
from pylab import *

pylab has a "load" function which is not the same as the numpy "load"  
function.

I always use
import numpy as N

Regards Markus


From gael.varoquaux at normalesup.org  Sun Feb 22 18:51:18 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 23 Feb 2009 00:51:18 +0100
Subject: [Numpy-discussion] Import NumPy in Self-defined function	script
In-Reply-To: <30376402-EB69-4DF5-82C2-F2E2572E55C0@physik.tu-darmstadt.de>
References: <69639.66983.qm@web63101.mail.re1.yahoo.com>
	<30376402-EB69-4DF5-82C2-F2E2572E55C0@physik.tu-darmstadt.de>
Message-ID: <20090222235118.GY6701@phare.normalesup.org>

On Mon, Feb 23, 2009 at 12:47:41AM +0100, Markus Rosenstihl wrote:

You are definitely right with your suggestion not to use 'from foo import *'
It took me a while to realise that the gain of explicite namespace was
more than the cost of a few characters, but I definitely stand by this.

> I always use
> import numpy as N

We try to encourage using 'import numpy as np'. I wasn't so exited to
push this convention on every body, but itis true that it is really
convenient to be able to edit a file, or copy-paste code, without
worrying how numpy is imported.

Cheers,

Ga?l


From dsdale24 at gmail.com  Sun Feb 22 18:51:20 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 22 Feb 2009 18:51:20 -0500
Subject: [Numpy-discussion] small suggestion for numpy.testing utils
In-Reply-To: <9457e7c80902221412t10e9a3f3pb5a20f1a80023e81@mail.gmail.com>
References: <a08e5f80902221217r50e76bcbk2586612e7c97e9a1@mail.gmail.com>
	<9457e7c80902221412t10e9a3f3pb5a20f1a80023e81@mail.gmail.com>
Message-ID: <a08e5f80902221551h29dbb81ened97cab189172758@mail.gmail.com>

On Sun, Feb 22, 2009 at 5:12 PM, St?fan van der Walt <stefan at sun.ac.za>wrote:

> Hi Darren
>
> 2009/2/22 Darren Dale <dsdale24 at gmail.com>:
> > I am using numpy's assert_array_equal and assert_array_almost_equal to
> unit
> > test my physical quantities package. I made a single minor change to
> > assert_array_compare that I think might make these functions more useful
> to
> > ndarray subclasses, and thought maybe they could be useful to numpy
> itself.
>
> Your patch makes good sense.  I applied it in r6457.  I'll keep an eye
> on the thread to see if anyone else has further comments.
>

Pierre just drew my attention to asanyarray, I think that would have been a
better choice for my patch.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/3f2bc1dd/attachment.html>

From strawman at astraw.com  Sun Feb 22 18:52:48 2009
From: strawman at astraw.com (Andrew Straw)
Date: Sun, 22 Feb 2009 15:52:48 -0800
Subject: [Numpy-discussion] small suggestion for numpy.testing utils
In-Reply-To: <a08e5f80902221238o257b5995x126c6588114fd7e0@mail.gmail.com>
References: <a08e5f80902221217r50e76bcbk2586612e7c97e9a1@mail.gmail.com>	<a08e5f80902221222j63a3ce21s74122a9d6932c0be@mail.gmail.com>
	<a08e5f80902221238o257b5995x126c6588114fd7e0@mail.gmail.com>
Message-ID: <49A1E550.6070002@astraw.com>

Darren,

What's the difference between asanyarray(y) and array(y, copy=False, 
subok=True)? I thought asanyarray would also do what you want.

-Andrew

Darren Dale wrote:
> On Sun, Feb 22, 2009 at 3:22 PM, Darren Dale <dsdale24 at gmail.com 
> <mailto:dsdale24 at gmail.com>> wrote:
>
>     On Sun, Feb 22, 2009 at 3:17 PM, Darren Dale <dsdale24 at gmail.com
>     <mailto:dsdale24 at gmail.com>> wrote:
>
>         Hello,
>
>         I am using numpy's assert_array_equal and
>         assert_array_almost_equal to unit test my physical quantities
>         package. I made a single minor change to assert_array_compare
>         that I think might make these functions more useful to ndarray
>         subclasses, and thought maybe they could be useful to numpy
>         itself. I tried applying this diff to numpy and running the
>         test suite, and instead of 9 known failures I got 1 known
>         failure, 11 skips, 2 errors and 2 failures. Perhaps it is
>         possible that by not forcing the input arrays to be ndarray
>         instances, some additional numpy features are exposed.
>
>         Thanks,
>         Darren
>
>         $ svn diff
>         Index: numpy/testing/utils.py
>         ===================================================================
>         --- numpy/testing/utils.py      (revision 6370)
>         +++ numpy/testing/utils.py      (working copy)
>         @@ -240,9 +240,9 @@
>
>          def assert_array_compare(comparison, x, y, err_msg='',
>         verbose=True,
>                                   header=''):
>         -    from numpy.core import asarray, isnan, any
>         -    x = asarray(x)
>         -    y = asarray(y)
>         +    from numpy.core import array, isnan, any
>         +    x = array(x, copy=False, subok=True)
>         +    y = array(y, copy=False, subok=True)
>
>              def isnumber(x):
>                  return x.dtype.char in '?bhilqpBHILQPfdgFDG'
>
>
>     Actually, my svn checkout was not up to date. With this patch
>     applied, I get 1 known failure and 11 skips.
>
>
> I just double checked and I think I get the same results running the 
> svn 6456 test suite with and without this patch applied. I tried 
> posting an enhancement request at the trac website, but I cant file 
> the ticket because I get "500 Internal Server Error", so I'm posting 
> it here.
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From dsdale24 at gmail.com  Sun Feb 22 18:55:49 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 22 Feb 2009 18:55:49 -0500
Subject: [Numpy-discussion] small suggestion for numpy.testing utils
In-Reply-To: <49A1E550.6070002@astraw.com>
References: <a08e5f80902221217r50e76bcbk2586612e7c97e9a1@mail.gmail.com>
	<a08e5f80902221222j63a3ce21s74122a9d6932c0be@mail.gmail.com>
	<a08e5f80902221238o257b5995x126c6588114fd7e0@mail.gmail.com>
	<49A1E550.6070002@astraw.com>
Message-ID: <a08e5f80902221555j164d4b8qdd070bef3011f50d@mail.gmail.com>

I think they are identical, its just that asanyarray appears to be targeted
for exactly this use-case, so perhaps it is a little faster. I just posted
that asanyarray would probably have been a better choice, the posts must
have crossed.

On Sun, Feb 22, 2009 at 6:52 PM, Andrew Straw <strawman at astraw.com> wrote:

> Darren,
>
> What's the difference between asanyarray(y) and array(y, copy=False,
> subok=True)? I thought asanyarray would also do what you want.
>
> -Andrew
>
> Darren Dale wrote:
> > On Sun, Feb 22, 2009 at 3:22 PM, Darren Dale <dsdale24 at gmail.com
> > <mailto:dsdale24 at gmail.com>> wrote:
> >
> >     On Sun, Feb 22, 2009 at 3:17 PM, Darren Dale <dsdale24 at gmail.com
> >     <mailto:dsdale24 at gmail.com>> wrote:
> >
> >         Hello,
> >
> >         I am using numpy's assert_array_equal and
> >         assert_array_almost_equal to unit test my physical quantities
> >         package. I made a single minor change to assert_array_compare
> >         that I think might make these functions more useful to ndarray
> >         subclasses, and thought maybe they could be useful to numpy
> >         itself. I tried applying this diff to numpy and running the
> >         test suite, and instead of 9 known failures I got 1 known
> >         failure, 11 skips, 2 errors and 2 failures. Perhaps it is
> >         possible that by not forcing the input arrays to be ndarray
> >         instances, some additional numpy features are exposed.
> >
> >         Thanks,
> >         Darren
> >
> >         $ svn diff
> >         Index: numpy/testing/utils.py
> >
> ===================================================================
> >         --- numpy/testing/utils.py      (revision 6370)
> >         +++ numpy/testing/utils.py      (working copy)
> >         @@ -240,9 +240,9 @@
> >
> >          def assert_array_compare(comparison, x, y, err_msg='',
> >         verbose=True,
> >                                   header=''):
> >         -    from numpy.core import asarray, isnan, any
> >         -    x = asarray(x)
> >         -    y = asarray(y)
> >         +    from numpy.core import array, isnan, any
> >         +    x = array(x, copy=False, subok=True)
> >         +    y = array(y, copy=False, subok=True)
> >
> >              def isnumber(x):
> >                  return x.dtype.char in '?bhilqpBHILQPfdgFDG'
> >
> >
> >     Actually, my svn checkout was not up to date. With this patch
> >     applied, I get 1 known failure and 11 skips.
> >
> >
> > I just double checked and I think I get the same results running the
> > svn 6456 test suite with and without this patch applied. I tried
> > posting an enhancement request at the trac website, but I cant file
> > the ticket because I get "500 Internal Server Error", so I'm posting
> > it here.
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/5a8dbad7/attachment.html>

From stefan at sun.ac.za  Sun Feb 22 18:59:25 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 23 Feb 2009 01:59:25 +0200
Subject: [Numpy-discussion] Import NumPy in Self-defined function script
In-Reply-To: <30376402-EB69-4DF5-82C2-F2E2572E55C0@physik.tu-darmstadt.de>
References: <69639.66983.qm@web63101.mail.re1.yahoo.com>
	<30376402-EB69-4DF5-82C2-F2E2572E55C0@physik.tu-darmstadt.de>
Message-ID: <9457e7c80902221559s7ab5c70t5c4bab243f91e026@mail.gmail.com>

2009/2/23 Markus Rosenstihl <markus.rosenstihl at physik.tu-darmstadt.de>:
>> Another script "controller.py" calls this function as:
>>
>> [code]#! /usr/bin/env python
>> # from numpy import *
>     ^ This is a comment, you are not importing numpy!

But he is also not using numpy in controller.

Cheers
St?fan


From dsdale24 at gmail.com  Sun Feb 22 19:01:44 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 22 Feb 2009 19:01:44 -0500
Subject: [Numpy-discussion] possible bug: __array_wrap__ is not called
	during arithmetic operations in some cases
In-Reply-To: <a08e5f80902221535l3db78a66j81967dd6cca8d3e5@mail.gmail.com>
References: <a08e5f80902221451q7aaa220cua3c13072f1993398@mail.gmail.com>
	<49A1DE17.4070001@hawaii.edu>
	<444F6AEF-9255-4350-AE1B-B9494A039D43@gmail.com>
	<a08e5f80902221535l3db78a66j81967dd6cca8d3e5@mail.gmail.com>
Message-ID: <a08e5f80902221601i5efb179sd0315553b4f0fe07@mail.gmail.com>

On Sun, Feb 22, 2009 at 6:35 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> On Sun, Feb 22, 2009 at 6:28 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
>
>>
>> On Feb 22, 2009, at 6:21 PM, Eric Firing wrote:
>>
>> > Darren Dale wrote:
>> >> Does anyone know why __array_wrap__ is not called for subclasses
>> >> during
>> >> arithmetic operations where an iterable like a list or tuple
>> >> appears to
>> >> the right of the subclass? When I do "mine*[1,2,3]", array_wrap is
>> >> not
>> >> called and I get an ndarray instead of a MyArray. "[1,2,3]*mine" is
>> >> fine, as is "mine*array([1,2,3])". I see the same issue with
>> >> division,
>> >
>> > The masked array subclass does not show this behavior:
>>
>> Because MaskedArray.__mul__ and others are redefined.
>>
>> Darren, you can fix your problem by redefining MyArray.__mul__ as:
>>
>>     def __mul__(self, other):
>>         return np.ndarray.__mul__(self, np.asanyarray(other))
>>
>> forcing the second term to be a ndarray (or a subclass of). You can do
>> the same thing for the other functions (__add__, __radd__, ...)
>
>
> Thanks for the suggestion. I know this can be done, but ufuncs like
> np.multiply(mine,[1,2,3]) will still not work. Plus, if I reimplement these
> methods, I take some small performance hit. I've been putting a lot of work
> in lately to get quantities to work with numpy's stock ufuncs.
>

I should point out:

import numpy as np

a=np.array([1,2,3,4])
b=np.ma.masked_where(a>2,a)
np.multiply([1,2,3,4],b) # yields a masked array
np.multiply(b,[1,2,3,4]) # yields an ndarray
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/0869bfbb/attachment.html>

From stefan at sun.ac.za  Sun Feb 22 19:02:25 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 23 Feb 2009 02:02:25 +0200
Subject: [Numpy-discussion] small suggestion for numpy.testing utils
In-Reply-To: <a08e5f80902221551h29dbb81ened97cab189172758@mail.gmail.com>
References: <a08e5f80902221217r50e76bcbk2586612e7c97e9a1@mail.gmail.com>
	<9457e7c80902221412t10e9a3f3pb5a20f1a80023e81@mail.gmail.com>
	<a08e5f80902221551h29dbb81ened97cab189172758@mail.gmail.com>
Message-ID: <9457e7c80902221602g792da010jfd389bebf75474d7@mail.gmail.com>

2009/2/23 Darren Dale <dsdale24 at gmail.com>:
> On Sun, Feb 22, 2009 at 5:12 PM, St?fan van der Walt <stefan at sun.ac.za>
> wrote:
>>
>> Hi Darren
>>
>> 2009/2/22 Darren Dale <dsdale24 at gmail.com>:
>> > I am using numpy's assert_array_equal and assert_array_almost_equal to
>> > unit
>> > test my physical quantities package. I made a single minor change to
>> > assert_array_compare that I think might make these functions more useful
>> > to
>> > ndarray subclasses, and thought maybe they could be useful to numpy
>> > itself.
>>
>> Your patch makes good sense.  I applied it in r6457.  I'll keep an eye
>> on the thread to see if anyone else has further comments.
>
> Pierre just drew my attention to asanyarray, I think that would have been a
> better choice for my patch.

Asanyarray is just

    return array(a, dtype, copy=False, order=order, subok=True)

with default values for dtype=None and order=None.  I think your
original suggestion more clearly shows what the function is meant to
achieve.

Cheers
St?fan


From markus.rosenstihl at physik.tu-darmstadt.de  Sun Feb 22 19:05:47 2009
From: markus.rosenstihl at physik.tu-darmstadt.de (Markus Rosenstihl)
Date: Mon, 23 Feb 2009 01:05:47 +0100
Subject: [Numpy-discussion] Import NumPy in Self-defined function script
In-Reply-To: <9457e7c80902221559s7ab5c70t5c4bab243f91e026@mail.gmail.com>
References: <69639.66983.qm@web63101.mail.re1.yahoo.com>
	<30376402-EB69-4DF5-82C2-F2E2572E55C0@physik.tu-darmstadt.de>
	<9457e7c80902221559s7ab5c70t5c4bab243f91e026@mail.gmail.com>
Message-ID: <9F54075E-3868-41CE-BCB6-2091DB9C8113@physik.tu-darmstadt.de>

Am 23.02.2009 um 00:59 schrieb St?fan van der Walt:

> 2009/2/23 Markus Rosenstihl <markus.rosenstihl at physik.tu- 
> darmstadt.de>:
>>> Another script "controller.py" calls this function as:
>>>
>>> [code]#! /usr/bin/env python
>>> # from numpy import *
>>    ^ This is a comment, you are not importing numpy!
>
> But he is also not using numpy in controller.

Yes, the moment I sent the mail I realised that it was not the  
culprit. In fact, on my Mac,
the program runs just fine (considering I use an arbitrary file): it  
does not complain about "zeros"





From xavier.gnata at gmail.com  Sun Feb 22 20:39:34 2009
From: xavier.gnata at gmail.com (Xavier Gnata)
Date: Mon, 23 Feb 2009 02:39:34 +0100
Subject: [Numpy-discussion] block matrix and sums of blocks
Message-ID: <49A1FE56.3070505@gmail.com>

Hi,

Let us consider one kN x kM array.
What is the fastest way to sum each k x k square block of A and to put 
all these results into a NxM array B?

For instance:
If A =
[112233
 112233
 223311
 223311]
then B =
[4 8 12
 4 12 4]

No sanity checks on the arrays shapes are requiered. Only speed matters ;)

Xavier


From charlesr.harris at gmail.com  Sun Feb 22 20:51:48 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 22 Feb 2009 18:51:48 -0700
Subject: [Numpy-discussion] block matrix and sums of blocks
In-Reply-To: <49A1FE56.3070505@gmail.com>
References: <49A1FE56.3070505@gmail.com>
Message-ID: <e06186140902221751y5f67fb7bp3db801c52aaa17b5@mail.gmail.com>

On Sun, Feb 22, 2009 at 6:39 PM, Xavier Gnata <xavier.gnata at gmail.com>wrote:

> Hi,
>
> Let us consider one kN x kM array.
> What is the fastest way to sum each k x k square block of A and to put
> all these results into a NxM array B?
>
> For instance:
> If A =
> [112233
>  112233
>  223311
>  223311]
> then B =
> [4 8 12
>  4 12 4]
>
> No sanity checks on the arrays shapes are requiered. Only speed matters ;)
>
> Xavier
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/86b6454f/attachment.html>

From charlesr.harris at gmail.com  Sun Feb 22 20:55:39 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 22 Feb 2009 18:55:39 -0700
Subject: [Numpy-discussion] block matrix and sums of blocks
In-Reply-To: <49A1FE56.3070505@gmail.com>
References: <49A1FE56.3070505@gmail.com>
Message-ID: <e06186140902221755g72283d91y8ddc93389fd9fdfe@mail.gmail.com>

On Sun, Feb 22, 2009 at 6:39 PM, Xavier Gnata <xavier.gnata at gmail.com>wrote:

> Hi,
>
> Let us consider one kN x kM array.
> What is the fastest way to sum each k x k square block of A and to put
> all these results into a NxM array B?
>
> For instance:
> If A =
> [112233
>  112233
>  223311
>  223311]
> then B =
> [4 8 12
>  4 12 4]
>
> No sanity checks on the arrays shapes are requiered. Only speed matters ;)
>

An example with real numpy arrays would help ;) But basically you will need
to reshape the matrix and sum along the last axis.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/441d9ffd/attachment.html>

From dsdale24 at gmail.com  Sun Feb 22 22:35:41 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 22 Feb 2009 22:35:41 -0500
Subject: [Numpy-discussion] numpy.fix and subclasses
Message-ID: <a08e5f80902221935v62a415b5xb4b0776df5b0b16c@mail.gmail.com>

I've been finding some numpy functions that could maybe be improved to work
better with ndarray subclasses. For example:

def fix(x, y=None):
    x = nx.asanyarray(x)
    if y is None:
        y = nx.zeros_like(x)
    y1 = nx.floor(x)
    y2 = nx.ceil(x)
    y[...] = nx.where(x >= 0, y1, y2)
    return y

This implementation is a problematic for subclasses, since it does not allow
metadata to propagate using the usual ufunc machinery of __array_wrap__,
like ceil and floor do. nx.zeros_like does yield another instance of
type(x), but y does not get x's metadata (such as units or a mask). Would it
be possible to do something like:

    if y is None:
        y = x*0

"where" is another function that could maybe be improved to work with the
rules established by array_priority, but I'm a lousy C programmer and I
haven't actually looked into how this would work. If "where" respected
array_priority, fix could be implemented as:

def fix(x, y=None):
    x = nx.asanyarray(x)
    y1 = nx.floor(x)
    y2 = nx.ceil(x)
    if y is None:
        return nx.where(x >= 0, y1, y2)
    y[...] = nx.where(x >= 0, y1, y2)
    return y

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/39e627c6/attachment.html>

From dsdale24 at gmail.com  Sun Feb 22 22:49:33 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 22 Feb 2009 22:49:33 -0500
Subject: [Numpy-discussion] numpy.fix and subclasses
In-Reply-To: <a08e5f80902221935v62a415b5xb4b0776df5b0b16c@mail.gmail.com>
References: <a08e5f80902221935v62a415b5xb4b0776df5b0b16c@mail.gmail.com>
Message-ID: <a08e5f80902221949x1f08f11q11c7286497ab1596@mail.gmail.com>

On Sun, Feb 22, 2009 at 10:35 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> I've been finding some numpy functions that could maybe be improved to work
> better with ndarray subclasses. For example:
>
> def fix(x, y=None):
>     x = nx.asanyarray(x)
>     if y is None:
>         y = nx.zeros_like(x)
>     y1 = nx.floor(x)
>     y2 = nx.ceil(x)
>     y[...] = nx.where(x >= 0, y1, y2)
>     return y
>
> This implementation is a problematic for subclasses, since it does not
> allow metadata to propagate using the usual ufunc machinery of
> __array_wrap__, like ceil and floor do. nx.zeros_like does yield another
> instance of type(x), but y does not get x's metadata (such as units or a
> mask). Would it be possible to do something like:
>
>     if y is None:
>         y = x*0
>
> "where" is another function that could maybe be improved to work with the
> rules established by array_priority, but I'm a lousy C programmer and I
> haven't actually looked into how this would work. If "where" respected
> array_priority, fix could be implemented as:
>
> def fix(x, y=None):
>     x = nx.asanyarray(x)
>     y1 = nx.floor(x)
>     y2 = nx.ceil(x)
>     if y is None:
>         return nx.where(x >= 0, y1, y2)
>     y[...] = nx.where(x >= 0, y1, y2)
>     return y


Actually, I just remembered that quantities tries to prevent things like
([1,2,3,4]*m)[:2] = [0,1], since the units dont match, so setting y=x*0 and
then setting data to a slice of y would be problematic. It would be most
desirable for "where" to respect __array_priority__, if possible. Any
comments?

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090222/585617d8/attachment.html>

From robert.kern at gmail.com  Sun Feb 22 23:26:35 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 22 Feb 2009 22:26:35 -0600
Subject: [Numpy-discussion] block matrix and sums of blocks
In-Reply-To: <49A1FE56.3070505@gmail.com>
References: <49A1FE56.3070505@gmail.com>
Message-ID: <3d375d730902222026r1ea7ef31y2ecba426b11564e1@mail.gmail.com>

On Sun, Feb 22, 2009 at 19:39, Xavier Gnata <xavier.gnata at gmail.com> wrote:
> Hi,
>
> Let us consider one kN x kM array.
> What is the fastest way to sum each k x k square block of A and to put
> all these results into a NxM array B?
>
> For instance:
> If A =
> [112233
>  112233
>  223311
>  223311]
> then B =
> [4 8 12
>  4 12 4]
>
> No sanity checks on the arrays shapes are requiered. Only speed matters ;)

In [6]: A
Out[6]:
array([[1, 1, 2, 2, 3, 3],
       [1, 1, 2, 2, 3, 3],
       [2, 2, 3, 3, 1, 1],
       [2, 2, 3, 3, 1, 1]])

In [7]: k = 2

In [8]: add.reduceat(add.reduceat(A, arange(0, A.shape[0], k),
axis=0), arange(0, A.shape[1], k), axis=1)
Out[8]:
array([[ 4,  8, 12],
       [ 8, 12,  4]])

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From pinto at mit.edu  Mon Feb 23 07:11:49 2009
From: pinto at mit.edu (Nicolas Pinto)
Date: Mon, 23 Feb 2009 07:11:49 -0500
Subject: [Numpy-discussion] How to generate equivalent "random" numbers in
	matlab and numpy?
Message-ID: <954ae5aa0902230411q60cd26b8qb717ba00b35ab32@mail.gmail.com>

Dear all,

I'd like to generate equivalent sequences of 'random' numbers in matlab and
numpy, is there any way I can do that? I tried to fix the seed (see below)
but it doesn't work.

# numpy
In [29]: np.random.seed(1); np.random.permutation(5)+1
Out[29]: array([3, 2, 5, 1, 4])

% matlab
>> rand('seed', 1); randperm(5)

ans =

     4     3     5     2     1


Thanks for your time.

Best regards,

-- 
Nicolas Pinto
Ph.D. Candidate, Brain & Computer Sciences
Massachusetts Institute of Technology, USA
http://web.mit.edu/pinto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090223/43e0b23a/attachment.html>

From sturla at molden.no  Mon Feb 23 07:51:25 2009
From: sturla at molden.no (Sturla Molden)
Date: Mon, 23 Feb 2009 13:51:25 +0100
Subject: [Numpy-discussion] How to generate equivalent "random" numbers
 in	matlab and numpy?
In-Reply-To: <954ae5aa0902230411q60cd26b8qb717ba00b35ab32@mail.gmail.com>
References: <954ae5aa0902230411q60cd26b8qb717ba00b35ab32@mail.gmail.com>
Message-ID: <49A29BCD.2090307@molden.no>





On 2/23/2009 1:11 PM, Nicolas Pinto wrote:
> Dear all,
> 
> I'd like to generate equivalent sequences of 'random' numbers in matlab 
> and numpy, is there any way I can do that? ...

Asked and answered on scipy-user.


S.M.


From timlee126 at yahoo.com  Mon Feb 23 09:35:11 2009
From: timlee126 at yahoo.com (Tim)
Date: Mon, 23 Feb 2009 06:35:11 -0800 (PST)
Subject: [Numpy-discussion] Import NumPy in Self-defined function script
In-Reply-To: <9457e7c80902221545w72ae315dp3c575f7cc0319ba6@mail.gmail.com>
Message-ID: <554610.57784.qm@web63101.mail.re1.yahoo.com>

Hi,
My Python is Python 2.5.2 (r252:60911, Oct? 5 2008, 19:24:49)
Here is the output:
>>> print numpy
<module 'numpy' from '/usr/lib/python2.5/site-packages/numpy/__init__.pyc'>
>>> print numpy.__version__
1.1.1
Are these compatible with each other?
I just found the error goes away, after I restart my ubuntu 8.10.
Thanks,
-Tim



--- On Sun, 2/22/09, St?fan van der Walt <stefan at sun.ac.za> wrote:
From: St?fan van der Walt <stefan at sun.ac.za>
Subject: Re: [Numpy-discussion] Import NumPy in Self-defined function script
To: "Discussion of Numerical Python" <numpy-discussion at scipy.org>
Date: Sunday, February 22, 2009, 6:45 PM

Hi Tim

2009/2/23 Tim <timlee126 at yahoo.com>:
> I think zeros belongs to Module numpy, and I import it in my function file
> "trainer.py". So I was wonering what's the mistake I am
making?  Thanks for
> help!

What is the output of the following (please execute this inside Python
inside trainer.py's directory):

import numpy
print numpy
print numpy.__version__

Cheers
St?fan
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090223/24ff7c30/attachment.html>

From mdroe at stsci.edu  Mon Feb 23 10:00:03 2009
From: mdroe at stsci.edu (Michael Droettboom)
Date: Mon, 23 Feb 2009 10:00:03 -0500
Subject: [Numpy-discussion] [matplotlib-devel] [Nipy-devel] Sphinx
 custom extension mess, and patches
In-Reply-To: <db6b5ecc0902201702u3c843ae3ga7f24ad81ec7aeb0@mail.gmail.com>
References: <20090215140400.GH20403@phare.normalesup.org>	<4999F3FD.4020806@python.org>	<20090216232106.GC24472@phare.normalesup.org>
	<db6b5ecc0902201702u3c843ae3ga7f24ad81ec7aeb0@mail.gmail.com>
Message-ID: <49A2B9F3.4070703@stsci.edu>

Thanks, Fernando.  I've applied your patch to matplotlib (branch and trunk).

Mike

Fernando Perez wrote:
> On Mon, Feb 16, 2009 at 3:21 PM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
>
>   
>> I am not blaming anyone, just pointing out a non ideal situation. It has
>> already improved a lot with the matplotlib guys and the scipy guys
>> merging some changes in extensions and publishing the extensions in an
>> importable part of their source tree.
>>     
>
> In keeping with the spirit of trying to get all of these extension
> changes upstream so that we can all eventually stop carrying our own
> copies, below is a tiny change I just made to the inheritance diagram
> one.  This is needed to ensure that the figure is separated from any
> surrounding text, since otherwise you get hideous off-screen diagrams
> in the rendered PDF.
>
> This has been committed to the nipy trunk already.
>
> Similarly (for the pymvpa crowd), the api autogen code is now a
> module, and it also contains a few small fixes, in particular
> regarding chapter titles.  Feel free to grab and update your copy:
>
> http://bazaar.launchpad.net/~nipy-developers/nipy/trunk/annotate/head%3A/tools/apigen.py
>
> I've been told the gods of numpy/sphinx don't like auto-generated
> docs, but I think there's a valid use case for these tools, so
> hopefully in the future it will be possible to include them upstream
> for us lesser mortals to use.  If not, I guess we'll just continue to
> carry our copies around :)
>
> Cheers,
>
> f
>
> # diff, inline because it's so trivial:
>
> === modified file 'doc/sphinxext/inheritance_diagram.py'
> --- doc/sphinxext/inheritance_diagram.py	2009-01-30 02:00:57 +0000
> +++ doc/sphinxext/inheritance_diagram.py	2009-02-20 21:11:38 +0000
> @@ -370,7 +370,7 @@
>
>      graph.run_dot(['-Tpdf', '-o%s' % pdf_path],
>                    name, parts, graph_options={'size': '"6.0,6.0"'})
> -    return '\\includegraphics{%s}' % pdf_path
> +    return '\n\\includegraphics{%s}\n\n' % pdf_path
>
>  def visit_inheritance_diagram(inner_func):
>      """
>
> ------------------------------------------------------------------------------
> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
> -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
> -Strategies to boost innovation and cut costs with open source participation
> -Receive a $600 discount off the registration fee with the source code: SFAD
> http://p.sf.net/sfu/XcvMzF8H
> _______________________________________________
> Matplotlib-devel mailing list
> Matplotlib-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>   

-- 
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA



From xavier.gnata at gmail.com  Mon Feb 23 15:06:06 2009
From: xavier.gnata at gmail.com (Xavier Gnata)
Date: Mon, 23 Feb 2009 21:06:06 +0100
Subject: [Numpy-discussion] block matrix and sums of blocks
In-Reply-To: <3d375d730902222026r1ea7ef31y2ecba426b11564e1@mail.gmail.com>
References: <49A1FE56.3070505@gmail.com>
	<3d375d730902222026r1ea7ef31y2ecba426b11564e1@mail.gmail.com>
Message-ID: <49A301AE.9010203@gmail.com>

Robert Kern wrote:
> On Sun, Feb 22, 2009 at 19:39, Xavier Gnata <xavier.gnata at gmail.com> wrote:
>   
>> Hi,
>>
>> Let us consider one kN x kM array.
>> What is the fastest way to sum each k x k square block of A and to put
>> all these results into a NxM array B?
>>
>> For instance:
>> If A =
>> [112233
>>  112233
>>  223311
>>  223311]
>> then B =
>> [4 8 12
>>  4 12 4]
>>
>> No sanity checks on the arrays shapes are requiered. Only speed matters ;)
>>     
>
> In [6]: A
> Out[6]:
> array([[1, 1, 2, 2, 3, 3],
>        [1, 1, 2, 2, 3, 3],
>        [2, 2, 3, 3, 1, 1],
>        [2, 2, 3, 3, 1, 1]])
>
> In [7]: k = 2
>
> In [8]: add.reduceat(add.reduceat(A, arange(0, A.shape[0], k),
> axis=0), arange(0, A.shape[1], k), axis=1)
> Out[8]:
> array([[ 4,  8, 12],
>        [ 8, 12,  4]])
>
>   
It does the job and I have learned .reduceat :)

Thanks,
Xavier


From cournape at gmail.com  Mon Feb 23 16:02:23 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 24 Feb 2009 06:02:23 +0900
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <gnlpfb$hih$1@ger.gmane.org>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org>
	<5b8d13220902190805s2d12ac96va40b6062959fc53@mail.gmail.com>
	<gnlpfb$hih$1@ger.gmane.org>
Message-ID: <5b8d13220902231302x42ede056i63a72ce968da9c8b@mail.gmail.com>

On Fri, Feb 20, 2009 at 5:26 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Fri, 20 Feb 2009 01:05:03 +0900, David Cournapeau wrote:
> [clip]
>>> I think they should be. Then we could easily use C99 complex math
>>> functions on plaforms on which they are available (and so get the
>>> "correct" corner-case semantics for free on these platforms).
>>
>> maybe we could change the names, then ? nc is not very clear IMHO (and
>> since they were static up to now, we are free to change them I believe).
>
> I think it would make sense to change them to follow C99 function names,
> with a npy_ prefix.

The problem of complex functions is that they don't follow the C99
conventions at all. IN particular, they accept pointers instead of
values. I don't know whether the rational for using pointers is still
valid (it is mentioned that how to pass structure is compiler
dependent - I would guess this is true mainly for fortran, I would
expect the C ABI of every platform to fix those kind of issues ?)

Using C99 names for functions with different prototypes may be confusing,

cheers,

David


From charlesr.harris at gmail.com  Mon Feb 23 17:01:41 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 23 Feb 2009 15:01:41 -0700
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <5b8d13220902231302x42ede056i63a72ce968da9c8b@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org>
	<5b8d13220902190805s2d12ac96va40b6062959fc53@mail.gmail.com>
	<gnlpfb$hih$1@ger.gmane.org>
	<5b8d13220902231302x42ede056i63a72ce968da9c8b@mail.gmail.com>
Message-ID: <e06186140902231401o5722ed1br48667a863904dd21@mail.gmail.com>

On Mon, Feb 23, 2009 at 2:02 PM, David Cournapeau <cournape at gmail.com>wrote:

> On Fri, Feb 20, 2009 at 5:26 PM, Pauli Virtanen <pav at iki.fi> wrote:
> > Fri, 20 Feb 2009 01:05:03 +0900, David Cournapeau wrote:
> > [clip]
> >>> I think they should be. Then we could easily use C99 complex math
> >>> functions on plaforms on which they are available (and so get the
> >>> "correct" corner-case semantics for free on these platforms).
> >>
> >> maybe we could change the names, then ? nc is not very clear IMHO (and
> >> since they were static up to now, we are free to change them I believe).
> >
> > I think it would make sense to change them to follow C99 function names,
> > with a npy_ prefix.
>
> The problem of complex functions is that they don't follow the C99
> conventions at all. IN particular, they accept pointers instead of
> values. I don't know whether the rational for using pointers is still
> valid (it is mentioned that how to pass structure is compiler


The usual rational is that it is more efficient to pass a pointer than to
push two floats on the stack. It is also an easier way to return values,
although recent versions of gcc do a good job of copying the return values
where they need to go. I would stick with the pointers, although we could
probably dispense with the structure and just use a pointer to the
underlying type with the assumption that the real & imaginary parts are
contiguous in memory. The ufuncs make that assumption in any case.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090223/a766cffd/attachment.html>

From washakie at gmail.com  Mon Feb 23 17:31:45 2009
From: washakie at gmail.com (John [H2O])
Date: Mon, 23 Feb 2009 14:31:45 -0800 (PST)
Subject: [Numpy-discussion]  probably simple, reverse a reshape...
Message-ID: <22171584.post@talk.nabble.com>


Hello, this is probably one of those questions that is going to seem simple
after reading responses...

I'm trying to get the original array indices out of a row number from a
reshaped array... shouldn't this be possible somehow? 

import numpy as np
a = np.ones((3,5,10))
#for testing, I've done the following:
a[0,:,:5] = np.eye(5,5)*1
a[1,:,:5] = np.eye(5,5)*10
a[2,:,:5] = np.eye(5,5)*20

#Ok, so I reshape the array:
b = a.reshape(15,10)

#now I want to be able to 'back out' the original shape indices from the row
number of the new array.

#I thought I could do something like this:
d0,d1,d2 = a.shape
new_d0 = row/d1
new_d1 = row/d0

#But it doesn't quite work... any suggestions as to what I am missing?

-- 
View this message in context: http://www.nabble.com/probably-simple%2C-reverse-a-reshape...-tp22171584p22171584.html
Sent from the Numpy-discussion mailing list archive at Nabble.com.



From washakie at gmail.com  Mon Feb 23 17:41:42 2009
From: washakie at gmail.com (John [H2O])
Date: Mon, 23 Feb 2009 14:41:42 -0800 (PST)
Subject: [Numpy-discussion] probably simple, reverse a reshape...
In-Reply-To: <22171584.post@talk.nabble.com>
References: <22171584.post@talk.nabble.com>
Message-ID: <22171790.post@talk.nabble.com>




John [H2O] wrote:
> 
> Hello, this is probably one of those questions that is going to seem
> simple after reading responses...
> 

and a few more minutes of thinking:
def row2shape(row,a):
      """ to get indices a 2d array row# to the 3d array 
      from which it was reshaped. ""
      d0,d1,d2 = a.shape;
      nd0 = row/d1;
      nd1 = (row-(nd0*d1));
      return nd0,nd1

-- 
View this message in context: http://www.nabble.com/probably-simple%2C-reverse-a-reshape...-tp22171584p22171790.html
Sent from the Numpy-discussion mailing list archive at Nabble.com.



From ndbecker2 at gmail.com  Tue Feb 24 09:04:42 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 24 Feb 2009 14:04:42 +0000 (UTC)
Subject: [Numpy-discussion] Core math library in numpy
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org>
Message-ID: <loom.20090224T140326-52@post.gmane.org>

Pauli Virtanen <pav <at> iki.fi> writes:

...
> One question: doesn't this add one extra function call to all umath 
> functions? Could we do '#define npy_XXX XXX' in the npy_math.h header 
> when the appropriate platform-specified functions are available?

There shouldn't be overhead on modern compilers for simple functions.  On gcc
use __inline__ should eliminate overhead (also, -O3 should automatically
inline).




From ralph at dont-mind.de  Tue Feb 24 09:45:39 2009
From: ralph at dont-mind.de (Ralph Heinkel)
Date: Tue, 24 Feb 2009 15:45:39 +0100
Subject: [Numpy-discussion] Creating arrays with 'titles' in dtype causes
	TypeError on data access
Message-ID: <49A40813.4090206@dont-mind.de>

Hi,

I'm trying to use the additional 'title' feature of dtypes when
creating arrays. The reason for using titles is that I want to store
meta data about the columns in them (so do-not-use-titles is not the
right solution for me...)

Everything works fine without titles:

>>> arr = array([('john', 4),('mark', 3)],
           dtype=[('name','O'),('id',int)])
>>> arr[0]
('john', 4)
>>> arr[0][0]
'john'


However here something is strange:

>>> arr = array([('john', 4),('mark', 3)],
     dtype=[(('source:yy', 'name'),'O'),(('source:xx','id'),int)])
>>> arr[0]
('john', 4)
>>> arr[0][0]
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: function takes at most 2 arguments (3 given)


Any ideas what I'm doing wrong? Any help would be appreciated.

Thanks,

Ralph


Using:
- Python V 2.6 (but same error with python 2.5)
- Numpy 1.2.1  (but same error on numpy 1.0.3.1)
- opensuse 10.3



From pav at iki.fi  Tue Feb 24 12:12:37 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 24 Feb 2009 17:12:37 +0000 (UTC)
Subject: [Numpy-discussion] Core math library in numpy
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org> <loom.20090224T140326-52@post.gmane.org>
Message-ID: <go19q5$f5u$1@ger.gmane.org>

Tue, 24 Feb 2009 14:04:42 +0000, Neal Becker wrote:
> Pauli Virtanen <pav <at> iki.fi> writes:
> 
> ...
>> One question: doesn't this add one extra function call to all umath
>> functions? Could we do '#define npy_XXX XXX' in the npy_math.h header
>> when the appropriate platform-specified functions are available?
> 
> There shouldn't be overhead on modern compilers for simple functions. 
> On gcc use __inline__ should eliminate overhead (also, -O3 should
> automatically inline).

Doesn't this require that the whole function body (not just the 
prototype) be included to the file using it? Or do current *linkers* do 
inlining? The core math library is compiled to a separate .o file, as the 
point is exactly to get rid of including c-files.

-- 
Pauli Virtanen



From charlesr.harris at gmail.com  Tue Feb 24 12:21:42 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 24 Feb 2009 10:21:42 -0700
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <loom.20090224T140326-52@post.gmane.org>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org> <loom.20090224T140326-52@post.gmane.org>
Message-ID: <e06186140902240921p3ba383elc40494863a40e832@mail.gmail.com>

On Tue, Feb 24, 2009 at 7:04 AM, Neal Becker <ndbecker2 at gmail.com> wrote:

> Pauli Virtanen <pav <at> iki.fi> writes:
>
> ...
> > One question: doesn't this add one extra function call to all umath
> > functions? Could we do '#define npy_XXX XXX' in the npy_math.h header
> > when the appropriate platform-specified functions are available?
>
> There shouldn't be overhead on modern compilers for simple functions.  On
> gcc
> use __inline__ should eliminate overhead (also, -O3 should automatically
> inline).
>

In order to inline, the function definition would need to be in the header
and a library version would still be required for passing functions by
address or in case the compiler decided *not* to inline.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090224/6d31910b/attachment.html>

From cournape at gmail.com  Tue Feb 24 12:37:31 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 25 Feb 2009 02:37:31 +0900
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <e06186140902240921p3ba383elc40494863a40e832@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org> <loom.20090224T140326-52@post.gmane.org>
	<e06186140902240921p3ba383elc40494863a40e832@mail.gmail.com>
Message-ID: <5b8d13220902240937s70617f9dw996b456eecd8d134@mail.gmail.com>

On Wed, Feb 25, 2009 at 2:21 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
> In order to inline, the function definition would need to be in the header
> and a library version would still be required for passing functions by
> address or in case the compiler decided *not* to inline.

I looked more into this while solving some nasty mingw FPU handling
bugs on windows 64. I learnt one thing I did not know about C: you can
define inline static functions in headers, and that's the approach
follows for very short functions. For example, FPU status flag code in
BSD is defined as follows:

fenv.h:

static __inline int feraiseexcept(int __excepts);
...

fenv.c:
int fereaiseexcept(int excepts);

we could follow this, maybe - once we solve the inline problem (many
compilers do not support it for C). Having optional, non inline is
important IMHO, for future SSE or other dynamically loaded
implementations (where inline is not an option).

David


From matthieu.brucher at gmail.com  Tue Feb 24 13:09:14 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 24 Feb 2009 19:09:14 +0100
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <5b8d13220902240937s70617f9dw996b456eecd8d134@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org> <loom.20090224T140326-52@post.gmane.org>
	<e06186140902240921p3ba383elc40494863a40e832@mail.gmail.com>
	<5b8d13220902240937s70617f9dw996b456eecd8d134@mail.gmail.com>
Message-ID: <e76aa17f0902241009s35a00885kabfa26f4b56c088d@mail.gmail.com>

In fact, the __inline is not helpful. It's the static keyword that
enables the compiler to inline the function if the function is small
enough. As the static indicates that the function will not be seen
from the outside, it can do this.

Matthieu

2009/2/24 David Cournapeau <cournape at gmail.com>:
> On Wed, Feb 25, 2009 at 2:21 AM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>
>> In order to inline, the function definition would need to be in the header
>> and a library version would still be required for passing functions by
>> address or in case the compiler decided *not* to inline.
>
> I looked more into this while solving some nasty mingw FPU handling
> bugs on windows 64. I learnt one thing I did not know about C: you can
> define inline static functions in headers, and that's the approach
> follows for very short functions. For example, FPU status flag code in
> BSD is defined as follows:
>
> fenv.h:
>
> static __inline int feraiseexcept(int __excepts);
> ...
>
> fenv.c:
> int fereaiseexcept(int excepts);
>
> we could follow this, maybe - once we solve the inline problem (many
> compilers do not support it for C). Having optional, non inline is
> important IMHO, for future SSE or other dynamically loaded
> implementations (where inline is not an option).
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From mtrumpis at berkeley.edu  Tue Feb 24 13:11:17 2009
From: mtrumpis at berkeley.edu (M Trumpis)
Date: Tue, 24 Feb 2009 10:11:17 -0800
Subject: [Numpy-discussion] Summary of ticket 937
In-Reply-To: <web-117210804@uni-stuttgart.de>
References: <web-117210804@uni-stuttgart.de>
Message-ID: <be3bc680902241011s7e18884cx62ac51ec72df5315@mail.gmail.com>

I ran into this problem as well a few months back.

The reason for the empty residual array when M==N is that the LAPACK
routine for Ax = b puts the solution for x in b. When M>N, the
norm-squared is parceled out into the unused (M-N) points in the b
array. When M==N, there's no room for the resids. Numpy could always
return the optimal norm-squared, but at the expense of another
matrix-matrix multiply.

The other error (complex norms) can be fixed pretty easily this way.

--- linalg.py   (revision 6436)
+++ linalg.py   (working copy)
@@ -1316,11 +1316,11 @@
     if is_1d:
         x = array(ravel(bstar)[:n], dtype=result_t, copy=True)
         if results['rank'] == n and m > n:
-            resids = array([sum((ravel(bstar)[n:])**2)], dtype=result_t)
+            resids = array([norm(ravel(bstar)[n:], 2)**2])
     else:
         x = array(transpose(bstar)[:n,:], dtype=result_t, copy=True)
         if results['rank'] == n and m > n:
-            resids = sum((transpose(bstar)[n:,:])**2, axis=0).astype(result_t)
+            resids = array([norm(v[n:], 2)**2 for v in bstar])
     st = s[:min(n, m)].copy().astype(_realType(result_t))
     return wrap(x), wrap(resids), results['rank'], st


Mike


On Thu, Feb 19, 2009 at 12:51 PM, Nils Wagner
<nwagner at iam.uni-stuttgart.de> wrote:
> Hi all,
>
> The summary of ticket 937 is incomplete.
> It should be "Complex matrices and lstsq".
>
> http://projects.scipy.org/scipy/numpy/ticket/937
>
> Nils
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From ndbecker2 at gmail.com  Tue Feb 24 13:20:10 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 24 Feb 2009 13:20:10 -0500
Subject: [Numpy-discussion] Core math library in numpy
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org>
	<loom.20090224T140326-52@post.gmane.org>
	<e06186140902240921p3ba383elc40494863a40e832@mail.gmail.com>
	<5b8d13220902240937s70617f9dw996b456eecd8d134@mail.gmail.com>
	<e76aa17f0902241009s35a00885kabfa26f4b56c088d@mail.gmail.com>
Message-ID: <go1doq$ok$1@ger.gmane.org>

Matthieu Brucher wrote:

> In fact, the __inline is not helpful. It's the static keyword that
> enables the compiler to inline the function if the function is small
> enough. As the static indicates that the function will not be seen
> from the outside, it can do this.
> 

Depends what is meant by 'helpful'.  My understanding is that gcc will both 
inline the function within the current module and supply an external copy if 
__inline__ is used.

> Matthieu
> 
> 2009/2/24 David Cournapeau <cournape at gmail.com>:
>> On Wed, Feb 25, 2009 at 2:21 AM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>>>
>>> In order to inline, the function definition would need to be in the
>>> header and a library version would still be required for passing
>>> functions by address or in case the compiler decided *not* to inline.
>>
>> I looked more into this while solving some nasty mingw FPU handling
>> bugs on windows 64. I learnt one thing I did not know about C: you can
>> define inline static functions in headers, and that's the approach
>> follows for very short functions. For example, FPU status flag code in
>> BSD is defined as follows:
>>
>> fenv.h:
>>
>> static __inline int feraiseexcept(int __excepts);
>> ...
>>
>> fenv.c:
>> int fereaiseexcept(int excepts);
>>
>> we could follow this, maybe - once we solve the inline problem (many
>> compilers do not support it for C). Having optional, non inline is
>> important IMHO, for future SSE or other dynamically loaded
>> implementations (where inline is not an option).
>>
>> David
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
> 
> 
> 





From matthieu.brucher at gmail.com  Tue Feb 24 13:25:09 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 24 Feb 2009 19:25:09 +0100
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <go1doq$ok$1@ger.gmane.org>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org> <loom.20090224T140326-52@post.gmane.org>
	<e06186140902240921p3ba383elc40494863a40e832@mail.gmail.com>
	<5b8d13220902240937s70617f9dw996b456eecd8d134@mail.gmail.com>
	<e76aa17f0902241009s35a00885kabfa26f4b56c088d@mail.gmail.com>
	<go1doq$ok$1@ger.gmane.org>
Message-ID: <e76aa17f0902241025u17f5ed54wda394c90a2d65c7f@mail.gmail.com>

The inline keyword is never an obligation to inline, it's only a
proposal. And the compiler in fact doesn't care about it. When using
an optimization mode, the compiler will inline the function if it is
simple enough. It's its call. It will even be easier to do so if the
static keyword is used, as it knows that it must not supply an
external copy.

Matthieu

2009/2/24 Neal Becker <ndbecker2 at gmail.com>:
> Matthieu Brucher wrote:
>
>> In fact, the __inline is not helpful. It's the static keyword that
>> enables the compiler to inline the function if the function is small
>> enough. As the static indicates that the function will not be seen
>> from the outside, it can do this.
>>
>
> Depends what is meant by 'helpful'. ?My understanding is that gcc will both
> inline the function within the current module and supply an external copy if
> __inline__ is used.
>
>> Matthieu
>>
>> 2009/2/24 David Cournapeau <cournape at gmail.com>:
>>> On Wed, Feb 25, 2009 at 2:21 AM, Charles R Harris
>>> <charlesr.harris at gmail.com> wrote:
>>>>
>>>> In order to inline, the function definition would need to be in the
>>>> header and a library version would still be required for passing
>>>> functions by address or in case the compiler decided *not* to inline.
>>>
>>> I looked more into this while solving some nasty mingw FPU handling
>>> bugs on windows 64. I learnt one thing I did not know about C: you can
>>> define inline static functions in headers, and that's the approach
>>> follows for very short functions. For example, FPU status flag code in
>>> BSD is defined as follows:
>>>
>>> fenv.h:
>>>
>>> static __inline int feraiseexcept(int __excepts);
>>> ...
>>>
>>> fenv.c:
>>> int fereaiseexcept(int excepts);
>>>
>>> we could follow this, maybe - once we solve the inline problem (many
>>> compilers do not support it for C). Having optional, non inline is
>>> important IMHO, for future SSE or other dynamically loaded
>>> implementations (where inline is not an option).
>>>
>>> David
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Numpy-discussion at scipy.org
>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>>
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From charlesr.harris at gmail.com  Tue Feb 24 13:26:47 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 24 Feb 2009 11:26:47 -0700
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <e76aa17f0902241009s35a00885kabfa26f4b56c088d@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org> <loom.20090224T140326-52@post.gmane.org>
	<e06186140902240921p3ba383elc40494863a40e832@mail.gmail.com>
	<5b8d13220902240937s70617f9dw996b456eecd8d134@mail.gmail.com>
	<e76aa17f0902241009s35a00885kabfa26f4b56c088d@mail.gmail.com>
Message-ID: <e06186140902241026h33514e5eod0c94fae4622b6c5@mail.gmail.com>

On Tue, Feb 24, 2009 at 11:09 AM, Matthieu Brucher <
matthieu.brucher at gmail.com> wrote:

> In fact, the __inline is not helpful. It's the static keyword that
> enables the compiler to inline the function if the function is small
> enough. As the static indicates that the function will not be seen
> from the outside, it can do this.


Good point. However, most of the ufuncs involving standard functions like
sin, cos, etc. are implemented as generic loops that are passed a function
pointer and for such functions the call overhead is probably not significant
in the absence of intrinsic hardware implementations. The complex functions
could probably use some inlining as they call other functions. That could
implemented by using some local static functions in the library code that
could be inlined when the library is compiled.

I think that the first priority should be correctness and portability. Speed
optimizations can come later.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090224/a162d1e3/attachment.html>

From Chris.Barker at noaa.gov  Tue Feb 24 14:39:39 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 24 Feb 2009 11:39:39 -0800
Subject: [Numpy-discussion] Fancy indexing question:
Message-ID: <49A44CFB.5090907@noaa.gov>

HI all,

I'm having a bit of trouble getting fancy indexing to do what I want.

Say I have a 2-d array:

 >>> a
array([[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11],
        [12, 13, 14, 15],
        [16, 17, 18, 19],
        [20, 21, 22, 23]])

I want to extract a sub-array:

The 1st, 3rd, and 4th rows:
 >>> i
[1, 3, 4]

and the 1st and 3rd columns:
 >>> j
[1, 3]

so I should get a 3x2 array:

[[ 5,  7],
  [13, 15],
  [17, 19]]

The obvious (to me!) way to do this:

 >>> a[i,j]
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ValueError: shape mismatch: objects cannot be broadcast to a single shape

fails.

I can do:

extract the rows:

 >>> rows = a[i]
 >>> rows
array([[ 4,  5,  6,  7],
        [12, 13, 14, 15],
        [16, 17, 18, 19]])

then the columns:

 >>> sub_array = rows[:,j]
 >>> sub_array
array([[ 5,  7],
        [13, 15],
        [17, 19]])

or, of course:
 >>> sub_array = a[i][:,j]
 >>> sub_array
array([[ 5,  7],
        [13, 15],
        [17, 19]])


however, it seems I am doing one more data copy and array creation than 
I want -- can this be done as a single indexing operation?

thanks,

Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From robert.kern at gmail.com  Tue Feb 24 14:43:49 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 24 Feb 2009 13:43:49 -0600
Subject: [Numpy-discussion] Fancy indexing question:
In-Reply-To: <49A44CFB.5090907@noaa.gov>
References: <49A44CFB.5090907@noaa.gov>
Message-ID: <3d375d730902241143u45723c40y2e926cba5fa18daf@mail.gmail.com>

On Tue, Feb 24, 2009 at 13:39, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> HI all,
>
> I'm having a bit of trouble getting fancy indexing to do what I want.
>
> Say I have a 2-d array:
>
> ?>>> a
> array([[ 0, ?1, ?2, ?3],
> ? ? ? ?[ 4, ?5, ?6, ?7],
> ? ? ? ?[ 8, ?9, 10, 11],
> ? ? ? ?[12, 13, 14, 15],
> ? ? ? ?[16, 17, 18, 19],
> ? ? ? ?[20, 21, 22, 23]])
>
> I want to extract a sub-array:
>
> The 1st, 3rd, and 4th rows:
> ?>>> i
> [1, 3, 4]
>
> and the 1st and 3rd columns:
> ?>>> j
> [1, 3]
>
> so I should get a 3x2 array:
>
> [[ 5, ?7],
> ?[13, 15],
> ?[17, 19]]
>
> The obvious (to me!) way to do this:
>
> ?>>> a[i,j]
> Traceback (most recent call last):
> ? File "<stdin>", line 1, in <module>
> ValueError: shape mismatch: objects cannot be broadcast to a single shape
>
> fails.

Please read the documentation:

  http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html

The short answer is that in multidimensional fancy indexing, the index
arrays are broadcasted against each other first, then the result array
is found by iterating over the broadcasted arrays in parallel. The
result array is the same shape as the broadcasted shape of the inde

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From borreguero at gmail.com  Tue Feb 24 14:55:47 2009
From: borreguero at gmail.com (Jose Borreguero)
Date: Tue, 24 Feb 2009 14:55:47 -0500
Subject: [Numpy-discussion] tensordot bug when summing over same axis
	indexes?
Message-ID: <7cced4ed0902241155x262af52eo64a8ceb757ce3e7a@mail.gmail.com>

The following example:

from numpy import *
a=arange(12).reshape(2,3,2)
b=arange(24).reshape(2,3,2,2)
c=tensordot( a,b,axes=([0,0],[1,1]) )

defaults:
c=tensordot( a,b,axes=([0,0],[1,1]) )
File "/usr/lib/python2.4/site-packages/numpy/core/numeric.py", line 359, in
tensordot
raise ValueError, "shape-mismatch for sum"
ValueError: shape-mismatch for sum

Am I doing something really stupid, or is this a bug?

-Jose
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090224/f513357d/attachment.html>

From Chris.Barker at noaa.gov  Tue Feb 24 15:06:05 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 24 Feb 2009 12:06:05 -0800
Subject: [Numpy-discussion] Fancy indexing question:
In-Reply-To: <3d375d730902241143u45723c40y2e926cba5fa18daf@mail.gmail.com>
References: <49A44CFB.5090907@noaa.gov>
	<3d375d730902241143u45723c40y2e926cba5fa18daf@mail.gmail.com>
Message-ID: <49A4532D.9040203@noaa.gov>

Robert Kern wrote:
> On Tue, Feb 24, 2009 at 13:39, Christopher Barker <Chris.Barker at noaa.gov> wrote:
>>  >>> a
>> array([[ 0,  1,  2,  3],
>>        [ 4,  5,  6,  7],
>>        [ 8,  9, 10, 11],
>>        [12, 13, 14, 15],
>>        [16, 17, 18, 19],
>>        [20, 21, 22, 23]])
>>
>> I want to extract a sub-array:

>> [[ 5,  7],
>>  [13, 15],
>>  [17, 19]]

> Please read the documentation:
> 
>   http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html

well, I did google a fair bit, though, oddly, I didn't find that page.

> The short answer is that in multidimensional fancy indexing, the index
> arrays are broadcasted against each other first, then the result array
> is found by iterating over the broadcasted arrays in parallel.

I did read this description before, and really didn't get it, or how to 
apply it to this problem. However, with your prodding, I thought about 
it some more, and I think I got it.

I suppose I should figure out how to contribute to the docs, but here is 
how I came to understand it, in the context of my example:

I want to extract a 3x2 subarray, with the 3 rows specified by i, and 
the two columns specified by j.

"""
All the integer indexing arrays must be broadcastable to the same shape.

The shape of the output (or the needed shape of the object to be used 
for setting) is the broadcasted shape.
"""

OK -- so I need my index arrays to broadcast to a 3x2 array -- this was 
my "light bulb" moment. I need to make the i index array a column:

 >>> i = np.array((1,3,4)).reshape((-1,1))

so now they will broadcast to a rectangle:

 >>> i * j
array([[ 1,  3],
        [ 3,  9],
        [ 4, 12]])

and the indexing works like I want:

 >>> a[i,j]
array([[ 5,  7],
        [13, 15],
        [17, 19]])


This makes so much sense when I think of it this way:

to extract a particular set of rows, I need a column of indexes:



a:

    [[ 0,  1,  2,  3],
     [ 4,  5,  6,  7],
     [ 8,  9, 10, 11],
     [12, 13, 14, 15],
     [16, 17, 18, 19],
     [20, 21, 22, 23]])

I want to extract a sub-array:

 >> [[ 5,  7],
 >>  [13, 15],
 >>  [17, 19]]

which are these columns:

           1       3

    [[ 0,  1,  2,  3],
     [ 4,  5,  6,  7],
     [ 8,  9, 10, 11],
     [12, 13, 14, 15],
     [16, 17, 18, 19],
     [20, 21, 22, 23]])

and these rows:

    [[ 0,  1,  2,  3],
1   [ 4,  5,  6,  7],
     [ 8,  9, 10, 11],
3   [12, 13, 14, 15],
4   [16, 17, 18, 19],
     [20, 21, 22, 23]])


so clearly, I need a (n,1) array to index the rows.

-Chris









-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From rmay31 at gmail.com  Tue Feb 24 15:13:00 2009
From: rmay31 at gmail.com (Ryan May)
Date: Tue, 24 Feb 2009 14:13:00 -0600
Subject: [Numpy-discussion] Fancy indexing question:
In-Reply-To: <49A44CFB.5090907@noaa.gov>
References: <49A44CFB.5090907@noaa.gov>
Message-ID: <c7b95a020902241213p65292dfk997e614b566e0eb4@mail.gmail.com>

On Tue, Feb 24, 2009 at 1:39 PM, Christopher Barker
<Chris.Barker at noaa.gov>wrote:

> HI all,
>
> I'm having a bit of trouble getting fancy indexing to do what I want.
>
> Say I have a 2-d array:
>
>  >>> a
> array([[ 0,  1,  2,  3],
>        [ 4,  5,  6,  7],
>        [ 8,  9, 10, 11],
>        [12, 13, 14, 15],
>        [16, 17, 18, 19],
>        [20, 21, 22, 23]])
>
> I want to extract a sub-array:
>
> The 1st, 3rd, and 4th rows:
>  >>> i
> [1, 3, 4]
>
> and the 1st and 3rd columns:
>  >>> j
> [1, 3]
>
> so I should get a 3x2 array:
>
> [[ 5,  7],
>  [13, 15],
>  [17, 19]]
>
> The obvious (to me!) way to do this:
>
>  >>> a[i,j]
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ValueError: shape mismatch: objects cannot be broadcast to a single shape
>

You need to listen more closely to the error message you get back. :)  It's
a broadcasting issue, so why not try this:

a = np.array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11],
       [12, 13, 14, 15],
       [16, 17, 18, 19],
       [20, 21, 22, 23]])
i = np.array([1, 3, 4]).reshape(-1,1)
j = np.array([1, 3])

a[i,j]

You need to make i,j conformable to the numpy broadcasting rules by manually
appending size 1 dimension.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
Sent from: Norman Oklahoma United States.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090224/ff6ed22d/attachment.html>

From josef.pktd at gmail.com  Tue Feb 24 15:20:36 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 24 Feb 2009 15:20:36 -0500
Subject: [Numpy-discussion] tensordot bug when summing over same axis
	indexes?
In-Reply-To: <7cced4ed0902241155x262af52eo64a8ceb757ce3e7a@mail.gmail.com>
References: <7cced4ed0902241155x262af52eo64a8ceb757ce3e7a@mail.gmail.com>
Message-ID: <1cd32cbb0902241220u4042a577tde7d224f263c1781@mail.gmail.com>

On Tue, Feb 24, 2009 at 2:55 PM, Jose Borreguero <borreguero at gmail.com> wrote:
> The following example:
>
> from numpy import *
> a=arange(12).reshape(2,3,2)
> b=arange(24).reshape(2,3,2,2)
> c=tensordot( a,b,axes=([0,0],[1,1]) )
>
> defaults:
> c=tensordot( a,b,axes=([0,0],[1,1]) )
> File "/usr/lib/python2.4/site-packages/numpy/core/numeric.py", line 359, in
> tensordot
> raise ValueError, "shape-mismatch for sum"
> ValueError: shape-mismatch for sum
>
> Am I doing something really stupid, or is this a bug?
>
> -Jose

did you want to do this?

>>> c=np.tensordot( a,b,axes=([0,1],[0,1]) )
>>> c
array([[[440, 470],
        [500, 530]],

       [[500, 536],
        [572, 608]]])

Josef


From lists_ravi at lavabit.com  Tue Feb 24 15:26:39 2009
From: lists_ravi at lavabit.com (Ravi)
Date: Tue, 24 Feb 2009 15:26:39 -0500
Subject: [Numpy-discussion] Fancy indexing question:
In-Reply-To: <49A44CFB.5090907@noaa.gov>
References: <49A44CFB.5090907@noaa.gov>
Message-ID: <200902241526.42724.lists_ravi@lavabit.com>

On Tuesday 24 February 2009 14:39:39 Christopher Barker wrote:
> I'm having a bit of trouble getting fancy indexing to do what I want.

Use ix_:

In [2]: a
Out[2]:
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11],
       [12, 13, 14, 15],
       [16, 17, 18, 19],
       [20, 21, 22, 23]])

In [3]: i = array( [1,3,4] )

In [4]: j = array( [1,3] )

In [5]: a[ ix_(i,j) ]
Out[5]:
array([[ 5,  7],
       [13, 15],
       [17, 19]])

Regards,
Ravi




From borreguero at gmail.com  Tue Feb 24 15:44:04 2009
From: borreguero at gmail.com (Jose Borreguero)
Date: Tue, 24 Feb 2009 15:44:04 -0500
Subject: [Numpy-discussion] tensordot bug when summing over same axis
	indexes?
In-Reply-To: <1cd32cbb0902241220u4042a577tde7d224f263c1781@mail.gmail.com>
References: <7cced4ed0902241155x262af52eo64a8ceb757ce3e7a@mail.gmail.com>
	<1cd32cbb0902241220u4042a577tde7d224f263c1781@mail.gmail.com>
Message-ID: <7cced4ed0902241244w3c304892i2b314ea1c6ea0a77@mail.gmail.com>

haha, so it was a stupid error...my stupid error. [?]

I incorrectly understood ([0,0],[1,1]) as index 0 of *a* summed with index 0
of *b*, and analogously for [1,1].

Gthanks, Josef

On Tue, Feb 24, 2009 at 3:20 PM, <josef.pktd at gmail.com> wrote:

> On Tue, Feb 24, 2009 at 2:55 PM, Jose Borreguero <borreguero at gmail.com>
> wrote:
> > The following example:
> >
> > from numpy import *
> > a=arange(12).reshape(2,3,2)
> > b=arange(24).reshape(2,3,2,2)
> > c=tensordot( a,b,axes=([0,0],[1,1]) )
> >
> > defaults:
> > c=tensordot( a,b,axes=([0,0],[1,1]) )
> > File "/usr/lib/python2.4/site-packages/numpy/core/numeric.py", line 359,
> in
> > tensordot
> > raise ValueError, "shape-mismatch for sum"
> > ValueError: shape-mismatch for sum
> >
> > Am I doing something really stupid, or is this a bug?
> >
> > -Jose
>
> did you want to do this?
>
> >>> c=np.tensordot( a,b,axes=([0,1],[0,1]) )
> >>> c
> array([[[440, 470],
>        [500, 530]],
>
>       [[500, 536],
>        [572, 608]]])
>
> Josef
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090224/b95e6faa/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 32F.gif
Type: image/gif
Size: 104 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090224/b95e6faa/attachment.gif>

From matthew.brett at gmail.com  Tue Feb 24 16:04:54 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Tue, 24 Feb 2009 13:04:54 -0800
Subject: [Numpy-discussion] isbuiltin - failure of understanding
Message-ID: <1e2af89e0902241304p707239c6m17b138cd5f261f72@mail.gmail.com>

Hi,

I was just trying to write a docstring for np.dtype.isbuiltin, when I
realized I didn't understand it.

As far as I can see, isbuitin should return:

0 for structured array dtypes
1 for types compiled into numpy
2 for extension types using the numpy C-API type extension machinery.

Here's the C code:

static PyObject *
arraydescr_isbuiltin_get(PyArray_Descr *self)
{
    long val;
    val = 0;
    if (self->fields == Py_None) val = 1;
    if (PyTypeNum_ISUSERDEF(self->type_num)) val = 2;
    return PyInt_FromLong(val);
}

But, why is this?

In [2]: dt = np.dtype('S1')

In [3]: dt
Out[3]: dtype('|S1')

In [4]: dt.isbuiltin
Out[4]: 0

In [5]: print dt.fields
None

In [6]: print dt.fields == None
True

Same for np.dtype('U1')

Best,

Matthew


From robert.kern at gmail.com  Tue Feb 24 16:10:11 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 24 Feb 2009 15:10:11 -0600
Subject: [Numpy-discussion] isbuiltin - failure of understanding
In-Reply-To: <1e2af89e0902241304p707239c6m17b138cd5f261f72@mail.gmail.com>
References: <1e2af89e0902241304p707239c6m17b138cd5f261f72@mail.gmail.com>
Message-ID: <3d375d730902241310u3ca6a1fdibc0cc47b2e9e308f@mail.gmail.com>

On Tue, Feb 24, 2009 at 15:04, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> I was just trying to write a docstring for np.dtype.isbuiltin, when I
> realized I didn't understand it.
>
> As far as I can see, isbuitin should return:
>
> 0 for structured array dtypes
> 1 for types compiled into numpy
> 2 for extension types using the numpy C-API type extension machinery.
>
> Here's the C code:
>
> static PyObject *
> arraydescr_isbuiltin_get(PyArray_Descr *self)
> {
> ? ?long val;
> ? ?val = 0;
> ? ?if (self->fields == Py_None) val = 1;
> ? ?if (PyTypeNum_ISUSERDEF(self->type_num)) val = 2;
> ? ?return PyInt_FromLong(val);
> }
>
> But, why is this?
>
> In [2]: dt = np.dtype('S1')
>
> In [3]: dt
> Out[3]: dtype('|S1')
>
> In [4]: dt.isbuiltin
> Out[4]: 0
>
> In [5]: print dt.fields
> None
>
> In [6]: print dt.fields == None
> True
>
> Same for np.dtype('U1')

The variable-length string dtypes are not builtin. The user-defined
lengths make them user-defined dtypes.

If you want a real kick in the pants, try playing with dtype(str). And
then try eval'ing its repr.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From matthew.brett at gmail.com  Tue Feb 24 18:08:31 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Tue, 24 Feb 2009 15:08:31 -0800
Subject: [Numpy-discussion] isbuiltin - failure of understanding
In-Reply-To: <3d375d730902241310u3ca6a1fdibc0cc47b2e9e308f@mail.gmail.com>
References: <1e2af89e0902241304p707239c6m17b138cd5f261f72@mail.gmail.com>
	<3d375d730902241310u3ca6a1fdibc0cc47b2e9e308f@mail.gmail.com>
Message-ID: <1e2af89e0902241508n1625a87cqaa1ba20e68bc6344@mail.gmail.com>

Hi,

> The variable-length string dtypes are not builtin. The user-defined
> lengths make them user-defined dtypes.

Right - but I was failing to understand, from the code, how '0' rather
than '2' could result.  Have I missed something?

> If you want a real kick in the pants, try playing with dtype(str). And
> then try eval'ing its repr.

I am trying to decide how to answer the first part of that sentence!

Matthew


From sccolbert at gmail.com  Tue Feb 24 19:15:20 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 24 Feb 2009 19:15:20 -0500
Subject: [Numpy-discussion] is there a faster way to get a buffer interface
	than ndarray.tostring()?
Message-ID: <7f014ea60902241615l18ddd46am541af69b12ac381e@mail.gmail.com>

Hi all,

 I'm new to mailing list and relatively new (~1 year) to python/numpy. I
would appreciate any insight any of you may have here. The last 8 hours of
digging through the docs has left me, finally, stuck.

I am making a wxPython program that includes webcam functionality. The
script below just brings up a simple display window that shows the webcam
feed. The problem I am running into is in the WebCamWorker.run() method.
This method reads the raw pixel data (in string form) from the webcam
buffer. This data (thank you microsoft!) comes in BGR and bottom-to-top. I
feed this buffer to the array constructor and then swap the pixel order to
RGB and top-to-bottom. This all happens very fast, ~1ms on a Quad Core, and
the majority of that time is spent constructing the array (the numpy pixel
swapping is on the order of E-5s !). The tostring() method, however, takes a
whopping 10ms to execute. Unfortunately, the wx.BitmapFromBuffer needs a
single segment buffer interface as an argument. Is there a way to expose my
numpy array as a buffer to cut down on this conversion time?

I have tried sending the array to the PIL Image.fromarray(), but that
eventually requires me to do a Image.tostring() anyway, which negates any
benefit.

Thanks in advance for the help!

S. Chris Colbert
Rehabilitation Robotics Laboratory
University of South Florida


####  Code #######

import VideoCapture
import wx
import time
import threading
import numpy as np
import Image


class WebCamWorker(threading.Thread):
    _abort = 0

    def __init__(self, parent):
        super(WebCamWorker, self).__init__()

        self._parent = parent
        self.panel = wx.Panel(parent)

    def run(self):

        while not self._abort:

            #numpy arrays reverse pixel data that comes in from CCD as BGR
and bottom to top

            pixeldata = np.ndarray((480,640,3),
buffer=webcam.getBuffer()[0], dtype='u1')
            revpixels = pixeldata[::-1,:,::-1].tostring()      #tostring is
an order of magnitude slower than the entire array manipulation. need a
faster method.

            rgbBMP = wx.BitmapFromBuffer(640, 480, revpixels)
            dc = wx.ClientDC(self.panel)
            dc.DrawBitmap(rgbBMP, 0, 0)
            #b = time.clock()
            #print b-a
            time.sleep(0.015)


        return



class TestWxWebCam(wx.Frame):
    def __init__(self, parent):
        wx.Frame.__init__(self, parent, -1, size=(640,480))

        self.worker = WebCamWorker(self)
        self.worker.start()



class TestApp(wx.App):
    def OnInit(self):
        self.frame = TestWxWebCam(None)
        self.frame.Show()

        return True

    def OnExit(self):
        WebCamWorker._abort = 1 #this is a hack


if __name__ == '__main__':
    webcam = VideoCapture.Device()
    webcam.setResolution(640, 480)
    webcam.displayCapturePinProperties()
    app = TestApp()
    app.MainLoop()
    del webcam


### /Code ######
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090224/3df3d854/attachment.html>

From robert.kern at gmail.com  Tue Feb 24 19:47:53 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 24 Feb 2009 18:47:53 -0600
Subject: [Numpy-discussion] is there a faster way to get a buffer
	interface than ndarray.tostring()?
In-Reply-To: <7f014ea60902241615l18ddd46am541af69b12ac381e@mail.gmail.com>
References: <7f014ea60902241615l18ddd46am541af69b12ac381e@mail.gmail.com>
Message-ID: <3d375d730902241647sd1132f7va61b6ef96185e129@mail.gmail.com>

On Tue, Feb 24, 2009 at 18:15, Chris Colbert <sccolbert at gmail.com> wrote:
> Hi all,
>
> ?I'm new to mailing list and relatively new (~1 year) to python/numpy. I
> would appreciate any insight any of you may have here. The last 8 hours of
> digging through the docs has left me, finally, stuck.
>
> I am making a wxPython program that includes webcam functionality. The
> script below just brings up a simple display window that shows the webcam
> feed. The problem I am running into is in the WebCamWorker.run() method.
> This method reads the raw pixel data (in string form) from the webcam
> buffer. This data (thank you microsoft!) comes in BGR and bottom-to-top. I
> feed this buffer to the array constructor and then swap the pixel order to
> RGB and top-to-bottom. This all happens very fast, ~1ms on a Quad Core, and
> the majority of that time is spent constructing the array (the numpy pixel
> swapping is on the order of E-5s !). The tostring() method, however, takes a
> whopping 10ms to execute. Unfortunately, the wx.BitmapFromBuffer needs a
> single segment buffer interface as an argument. Is there a way to expose my
> numpy array as a buffer to cut down on this conversion time?

buffer(myarray)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From dalcinl at gmail.com  Tue Feb 24 20:06:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 24 Feb 2009 22:06:21 -0300
Subject: [Numpy-discussion] is there a faster way to get a buffer
	interface than ndarray.tostring()?
In-Reply-To: <7f014ea60902241615l18ddd46am541af69b12ac381e@mail.gmail.com>
References: <7f014ea60902241615l18ddd46am541af69b12ac381e@mail.gmail.com>
Message-ID: <e7ba66e40902241706v5ea2782btaf743b405ae976b0@mail.gmail.com>

When you do pixeldata[::-1,:,::-1], you just got a new array with
different strides, but now non-contiguous... So I believe you really
need a fresh copy of the data... tostring() copies, but could be
slow... try to use

revpixels = pixeldata[::-1,:,::-1].copy()

...

rgbBMP = wx.BitmapFromBuffer(640, 480, buffer(revpixels))


Perhaps you could optimize this by declaring revpixels outside de
loop, and then inside de loop doing

revpixels[...] = pixeldata[::-1,:,::-1]


This way you will save the mem allocation of revpixels at each step of the loop.





On Tue, Feb 24, 2009 at 9:15 PM, Chris Colbert <sccolbert at gmail.com> wrote:
> Hi all,
>
> ?I'm new to mailing list and relatively new (~1 year) to python/numpy. I
> would appreciate any insight any of you may have here. The last 8 hours of
> digging through the docs has left me, finally, stuck.
>
> I am making a wxPython program that includes webcam functionality. The
> script below just brings up a simple display window that shows the webcam
> feed. The problem I am running into is in the WebCamWorker.run() method.
> This method reads the raw pixel data (in string form) from the webcam
> buffer. This data (thank you microsoft!) comes in BGR and bottom-to-top. I
> feed this buffer to the array constructor and then swap the pixel order to
> RGB and top-to-bottom. This all happens very fast, ~1ms on a Quad Core, and
> the majority of that time is spent constructing the array (the numpy pixel
> swapping is on the order of E-5s !). The tostring() method, however, takes a
> whopping 10ms to execute. Unfortunately, the wx.BitmapFromBuffer needs a
> single segment buffer interface as an argument. Is there a way to expose my
> numpy array as a buffer to cut down on this conversion time?
>
> I have tried sending the array to the PIL Image.fromarray(), but that
> eventually requires me to do a Image.tostring() anyway, which negates any
> benefit.
>
> Thanks in advance for the help!
>
> S. Chris Colbert
> Rehabilitation Robotics Laboratory
> University of South Florida
>
>
> ####? Code #######
>
> import VideoCapture
> import wx
> import time
> import threading
> import numpy as np
> import Image
>
>
> class WebCamWorker(threading.Thread):
> ??? _abort = 0
>
> ??? def __init__(self, parent):
> ??????? super(WebCamWorker, self).__init__()
>
> ??????? self._parent = parent
> ??????? self.panel = wx.Panel(parent)
>
> ??? def run(self):
>
> ??????? while not self._abort:
>
> ??????????? #numpy arrays reverse pixel data that comes in from CCD as BGR
> and bottom to top
>
> ??????????? pixeldata = np.ndarray((480,640,3),
> buffer=webcam.getBuffer()[0], dtype='u1')
> ??????????? revpixels = pixeldata[::-1,:,::-1].tostring()????? #tostring is
> an order of magnitude slower than the entire array manipulation. need a
> faster method.
>
> ??????????? rgbBMP = wx.BitmapFromBuffer(640, 480, revpixels)
> ??????????? dc = wx.ClientDC(self.panel)
> ??????????? dc.DrawBitmap(rgbBMP, 0, 0)
> ??????????? #b = time.clock()
> ??????????? #print b-a
> ??????????? time.sleep(0.015)
>
>
> ??????? return
>
>
>
> class TestWxWebCam(wx.Frame):
> ??? def __init__(self, parent):
> ??????? wx.Frame.__init__(self, parent, -1, size=(640,480))
>
> ??????? self.worker = WebCamWorker(self)
> ??????? self.worker.start()
>
>
>
> class TestApp(wx.App):
> ??? def OnInit(self):
> ??????? self.frame = TestWxWebCam(None)
> ??????? self.frame.Show()
>
> ??????? return True
>
> ??? def OnExit(self):
> ??????? WebCamWorker._abort = 1 #this is a hack
>
>
> if __name__ == '__main__':
> ??? webcam = VideoCapture.Device()
> ??? webcam.setResolution(640, 480)
> ??? webcam.displayCapturePinProperties()
> ??? app = TestApp()
> ??? app.MainLoop()
> ??? del webcam
>
>
> ### /Code ######
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594


From sccolbert at gmail.com  Tue Feb 24 21:27:31 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 24 Feb 2009 21:27:31 -0500
Subject: [Numpy-discussion] is there a faster way to get a buffer
	interface than ndarray.tostring()?
In-Reply-To: <e7ba66e40902241706v5ea2782btaf743b405ae976b0@mail.gmail.com>
References: <7f014ea60902241615l18ddd46am541af69b12ac381e@mail.gmail.com>
	<e7ba66e40902241706v5ea2782btaf743b405ae976b0@mail.gmail.com>
Message-ID: <7f014ea60902241827n4a453c8fn48e9836e71ad05bf@mail.gmail.com>

thanks for both answers!

Lisandro, you're right, I should have declared the array outside the loop.
Thanks for catching that!

Robert, as always, thanks for the answer. Quick and to the point! You've
helped me more than once on the enthought list :)



On Tue, Feb 24, 2009 at 8:06 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> When you do pixeldata[::-1,:,::-1], you just got a new array with
> different strides, but now non-contiguous... So I believe you really
> need a fresh copy of the data... tostring() copies, but could be
> slow... try to use
>
> revpixels = pixeldata[::-1,:,::-1].copy()
>
> ...
>
> rgbBMP = wx.BitmapFromBuffer(640, 480, buffer(revpixels))
>
>
> Perhaps you could optimize this by declaring revpixels outside de
> loop, and then inside de loop doing
>
> revpixels[...] = pixeldata[::-1,:,::-1]
>
>
> This way you will save the mem allocation of revpixels at each step of the
> loop.
>
>
>
>
>
> On Tue, Feb 24, 2009 at 9:15 PM, Chris Colbert <sccolbert at gmail.com>
> wrote:
> > Hi all,
> >
> >  I'm new to mailing list and relatively new (~1 year) to python/numpy. I
> > would appreciate any insight any of you may have here. The last 8 hours
> of
> > digging through the docs has left me, finally, stuck.
> >
> > I am making a wxPython program that includes webcam functionality. The
> > script below just brings up a simple display window that shows the webcam
> > feed. The problem I am running into is in the WebCamWorker.run() method.
> > This method reads the raw pixel data (in string form) from the webcam
> > buffer. This data (thank you microsoft!) comes in BGR and bottom-to-top.
> I
> > feed this buffer to the array constructor and then swap the pixel order
> to
> > RGB and top-to-bottom. This all happens very fast, ~1ms on a Quad Core,
> and
> > the majority of that time is spent constructing the array (the numpy
> pixel
> > swapping is on the order of E-5s !). The tostring() method, however,
> takes a
> > whopping 10ms to execute. Unfortunately, the wx.BitmapFromBuffer needs a
> > single segment buffer interface as an argument. Is there a way to expose
> my
> > numpy array as a buffer to cut down on this conversion time?
> >
> > I have tried sending the array to the PIL Image.fromarray(), but that
> > eventually requires me to do a Image.tostring() anyway, which negates any
> > benefit.
> >
> > Thanks in advance for the help!
> >
> > S. Chris Colbert
> > Rehabilitation Robotics Laboratory
> > University of South Florida
> >
> >
> > ####  Code #######
> >
> > import VideoCapture
> > import wx
> > import time
> > import threading
> > import numpy as np
> > import Image
> >
> >
> > class WebCamWorker(threading.Thread):
> >     _abort = 0
> >
> >     def __init__(self, parent):
> >         super(WebCamWorker, self).__init__()
> >
> >         self._parent = parent
> >         self.panel = wx.Panel(parent)
> >
> >     def run(self):
> >
> >         while not self._abort:
> >
> >             #numpy arrays reverse pixel data that comes in from CCD as
> BGR
> > and bottom to top
> >
> >             pixeldata = np.ndarray((480,640,3),
> > buffer=webcam.getBuffer()[0], dtype='u1')
> >             revpixels = pixeldata[::-1,:,::-1].tostring()      #tostring
> is
> > an order of magnitude slower than the entire array manipulation. need a
> > faster method.
> >
> >             rgbBMP = wx.BitmapFromBuffer(640, 480, revpixels)
> >             dc = wx.ClientDC(self.panel)
> >             dc.DrawBitmap(rgbBMP, 0, 0)
> >             #b = time.clock()
> >             #print b-a
> >             time.sleep(0.015)
> >
> >
> >         return
> >
> >
> >
> > class TestWxWebCam(wx.Frame):
> >     def __init__(self, parent):
> >         wx.Frame.__init__(self, parent, -1, size=(640,480))
> >
> >         self.worker = WebCamWorker(self)
> >         self.worker.start()
> >
> >
> >
> > class TestApp(wx.App):
> >     def OnInit(self):
> >         self.frame = TestWxWebCam(None)
> >         self.frame.Show()
> >
> >         return True
> >
> >     def OnExit(self):
> >         WebCamWorker._abort = 1 #this is a hack
> >
> >
> > if __name__ == '__main__':
> >     webcam = VideoCapture.Device()
> >     webcam.setResolution(640, 480)
> >     webcam.displayCapturePinProperties()
> >     app = TestApp()
> >     app.MainLoop()
> >     del webcam
> >
> >
> > ### /Code ######
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090224/30426c90/attachment.html>

From sccolbert at gmail.com  Tue Feb 24 21:27:31 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 24 Feb 2009 21:27:31 -0500
Subject: [Numpy-discussion] is there a faster way to get a buffer
	interface than ndarray.tostring()?
In-Reply-To: <e7ba66e40902241706v5ea2782btaf743b405ae976b0@mail.gmail.com>
References: <7f014ea60902241615l18ddd46am541af69b12ac381e@mail.gmail.com>
	<e7ba66e40902241706v5ea2782btaf743b405ae976b0@mail.gmail.com>
Message-ID: <7f014ea60902241827n4a453c8fn48e9836e71ad05bf@mail.gmail.com>

thanks for both answers!

Lisandro, you're right, I should have declared the array outside the loop.
Thanks for catching that!

Robert, as always, thanks for the answer. Quick and to the point! You've
helped me more than once on the enthought list :)



On Tue, Feb 24, 2009 at 8:06 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> When you do pixeldata[::-1,:,::-1], you just got a new array with
> different strides, but now non-contiguous... So I believe you really
> need a fresh copy of the data... tostring() copies, but could be
> slow... try to use
>
> revpixels = pixeldata[::-1,:,::-1].copy()
>
> ...
>
> rgbBMP = wx.BitmapFromBuffer(640, 480, buffer(revpixels))
>
>
> Perhaps you could optimize this by declaring revpixels outside de
> loop, and then inside de loop doing
>
> revpixels[...] = pixeldata[::-1,:,::-1]
>
>
> This way you will save the mem allocation of revpixels at each step of the
> loop.
>
>
>
>
>
> On Tue, Feb 24, 2009 at 9:15 PM, Chris Colbert <sccolbert at gmail.com>
> wrote:
> > Hi all,
> >
> >  I'm new to mailing list and relatively new (~1 year) to python/numpy. I
> > would appreciate any insight any of you may have here. The last 8 hours
> of
> > digging through the docs has left me, finally, stuck.
> >
> > I am making a wxPython program that includes webcam functionality. The
> > script below just brings up a simple display window that shows the webcam
> > feed. The problem I am running into is in the WebCamWorker.run() method.
> > This method reads the raw pixel data (in string form) from the webcam
> > buffer. This data (thank you microsoft!) comes in BGR and bottom-to-top.
> I
> > feed this buffer to the array constructor and then swap the pixel order
> to
> > RGB and top-to-bottom. This all happens very fast, ~1ms on a Quad Core,
> and
> > the majority of that time is spent constructing the array (the numpy
> pixel
> > swapping is on the order of E-5s !). The tostring() method, however,
> takes a
> > whopping 10ms to execute. Unfortunately, the wx.BitmapFromBuffer needs a
> > single segment buffer interface as an argument. Is there a way to expose
> my
> > numpy array as a buffer to cut down on this conversion time?
> >
> > I have tried sending the array to the PIL Image.fromarray(), but that
> > eventually requires me to do a Image.tostring() anyway, which negates any
> > benefit.
> >
> > Thanks in advance for the help!
> >
> > S. Chris Colbert
> > Rehabilitation Robotics Laboratory
> > University of South Florida
> >
> >
> > ####  Code #######
> >
> > import VideoCapture
> > import wx
> > import time
> > import threading
> > import numpy as np
> > import Image
> >
> >
> > class WebCamWorker(threading.Thread):
> >     _abort = 0
> >
> >     def __init__(self, parent):
> >         super(WebCamWorker, self).__init__()
> >
> >         self._parent = parent
> >         self.panel = wx.Panel(parent)
> >
> >     def run(self):
> >
> >         while not self._abort:
> >
> >             #numpy arrays reverse pixel data that comes in from CCD as
> BGR
> > and bottom to top
> >
> >             pixeldata = np.ndarray((480,640,3),
> > buffer=webcam.getBuffer()[0], dtype='u1')
> >             revpixels = pixeldata[::-1,:,::-1].tostring()      #tostring
> is
> > an order of magnitude slower than the entire array manipulation. need a
> > faster method.
> >
> >             rgbBMP = wx.BitmapFromBuffer(640, 480, revpixels)
> >             dc = wx.ClientDC(self.panel)
> >             dc.DrawBitmap(rgbBMP, 0, 0)
> >             #b = time.clock()
> >             #print b-a
> >             time.sleep(0.015)
> >
> >
> >         return
> >
> >
> >
> > class TestWxWebCam(wx.Frame):
> >     def __init__(self, parent):
> >         wx.Frame.__init__(self, parent, -1, size=(640,480))
> >
> >         self.worker = WebCamWorker(self)
> >         self.worker.start()
> >
> >
> >
> > class TestApp(wx.App):
> >     def OnInit(self):
> >         self.frame = TestWxWebCam(None)
> >         self.frame.Show()
> >
> >         return True
> >
> >     def OnExit(self):
> >         WebCamWorker._abort = 1 #this is a hack
> >
> >
> > if __name__ == '__main__':
> >     webcam = VideoCapture.Device()
> >     webcam.setResolution(640, 480)
> >     webcam.displayCapturePinProperties()
> >     app = TestApp()
> >     app.MainLoop()
> >     del webcam
> >
> >
> > ### /Code ######
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090224/30426c90/attachment-0001.html>

From sccolbert at gmail.com  Tue Feb 24 21:49:37 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 24 Feb 2009 21:49:37 -0500
Subject: [Numpy-discussion] is there a faster way to get a buffer
	interface than ndarray.tostring()?
In-Reply-To: <7f014ea60902241827n4a453c8fn48e9836e71ad05bf@mail.gmail.com>
References: <7f014ea60902241615l18ddd46am541af69b12ac381e@mail.gmail.com>
	<e7ba66e40902241706v5ea2782btaf743b405ae976b0@mail.gmail.com>
	<7f014ea60902241827n4a453c8fn48e9836e71ad05bf@mail.gmail.com>
Message-ID: <7f014ea60902241849s49e929dfsf8fc8201290a1feb@mail.gmail.com>

As an update for any future googlers:
the problem was with revpixels = pixeldata[::-1,:,;:-1] which apparently
returns an array that is discontinuous in memory. What Lisandro suggested
worked. pixeldata[::-1,:,;:-1].copy() returns a continuous array object
which natively implements a single-segment buffer interface. i.e. no
buffer(revpixels) is needed; just simply revpixels.

array.copy() is also 50% faster than array.tostring() on my machine.

Chris

On Tue, Feb 24, 2009 at 9:27 PM, Chris Colbert <sccolbert at gmail.com> wrote:

> thanks for both answers!
>
> Lisandro, you're right, I should have declared the array outside the loop.
> Thanks for catching that!
>
> Robert, as always, thanks for the answer. Quick and to the point! You've
> helped me more than once on the enthought list :)
>
>
>
> On Tue, Feb 24, 2009 at 8:06 PM, Lisandro Dalcin <dalcinl at gmail.com>wrote:
>
>> When you do pixeldata[::-1,:,::-1], you just got a new array with
>> different strides, but now non-contiguous... So I believe you really
>> need a fresh copy of the data... tostring() copies, but could be
>> slow... try to use
>>
>> revpixels = pixeldata[::-1,:,::-1].copy()
>>
>> ...
>>
>> rgbBMP = wx.BitmapFromBuffer(640, 480, buffer(revpixels))
>>
>>
>> Perhaps you could optimize this by declaring revpixels outside de
>> loop, and then inside de loop doing
>>
>> revpixels[...] = pixeldata[::-1,:,::-1]
>>
>>
>> This way you will save the mem allocation of revpixels at each step of the
>> loop.
>>
>>
>>
>>
>>
>> On Tue, Feb 24, 2009 at 9:15 PM, Chris Colbert <sccolbert at gmail.com>
>> wrote:
>> > Hi all,
>> >
>> >  I'm new to mailing list and relatively new (~1 year) to python/numpy. I
>> > would appreciate any insight any of you may have here. The last 8 hours
>> of
>> > digging through the docs has left me, finally, stuck.
>> >
>> > I am making a wxPython program that includes webcam functionality. The
>> > script below just brings up a simple display window that shows the
>> webcam
>> > feed. The problem I am running into is in the WebCamWorker.run() method.
>> > This method reads the raw pixel data (in string form) from the webcam
>> > buffer. This data (thank you microsoft!) comes in BGR and bottom-to-top.
>> I
>> > feed this buffer to the array constructor and then swap the pixel order
>> to
>> > RGB and top-to-bottom. This all happens very fast, ~1ms on a Quad Core,
>> and
>> > the majority of that time is spent constructing the array (the numpy
>> pixel
>> > swapping is on the order of E-5s !). The tostring() method, however,
>> takes a
>> > whopping 10ms to execute. Unfortunately, the wx.BitmapFromBuffer needs a
>> > single segment buffer interface as an argument. Is there a way to expose
>> my
>> > numpy array as a buffer to cut down on this conversion time?
>> >
>> > I have tried sending the array to the PIL Image.fromarray(), but that
>> > eventually requires me to do a Image.tostring() anyway, which negates
>> any
>> > benefit.
>> >
>> > Thanks in advance for the help!
>> >
>> > S. Chris Colbert
>> > Rehabilitation Robotics Laboratory
>> > University of South Florida
>> >
>> >
>> > ####  Code #######
>> >
>> > import VideoCapture
>> > import wx
>> > import time
>> > import threading
>> > import numpy as np
>> > import Image
>> >
>> >
>> > class WebCamWorker(threading.Thread):
>> >     _abort = 0
>> >
>> >     def __init__(self, parent):
>> >         super(WebCamWorker, self).__init__()
>> >
>> >         self._parent = parent
>> >         self.panel = wx.Panel(parent)
>> >
>> >     def run(self):
>> >
>> >         while not self._abort:
>> >
>> >             #numpy arrays reverse pixel data that comes in from CCD as
>> BGR
>> > and bottom to top
>> >
>> >             pixeldata = np.ndarray((480,640,3),
>> > buffer=webcam.getBuffer()[0], dtype='u1')
>> >             revpixels = pixeldata[::-1,:,::-1].tostring()      #tostring
>> is
>> > an order of magnitude slower than the entire array manipulation. need a
>> > faster method.
>> >
>> >             rgbBMP = wx.BitmapFromBuffer(640, 480, revpixels)
>> >             dc = wx.ClientDC(self.panel)
>> >             dc.DrawBitmap(rgbBMP, 0, 0)
>> >             #b = time.clock()
>> >             #print b-a
>> >             time.sleep(0.015)
>> >
>> >
>> >         return
>> >
>> >
>> >
>> > class TestWxWebCam(wx.Frame):
>> >     def __init__(self, parent):
>> >         wx.Frame.__init__(self, parent, -1, size=(640,480))
>> >
>> >         self.worker = WebCamWorker(self)
>> >         self.worker.start()
>> >
>> >
>> >
>> > class TestApp(wx.App):
>> >     def OnInit(self):
>> >         self.frame = TestWxWebCam(None)
>> >         self.frame.Show()
>> >
>> >         return True
>> >
>> >     def OnExit(self):
>> >         WebCamWorker._abort = 1 #this is a hack
>> >
>> >
>> > if __name__ == '__main__':
>> >     webcam = VideoCapture.Device()
>> >     webcam.setResolution(640, 480)
>> >     webcam.displayCapturePinProperties()
>> >     app = TestApp()
>> >     app.MainLoop()
>> >     del webcam
>> >
>> >
>> > ### /Code ######
>> >
>> > _______________________________________________
>> > Numpy-discussion mailing list
>> > Numpy-discussion at scipy.org
>> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> >
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090224/4da19b58/attachment.html>

From cournape at gmail.com  Tue Feb 24 22:39:33 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 25 Feb 2009 12:39:33 +0900
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <e06186140902241026h33514e5eod0c94fae4622b6c5@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org> <loom.20090224T140326-52@post.gmane.org>
	<e06186140902240921p3ba383elc40494863a40e832@mail.gmail.com>
	<5b8d13220902240937s70617f9dw996b456eecd8d134@mail.gmail.com>
	<e76aa17f0902241009s35a00885kabfa26f4b56c088d@mail.gmail.com>
	<e06186140902241026h33514e5eod0c94fae4622b6c5@mail.gmail.com>
Message-ID: <5b8d13220902241939n51ecc412v476daaf056441d14@mail.gmail.com>

On Wed, Feb 25, 2009 at 3:26 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:

>
> Good point. However, most of the ufuncs involving standard functions like
> sin, cos, etc. are implemented as generic loops that are passed a function
> pointer and for such functions the call overhead is probably not significant
> in the absence of intrinsic hardware implementations. The complex functions
> could probably use some inlining as they call other functions. That could
> implemented by using some local static functions in the library code that
> could be inlined when the library is compiled.
>
> I think that the first priority should be correctness and portability. Speed
> optimizations can come later.

I agree. I can see plenty of advantages to force a function call - and
I have not seen any clear estimation of the function call cost. If it
is a burden, this can be improved later (AFAIK, inline is not a
modification of the signature).

David


From oliphant at enthought.com  Tue Feb 24 23:58:51 2009
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Tue, 24 Feb 2009 22:58:51 -0600
Subject: [Numpy-discussion] Creating arrays with 'titles' in dtype
 causes TypeError on data access
In-Reply-To: <49A40813.4090206@dont-mind.de>
References: <49A40813.4090206@dont-mind.de>
Message-ID: <49A4D00B.1010109@enthought.com>

Ralph Heinkel wrote:
> However here something is strange:
>
>   
>>>> arr = array([('john', 4),('mark', 3)],
>>>>         
>      dtype=[(('source:yy', 'name'),'O'),(('source:xx','id'),int)])
>   
>>>> arr[0]
>>>>         
> ('john', 4)
>   
>>>> arr[0][0]
>>>>         
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> TypeError: function takes at most 2 arguments (3 given)
>
>
> Any ideas what I'm doing wrong? Any help would be appreciated.
>
>   
This is a bug in NumPy.     I will fix it in the trunk tonight.

-Travis


-- 

Travis Oliphant
Enthought, Inc.
(512) 536-1057 (office)
(512) 536-1059 (fax)
http://www.enthought.com
oliphant at enthought.com



From strawman at astraw.com  Wed Feb 25 01:22:05 2009
From: strawman at astraw.com (Andrew Straw)
Date: Tue, 24 Feb 2009 22:22:05 -0800
Subject: [Numpy-discussion] is there a faster way to get a
 buffer	interface than ndarray.tostring()?
In-Reply-To: <7f014ea60902241849s49e929dfsf8fc8201290a1feb@mail.gmail.com>
References: <7f014ea60902241615l18ddd46am541af69b12ac381e@mail.gmail.com>	<e7ba66e40902241706v5ea2782btaf743b405ae976b0@mail.gmail.com>	<7f014ea60902241827n4a453c8fn48e9836e71ad05bf@mail.gmail.com>
	<7f014ea60902241849s49e929dfsf8fc8201290a1feb@mail.gmail.com>
Message-ID: <49A4E38D.4030204@astraw.com>

Given what you're doing, may I also suggest having a look at
http://code.astraw.com/projects/motmot/wxglvideo.html

-Andrew

Chris Colbert wrote:
> As an update for any future googlers: 
> 
> the problem was with revpixels = pixeldata[::-1,:,;:-1] which apparently
> returns an array that is discontinuous in memory. What Lisandro
> suggested worked. pixeldata[::-1,:,;:-1].copy() returns a continuous
> array object which natively implements a single-segment buffer
> interface. i.e. no buffer(revpixels) is needed; just simply revpixels. 
> 
> array.copy() is also 50% faster than array.tostring() on my machine. 
> 
> Chris
> 
> On Tue, Feb 24, 2009 at 9:27 PM, Chris Colbert <sccolbert at gmail.com
> <mailto:sccolbert at gmail.com>> wrote:
> 
>     thanks for both answers! 
> 
> 
>     Lisandro, you're right, I should have declared the array outside the
>     loop. Thanks for catching that!
> 
>     Robert, as always, thanks for the answer. Quick and to the point!
>     You've helped me more than once on the enthought list :)
> 
> 
> 
>     On Tue, Feb 24, 2009 at 8:06 PM, Lisandro Dalcin <dalcinl at gmail.com
>     <mailto:dalcinl at gmail.com>> wrote:
> 
>         When you do pixeldata[::-1,:,::-1], you just got a new array with
>         different strides, but now non-contiguous... So I believe you really
>         need a fresh copy of the data... tostring() copies, but could be
>         slow... try to use
> 
>         revpixels = pixeldata[::-1,:,::-1].copy()
> 
>         ...
> 
>         rgbBMP = wx.BitmapFromBuffer(640, 480, buffer(revpixels))
> 
> 
>         Perhaps you could optimize this by declaring revpixels outside de
>         loop, and then inside de loop doing
> 
>         revpixels[...] = pixeldata[::-1,:,::-1]
> 
> 
>         This way you will save the mem allocation of revpixels at each
>         step of the loop.
> 
> 
> 
> 
> 
>         On Tue, Feb 24, 2009 at 9:15 PM, Chris Colbert
>         <sccolbert at gmail.com <mailto:sccolbert at gmail.com>> wrote:
>         > Hi all,
>         >
>         >  I'm new to mailing list and relatively new (~1 year) to
>         python/numpy. I
>         > would appreciate any insight any of you may have here. The
>         last 8 hours of
>         > digging through the docs has left me, finally, stuck.
>         >
>         > I am making a wxPython program that includes webcam
>         functionality. The
>         > script below just brings up a simple display window that shows
>         the webcam
>         > feed. The problem I am running into is in the
>         WebCamWorker.run() method.
>         > This method reads the raw pixel data (in string form) from the
>         webcam
>         > buffer. This data (thank you microsoft!) comes in BGR and
>         bottom-to-top. I
>         > feed this buffer to the array constructor and then swap the
>         pixel order to
>         > RGB and top-to-bottom. This all happens very fast, ~1ms on a
>         Quad Core, and
>         > the majority of that time is spent constructing the array (the
>         numpy pixel
>         > swapping is on the order of E-5s !). The tostring() method,
>         however, takes a
>         > whopping 10ms to execute. Unfortunately, the
>         wx.BitmapFromBuffer needs a
>         > single segment buffer interface as an argument. Is there a way
>         to expose my
>         > numpy array as a buffer to cut down on this conversion time?
>         >
>         > I have tried sending the array to the PIL Image.fromarray(),
>         but that
>         > eventually requires me to do a Image.tostring() anyway, which
>         negates any
>         > benefit.
>         >
>         > Thanks in advance for the help!
>         >
>         > S. Chris Colbert
>         > Rehabilitation Robotics Laboratory
>         > University of South Florida
>         >
>         >
>         > ####  Code #######
>         >
>         > import VideoCapture
>         > import wx
>         > import time
>         > import threading
>         > import numpy as np
>         > import Image
>         >
>         >
>         > class WebCamWorker(threading.Thread):
>         >     _abort = 0
>         >
>         >     def __init__(self, parent):
>         >         super(WebCamWorker, self).__init__()
>         >
>         >         self._parent = parent
>         >         self.panel = wx.Panel(parent)
>         >
>         >     def run(self):
>         >
>         >         while not self._abort:
>         >
>         >             #numpy arrays reverse pixel data that comes in
>         from CCD as BGR
>         > and bottom to top
>         >
>         >             pixeldata = np.ndarray((480,640,3),
>         > buffer=webcam.getBuffer()[0], dtype='u1')
>         >             revpixels = pixeldata[::-1,:,::-1].tostring()     
>         #tostring is
>         > an order of magnitude slower than the entire array
>         manipulation. need a
>         > faster method.
>         >
>         >             rgbBMP = wx.BitmapFromBuffer(640, 480, revpixels)
>         >             dc = wx.ClientDC(self.panel)
>         >             dc.DrawBitmap(rgbBMP, 0, 0)
>         >             #b = time.clock()
>         >             #print b-a
>         >             time.sleep(0.015)
>         >
>         >
>         >         return
>         >
>         >
>         >
>         > class TestWxWebCam(wx.Frame):
>         >     def __init__(self, parent):
>         >         wx.Frame.__init__(self, parent, -1, size=(640,480))
>         >
>         >         self.worker = WebCamWorker(self)
>         >         self.worker.start()
>         >
>         >
>         >
>         > class TestApp(wx.App):
>         >     def OnInit(self):
>         >         self.frame = TestWxWebCam(None)
>         >         self.frame.Show()
>         >
>         >         return True
>         >
>         >     def OnExit(self):
>         >         WebCamWorker._abort = 1 #this is a hack
>         >
>         >
>         > if __name__ == '__main__':
>         >     webcam = VideoCapture.Device()
>         >     webcam.setResolution(640, 480)
>         >     webcam.displayCapturePinProperties()
>         >     app = TestApp()
>         >     app.MainLoop()
>         >     del webcam
>         >
>         >
>         > ### /Code ######
>         >
>         > _______________________________________________
>         > Numpy-discussion mailing list
>         > Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
>         > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>         >
>         >
> 
> 
> 
>         --
>         Lisandro Dalc?n
>         ---------------
>         Centro Internacional de M?todos Computacionales en Ingenier?a
>         (CIMEC)
>         Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica
>         (INTEC)
>         Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>         PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>         Tel/Fax: +54-(0)342-451.1594
>         _______________________________________________
>         Numpy-discussion mailing list
>         Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
>         http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From sccolbert at gmail.com  Wed Feb 25 01:50:18 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 25 Feb 2009 01:50:18 -0500
Subject: [Numpy-discussion] is there a faster way to get a buffer
	interface than ndarray.tostring()?
In-Reply-To: <49A4E38D.4030204@astraw.com>
References: <7f014ea60902241615l18ddd46am541af69b12ac381e@mail.gmail.com>
	<e7ba66e40902241706v5ea2782btaf743b405ae976b0@mail.gmail.com>
	<7f014ea60902241827n4a453c8fn48e9836e71ad05bf@mail.gmail.com>
	<7f014ea60902241849s49e929dfsf8fc8201290a1feb@mail.gmail.com>
	<49A4E38D.4030204@astraw.com>
Message-ID: <7f014ea60902242250x28c968d5i788a44385c5397cc@mail.gmail.com>

thanks!

On Wed, Feb 25, 2009 at 1:22 AM, Andrew Straw <strawman at astraw.com> wrote:

> Given what you're doing, may I also suggest having a look at
> http://code.astraw.com/projects/motmot/wxglvideo.html
>
> -Andrew
>
> Chris Colbert wrote:
> > As an update for any future googlers:
> >
> > the problem was with revpixels = pixeldata[::-1,:,;:-1] which apparently
> > returns an array that is discontinuous in memory. What Lisandro
> > suggested worked. pixeldata[::-1,:,;:-1].copy() returns a continuous
> > array object which natively implements a single-segment buffer
> > interface. i.e. no buffer(revpixels) is needed; just simply revpixels.
> >
> > array.copy() is also 50% faster than array.tostring() on my machine.
> >
> > Chris
> >
> > On Tue, Feb 24, 2009 at 9:27 PM, Chris Colbert <sccolbert at gmail.com
> > <mailto:sccolbert at gmail.com>> wrote:
> >
> >     thanks for both answers!
> >
> >
> >     Lisandro, you're right, I should have declared the array outside the
> >     loop. Thanks for catching that!
> >
> >     Robert, as always, thanks for the answer. Quick and to the point!
> >     You've helped me more than once on the enthought list :)
> >
> >
> >
> >     On Tue, Feb 24, 2009 at 8:06 PM, Lisandro Dalcin <dalcinl at gmail.com
> >     <mailto:dalcinl at gmail.com>> wrote:
> >
> >         When you do pixeldata[::-1,:,::-1], you just got a new array with
> >         different strides, but now non-contiguous... So I believe you
> really
> >         need a fresh copy of the data... tostring() copies, but could be
> >         slow... try to use
> >
> >         revpixels = pixeldata[::-1,:,::-1].copy()
> >
> >         ...
> >
> >         rgbBMP = wx.BitmapFromBuffer(640, 480, buffer(revpixels))
> >
> >
> >         Perhaps you could optimize this by declaring revpixels outside de
> >         loop, and then inside de loop doing
> >
> >         revpixels[...] = pixeldata[::-1,:,::-1]
> >
> >
> >         This way you will save the mem allocation of revpixels at each
> >         step of the loop.
> >
> >
> >
> >
> >
> >         On Tue, Feb 24, 2009 at 9:15 PM, Chris Colbert
> >         <sccolbert at gmail.com <mailto:sccolbert at gmail.com>> wrote:
> >         > Hi all,
> >         >
> >         >  I'm new to mailing list and relatively new (~1 year) to
> >         python/numpy. I
> >         > would appreciate any insight any of you may have here. The
> >         last 8 hours of
> >         > digging through the docs has left me, finally, stuck.
> >         >
> >         > I am making a wxPython program that includes webcam
> >         functionality. The
> >         > script below just brings up a simple display window that shows
> >         the webcam
> >         > feed. The problem I am running into is in the
> >         WebCamWorker.run() method.
> >         > This method reads the raw pixel data (in string form) from the
> >         webcam
> >         > buffer. This data (thank you microsoft!) comes in BGR and
> >         bottom-to-top. I
> >         > feed this buffer to the array constructor and then swap the
> >         pixel order to
> >         > RGB and top-to-bottom. This all happens very fast, ~1ms on a
> >         Quad Core, and
> >         > the majority of that time is spent constructing the array (the
> >         numpy pixel
> >         > swapping is on the order of E-5s !). The tostring() method,
> >         however, takes a
> >         > whopping 10ms to execute. Unfortunately, the
> >         wx.BitmapFromBuffer needs a
> >         > single segment buffer interface as an argument. Is there a way
> >         to expose my
> >         > numpy array as a buffer to cut down on this conversion time?
> >         >
> >         > I have tried sending the array to the PIL Image.fromarray(),
> >         but that
> >         > eventually requires me to do a Image.tostring() anyway, which
> >         negates any
> >         > benefit.
> >         >
> >         > Thanks in advance for the help!
> >         >
> >         > S. Chris Colbert
> >         > Rehabilitation Robotics Laboratory
> >         > University of South Florida
> >         >
> >         >
> >         > ####  Code #######
> >         >
> >         > import VideoCapture
> >         > import wx
> >         > import time
> >         > import threading
> >         > import numpy as np
> >         > import Image
> >         >
> >         >
> >         > class WebCamWorker(threading.Thread):
> >         >     _abort = 0
> >         >
> >         >     def __init__(self, parent):
> >         >         super(WebCamWorker, self).__init__()
> >         >
> >         >         self._parent = parent
> >         >         self.panel = wx.Panel(parent)
> >         >
> >         >     def run(self):
> >         >
> >         >         while not self._abort:
> >         >
> >         >             #numpy arrays reverse pixel data that comes in
> >         from CCD as BGR
> >         > and bottom to top
> >         >
> >         >             pixeldata = np.ndarray((480,640,3),
> >         > buffer=webcam.getBuffer()[0], dtype='u1')
> >         >             revpixels = pixeldata[::-1,:,::-1].tostring()
> >         #tostring is
> >         > an order of magnitude slower than the entire array
> >         manipulation. need a
> >         > faster method.
> >         >
> >         >             rgbBMP = wx.BitmapFromBuffer(640, 480, revpixels)
> >         >             dc = wx.ClientDC(self.panel)
> >         >             dc.DrawBitmap(rgbBMP, 0, 0)
> >         >             #b = time.clock()
> >         >             #print b-a
> >         >             time.sleep(0.015)
> >         >
> >         >
> >         >         return
> >         >
> >         >
> >         >
> >         > class TestWxWebCam(wx.Frame):
> >         >     def __init__(self, parent):
> >         >         wx.Frame.__init__(self, parent, -1, size=(640,480))
> >         >
> >         >         self.worker = WebCamWorker(self)
> >         >         self.worker.start()
> >         >
> >         >
> >         >
> >         > class TestApp(wx.App):
> >         >     def OnInit(self):
> >         >         self.frame = TestWxWebCam(None)
> >         >         self.frame.Show()
> >         >
> >         >         return True
> >         >
> >         >     def OnExit(self):
> >         >         WebCamWorker._abort = 1 #this is a hack
> >         >
> >         >
> >         > if __name__ == '__main__':
> >         >     webcam = VideoCapture.Device()
> >         >     webcam.setResolution(640, 480)
> >         >     webcam.displayCapturePinProperties()
> >         >     app = TestApp()
> >         >     app.MainLoop()
> >         >     del webcam
> >         >
> >         >
> >         > ### /Code ######
> >         >
> >         > _______________________________________________
> >         > Numpy-discussion mailing list
> >         > Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
> >         > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >         >
> >         >
> >
> >
> >
> >         --
> >         Lisandro Dalc?n
> >         ---------------
> >         Centro Internacional de M?todos Computacionales en Ingenier?a
> >         (CIMEC)
> >         Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica
> >         (INTEC)
> >         Consejo Nacional de Investigaciones Cient?ficas y T?cnicas
> (CONICET)
> >         PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> >         Tel/Fax: +54-(0)342-451.1594
> >         _______________________________________________
> >         Numpy-discussion mailing list
> >         Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
> >         http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090225/1843c34a/attachment.html>

From matthieu.brucher at gmail.com  Wed Feb 25 03:27:32 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 25 Feb 2009 09:27:32 +0100
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <5b8d13220902241939n51ecc412v476daaf056441d14@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org> <loom.20090224T140326-52@post.gmane.org>
	<e06186140902240921p3ba383elc40494863a40e832@mail.gmail.com>
	<5b8d13220902240937s70617f9dw996b456eecd8d134@mail.gmail.com>
	<e76aa17f0902241009s35a00885kabfa26f4b56c088d@mail.gmail.com>
	<e06186140902241026h33514e5eod0c94fae4622b6c5@mail.gmail.com>
	<5b8d13220902241939n51ecc412v476daaf056441d14@mail.gmail.com>
Message-ID: <e76aa17f0902250027u2f94fd96yb2817f3a58421a90@mail.gmail.com>

(AFAIK, inline is not a
> modification of the signature).

Indeed, even more so for C.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From ondrej at certik.cz  Wed Feb 25 04:47:41 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 25 Feb 2009 01:47:41 -0800
Subject: [Numpy-discussion] Problem looping over numpy array in C
In-Reply-To: <35905.128.111.8.119.1235161164.squirrel@webmail.xs4all.nl>
References: <35905.128.111.8.119.1235161164.squirrel@webmail.xs4all.nl>
Message-ID: <85b5c3130902250147y4f47a185hcae0f4f93fa9d96c@mail.gmail.com>

On Fri, Feb 20, 2009 at 12:19 PM, Sumant S.R. Oemrawsingh
<soemraws at xs4all.nl> wrote:
> Hi guys,
>
> I have a problem with looping over numpy arrays in C. I modify the array
> in-place (and return None), but after modification, the array doesn't seem
> to play nice any more.
>
> Below, I have the C code for a function do_something (a stripped version
> of my original function), which has as arguments a 1D numpy array (float
> or complex) and either an int or a sequence of ints.
>
> In python, I do the following using only the integer:
>
>>>> a = array([0., 1., 2., 3., 2., 1., 0.])
>>>> do_something(a,3)
>>>> savetxt('bla',a)
>>>>
>
> Which works fine. However, when I do the same, but with a list of any
> length larger than 0:
>
>>>> a = array([0., 1., 2., 3., 2., 1., 0.])
>>>> do_something(a,[3])
>>>> savetxt('bla',a)
> Traceback (most recent call last):
> ?File "<stdin>", line 1, in <module>
> ?File "/usr/lib/python2.5/site-packages/numpy/core/numeric.py", line 767,
> in savetxt
> ? ?X = asarray(X)
> ?File "/usr/lib/python2.5/site-packages/numpy/core/numeric.py", line 132,
> in asarray
> ? ?return array(a, dtype, copy=False, order=order)
> TypeError: an integer is required
>>>> savetxt('bla',a)
>>>>
>
> For some reason, the first time it doesn't work; the asarray fails but
> then succeeds on a second try. The resulting file is identical to when I
> would have looped over the list in python, and called the do_something
> function with each integer separately. For instance:
>
> for i in [3,1,0]:
> ?do_something(a, i)
>
> works fine as well. So apparently looping in C temporarily leaves the
> array in a weird state but manages to automatically be restored after one
> exception. I've checked that type(a), a.dtype, a.shape and a.ndim remain
> the same before and after calling do_something with a sequence or with an
> integer as second argument. That doesn't seem to be the problem.
>
> The reason that I want do the loop in C is that I need some
> precalculations, before being able to do the actual loop. But they might
> not be time-consuming enough to warrant writing it in C, so I could just
> do the loop in python and not have this problem any more. However, if the
> loop in C manages to somehow (temporarily) corrupt the array, how can I be
> sure that the single-shot call doesn't succeed by accident?
>
> If anyone can help, suggest something to try or spot my mistake, I'd
> appreciate it.

I don't have time to go through your code, but I suggest you try
Cython, that plays very nice with numpy.

Ondrej


From ondrej at certik.cz  Wed Feb 25 04:49:29 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 25 Feb 2009 01:49:29 -0800
Subject: [Numpy-discussion] denormal test: what is it for ?
In-Reply-To: <5b8d13220902170808p7848b42eo2e8a47257478b79@mail.gmail.com>
References: <5b8d13220902170808p7848b42eo2e8a47257478b79@mail.gmail.com>
Message-ID: <85b5c3130902250149h51e7716es84b997cd941b51a1@mail.gmail.com>

On Tue, Feb 17, 2009 at 8:08 AM, David Cournapeau <cournape at gmail.com> wrote:
> Hi,
>
> I would like to continue cleaning the setup.py from numpy/core, and
> there is one test that I can't make sense of: the denormal thing
> (testcode_mathlib function). The svn log has no information on it (its
> presence goes back to the first revision of the file). What is it
> useful for ? Denormal support ? Whether exp works ?

I guess noone knows. :)

Ondrej


From slaunger at gmail.com  Wed Feb 25 07:28:26 2009
From: slaunger at gmail.com (Kim Hansen)
Date: Wed, 25 Feb 2009 13:28:26 +0100
Subject: [Numpy-discussion] Numpy array in iterable
Message-ID: <f2e7a4240902250428ufd52ae3ye746164d46e99546@mail.gmail.com>

Hi Numpy discussions
Quite often I find myself wanting to generate a boolean mask for fancy
slicing of some array, where the mask itself is generated by checking
if its value has one of several relevant values (corresponding to
states)
So at the the element level thsi corresponds to checking if
element in iterable
But I can't use the in operator on a numpy array:

In [1]: test = arange(5)
In [2]: states = [0, 2]
In [3]: mask = test in states
---------------------------------------------------------------------------
ValueError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Traceback (most recent call last)
C:\Documents and Settings\kha\<ipython console> in <module>()
ValueError: The truth value of an array with more than one element is ambiguous.
Use a.any() or a.all()

I can however make my own utility function which works effectively the
same way by iterating through the states

In [4]: for i, state in enumerate(states):
? ...: ? ? if i == 0:
? ...: ? ? ? ? result = test == state
? ...: ? ? else:
? ...: ? ? ? ? result |= test == state
? ...:
? ...:
In [5]: result
Out[5]: array([ True, False, ?True, False, False], dtype=bool)

However, I would have thought such an "array.is_in()" utility function
was already available in the numpy package?

But I can't find it, and I am curious to hear if it is there or if it
just available in another form which I have simply overlooked.

If it is not there I think it could be a nice extra utility funtion
for the ndarray object.

--Slaunger


From josef.pktd at gmail.com  Wed Feb 25 08:40:47 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 25 Feb 2009 08:40:47 -0500
Subject: [Numpy-discussion] Numpy array in iterable
In-Reply-To: <f2e7a4240902250428ufd52ae3ye746164d46e99546@mail.gmail.com>
References: <f2e7a4240902250428ufd52ae3ye746164d46e99546@mail.gmail.com>
Message-ID: <1cd32cbb0902250540u66da1969pfe481dafbb061cea@mail.gmail.com>

On Wed, Feb 25, 2009 at 7:28 AM, Kim Hansen <slaunger at gmail.com> wrote:
> Hi Numpy discussions
> Quite often I find myself wanting to generate a boolean mask for fancy
> slicing of some array, where the mask itself is generated by checking
> if its value has one of several relevant values (corresponding to
> states)
> So at the the element level thsi corresponds to checking if
> element in iterable
> But I can't use the in operator on a numpy array:
>
> In [1]: test = arange(5)
> In [2]: states = [0, 2]
> In [3]: mask = test in states
> ---------------------------------------------------------------------------
> ValueError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Traceback (most recent call last)
> C:\Documents and Settings\kha\<ipython console> in <module>()
> ValueError: The truth value of an array with more than one element is ambiguous.
> Use a.any() or a.all()
>
> I can however make my own utility function which works effectively the
> same way by iterating through the states
>
> In [4]: for i, state in enumerate(states):
> ? ...: ? ? if i == 0:
> ? ...: ? ? ? ? result = test == state
> ? ...: ? ? else:
> ? ...: ? ? ? ? result |= test == state
> ? ...:
> ? ...:
> In [5]: result
> Out[5]: array([ True, False, ?True, False, False], dtype=bool)
>
> However, I would have thought such an "array.is_in()" utility function
> was already available in the numpy package?
>
> But I can't find it, and I am curious to hear if it is there or if it
> just available in another form which I have simply overlooked.
>
> If it is not there I think it could be a nice extra utility funtion
> for the ndarray object.
>
> --Slaunger
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

does this help:

>>> np.setmember1d(test,states)
array([ True, False,  True, False, False], dtype=bool)

Josef


From slaunger at gmail.com  Wed Feb 25 09:02:24 2009
From: slaunger at gmail.com (Kim Hansen)
Date: Wed, 25 Feb 2009 15:02:24 +0100
Subject: [Numpy-discussion] Numpy array in iterable
In-Reply-To: <1cd32cbb0902250540u66da1969pfe481dafbb061cea@mail.gmail.com>
References: <f2e7a4240902250428ufd52ae3ye746164d46e99546@mail.gmail.com>
	<1cd32cbb0902250540u66da1969pfe481dafbb061cea@mail.gmail.com>
Message-ID: <f2e7a4240902250602h7df75f3an324e45ad9a81bbf8@mail.gmail.com>

Yes, this is exactly what I was after, only the function name did not
ring a bell (I still cannot associate it with something meaningful for
my use case). Thanks!

-- Slaunger

2009/2/25  <josef.pktd at gmail.com>:
> On Wed, Feb 25, 2009 at 7:28 AM, Kim Hansen <slaunger at gmail.com> wrote:
>> Hi Numpy discussions
>> Quite often I find myself wanting to generate a boolean mask for fancy
>> slicing of some array, where the mask itself is generated by checking
>> if its value has one of several relevant values (corresponding to
>> states)
>> So at the the element level thsi corresponds to checking if
>> element in iterable
>> But I can't use the in operator on a numpy array:
>>
>> In [1]: test = arange(5)
>> In [2]: states = [0, 2]
>> In [3]: mask = test in states
>> ---------------------------------------------------------------------------
>> ValueError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Traceback (most recent call last)
>> C:\Documents and Settings\kha\<ipython console> in <module>()
>> ValueError: The truth value of an array with more than one element is ambiguous.
>> Use a.any() or a.all()
>>
>> I can however make my own utility function which works effectively the
>> same way by iterating through the states
>>
>> In [4]: for i, state in enumerate(states):
>> ? ...: ? ? if i == 0:
>> ? ...: ? ? ? ? result = test == state
>> ? ...: ? ? else:
>> ? ...: ? ? ? ? result |= test == state
>> ? ...:
>> ? ...:
>> In [5]: result
>> Out[5]: array([ True, False, ?True, False, False], dtype=bool)
>>
>> However, I would have thought such an "array.is_in()" utility function
>> was already available in the numpy package?
>>
>> But I can't find it, and I am curious to hear if it is there or if it
>> just available in another form which I have simply overlooked.
>>
>> If it is not there I think it could be a nice extra utility funtion
>> for the ndarray object.
>>
>> --Slaunger
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>
> does this help:
>
>>>> np.setmember1d(test,states)
> array([ True, False, ?True, False, False], dtype=bool)
>
> Josef
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From josef.pktd at gmail.com  Wed Feb 25 09:33:11 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 25 Feb 2009 09:33:11 -0500
Subject: [Numpy-discussion] Numpy array in iterable
In-Reply-To: <f2e7a4240902250602h7df75f3an324e45ad9a81bbf8@mail.gmail.com>
References: <f2e7a4240902250428ufd52ae3ye746164d46e99546@mail.gmail.com>
	<1cd32cbb0902250540u66da1969pfe481dafbb061cea@mail.gmail.com>
	<f2e7a4240902250602h7df75f3an324e45ad9a81bbf8@mail.gmail.com>
Message-ID: <1cd32cbb0902250633y121e134fv2ade89ee200875df@mail.gmail.com>

On Wed, Feb 25, 2009 at 9:02 AM, Kim Hansen <slaunger at gmail.com> wrote:
> Yes, this is exactly what I was after, only the function name did not
> ring a bell (I still cannot associate it with something meaningful for
> my use case). Thanks!
>
> -- Slaunger
>

I just looked under "set routines" in the help file. I really like the
speed of the windows help file.

Josef


From slaunger at gmail.com  Wed Feb 25 13:37:16 2009
From: slaunger at gmail.com (Kim Hansen)
Date: Wed, 25 Feb 2009 19:37:16 +0100
Subject: [Numpy-discussion] Numpy array in iterable
In-Reply-To: <1cd32cbb0902250633y121e134fv2ade89ee200875df@mail.gmail.com>
References: <f2e7a4240902250428ufd52ae3ye746164d46e99546@mail.gmail.com>
	<1cd32cbb0902250540u66da1969pfe481dafbb061cea@mail.gmail.com>
	<f2e7a4240902250602h7df75f3an324e45ad9a81bbf8@mail.gmail.com>
	<1cd32cbb0902250633y121e134fv2ade89ee200875df@mail.gmail.com>
Message-ID: <f2e7a4240902251037q2d1645c8w60927166c29b8d8f@mail.gmail.com>

> I just looked under "set routines" in the help file. I really like the
> speed of the windows help file.

Is there a Numpy windows help file?

Cool!

But where is it? I can't find it in  my numpy 1.2.1 installation?!?

I like the Python 2.5 Windows help file too and I agree it is a fast
and efficient way to find what you need, fast.

--Slaunger


From josef.pktd at gmail.com  Wed Feb 25 14:21:40 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 25 Feb 2009 14:21:40 -0500
Subject: [Numpy-discussion] Numpy array in iterable
In-Reply-To: <f2e7a4240902251037q2d1645c8w60927166c29b8d8f@mail.gmail.com>
References: <f2e7a4240902250428ufd52ae3ye746164d46e99546@mail.gmail.com>
	<1cd32cbb0902250540u66da1969pfe481dafbb061cea@mail.gmail.com>
	<f2e7a4240902250602h7df75f3an324e45ad9a81bbf8@mail.gmail.com>
	<1cd32cbb0902250633y121e134fv2ade89ee200875df@mail.gmail.com>
	<f2e7a4240902251037q2d1645c8w60927166c29b8d8f@mail.gmail.com>
Message-ID: <1cd32cbb0902251121o15879bc7hd1a3f8f1f718572b@mail.gmail.com>

On Wed, Feb 25, 2009 at 1:37 PM, Kim Hansen <slaunger at gmail.com> wrote:
>> I just looked under "set routines" in the help file. I really like the
>> speed of the windows help file.
>
> Is there a Numpy windows help file?
>
> Cool!
>
> But where is it? I can't find it in ?my numpy 1.2.1 installation?!?
>
> I like the Python 2.5 Windows help file too and I agree it is a fast
> and efficient way to find what you need, fast.
>
> --Slaunger

You can download them from here: http://docs.scipy.org/doc/
But since the docs are continuously improved, it is also worth to
check the online documentation if the help file is not very
informative about something.

Josef


From Anthony.Kong at macquarie.com  Wed Feb 25 18:21:48 2009
From: Anthony.Kong at macquarie.com (Anthony Kong)
Date: Thu, 26 Feb 2009 10:21:48 +1100
Subject: [Numpy-discussion] Question on lstsq and correlation coeff
Message-ID: <90CBFFFE6273484B9579400AC950800502024765@ntsydexm01.pc.internal.macquarie.com>

Hi, all,
 
It is probably a newbie question. 
 
I trying to use scipy/numpy in a finanical context. I want to compute
the correlation coeff of two series (returns vs index returns). I tried
two appoarches
 
Firstly,
 
from scipy.linalg import lstsq
coeffs,a,b,c = lstsq(matrix, returns) # matrix contains index returns
 
then I tried,
 
import numpy as np
cov = np.cov(idx1, returns)
print cov.tolist()
stddev_x = np.std(returns, ddof=1)
stddev_y = np.std(idx1, ddof=1)
print "cor = %s" % (cov.tolist()[:-1] /(stddev_x * stddev_y))

They differ from each other.
 
As you can see from the numpy example, I am trying to find cor coeff for
a sample. (ddof=1)
 
So, my question is: is the discrepency caused by the fact that I am
trying to use lstsq() on a 'sample popluation' (i.e. I am not regressing
a full return series)? Is it correct to use lstsq() this way?
 
Cheers, Anthony

NOTICE
This e-mail and any attachments are confidential and may contain copyright material of Macquarie Group Limited or third parties. If you are not the intended recipient of this email you should not read, print, re-transmit, store or act in reliance on this e-mail or any attachments, and should destroy all copies of them. Macquarie Group Limited does not guarantee the integrity of any emails or any attached files. The views or opinions expressed are the author's own and may not reflect the views or opinions of Macquarie Group Limited.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090226/b89d1040/attachment.html>

From josef.pktd at gmail.com  Wed Feb 25 19:08:48 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 25 Feb 2009 19:08:48 -0500
Subject: [Numpy-discussion] Question on lstsq and correlation coeff
In-Reply-To: <90CBFFFE6273484B9579400AC950800502024765@ntsydexm01.pc.internal.macquarie.com>
References: <90CBFFFE6273484B9579400AC950800502024765@ntsydexm01.pc.internal.macquarie.com>
Message-ID: <1cd32cbb0902251608x5b407a4dv9ec0dccf55d7d3ca@mail.gmail.com>

On Wed, Feb 25, 2009 at 6:21 PM, Anthony Kong
<Anthony.Kong at macquarie.com> wrote:
> Hi, all,
>
> It is probably a newbie question.
>
> I?trying to use?scipy/numpy in a finanical context. I want to compute the
> correlation coeff of two series (returns vs index returns). I tried two
> appoarches
>
> Firstly,
>
> from scipy.linalg import lstsq
> coeffs,a,b,c = lstsq(matrix, returns) # matrix contains index returns
>
> then I tried,
>
> import numpy as np
> cov = np.cov(idx1, returns)
> print cov.tolist()
> stddev_x = np.std(returns, ddof=1)
> stddev_y = np.std(idx1, ddof=1)
> print "cor = %s" % (cov.tolist()[:-1] /(stddev_x * stddev_y))
> They differ from each other.
>
> As you can see from the numpy example, I am trying to?find cor?coeff for?a
> sample.?(ddof=1)
>
> So, my question is: is the discrepency caused by the fact that I am trying
> to use lstsq() on a 'sample popluation' (i.e. I am not regressing a full
> return series)? Is it correct to use lstsq() this way?
>

the most direct way to calculate the correlation matrix, use index
[0,1] to get coefficient.

numpy.corrcoef(x, y=None, rowvar=1, bias=0)

np.cov, that you used, uses biased estimator, denominator = N by
default, but for std you used N-1

Josef


From Anthony.Kong at macquarie.com  Wed Feb 25 19:31:37 2009
From: Anthony.Kong at macquarie.com (Anthony Kong)
Date: Thu, 26 Feb 2009 11:31:37 +1100
Subject: [Numpy-discussion] Question on lstsq and correlation coeff
In-Reply-To: <1cd32cbb0902251608x5b407a4dv9ec0dccf55d7d3ca@mail.gmail.com>
References: <90CBFFFE6273484B9579400AC950800502024765@ntsydexm01.pc.internal.macquarie.com>
	<1cd32cbb0902251608x5b407a4dv9ec0dccf55d7d3ca@mail.gmail.com>
Message-ID: <90CBFFFE6273484B9579400AC950800502024896@ntsydexm01.pc.internal.macquarie.com>


 Hi, Josef,

Thanks very much for the quick and helpful response.

Could you also comment on the use of lstsq(): Why it leads to inconsistent result?

Cheers, Anthony

-----Original Message-----
From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of josef.pktd at gmail.com
Sent: Thursday, 26 February 2009 11:09 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Question on lstsq and correlation coeff

On Wed, Feb 25, 2009 at 6:21 PM, Anthony Kong <Anthony.Kong at macquarie.com> wrote:
> Hi, all,
>
> It is probably a newbie question.
>
> I?trying to use?scipy/numpy in a finanical context. I want to compute 
> the correlation coeff of two series (returns vs index returns). I 
> tried two appoarches
>
> Firstly,
>
> from scipy.linalg import lstsq
> coeffs,a,b,c = lstsq(matrix, returns) # matrix contains index returns
>
> then I tried,
>
> import numpy as np
> cov = np.cov(idx1, returns)
> print cov.tolist()
> stddev_x = np.std(returns, ddof=1)
> stddev_y = np.std(idx1, ddof=1)
> print "cor = %s" % (cov.tolist()[:-1] /(stddev_x * stddev_y)) They 
> differ from each other.
>
> As you can see from the numpy example, I am trying to?find cor?coeff 
> for?a sample.?(ddof=1)
>
> So, my question is: is the discrepency caused by the fact that I am 
> trying to use lstsq() on a 'sample popluation' (i.e. I am not 
> regressing a full return series)? Is it correct to use lstsq() this way?
>

the most direct way to calculate the correlation matrix, use index [0,1] to get coefficient.

numpy.corrcoef(x, y=None, rowvar=1, bias=0)

np.cov, that you used, uses biased estimator, denominator = N by default, but for std you used N-1

Josef
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

NOTICE
This e-mail and any attachments are confidential and may contain copyright material of Macquarie Group Limited or third parties. If you are not the intended recipient of this email you should not read, print, re-transmit, store or act in reliance on this e-mail or any attachments, and should destroy all copies of them. Macquarie Group Limited does not guarantee the integrity of any emails or any attached files. The views or opinions expressed are the author's own and may not reflect the views or opinions of Macquarie Group Limited.



From kwgoodman at gmail.com  Wed Feb 25 19:48:08 2009
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 25 Feb 2009 16:48:08 -0800
Subject: [Numpy-discussion] Question on lstsq and correlation coeff
In-Reply-To: <90CBFFFE6273484B9579400AC950800502024765@ntsydexm01.pc.internal.macquarie.com>
References: <90CBFFFE6273484B9579400AC950800502024765@ntsydexm01.pc.internal.macquarie.com>
Message-ID: <f4f93d420902251648k4cc67d41lc858048db641b590@mail.gmail.com>

On Wed, Feb 25, 2009 at 3:21 PM, Anthony Kong
<Anthony.Kong at macquarie.com> wrote:
> I?trying to use?scipy/numpy in a finanical context. I want to compute the
> correlation coeff of two series (returns vs index returns). I tried two
> appoarches
>
> Firstly,
>
> from scipy.linalg import lstsq
> coeffs,a,b,c = lstsq(matrix, returns) # matrix contains index returns
>
> then I tried,
>
> import numpy as np
> cov = np.cov(idx1, returns)
> print cov.tolist()
> stddev_x = np.std(returns, ddof=1)
> stddev_y = np.std(idx1, ddof=1)
> print "cor = %s" % (cov.tolist()[:-1] /(stddev_x * stddev_y))
> They differ from each other.

coeffs in

coeffs,a,b,c = lstsq(matrix, returns) # matrix contains index returns

is the beta of the stock with respect to the index, not the correlation.


From cournape at gmail.com  Thu Feb 26 01:54:33 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 26 Feb 2009 15:54:33 +0900
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <e06186140902231401o5722ed1br48667a863904dd21@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org>
	<5b8d13220902190805s2d12ac96va40b6062959fc53@mail.gmail.com>
	<gnlpfb$hih$1@ger.gmane.org>
	<5b8d13220902231302x42ede056i63a72ce968da9c8b@mail.gmail.com>
	<e06186140902231401o5722ed1br48667a863904dd21@mail.gmail.com>
Message-ID: <5b8d13220902252254s6e12f5b0y2f032bbabbe3b843@mail.gmail.com>

On Tue, Feb 24, 2009 at 7:01 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Mon, Feb 23, 2009 at 2:02 PM, David Cournapeau <cournape at gmail.com>
> wrote:
>>
>> On Fri, Feb 20, 2009 at 5:26 PM, Pauli Virtanen <pav at iki.fi> wrote:
>> > Fri, 20 Feb 2009 01:05:03 +0900, David Cournapeau wrote:
>> > [clip]
>> >>> I think they should be. Then we could easily use C99 complex math
>> >>> functions on plaforms on which they are available (and so get the
>> >>> "correct" corner-case semantics for free on these platforms).
>> >>
>> >> maybe we could change the names, then ? nc is not very clear IMHO (and
>> >> since they were static up to now, we are free to change them I
>> >> believe).
>> >
>> > I think it would make sense to change them to follow C99 function names,
>> > with a npy_ prefix.
>>
>> The problem of complex functions is that they don't follow the C99
>> conventions at all. IN particular, they accept pointers instead of
>> values. I don't know whether the rational for using pointers is still
>> valid (it is mentioned that how to pass structure is compiler
>
> The usual rational is that it is more efficient to pass a pointer than to
> push two floats on the stack. It is also an easier way to return values,
> although recent versions of gcc do a good job of copying the return values
> where they need to go. I would stick with the pointers, although we could
> probably dispense with the structure and just use a pointer to the
> underlying type with the assumption that the real & imaginary parts are
> contiguous in memory. The ufuncs make that assumption in any case.

Ok, what about making the decision for complex functions later, and
merging now the the real functions (the one with clear API) ? Having
coremath in trunk would help me tracking down crashes on windows x64
(the trunk does not build right now because of some configuration
problems which are not a concern anymore witht the coremath branch),

cheers,

David


From charlesr.harris at gmail.com  Thu Feb 26 02:32:25 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 26 Feb 2009 00:32:25 -0700
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <5b8d13220902252254s6e12f5b0y2f032bbabbe3b843@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org>
	<5b8d13220902190805s2d12ac96va40b6062959fc53@mail.gmail.com>
	<gnlpfb$hih$1@ger.gmane.org>
	<5b8d13220902231302x42ede056i63a72ce968da9c8b@mail.gmail.com>
	<e06186140902231401o5722ed1br48667a863904dd21@mail.gmail.com>
	<5b8d13220902252254s6e12f5b0y2f032bbabbe3b843@mail.gmail.com>
Message-ID: <e06186140902252332s2cf094faj3a8dd83277118224@mail.gmail.com>

On Wed, Feb 25, 2009 at 11:54 PM, David Cournapeau <cournape at gmail.com>wrote:

> On Tue, Feb 24, 2009 at 7:01 AM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> >
> > On Mon, Feb 23, 2009 at 2:02 PM, David Cournapeau <cournape at gmail.com>
> > wrote:
> >>
> >> On Fri, Feb 20, 2009 at 5:26 PM, Pauli Virtanen <pav at iki.fi> wrote:
> >> > Fri, 20 Feb 2009 01:05:03 +0900, David Cournapeau wrote:
> >> > [clip]
> >> >>> I think they should be. Then we could easily use C99 complex math
> >> >>> functions on plaforms on which they are available (and so get the
> >> >>> "correct" corner-case semantics for free on these platforms).
> >> >>
> >> >> maybe we could change the names, then ? nc is not very clear IMHO
> (and
> >> >> since they were static up to now, we are free to change them I
> >> >> believe).
> >> >
> >> > I think it would make sense to change them to follow C99 function
> names,
> >> > with a npy_ prefix.
> >>
> >> The problem of complex functions is that they don't follow the C99
> >> conventions at all. IN particular, they accept pointers instead of
> >> values. I don't know whether the rational for using pointers is still
> >> valid (it is mentioned that how to pass structure is compiler
> >
> > The usual rational is that it is more efficient to pass a pointer than to
> > push two floats on the stack. It is also an easier way to return values,
> > although recent versions of gcc do a good job of copying the return
> values
> > where they need to go. I would stick with the pointers, although we could
> > probably dispense with the structure and just use a pointer to the
> > underlying type with the assumption that the real & imaginary parts are
> > contiguous in memory. The ufuncs make that assumption in any case.
>
> Ok, what about making the decision for complex functions later, and
> merging now the the real functions (the one with clear API) ? Having
> coremath in trunk would help me tracking down crashes on windows x64
> (the trunk does not build right now because of some configuration
> problems which are not a concern anymore witht the coremath branch),
>

Sounds good, I don't think the complex functions are a pressing concern. But
I suspect we should start looking forward to a code freeze in a month or so
and getting the build working is a clear priority. Maybe we should start the
release process? I have a short list of things I think I should finish up
and now might be a good time to put together a list of things to do.

I've been thinking a bit about the complex functions. It might be worth
benchmarking a few to see how much it costs to pass full complex numbers
back forth. If the functions were inlined the optimizer could probably do
good things with the FPU registers, but how things will work when the
functions are in a library is another question. And rewriting the functions
will be a lot of work unless we can copy a bunch of them from someone else.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090226/618ba396/attachment.html>

From cournape at gmail.com  Thu Feb 26 03:23:46 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 26 Feb 2009 17:23:46 +0900
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <e06186140902252332s2cf094faj3a8dd83277118224@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<3d375d730902150009geb44dc5x6bb370cb47975717@mail.gmail.com>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org>
	<5b8d13220902190805s2d12ac96va40b6062959fc53@mail.gmail.com>
	<gnlpfb$hih$1@ger.gmane.org>
	<5b8d13220902231302x42ede056i63a72ce968da9c8b@mail.gmail.com>
	<e06186140902231401o5722ed1br48667a863904dd21@mail.gmail.com>
	<5b8d13220902252254s6e12f5b0y2f032bbabbe3b843@mail.gmail.com>
	<e06186140902252332s2cf094faj3a8dd83277118224@mail.gmail.com>
Message-ID: <5b8d13220902260023u5f173705x8f55af7c86c1df32@mail.gmail.com>

On Thu, Feb 26, 2009 at 4:32 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:

> Sounds good, I don't think the complex functions are a pressing concern. But
> I suspect we should start looking forward to a code freeze in a month or so
> and getting the build working is a clear priority.

The build is working more or less. Building with VS 2008 works (by
works I mean build and run the testsuite with a couple of failures),
but building mingw-w64 still does not (it builds, but the test suite
crashes at several points - up to now, it was mostly bugs in the
mingw-w64 tools, but I think some errors are ours now). Building with
mingw-w64 is necessary IMO, because I don't see an easy way to build
scipy wo a free fortran compiler, and gfortran is the only one I know
on windows 64 (VS 2008 + gfortran breaks horribly on 64 bits ATM).
Restricting numpy wo BLAS/LAPACK for windows 64 bits may be another
option: in that case, the infrastructure can be considered ready. In
any case, the windows 64 bits build would be very experimental.

Building the 32 bits version with python 2.6 should work well, too - I
am confident we can release binaries for 2.6 on windows and mac os X
for 1.3, even if it is released in a short time.

> Maybe we should start the
> release process? I have a short list of things I think I should finish up
> and now might be a good time to put together a list of things to do.

Yes, I was thinking about that too. There was a 1.3 thread a couple of
weeks ago, we should summarize it, and set a timeline for 1.3 really
soon. I can do it, unless you want do it it,

David


From charlesr.harris at gmail.com  Thu Feb 26 03:41:38 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 26 Feb 2009 01:41:38 -0700
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <5b8d13220902260023u5f173705x8f55af7c86c1df32@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>
	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>
	<gnjtra$1bo$1@ger.gmane.org>
	<5b8d13220902190805s2d12ac96va40b6062959fc53@mail.gmail.com>
	<gnlpfb$hih$1@ger.gmane.org>
	<5b8d13220902231302x42ede056i63a72ce968da9c8b@mail.gmail.com>
	<e06186140902231401o5722ed1br48667a863904dd21@mail.gmail.com>
	<5b8d13220902252254s6e12f5b0y2f032bbabbe3b843@mail.gmail.com>
	<e06186140902252332s2cf094faj3a8dd83277118224@mail.gmail.com>
	<5b8d13220902260023u5f173705x8f55af7c86c1df32@mail.gmail.com>
Message-ID: <e06186140902260041y1846895doe34a396b1d562a95@mail.gmail.com>

On Thu, Feb 26, 2009 at 1:23 AM, David Cournapeau <cournape at gmail.com>wrote:

> On Thu, Feb 26, 2009 at 4:32 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>
> > Sounds good, I don't think the complex functions are a pressing concern.
> But
> > I suspect we should start looking forward to a code freeze in a month or
> so
> > and getting the build working is a clear priority.
>
> The build is working more or less. Building with VS 2008 works (by
> works I mean build and run the testsuite with a couple of failures),
> but building mingw-w64 still does not (it builds, but the test suite
> crashes at several points - up to now, it was mostly bugs in the
> mingw-w64 tools, but I think some errors are ours now). Building with
> mingw-w64 is necessary IMO, because I don't see an easy way to build
> scipy wo a free fortran compiler, and gfortran is the only one I know
> on windows 64 (VS 2008 + gfortran breaks horribly on 64 bits ATM).
> Restricting numpy wo BLAS/LAPACK for windows 64 bits may be another
> option: in that case, the infrastructure can be considered ready. In
> any case, the windows 64 bits build would be very experimental.
>
> Building the 32 bits version with python 2.6 should work well, too - I
> am confident we can release binaries for 2.6 on windows and mac os X
> for 1.3, even if it is released in a short time.
>
> > Maybe we should start the
> > release process? I have a short list of things I think I should finish up
> > and now might be a good time to put together a list of things to do.
>
> Yes, I was thinking about that too. There was a 1.3 thread a couple of
> weeks ago, we should summarize it, and set a timeline for 1.3 really
> soon. I can do it, unless you want do it it,
>

Why don't you make a start. I can help go through the tickets this weekend
to pick out the ones that need to get fixed up. Umm... come to think of it,
I'll be out of town for a few days starting Sunday, but I'll do my best.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090226/33f97ba1/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Thu Feb 26 03:45:59 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 26 Feb 2009 17:45:59 +0900
Subject: [Numpy-discussion] Core math library in numpy
In-Reply-To: <e06186140902260041y1846895doe34a396b1d562a95@mail.gmail.com>
References: <4997C8E5.1040105@ar.media.kyoto-u.ac.jp>	<5b8d13220902190716h3d3083cdg30f692276ef91b3f@mail.gmail.com>	<gnjtra$1bo$1@ger.gmane.org>	<5b8d13220902190805s2d12ac96va40b6062959fc53@mail.gmail.com>	<gnlpfb$hih$1@ger.gmane.org>	<5b8d13220902231302x42ede056i63a72ce968da9c8b@mail.gmail.com>	<e06186140902231401o5722ed1br48667a863904dd21@mail.gmail.com>	<5b8d13220902252254s6e12f5b0y2f032bbabbe3b843@mail.gmail.com>	<e06186140902252332s2cf094faj3a8dd83277118224@mail.gmail.com>	<5b8d13220902260023u5f173705x8f55af7c86c1df32@mail.gmail.com>
	<e06186140902260041y1846895doe34a396b1d562a95@mail.gmail.com>
Message-ID: <49A656C7.2070202@ar.media.kyoto-u.ac.jp>

Charles R Harris wrote:
>
>
>
>     Yes, I was thinking about that too. There was a 1.3 thread a couple of
>     weeks ago, we should summarize it, and set a timeline for 1.3 really
>     soon. I can do it, unless you want do it it,
>
>
> Why don't you make a start.

Ok,

David


From david at ar.media.kyoto-u.ac.jp  Thu Feb 26 05:29:42 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 26 Feb 2009 19:29:42 +0900
Subject: [Numpy-discussion] coremath branch merged
Message-ID: <49A66F16.9030706@ar.media.kyoto-u.ac.jp>

Hi,

    I've just merged the work on the coremath branch into the trunk. Let
me know if you see some problems - I did a quick sanity check on Linux
after the merge, but I have not tested across many platforms,

cheers,

David


From dsdale24 at gmail.com  Thu Feb 26 09:18:34 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Thu, 26 Feb 2009 09:18:34 -0500
Subject: [Numpy-discussion] coremath branch merged
In-Reply-To: <49A66F16.9030706@ar.media.kyoto-u.ac.jp>
References: <49A66F16.9030706@ar.media.kyoto-u.ac.jp>
Message-ID: <a08e5f80902260618t7f524324i82f76cf2a34d85f8@mail.gmail.com>

On Thu, Feb 26, 2009 at 5:29 AM, David Cournapeau <
david at ar.media.kyoto-u.ac.jp> wrote:

> Hi,
>
>    I've just merged the work on the coremath branch into the trunk. Let
> me know if you see some problems - I did a quick sanity check on Linux
> after the merge, but I have not tested across many platforms,
>

I just ran the tests on 64-bit gentoo linux with gcc-4.3.3 and
glibc-2.9_p20081201. Looks good here, all tests pass with the exception of
one known failure.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090226/484980c2/attachment.html>

From cournape at gmail.com  Thu Feb 26 09:31:53 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 26 Feb 2009 23:31:53 +0900
Subject: [Numpy-discussion] CROSS-COMPILING NUMPY
In-Reply-To: <5b8d13220902161937x5f2b1932i3412c0e90c8c6108@mail.gmail.com>
References: <4997C4A1.8090307@ar.media.kyoto-u.ac.jp>
	<C5BED2F7.E71%esyepez@sandia.gov>
	<5b8d13220902161937x5f2b1932i3412c0e90c8c6108@mail.gmail.com>
Message-ID: <5b8d13220902260631j3e31105dmd14c48eca362c08d@mail.gmail.com>

On Tue, Feb 17, 2009 at 12:37 PM, David Cournapeau <cournape at gmail.com> wrote:

> Also, some configuration tests need to be executed on the target
> machine: this needs to be fixed.

Ok, that one at least is fixed. Now, the numpy configuration stage
does not need to execute anything on the target platform. All the
tests are purely compile/link only, no execution. Of course, I cannot
check it works in cross-compilation environment, because that's only
one step - but the particular case of windows x86 -> windows amd64
should now work.

cheers,

David


From mudit_19a at yahoo.com  Thu Feb 26 12:48:52 2009
From: mudit_19a at yahoo.com (mudit sharma)
Date: Thu, 26 Feb 2009 23:18:52 +0530 (IST)
Subject: [Numpy-discussion] intersect1d and setmember1d
In-Reply-To: <f4f93d420902251648k4cc67d41lc858048db641b590@mail.gmail.com>
References: <90CBFFFE6273484B9579400AC950800502024765@ntsydexm01.pc.internal.macquarie.com>
	<f4f93d420902251648k4cc67d41lc858048db641b590@mail.gmail.com>
Message-ID: <243385.2089.qm@web94910.mail.in2.yahoo.com>


intersect1d and setmember1d doesn't give expected results in case there are duplicate values in either array becuase it works by sorting data and substracting previous value. Is there an alternative in numpy to get indices of intersected values.

In [31]: p nonzero(setmember1d(v1.Id, v2.Id))[0]
[ 0  1  2  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25                        <-------------- index 2 shouldn't be here look at the data below.
 26 27 28 29]

In [32]: p v1.Id[:10]
[ 232.  232.  233.  233.  234.  234.  235.  235.  237.  237.]

In [33]: p v2.Id[:10]
[ 232.  232.  234.  234.  235.  235.  236.  236.  237.  237.]

In [34]: p setmember1d(v1.Id, v2.Id)
[ True  True  True False  True  True  True  True  True  True  True  True                       <-------------- index 2 shouldn't be True
  True  True  True  True  True  True  True  True  True  True  True  True
  True  True  True  True  True  True]

In [35]: p setmember1d(v1.Id[:10], v2.Id[:10])
[ True  True  True False  True  True  True  True  True  True]


      



From zachary.pincus at yale.edu  Thu Feb 26 13:07:21 2009
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Thu, 26 Feb 2009 13:07:21 -0500
Subject: [Numpy-discussion] intersect1d and setmember1d
In-Reply-To: <243385.2089.qm@web94910.mail.in2.yahoo.com>
References: <90CBFFFE6273484B9579400AC950800502024765@ntsydexm01.pc.internal.macquarie.com>
	<f4f93d420902251648k4cc67d41lc858048db641b590@mail.gmail.com>
	<243385.2089.qm@web94910.mail.in2.yahoo.com>
Message-ID: <17AFFA4C-86B5-4C4E-87EA-E71F2E1C67A3@yale.edu>

Hi,

> intersect1d and setmember1d doesn't give expected results in case  
> there are duplicate values in either array becuase it works by  
> sorting data and substracting previous value. Is there an  
> alternative in numpy to get indices of intersected values.

 From the docstring for setmember1d (and other set operations), you  
are only supposed to pass it arrays with unique values (i.e. arrays  
that represent sets in the mathematical sense):

 >>> print numpy.setmember1d.__doc__
     Return a boolean array set True where first element is in second  
array.

     Boolean array is the shape of `ar1` containing True where the  
elements
     of `ar1` are in `ar2` and False otherwise.

     Use unique1d() to generate arrays with only unique elements to  
use as
     inputs to this function.
     [...]

As stated, use unique1d to generate set-arrays from your input.

On the other hand, intersect1d is supposed to work with repeated  
elements:
 >>> print numpy.intersect1d.__doc__
     Intersection returning repeated or unique elements common to both  
arrays.

     Parameters
     ----------
     ar1,ar2 : array_like
         Input arrays.

     Returns
     -------
     out : ndarray, shape(N,)
         Sorted 1D array of common elements with repeating elements.

     See Also
     --------
     intersect1d_nu : Returns only unique common elements.
     [...]

Do you have an example of intersect1d not working right? If so, what  
version of numpy are you using (print numpy.version.version)?

Zach




On Feb 26, 2009, at 12:48 PM, mudit sharma wrote:

>
> intersect1d and setmember1d doesn't give expected results in case  
> there are duplicate values in either array becuase it works by  
> sorting data and substracting previous value. Is there an  
> alternative in numpy to get indices of intersected values.
>
> In [31]: p nonzero(setmember1d(v1.Id, v2.Id))[0]
> [ 0  1  2  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22  
> 23 24 25                        <-------------- index 2 shouldn't be  
> here look at the data below.
> 26 27 28 29]
>
> In [32]: p v1.Id[:10]
> [ 232.  232.  233.  233.  234.  234.  235.  235.  237.  237.]
>
> In [33]: p v2.Id[:10]
> [ 232.  232.  234.  234.  235.  235.  236.  236.  237.  237.]
>
> In [34]: p setmember1d(v1.Id, v2.Id)
> [ True  True  True False  True  True  True  True  True  True  True   
> True                       <-------------- index 2 shouldn't be True
>  True  True  True  True  True  True  True  True  True  True  True   
> True
>  True  True  True  True  True  True]
>
> In [35]: p setmember1d(v1.Id[:10], v2.Id[:10])
> [ True  True  True False  True  True  True  True  True  True]
>
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From ralph at dont-mind.de  Thu Feb 26 13:42:44 2009
From: ralph at dont-mind.de (Ralph Heinkel)
Date: Thu, 26 Feb 2009 19:42:44 +0100
Subject: [Numpy-discussion] What's wrong with floatint.c example code?
Message-ID: <49A6E2A4.2090401@dont-mind.de>

Hi,

I'm trying to get into the realm of implementing my own numpy data
types in numpy, and doing so I had a look at the floatint.c example
coming from the numpy/doc/newdtype_example directory.

Obviously it is not possible to create an array with the new floatint
type by doing

array([1,2,3,4,5,6,7,8], dtype=floatint)

but instead this works:

array([1,2,3,4,5,6,7,8]).view(ff.floatint_type)

Would anybody mind helping me to understand this behavior? Is the
reason for this that the floatint.c stuff is incomplete and would need
some more elaboration in order to function as a proper type?

Thanks in advance

Ralph



From Chris.Barker at noaa.gov  Thu Feb 26 16:29:32 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 26 Feb 2009 13:29:32 -0800
Subject: [Numpy-discussion] Fancy indexing question:
In-Reply-To: <200902241526.42724.lists_ravi@lavabit.com>
References: <49A44CFB.5090907@noaa.gov>
	<200902241526.42724.lists_ravi@lavabit.com>
Message-ID: <49A709BC.2070200@noaa.gov>

Ravi wrote:
> Use ix_:
> 
> In [5]: a[ ix_(i,j) ]
> Out[5]:
> array([[ 5,  7],
>        [13, 15],
>        [17, 19]])

Very nice!


thanks,
-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From simpson at math.toronto.edu  Thu Feb 26 17:53:28 2009
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Thu, 26 Feb 2009 17:53:28 -0500
Subject: [Numpy-discussion] casting integers to reals
Message-ID: <D518FB09-7C9F-48F8-8F54-E3FD9841EBF3@math.toronto.edu>

I want to do:

numpy.float(numpy.arange(0, 10))

but get the error:

Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: only length-1 arrays can be converted to Python scalars

How should I do this?

-gideon



From gael.varoquaux at normalesup.org  Thu Feb 26 17:55:10 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 26 Feb 2009 23:55:10 +0100
Subject: [Numpy-discussion] casting integers to reals
In-Reply-To: <D518FB09-7C9F-48F8-8F54-E3FD9841EBF3@math.toronto.edu>
References: <D518FB09-7C9F-48F8-8F54-E3FD9841EBF3@math.toronto.edu>
Message-ID: <20090226225510.GE23810@phare.normalesup.org>

On Thu, Feb 26, 2009 at 05:53:28PM -0500, Gideon Simpson wrote:
> I want to do:

> numpy.float(numpy.arange(0, 10))

> but get the error:

> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> TypeError: only length-1 arrays can be converted to Python scalars

> How should I do this?

nump.arange(0, 10.astype(numpy.float)

but in this special case you can do:

numpy.arange(0., 10.)

Ga?l


From jdh2358 at gmail.com  Thu Feb 26 18:05:18 2009
From: jdh2358 at gmail.com (John Hunter)
Date: Thu, 26 Feb 2009 17:05:18 -0600
Subject: [Numpy-discussion] numpy save files from C
Message-ID: <88e473830902261505la1f1ec6n41544f34b0aa9297@mail.gmail.com>

A colleague of mine has a bunch of numpy arrays saved with np.save and
he now wants to access them directly in C, with or w/o the numpy C API
doesn't matter.  Does anyone have any sample code lying around which
he can borrow from?  The array is a structured array with an otherwise
plain vanilla dtype (ints and floats).

I've referred him to the npy-format NEP document, as well as the
format.py implementation, so he can roll his own if need be, but if
someone has a head start code example that would be great.

Thanks,
JDH


From robert.kern at gmail.com  Thu Feb 26 18:14:57 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 26 Feb 2009 17:14:57 -0600
Subject: [Numpy-discussion] numpy save files from C
In-Reply-To: <88e473830902261505la1f1ec6n41544f34b0aa9297@mail.gmail.com>
References: <88e473830902261505la1f1ec6n41544f34b0aa9297@mail.gmail.com>
Message-ID: <3d375d730902261514w52d97ca4p1c213651932dc853@mail.gmail.com>

On Thu, Feb 26, 2009 at 17:05, John Hunter <jdh2358 at gmail.com> wrote:
> A colleague of mine has a bunch of numpy arrays saved with np.save and
> he now wants to access them directly in C, with or w/o the numpy C API
> doesn't matter. ?Does anyone have any sample code lying around which
> he can borrow from? ?The array is a structured array with an otherwise
> plain vanilla dtype (ints and floats).
>
> I've referred him to the npy-format NEP document, as well as the
> format.py implementation, so he can roll his own if need be, but if
> someone has a head start code example that would be great.

If anyone does, it would probably be me, but I don't. If you already
know the dtype beforehand, reading the header offset and skipping to
the data is probably the easiest.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Thu Feb 26 18:23:30 2009
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 26 Feb 2009 15:23:30 -0800
Subject: [Numpy-discussion] casting integers to reals
In-Reply-To: <20090226225510.GE23810@phare.normalesup.org>
References: <D518FB09-7C9F-48F8-8F54-E3FD9841EBF3@math.toronto.edu>
	<20090226225510.GE23810@phare.normalesup.org>
Message-ID: <49A72472.8040808@noaa.gov>

Gael Varoquaux wrote:
> nump.arange(0, 10.astype(numpy.float)

I think you meant:

np.arange(0, 10).astype(np.float)

but:

np.arange(0, 10, dtype=np.float)

is a better bet.

> but in this special case you can do:
> 
> numpy.arange(0., 10.)

yup -- however, beware, using arange() with floating point numbers 
doesn't necessarily work as you would like it to, die to floating point 
rounding. np.linspace may be a better bet:

 >>> np.linspace(0, 9, 10)
array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])


-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From efiring at hawaii.edu  Thu Feb 26 18:24:22 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Thu, 26 Feb 2009 13:24:22 -1000
Subject: [Numpy-discussion] casting integers to reals
In-Reply-To: <D518FB09-7C9F-48F8-8F54-E3FD9841EBF3@math.toronto.edu>
References: <D518FB09-7C9F-48F8-8F54-E3FD9841EBF3@math.toronto.edu>
Message-ID: <49A724A6.8050109@hawaii.edu>

Gideon Simpson wrote:
> I want to do:
> 
> numpy.float(numpy.arange(0, 10))
> 
> but get the error:
> 
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> TypeError: only length-1 arrays can be converted to Python scalars
> 
> How should I do this?

numpy.arange(0,10, dtype=float)
or
numpy.arange(0.0, 10.0)

The first is nice because it is very explicit, and it works when the 
limits may be variables of any numeric type.

Eric

> 
> -gideon
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From jonathan.taylor at utoronto.ca  Thu Feb 26 22:00:24 2009
From: jonathan.taylor at utoronto.ca (Jonathan Taylor)
Date: Thu, 26 Feb 2009 22:00:24 -0500
Subject: [Numpy-discussion] Slicing/selection in multiple dimensions
	simultaneously
In-Reply-To: <46E72116.8040408@enthought.com>
References: <268febdf0709111511n3ca15d42o85d31831178d96a@mail.gmail.com>
	<46E71591.20802@gmail.com>
	<e4412d6b0709111542m355ad6a6w21784b5a2108c6c5@mail.gmail.com>
	<46E72116.8040408@enthought.com>
Message-ID: <463e11f90902261900o748940b6yf8410abda82524cc@mail.gmail.com>

Am I right to assume that there is no way elegant way to interact with
slices.  i.e. Is there anyway to get

a[ix_([2,3,6],:,[3,2])]

to work?  So that the dimension is completely specified?  Or perhaps
the only way to do this is via

a[ix_([2,3,6],range(a.shape[1]),[3,2])]

If anyone knows a better way?

Thanks,
Jonathan.

On Tue, Sep 11, 2007 at 6:13 PM, Travis E. Oliphant
<oliphant at enthought.com> wrote:
> Timothy Hochberg wrote:
>>
>>
>> On 9/11/07, *Robert Kern* <robert.kern at gmail.com
>> <mailto:robert.kern at gmail.com>> wrote:
>>
>> ? ? Mike Ressler wrote:
>> ? ? > The following seems to be a wart: is it expected?
>> ? ? >
>> ? ? > Set up a 10x10 array and some indexing arrays:
>> ? ? >
>> ? ? > a=arange(100)
>> ? ? > a.shape=(10,10)
>> ? ? > q=array([0,2,4,6,8])
>> ? ? > r=array([0,5])
>> ? ? >
>> ? ? > Suppose I want to extract only the "even" numbered rows from a -
>> ? ? then
>> ? ? >
>> ? ? > print a[q,:]
>> ? ? >
>> ? ? > <works - output deleted>
>> ? ? >
>> ? ? > Every fifth column:
>> ? ? >
>> ? ? > print a[:,r]
>> ? ? >
>> ? ? > <works - output deleted>
>> ? ? >
>> ? ? > Only the even rows of every fifth column:
>> ? ? >
>> ? ? > print a[q,r]
>> ? ? >
>> ? ? >
>> ? ? ---------------------------------------------------------------------------
>>
>> ? ? > <type 'exceptions.ValueError'> ? ? ? ? ? ?Traceback (most recent
>> ? ? call last)
>> ? ? >
>> ? ? > /.../.../.../<ipython console> in <module>()
>> ? ? >
>> ? ? > <type 'exceptions.ValueError '>: shape mismatch: objects cannot be
>> ? ? > broadcast to a single shape
>> ? ? >
>> ? ? > But, this works:
>> ? ? >
>> ? ? > print a[q,:][:,r]
>> ? ? >
>> ? ? > [[ 0 ?5]
>> ? ? > ?[20 25]
>> ? ? > ?[40 45]
>> ? ? > ?[60 65]
>> ? ? > ?[80 85]]
>> ? ? >
>> ? ? > So why does the a[q,r] form have problems? Thanks for your insights.
>>
>> ? ? It is intended that the form a[q,r] be the general case: q and r
>> ? ? are broadcasted
>> ? ? against each other to a single shape. The result of the indexing
>> ? ? is an array of
>> ? ? that broadcasted shape with elements found by using each pair of
>> ? ? elements in the
>> ? ? broadcasted q and r arrays as indices.
>>
>> ? ? There are operations you can express with this form that you
>> ? ? couldn't if the
>> ? ? behavior that you expected were the case whereas you can get the
>> ? ? result you want
>> ? ? relatively straightforwardly.
>>
>> ? ? In [6]: a[q[:,newaxis], r]
>> ? ? Out[6]:
>> ? ? array([[ 0, ?5],
>> ? ? ? ? ? ?[20, 25],
>> ? ? ? ? ? ?[40, 45],
>> ? ? ? ? ? ?[60, 65],
>> ? ? ? ? ? ?[80, 85]])
>>
>>
>>
>> At the risk of making Robert grumpy: while it is true the form we
>> ended up with is more general I've come to the conclusion that it was
>> a bit of a mistake. In the spirit of making simple things simple and
>> complex things possible, I suspect that having fancy-indexing do the
>> obvious thing here[1] and delegating the more powerful but also more
>> difficult to understand case to a function or method would have been
>> overall more useful. Cases where the multidimensional features of
>> fancy-indexing get used are messy enough that they don't benefit much
>> from the conciseness of the indexing notation, at least in my experience.
> This is a reasonable argument. ? ?It is reasonable enough that I
> intentionally made an ix_ function to do what you want.
>
> a[ix_(q,r)]
>
> does as originally expected if a bit more line-noise.
>
> -Travis
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From ddf at iqdotdt.com  Fri Feb 27 00:27:57 2009
From: ddf at iqdotdt.com (Delbert Franz)
Date: Thu, 26 Feb 2009 21:27:57 -0800 (PST)
Subject: [Numpy-discussion] Speedup creation of a 3-color array from a 2-d
 color-index array a color lut
Message-ID: <22236421.post@talk.nabble.com>


I have  geotiff files of scanned paper maps that use an indexed color scheme
with a 256-element
color lookup table (color lut) and a 9252 by 7420 array  of uint8 elements. 
The color is given by 
three values.  I want to create an array with shape: (9252, 7420, 3) so that
I can display the 
image without creating internal array working space in Matplotlib that
exeeds 2^31 bytes. 

The following three approaches work in that the correct image is displayed,
but all of them 
are waaaaay too slow:)

 Let
   doq have shape (9252, 7420) and have uint8 elements
   ctab have shape (256, 3) and have uint8 elements.
   doqq have shape (9252, 7420, 3) and have unit8 elements

#1  The way it would be done in a compiled language, like Fortran where it
would run in a small fraction of 
a second instead of several minutes:)  But what I want to ultimately
accomplish is hard to do in Fortran!

 for i in range(9252):
    for j in range(7420):
        for k in range(3):
            doqq[i,j,k] = ctab[doq[i,j],k]

#2  Try to use some special numpy features:

for i in doq.flat:
    doqq.flat = ctab[i,0:3]


#3 Variation of #1

for i in range(9252):
   for j in range(7420):
      doqq[i,j,] = ctab[doq[i,j],]

All of these use too many element accesses, which are slow.  My searching
and reading the
documentation has not given me an approach that avoids these low-level and
slow
(in python/numpy) accesses.  I suspect there is a clever way to do this but
as a newbie
I'm having some rough going on getting this process to be much faster. 

Thanks,

       Delbert Franz

            

-- 
View this message in context: http://www.nabble.com/Speedup-creation-of-a-3-color-array-from-a-2-d-color-index-array-a-color-lut-tp22236421p22236421.html
Sent from the Numpy-discussion mailing list archive at Nabble.com.



From robert.kern at gmail.com  Fri Feb 27 00:32:35 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 26 Feb 2009 23:32:35 -0600
Subject: [Numpy-discussion] Speedup creation of a 3-color array from a
	2-d color-index array a color lut
In-Reply-To: <22236421.post@talk.nabble.com>
References: <22236421.post@talk.nabble.com>
Message-ID: <3d375d730902262132k5a011d14t84317458f9b6cf87@mail.gmail.com>

On Thu, Feb 26, 2009 at 23:27, Delbert Franz <ddf at iqdotdt.com> wrote:
>
> I have ?geotiff files of scanned paper maps that use an indexed color scheme
> with a 256-element
> color lookup table (color lut) and a 9252 by 7420 array ?of uint8 elements.
> The color is given by
> three values. ?I want to create an array with shape: (9252, 7420, 3) so that
> I can display the
> image without creating internal array working space in Matplotlib that
> exeeds 2^31 bytes.
>
> The following three approaches work in that the correct image is displayed,
> but all of them
> are waaaaay too slow:)
>
> ?Let
> ? doq have shape (9252, 7420) and have uint8 elements
> ? ctab have shape (256, 3) and have uint8 elements.
> ? doqq have shape (9252, 7420, 3) and have unit8 elements

doqq = ctab[doq]

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From cimrman3 at ntc.zcu.cz  Fri Feb 27 04:15:59 2009
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Fri, 27 Feb 2009 10:15:59 +0100
Subject: [Numpy-discussion] intersect1d and setmember1d
In-Reply-To: <17AFFA4C-86B5-4C4E-87EA-E71F2E1C67A3@yale.edu>
References: <90CBFFFE6273484B9579400AC950800502024765@ntsydexm01.pc.internal.macquarie.com>	<f4f93d420902251648k4cc67d41lc858048db641b590@mail.gmail.com>	<243385.2089.qm@web94910.mail.in2.yahoo.com>
	<17AFFA4C-86B5-4C4E-87EA-E71F2E1C67A3@yale.edu>
Message-ID: <49A7AF4F.1050409@ntc.zcu.cz>

Zachary Pincus wrote:
> Hi,
> 
>> intersect1d and setmember1d doesn't give expected results in case
>>  there are duplicate values in either array becuase it works by 
>> sorting data and substracting previous value. Is there an 
>> alternative in numpy to get indices of intersected values.
> 
> From the docstring for setmember1d (and other set operations), you 
> are only supposed to pass it arrays with unique values (i.e. arrays
>  that represent sets in the mathematical sense):
> 
>>>> print numpy.setmember1d.__doc__
> Return a boolean array set True where first element is in second 
> array.
> 
> Boolean array is the shape of `ar1` containing True where the 
> elements of `ar1` are in `ar2` and False otherwise.
> 
> Use unique1d() to generate arrays with only unique elements to use as
>  inputs to this function. [...]
> 
> As stated, use unique1d to generate set-arrays from your input.
> 
> On the other hand, intersect1d is supposed to work with repeated 
> elements:
>>>> print numpy.intersect1d.__doc__
> Intersection returning repeated or unique elements common to both 
> arrays.
> 
> Parameters ---------- ar1,ar2 : array_like Input arrays.
> 
> Returns ------- out : ndarray, shape(N,) Sorted 1D array of common
> elements with repeating elements.
> 
> See Also -------- intersect1d_nu : Returns only unique common
> elements. [...]
> 
> Do you have an example of intersect1d not working right? If so, what
>  version of numpy are you using (print numpy.version.version)?
> 
> Zach

Hi,

yes, many functions in arraysetops.py ('intersect1d', 'setxor1d',
'setmember1d', 'union1d', 'setdiff1d') were originally meant to work 
with arrays of unique elements as inputs. I have just noticed, that the 
docstring of intersect1d says that it works for non-unique arrays and 
contains the following example:

 >>> np.intersect1d([1,3,3],[3,1,1])
     array([1, 1, 3, 3])

I am not sure if this is a useful behaviour - does anybody uses this 
"feature" (or better, side-effect)?

I would like to change the example to the usual use case:
In [9]: np.intersect1d([1,2,4,3],[3,1,5])
Out[9]: array([1, 3])

For arrays with non-unique elements, there is:

In [11]: np.intersect1d_nu([1,3,3],[3,1,1])
Out[11]: array([1, 3])

which just calls unique1d() for its arguments.

cheers,
r.


From nwagner at iam.uni-stuttgart.de  Fri Feb 27 10:20:46 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Fri, 27 Feb 2009 16:20:46 +0100
Subject: [Numpy-discussion] ValueError: invalid literal for float()
Message-ID: <web-117489235@uni-stuttgart.de>

Hi all,

Is it possible to modify the behaviour of float wrt
the following situation

  
>>> permas_M[0,2]
'1.5699999809265137D+01'
>>> float(permas_M[0,2])
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ValueError: invalid literal for float(): 
1.5699999809265137D+01

The following works.

>>> permas_M[0,2] = '1.5699999809265137E+01'
>>> permas_M[0,2]
'1.5699999809265137E+01'
>>> float(permas_M[0,2])
15.699999809265137

Nils


From charlesr.harris at gmail.com  Fri Feb 27 10:35:42 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 27 Feb 2009 08:35:42 -0700
Subject: [Numpy-discussion] ValueError: invalid literal for float()
In-Reply-To: <web-117489235@uni-stuttgart.de>
References: <web-117489235@uni-stuttgart.de>
Message-ID: <e06186140902270735q6dbcad25ia59d99763962f77@mail.gmail.com>

On Fri, Feb 27, 2009 at 8:20 AM, Nils Wagner
<nwagner at iam.uni-stuttgart.de>wrote:

> Hi all,
>
> Is it possible to modify the behaviour of float wrt
> the following situation
>
>
> >>> permas_M[0,2]
> '1.5699999809265137D+01'
> >>> float(permas_M[0,2])
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ValueError: invalid literal for float():
> 1.5699999809265137D+01
>

I think float is python, not numpy,  but numpy uses the same function
internally to convert from a string.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090227/5f00cc8e/attachment.html>

From animator333 at yahoo.com  Fri Feb 27 11:57:29 2009
From: animator333 at yahoo.com (Prashant Saxena)
Date: Fri, 27 Feb 2009 22:27:29 +0530 (IST)
Subject: [Numpy-discussion] image processing using numpy-scipy?
Message-ID: <332268.94796.qm@web94913.mail.in2.yahoo.com>

Hi,

This a little wiered problem. I am having a black and white image. (black background)
Entire image is filled with noisy white patterns of different size and shape. I need to fill the
white patches if there area is more then given one. Logically this could possible to use a quickfill algorithm
for every pixel and if the filled area generated by that pixel is more then given area(in pixel) then paint that
patch (filled area) with black.

I have read the docs of PIL but there is no function for this. Can I use numpy-scipy for the matter?
The image size is 1K.

Any help would be greatly apperciated.

Regards
Prashant



      Connect with friends all over the world. Get Yahoo! India Messenger at http://in.messenger.yahoo.com/?wm=n/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090227/f0be37c0/attachment.html>

From stefan at sun.ac.za  Fri Feb 27 12:26:23 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 27 Feb 2009 19:26:23 +0200
Subject: [Numpy-discussion] image processing using numpy-scipy?
In-Reply-To: <332268.94796.qm@web94913.mail.in2.yahoo.com>
References: <332268.94796.qm@web94913.mail.in2.yahoo.com>
Message-ID: <9457e7c80902270926i1904027eu9393ea7a4e478715@mail.gmail.com>

Hi Prashant

2009/2/27 Prashant Saxena <animator333 at yahoo.com>:
> This a little wiered problem. I am having a black and white image. (black
> background)
> Entire image is filled with noisy white patterns of different size and
> shape. I need to fill the
> white patches if there area is more then given one. Logically this could
> possible to use a quickfill algorithm
> for every pixel and if the filled area generated by that pixel is more then
> given area(in pixel) then paint that
> patch (filled area) with black.
>
> I have read the docs of PIL but there is no function for this. Can I use
> numpy-scipy for the matter?

Sure, there are a couple of options.  First, look at scipy.ndimage if
there is anything there you can use (i.e. maybe binary dilation is
sufficient).

Otherwise, I've got some connected component code at:

http://mentat.za.net/source/connected_components.tar.bz2

(The repository is at http://mentat.za.net/hg/ccomp if you prefer)

Using that code, you can identify connected regions, and then fill
them up as required.

Regards
St?fan


From david at ar.media.kyoto-u.ac.jp  Fri Feb 27 12:31:50 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 28 Feb 2009 02:31:50 +0900
Subject: [Numpy-discussion] Call for testing: full blas/lapack builds of
 numpy on windows 64 bits
Message-ID: <49A82386.9030208@ar.media.kyoto-u.ac.jp>

Hi,

    That's a call for testing for 64 bits windows users out there:
please try the following binary with the test suite:

http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.3.0.dev6517.win-amd64-py2.6.exe

python -c "import numpy; numpy.test()"

Report any crash. I am particularly interested in vista and higher
versions without visual studio installed - it looks like those cause a
lot of trouble wrt the C runtime. *THIS IS ONLY FOR TESTING* - it is
still far from being stable.

Details:

This one is built with free compilers (gcc + gfortran 4.4), and
BLAS/LAPACK built with gfortran. This should enable openmp usage too
(4.4 supports openmp3.0). It also means supporting scipy is not too far
away - there are still some issues with C++, but hopefully, those are
compiler bugs, and all numpy issues will have been sorted out.

cheers,

David


From zachary.pincus at yale.edu  Fri Feb 27 13:11:43 2009
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Fri, 27 Feb 2009 13:11:43 -0500
Subject: [Numpy-discussion] image processing using numpy-scipy?
In-Reply-To: <9457e7c80902270926i1904027eu9393ea7a4e478715@mail.gmail.com>
References: <332268.94796.qm@web94913.mail.in2.yahoo.com>
	<9457e7c80902270926i1904027eu9393ea7a4e478715@mail.gmail.com>
Message-ID: <B87B0040-477B-427C-B9A3-C228C6592892@yale.edu>

Hello,

>> This a little wiered problem. I am having a black and white image.  
>> (black
>> background)
>> Entire image is filled with noisy white patterns of different size  
>> and
>> shape. I need to fill the
>> white patches if there area is more then given one. Logically this  
>> could
>> possible to use a quickfill algorithm
>> for every pixel and if the filled area generated by that pixel is  
>> more then
>> given area(in pixel) then paint that
>> patch (filled area) with black.
>>
>> I have read the docs of PIL but there is no function for this. Can  
>> I use
>> numpy-scipy for the matter?
>
> Sure, there are a couple of options.  First, look at scipy.ndimage if
> there is anything there you can use (i.e. maybe binary dilation is
> sufficient).
>
> Otherwise, I've got some connected component code at:
>
> http://mentat.za.net/source/connected_components.tar.bz2
>
> (The repository is at http://mentat.za.net/hg/ccomp if you prefer)

I think that scipy.ndimage.label also provides connected-component  
labeling, with arbitrary connectivity. But this problem could also be  
solvable via dilation / erosion, if the area constraint is loose --  
e.g. if you want to quash all blobs smaller than, say, 5x5 pixels, you  
could just erode for 5 iterations, and then dilate the eroded image  
back for '-1' iterations (which causes the ndimage algorithm to  
iterate until no pixels change), using a mask of the original image  
(so that no pixels outside of the original blobs are turned on). This  
basically means that any object that survives the original erosion  
will be dilated back to its initial size. (Similar tricks can also be  
used to find / kill objects touching the edges -- use a non-zero  
constant out-of-bounds value and dilate an all-zeros array for -1  
iterations, using the original array as the mask. Then only objects  
touching the edge will get filled in...)

Alternately, if you need a very stringent area threshold -- e.g.  
remove all objects with five or fewer total pixels, regardless of  
their configuration -- then the connected-components approach is  
required. Below is a stab at it, though note that there's a slow step  
that I can't right now figure out how to avoid, short of coding just  
that in C or with cython or something...

Zach



import numpy
import scipy.ndimage as ndimage

_4_connected = numpy.array([[0, 1, 0],
                             [1, 1, 1],
                             [0, 1, 0]], dtype=bool)

def kill_small(binary_image, min_size, structure=_4_connected):
   label_image, num_objects = ndimage.label(binary_image, structure)
   # Label 0 is the background...
   object_labels = numpy.arange(1, num_objects+1)
   # Get the area of each labeled object by summing up the pixel  
values in the
   # binary image. (Divide the binary image by its max val to ensure  
that the image
   # consists of 1s and 0s, so that the sum equals the area in pixels.)
   areas = numpy.array(nd.sum(binary_image / binary_image.max(),  
label_image, object_labels))
   big_objects = object_labels[areas >= min_size]
   # This part will be pretty slow! But I can't think right now how to  
speed it up.
   # (If there are more big objects than small objects, it would be  
faster to
   # reverse this process and xor out the small objects from the  
binary image,
   # rather than the below, which or's up a new image of just the  
large objects.)
   big_object_image = numpy.zeros(binary_image.shape, dtype=bool)
   for bo in big_objects:
     big_object_image |= label_image == bo
   return big_object_image


From wnbell at gmail.com  Fri Feb 27 14:15:48 2009
From: wnbell at gmail.com (Nathan Bell)
Date: Fri, 27 Feb 2009 14:15:48 -0500
Subject: [Numpy-discussion] Call for testing: full blas/lapack builds of
	numpy on windows 64 bits
In-Reply-To: <49A82386.9030208@ar.media.kyoto-u.ac.jp>
References: <49A82386.9030208@ar.media.kyoto-u.ac.jp>
Message-ID: <d05265cb0902271115k533eddf5h167605701c22c446@mail.gmail.com>

On Fri, Feb 27, 2009 at 12:31 PM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Hi,
>
> ? ?That's a call for testing for 64 bits windows users out there:
> please try the following binary with the test suite:
>
> http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.3.0.dev6517.win-amd64-py2.6.exe
>
> python -c "import numpy; numpy.test()"
>

Running on Vista 64-bits.


C:\Python26>python -c "import numpy; numpy.test()"
Running unit tests for numpy
C:\Python26\lib\site-packages\nose\proxy.py:93: SyntaxWarning:
assertion is always true, perhaps remove parentheses?
  assert (test is self.test
NumPy version 1.3.0.dev6517
NumPy is installed in C:\Python26\lib\site-packages\numpy
Python version 2.6.1 (r26:67517, Dec  4 2008, 16:59:09) [MSC v.1500 64
bit (AMD64)]
nose version 0.10.4
Forcing DISTUTILS_USE_SDK=1

<snip lots of dots>

Ran 1787 tests in 8.916s

OK (KNOWNFAIL=6, SKIP=1)


Mission accomplished?

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From cournape at gmail.com  Fri Feb 27 14:33:43 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 28 Feb 2009 04:33:43 +0900
Subject: [Numpy-discussion] Call for testing: full blas/lapack builds of
	numpy on windows 64 bits
In-Reply-To: <d05265cb0902271115k533eddf5h167605701c22c446@mail.gmail.com>
References: <49A82386.9030208@ar.media.kyoto-u.ac.jp>
	<d05265cb0902271115k533eddf5h167605701c22c446@mail.gmail.com>
Message-ID: <5b8d13220902271133pe80530bj36200e42a1aeafd1@mail.gmail.com>

On Sat, Feb 28, 2009 at 4:15 AM, Nathan Bell <wnbell at gmail.com> wrote:
> On Fri, Feb 27, 2009 at 12:31 PM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
>> Hi,
>>
>> ? ?That's a call for testing for 64 bits windows users out there:
>> please try the following binary with the test suite:
>>
>> http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.3.0.dev6517.win-amd64-py2.6.exe
>>
>> python -c "import numpy; numpy.test()"
>>
>
> Running on Vista 64-bits.

Great, thanks. Do you have VS installed ? Did you install python for
all users (I would guess so, but I am not yet clear on all the details
on that matter).

cheers,

David


From animator333 at yahoo.com  Fri Feb 27 14:42:47 2009
From: animator333 at yahoo.com (Prashant Saxena)
Date: Sat, 28 Feb 2009 01:12:47 +0530 (IST)
Subject: [Numpy-discussion] image processing using numpy-scipy?
In-Reply-To: <B87B0040-477B-427C-B9A3-C228C6592892@yale.edu>
References: <332268.94796.qm@web94913.mail.in2.yahoo.com>
	<9457e7c80902270926i1904027eu9393ea7a4e478715@mail.gmail.com>
	<B87B0040-477B-427C-B9A3-C228C6592892@yale.edu>
Message-ID: <639132.82642.qm@web94910.mail.in2.yahoo.com>

Well,

I came up with a slightly different approach.

Get the first row on the image.
It would be something like this:
[1,1,1,0,0,0,1,1,0,0,1]
1 = white, 0 = black.
Run the floodfill on [0],[6] and [10] pixel
if floodfill area is smaller then given area then paint that area black
if floodfill area is larger then given area then copy that area to another image of same dimension and paint this area also black here.
This would save tons of per pixel iteration. Next row would surely be having less number of white pixels to do the iteration.

Now I need an efficient floodfill algorithm for above logic and I hope this will work.

Prashant




________________________________
From: Zachary Pincus <zachary.pincus at yale.edu>
To: Discussion of Numerical Python <numpy-discussion at scipy.org>
Sent: Friday, 27 February, 2009 11:41:43 PM
Subject: Re: [Numpy-discussion] image processing using numpy-scipy?

Hello,

>> This a little wiered problem. I am having a black and white image.  
>> (black
>> background)
>> Entire image is filled with noisy white patterns of different size  
>> and
>> shape. I need to fill the
>> white patches if there area is more then given one. Logically this  
>> could
>> possible to use a quickfill algorithm
>> for every pixel and if the filled area generated by that pixel is  
>> more then
>> given area(in pixel) then paint that
>> patch (filled area) with black.
>>
>> I have read the docs of PIL but there is no function for this. Can  
>> I use
>> numpy-scipy for the matter?
>
> Sure, there are a couple of options.  First, look at scipy.ndimage if
> there is anything there you can use (i.e. maybe binary dilation is
> sufficient).
>
> Otherwise, I've got some connected component code at:
>
> http://mentat.za.net/source/connected_components.tar.bz2
>
> (The repository is at http://mentat.za.net/hg/ccomp if you prefer)

I think that scipy.ndimage.label also provides connected-component  
labeling, with arbitrary connectivity. But this problem could also be  
solvable via dilation / erosion, if the area constraint is loose --  
e.g. if you want to quash all blobs smaller than, say, 5x5 pixels, you  
could just erode for 5 iterations, and then dilate the eroded image  
back for '-1' iterations (which causes the ndimage algorithm to  
iterate until no pixels change), using a mask of the original image  
(so that no pixels outside of the original blobs are turned on). This  
basically means that any object that survives the original erosion  
will be dilated back to its initial size. (Similar tricks can also be  
used to find / kill objects touching the edges -- use a non-zero  
constant out-of-bounds value and dilate an all-zeros array for -1  
iterations, using the original array as the mask. Then only objects  
touching the edge will get filled in...)

Alternately, if you need a very stringent area threshold -- e.g.  
remove all objects with five or fewer total pixels, regardless of  
their configuration -- then the connected-components approach is  
required. Below is a stab at it, though note that there's a slow step  
that I can't right now figure out how to avoid, short of coding just  
that in C or with cython or something...

Zach



import numpy
import scipy.ndimage as ndimage

_4_connected = numpy.array([[0, 1, 0],
                             [1, 1, 1],
                             [0, 1, 0]], dtype=bool)

def kill_small(binary_image, min_size, structure=_4_connected):
   label_image, num_objects = ndimage.label(binary_image, structure)
   # Label 0 is the background...
   object_labels = numpy.arange(1, num_objects+1)
   # Get the area of each labeled object by summing up the pixel  
values in the
   # binary image. (Divide the binary image by its max val to ensure  
that the image
   # consists of 1s and 0s, so that the sum equals the area in pixels.)
   areas = numpy.array(nd.sum(binary_image / binary_image.max(),  
label_image, object_labels))
   big_objects = object_labels[areas >= min_size]
   # This part will be pretty slow! But I can't think right now how to  
speed it up.
   # (If there are more big objects than small objects, it would be  
faster to
   # reverse this process and xor out the small objects from the  
binary image,
   # rather than the below, which or's up a new image of just the  
large objects.)
   big_object_image = numpy.zeros(binary_image.shape, dtype=bool)
   for bo in big_objects:
     big_object_image |= label_image == bo
   return big_object_image
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy..org/mailman/listinfo/numpy-discussion



      Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090228/cfb868fb/attachment.html>

From zachary.pincus at yale.edu  Fri Feb 27 15:26:55 2009
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Fri, 27 Feb 2009 15:26:55 -0500
Subject: [Numpy-discussion] Bilateral filter
In-Reply-To: <9457e7c80902141029w451160a2te6f7e6eb57db2d1e@mail.gmail.com>
References: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il>
	<9457e7c80902130150h4dc8755eqf6097fc0d20b6522@mail.gmail.com>
	<710F2847B0018641891D9A216027636029C432@ex3.envision.co.il>
	<9457e7c80902141029w451160a2te6f7e6eb57db2d1e@mail.gmail.com>
Message-ID: <2966A073-2846-4950-8A48-F170990503D8@yale.edu>

Hi all,

I just grabbed the latest bilateral filter from St?fan's repository,  
but I can't get it to work! I'm using a recent numpy SVN and the  
latest release of cython...

In [10]: bl = bilateral.bilateral(image, 2, 150)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call  
last)

/Users/zpincus/Documents/Research/Slack Lab/Experimental Data/Big Data/ 
2009-2-17/<ipython console> in <module>()

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
packages/bilateral/bilateral.pyc in bilateral(mat, xy_sig, z_sig,  
filter_size, mode)
      36     '''
      37
---> 38     filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size)
      39     size = filter_fcn.xy_size
      40     return generic_filter(mat, filter_fcn.cfilter, size=size,  
mode=mode)

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
packages/bilateral/bilateral_base.so in  
bilateral.bilateral_base.Bilat_fcn.__init__ (bilateral/ 
bilateral_base.c:590)()

NameError: np

I know very little about the internals of cython, so I can't figure  
this error out... it seems to be a problem with the 'cimport numpy as  
np' line in the pyx file, but beyond that I'm lost.


Let me know if I can provide any additional information,

Zach




On Feb 14, 2009, at 1:29 PM, St?fan van der Walt wrote:

> Hi Nadav
>
> 2009/2/13 Nadav Horesh <nadavh at visionsense.com>:
>> I tried the installer and it did not copy bilateral.py. I tried to  
>> improve it and the result is in the attached file.  I hope it would  
>> pass the mail filter, if not contact me directly to the email  
>> address below.
>
> Thanks!  I applied your changes and modified the setup.py to support
> in-place building.  Again available here:
>
> http://github.com/stefanv/bilateral/tree/master
>
> Cheers
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From dwf at cs.toronto.edu  Fri Feb 27 15:35:03 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Fri, 27 Feb 2009 15:35:03 -0500
Subject: [Numpy-discussion] Slicing/selection in multiple dimensions
	simultaneously
In-Reply-To: <463e11f90902261900o748940b6yf8410abda82524cc@mail.gmail.com>
References: <268febdf0709111511n3ca15d42o85d31831178d96a@mail.gmail.com>
	<46E71591.20802@gmail.com>
	<e4412d6b0709111542m355ad6a6w21784b5a2108c6c5@mail.gmail.com>
	<46E72116.8040408@enthought.com>
	<463e11f90902261900o748940b6yf8410abda82524cc@mail.gmail.com>
Message-ID: <069E94BE-B877-47C8-A723-703A7E3620B9@cs.toronto.edu>

Hey Jon,


On 26-Feb-09, at 10:00 PM, Jonathan Taylor wrote:

> Am I right to assume that there is no way elegant way to interact with
> slices.  i.e. Is there anyway to get
>
> a[ix_([2,3,6],:,[3,2])]
>
> to work?  So that the dimension is completely specified?  Or perhaps
> the only way to do this is via
>
> a[ix_([2,3,6],range(a.shape[1]),[3,2])]
>
> If anyone knows a better way?

a[[2,3,6],:,:][:,:,[3,2]] should do what you want.


In [21]: a = randn(50,50,50)

In [22]: all(a[ix_([2,3,6],range(a.shape[1]),[3,2])] == a[[2,3,6],:,:] 
[:,:,[3,2]])
Out[22]: True

David


From robert.kern at gmail.com  Fri Feb 27 15:38:52 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 27 Feb 2009 14:38:52 -0600
Subject: [Numpy-discussion] Slicing/selection in multiple dimensions
	simultaneously
In-Reply-To: <463e11f90902261900o748940b6yf8410abda82524cc@mail.gmail.com>
References: <268febdf0709111511n3ca15d42o85d31831178d96a@mail.gmail.com>
	<46E71591.20802@gmail.com>
	<e4412d6b0709111542m355ad6a6w21784b5a2108c6c5@mail.gmail.com>
	<46E72116.8040408@enthought.com>
	<463e11f90902261900o748940b6yf8410abda82524cc@mail.gmail.com>
Message-ID: <3d375d730902271238p7fe29192hb953df2c5f87c245@mail.gmail.com>

On Thu, Feb 26, 2009 at 21:00, Jonathan Taylor
<jonathan.taylor at utoronto.ca> wrote:
> Am I right to assume that there is no way elegant way to interact with
> slices. ?i.e. Is there anyway to get
>
> a[ix_([2,3,6],:,[3,2])]
>
> to work? ?So that the dimension is completely specified? ?Or perhaps
> the only way to do this is via
>
> a[ix_([2,3,6],range(a.shape[1]),[3,2])]
>
> If anyone knows a better way?

One could probably make ix_() take slice objects, too, to generate the
correct arange() in the appropriate place.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Fri Feb 27 15:44:57 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 27 Feb 2009 14:44:57 -0600
Subject: [Numpy-discussion] Speedup creation of a 3-color array from a
	2-d color-index array a color lut
In-Reply-To: <200902271241.51386.ddf@iqdotdt.com>
References: <mailman.7.1235757603.22736.numpy-discussion@scipy.org>
	<200902271241.51386.ddf@iqdotdt.com>
Message-ID: <3d375d730902271244n1402d1dk847b50a905ce7a82@mail.gmail.com>

On Fri, Feb 27, 2009 at 14:41, Delbert Franz <ddf at iqdotdt.com> wrote:
>
>> Message: 2
>> Date: Thu, 26 Feb 2009 23:32:35 -0600
>> From: Robert Kern <robert.kern at gmail.com>
>> Subject: Re: [Numpy-discussion] Speedup creation of a 3-color array
>> ? ? ? from a ?2-d color-index array a color lut
>> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
>> Message-ID:
>> ? ? ? <3d375d730902262132k5a011d14t84317458f9b6cf87 at mail.gmail.com>
>> Content-Type: text/plain; charset=UTF-8
>>
>> On Thu, Feb 26, 2009 at 23:27, Delbert Franz <ddf at iqdotdt.com> wrote:
>> >
>> > I have ?geotiff files of scanned paper maps that use an indexed color scheme
>> > with a 256-element
>> > color lookup table (color lut) and a 9252 by 7420 array ?of uint8 elements.
>> > The color is given by
>> > three values. ?I want to create an array with shape: (9252, 7420, 3) so that
>> > I can display the
>> > image without creating internal array working space in Matplotlib that
>> > exeeds 2^31 bytes.
>> >
>> > The following three approaches work in that the correct image is displayed,
>> > but all of them
>> > are waaaaay too slow:)
>> >
>> > ?Let
>> > ? doq have shape (9252, 7420) and have uint8 elements
>> > ? ctab have shape (256, 3) and have uint8 elements.
>> > ? doqq have shape (9252, 7420, 3) and have unit8 elements
>>
>> doqq = ctab[doq]
>>
> Thanks, Robert! ?It works just fine but it "blows my mind":) ?Never would of
> thought of trying that. ?I'll have to work hard at getting my "mind around" how
> Numpy "views the universe".

The full range of indexing is discussed here:

  http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html

>?I'm just starting to come to terms with what
> "object oriented" means in Python. ?In 46 years of developing software, one
> is always on a learning curve, sometimes relatively flat, but sometimes
> nearly vertical--this was a moment on the vertical:)

Good. That means you're learning quickly.  :-)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ddf at iqdotdt.com  Fri Feb 27 15:41:51 2009
From: ddf at iqdotdt.com (Delbert Franz)
Date: Fri, 27 Feb 2009 12:41:51 -0800
Subject: [Numpy-discussion]
	=?iso-8859-1?q?Speedup_creation_of_a_3-color_a?=
	=?iso-8859-1?q?rray_from_a=092-d_color-index_array_a_color_lut?=
In-Reply-To: <mailman.7.1235757603.22736.numpy-discussion@scipy.org>
References: <mailman.7.1235757603.22736.numpy-discussion@scipy.org>
Message-ID: <200902271241.51386.ddf@iqdotdt.com>



> Message: 2
> Date: Thu, 26 Feb 2009 23:32:35 -0600
> From: Robert Kern <robert.kern at gmail.com>
> Subject: Re: [Numpy-discussion] Speedup creation of a 3-color array
> 	from a	2-d color-index array a color lut
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Message-ID:
> 	<3d375d730902262132k5a011d14t84317458f9b6cf87 at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
> 
> On Thu, Feb 26, 2009 at 23:27, Delbert Franz <ddf at iqdotdt.com> wrote:
> >
> > I have ?geotiff files of scanned paper maps that use an indexed color scheme
> > with a 256-element
> > color lookup table (color lut) and a 9252 by 7420 array ?of uint8 elements.
> > The color is given by
> > three values. ?I want to create an array with shape: (9252, 7420, 3) so that
> > I can display the
> > image without creating internal array working space in Matplotlib that
> > exeeds 2^31 bytes.
> >
> > The following three approaches work in that the correct image is displayed,
> > but all of them
> > are waaaaay too slow:)
> >
> > ?Let
> > ? doq have shape (9252, 7420) and have uint8 elements
> > ? ctab have shape (256, 3) and have uint8 elements.
> > ? doqq have shape (9252, 7420, 3) and have unit8 elements
> 
> doqq = ctab[doq]
> 
Thanks, Robert!  It works just fine but it "blows my mind":)  Never would of 
thought of trying that.  I'll have to work hard at getting my "mind around" how
Numpy "views the universe".  I'm just starting to come to terms with what
"object oriented" means in Python.  In 46 years of developing software, one
is always on a learning curve, sometimes relatively flat, but sometimes
nearly vertical--this was a moment on the vertical:)

                              Delbert


From wnbell at gmail.com  Fri Feb 27 16:08:24 2009
From: wnbell at gmail.com (Nathan Bell)
Date: Fri, 27 Feb 2009 16:08:24 -0500
Subject: [Numpy-discussion] Call for testing: full blas/lapack builds of
	numpy on windows 64 bits
In-Reply-To: <5b8d13220902271133pe80530bj36200e42a1aeafd1@mail.gmail.com>
References: <49A82386.9030208@ar.media.kyoto-u.ac.jp>
	<d05265cb0902271115k533eddf5h167605701c22c446@mail.gmail.com>
	<5b8d13220902271133pe80530bj36200e42a1aeafd1@mail.gmail.com>
Message-ID: <d05265cb0902271308h41688f6dxfa7b6e9e11f10c84@mail.gmail.com>

On Fri, Feb 27, 2009 at 2:33 PM, David Cournapeau <cournape at gmail.com> wrote:
>
> Great, thanks. Do you have VS installed ? Did you install python for
> all users (I would guess so, but I am not yet clear on all the details
> on that matter).
>

I do not have VS installed.  I just downloaded the official Python
2.6.1 msi installer and blindly clicked 'next' a few times.  I then
installed nose from the source .tar.gz.

Python was installed to C:\Python26\, which I assume means all users.

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From dsdale24 at gmail.com  Fri Feb 27 16:46:48 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Fri, 27 Feb 2009 16:46:48 -0500
Subject: [Numpy-discussion] Call for testing: full blas/lapack builds of
	numpy on windows 64 bits
In-Reply-To: <d05265cb0902271308h41688f6dxfa7b6e9e11f10c84@mail.gmail.com>
References: <49A82386.9030208@ar.media.kyoto-u.ac.jp>
	<d05265cb0902271115k533eddf5h167605701c22c446@mail.gmail.com>
	<5b8d13220902271133pe80530bj36200e42a1aeafd1@mail.gmail.com>
	<d05265cb0902271308h41688f6dxfa7b6e9e11f10c84@mail.gmail.com>
Message-ID: <a08e5f80902271346m5f46ae64q31a4fda26c6c4c50@mail.gmail.com>

On Fri, Feb 27, 2009 at 4:08 PM, Nathan Bell <wnbell at gmail.com> wrote:

> On Fri, Feb 27, 2009 at 2:33 PM, David Cournapeau <cournape at gmail.com>
> wrote:
> >
> > Great, thanks. Do you have VS installed ? Did you install python for
> > all users (I would guess so, but I am not yet clear on all the details
> > on that matter).
> >
>
> I do not have VS installed.  I just downloaded the official Python
> 2.6.1 msi installer and blindly clicked 'next' a few times.  I then
> installed nose from the source .tar.gz.
>
> Python was installed to C:\Python26\, which I assume means all users.
>

Python-2.6.1 msi will only install for all users. "Install just for me" is
not selectable.

Your binary would not install against the 32-bit python-2.6.1. Thats good, I
forgot to check which python installer it gave me by default. I removed the
32 bit python and installed the 64 bit version. In the past I have always
had to run python module installer exe's with administrator privileges,
thats what I did this time with your numpy installer. I got the same results
as Nathan. However, I have the following packages installed:

VC++ 2005 redistributable (32 and 64 bit)
VC++ 2008 express edition sp1
VC++ 2008 redistributable x86
MVS 2008 remote debugger light (x74)
MS windows sdk for VS 2008 headers and libraries
MS windows sdk for VS 2008 SP1 express tools for .Net
MS windows sdk for VS 2008 SP1 express tools for win32

Have you considered distributing the windows installer as an msi? I dont
think you need to run those installers with admin privs on Vista.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090227/857a4cf3/attachment.html>

From bsouthey at gmail.com  Fri Feb 27 17:01:59 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Fri, 27 Feb 2009 16:01:59 -0600
Subject: [Numpy-discussion] Call for testing: full blas/lapack builds of
 numpy on windows 64 bits
In-Reply-To: <d05265cb0902271308h41688f6dxfa7b6e9e11f10c84@mail.gmail.com>
References: <49A82386.9030208@ar.media.kyoto-u.ac.jp>	<d05265cb0902271115k533eddf5h167605701c22c446@mail.gmail.com>	<5b8d13220902271133pe80530bj36200e42a1aeafd1@mail.gmail.com>
	<d05265cb0902271308h41688f6dxfa7b6e9e11f10c84@mail.gmail.com>
Message-ID: <49A862D7.60009@gmail.com>

Nathan Bell wrote:
> On Fri, Feb 27, 2009 at 2:33 PM, David Cournapeau <cournape at gmail.com> wrote:
>   
>> Great, thanks. Do you have VS installed ? Did you install python for
>> all users (I would guess so, but I am not yet clear on all the details
>> on that matter).
>>
>>     
>
> I do not have VS installed.  I just downloaded the official Python
> 2.6.1 msi installer and blindly clicked 'next' a few times.  I then
> installed nose from the source .tar.gz.
>
> Python was installed to C:\Python26\, which I assume means all users.
>
>   
Hi,
After much frustration and disabling my antivirus software (McAfee), I 
was able to get the 'import numpy' to work and tests to pass as 
expected. Of course after turning it back on and restarting the IDLE gui 
a few times, the 'import numpy' crashes. This also happens on the 
command line.

This is clearly related to the antivirus software because enabling or 
disabling the 'on-access scan' is usually sufficient to failure import 
failure or success, respectively. Perhaps some thing to do with MinGW on 
Vista?

I am running an Intel Quad system (QX6700) with Vista Business with 
service pack 1 - I don't think VS is installed.

Bruce



From bgerke at slac.stanford.edu  Fri Feb 27 19:26:27 2009
From: bgerke at slac.stanford.edu (Brian Gerke)
Date: Fri, 27 Feb 2009 16:26:27 -0800
Subject: [Numpy-discussion] problem with assigning to recarrays
Message-ID: <F0AD3910-E447-46AC-9285-F8000A74CF4A@slac.stanford.edu>


Hi-

I'm quite new to numpy and to python in general, so I apologize if I'm  
missing something obvious, but I've come across some seemingly nasty  
behavior when trying to assign values to the fields of an indexed  
subarray of a numpy record array.   Perhaps an example would explain  
it best.

First, I make a boring record array:

	In [124]: r = rec.fromarrays([zeros(5), zeros(5)],  
names='field1,field2')

This has five elements with two fields, all values are zero.
Now I can change the values for field1 for a few of the array elements:

	In [125]: r[1].field1=1

	In [126]: r[3].field1=1

Let's check and make sure that worked:

	In [127]: print r.field1
	[ 0. 1. 0. 1. 0.]

So far, so good.
Now I want to change the value of field2 for those same elements:

	In [128]: r[where(r.field1 == 1.)].field2 = 1

Ok, so now the values of field 2 have been changed, for those elements  
right?

	In [129]: r.field2

	Out[129]: array([ 0.,  0.,  0.,  0.,  0.])

Wait.  What?
That can't be right.  Let's check again:

	In [130]: print r[where(r.field1 == 1.)].field2
	[ 0. 0.]

Ok, so it appears that I can *access* fields in this array with an  
array of indices, but I can't assign new values to fields so  
accessed.  However, I *can* change the values if I use a scalar  
index.  This is different from the behavior of ordinary arrays, for  
which I can reassign elements' values either way.

Moreover, when I try to reassign record array fields by indexing with  
an array of indices, it would appear that nothing at all happens.    
This syntax is equivalent to the pass command.

So, my question is this:  is there some reason for this behavior in  
record arrays, which is unexpectedly different from the behavior of  
normal arrays, and rather confusing.   If so, why does the attempt to  
assign values to fields of an indexed subarray not raise some kind of  
error, rather than doing nothing?  I think it's unlikely that I've  
actually found a bug in numpy, but this behavior does not make sense  
to me.


Thanks for any insights,

Brian





From robert.kern at gmail.com  Fri Feb 27 19:30:24 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 27 Feb 2009 18:30:24 -0600
Subject: [Numpy-discussion] problem with assigning to recarrays
In-Reply-To: <F0AD3910-E447-46AC-9285-F8000A74CF4A@slac.stanford.edu>
References: <F0AD3910-E447-46AC-9285-F8000A74CF4A@slac.stanford.edu>
Message-ID: <3d375d730902271630w5f924425t9a88939273c32cce@mail.gmail.com>

On Fri, Feb 27, 2009 at 18:26, Brian Gerke <bgerke at slac.stanford.edu> wrote:
>
> Hi-
>
> I'm quite new to numpy and to python in general, so I apologize if I'm
> missing something obvious, but I've come across some seemingly nasty
> behavior when trying to assign values to the fields of an indexed
> subarray of a numpy record array. ? Perhaps an example would explain
> it best.
>
> First, I make a boring record array:
>
> ? ? ? ?In [124]: r = rec.fromarrays([zeros(5), zeros(5)],
> names='field1,field2')
>
> This has five elements with two fields, all values are zero.
> Now I can change the values for field1 for a few of the array elements:
>
> ? ? ? ?In [125]: r[1].field1=1
>
> ? ? ? ?In [126]: r[3].field1=1
>
> Let's check and make sure that worked:
>
> ? ? ? ?In [127]: print r.field1
> ? ? ? ?[ 0. 1. 0. 1. 0.]
>
> So far, so good.
> Now I want to change the value of field2 for those same elements:
>
> ? ? ? ?In [128]: r[where(r.field1 == 1.)].field2 = 1
>
> Ok, so now the values of field 2 have been changed, for those elements
> right?
>
> ? ? ? ?In [129]: r.field2
>
> ? ? ? ?Out[129]: array([ 0., ?0., ?0., ?0., ?0.])
>
> Wait. ?What?
> That can't be right. ?Let's check again:
>
> ? ? ? ?In [130]: print r[where(r.field1 == 1.)].field2
> ? ? ? ?[ 0. 0.]
>
> Ok, so it appears that I can *access* fields in this array with an
> array of indices, but I can't assign new values to fields so
> accessed. ?However, I *can* change the values if I use a scalar
> index. ?This is different from the behavior of ordinary arrays, for
> which I can reassign elements' values either way.
>
> Moreover, when I try to reassign record array fields by indexing with
> an array of indices, it would appear that nothing at all happens.
> This syntax is equivalent to the pass command.
>
> So, my question is this: ?is there some reason for this behavior in
> record arrays, which is unexpectedly different from the behavior of
> normal arrays, and rather confusing. ? If so, why does the attempt to
> assign values to fields of an indexed subarray not raise some kind of
> error, rather than doing nothing? ?I think it's unlikely that I've
> actually found a bug in numpy, but this behavior does not make sense
> to me.

r[where(r.field1 == 1.)] make a copy. There is no way for us to
construct a view onto the original memory for this circumstance given
numpy's memory model.

r[where(r.field1 == 1.)].field2 = 0.0 assigns to the copy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From pgmdevlist at gmail.com  Fri Feb 27 20:05:15 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Fri, 27 Feb 2009 20:05:15 -0500
Subject: [Numpy-discussion] problem with assigning to recarrays
In-Reply-To: <F0AD3910-E447-46AC-9285-F8000A74CF4A@slac.stanford.edu>
References: <F0AD3910-E447-46AC-9285-F8000A74CF4A@slac.stanford.edu>
Message-ID: <C96B3CD5-338F-435D-A2B7-EAAE35115CD0@gmail.com>

As a follow-up to Robert's answer:

 >>> r[r.field1 == 1].field2 = 1
doesn't work, but

 >>>r.field2[r.field1==1] = 1
does.


> So far, so good.
> Now I want to change the value of field2 for those same elements:
>
> 	In [128]: r[where(r.field1 == 1.)].field2 = 1
>






> Ok, so now the values of field 2 have been changed, for those elements
> right?
>
> 	In [129]: r.field2
>
> 	Out[129]: array([ 0.,  0.,  0.,  0.,  0.])
>
> Wait.  What?
> That can't be right.  Let's check again:
>
> 	In [130]: print r[where(r.field1 == 1.)].field2
> 	[ 0. 0.]
>
> Ok, so it appears that I can *access* fields in this array with an
> array of indices, but I can't assign new values to fields so
> accessed.  However, I *can* change the values if I use a scalar
> index.  This is different from the behavior of ordinary arrays, for
> which I can reassign elements' values either way.
>
> Moreover, when I try to reassign record array fields by indexing with
> an array of indices, it would appear that nothing at all happens.
> This syntax is equivalent to the pass command.
>
> So, my question is this:  is there some reason for this behavior in
> record arrays, which is unexpectedly different from the behavior of
> normal arrays, and rather confusing.   If so, why does the attempt to
> assign values to fields of an indexed subarray not raise some kind of
> error, rather than doing nothing?  I think it's unlikely that I've
> actually found a bug in numpy, but this behavior does not make sense
> to me.
>
>
> Thanks for any insights,
>
> Brian
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From bgerke at slac.stanford.edu  Fri Feb 27 20:06:33 2009
From: bgerke at slac.stanford.edu (Brian Gerke)
Date: Fri, 27 Feb 2009 17:06:33 -0800
Subject: [Numpy-discussion] problem with assigning to recarrays
In-Reply-To: <3d375d730902271630w5f924425t9a88939273c32cce@mail.gmail.com>
References: <F0AD3910-E447-46AC-9285-F8000A74CF4A@slac.stanford.edu>
	<3d375d730902271630w5f924425t9a88939273c32cce@mail.gmail.com>
Message-ID: <2E4F7165-0A1D-47EB-9AF5-64093D99056B@slac.stanford.edu>


On Feb 27, 2009, at 4:30 PM, Robert Kern wrote:
>>
> r[where(r.field1 == 1.)] make a copy. There is no way for us to
> construct a view onto the original memory for this circumstance given
> numpy's memory model.

Many thanks for the quick reply.  I assume that this is true only for  
record arrays, not for ordinary arrays?  Certainly I can make an  
assignment in this way with a normal array.

Also, if it is truly impossible to change this behavior, or to have it  
raise an error--then are there any best-practice suggestions for how  
to remember and avoid running into this non-obvious behavior?  If one  
thinks of record arrays as inheriting  from numpy arrays, then this  
problem is certainly unexpected.


Also, I've just found that the following syntax does do what is  
expected:

(r.field2)[where(field1 == 1.)] = 1.

It is at least a little aesthetically displeasing that the syntax  
works one way but not the other.  Perhaps my best bet is to stick with  
this syntax and forget that the other exists?  A less-than-satisfying  
solution, but workable.

Brian

>
>
> r[where(r.field1 == 1.)].field2 = 0.0 assigns to the copy.
>
> -- 
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From dwf at cs.toronto.edu  Fri Feb 27 23:54:42 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Fri, 27 Feb 2009 23:54:42 -0500
Subject: [Numpy-discussion] Slicing/selection in multiple dimensions
	simultaneously
In-Reply-To: <069E94BE-B877-47C8-A723-703A7E3620B9@cs.toronto.edu>
References: <268febdf0709111511n3ca15d42o85d31831178d96a@mail.gmail.com>
	<46E71591.20802@gmail.com>
	<e4412d6b0709111542m355ad6a6w21784b5a2108c6c5@mail.gmail.com>
	<46E72116.8040408@enthought.com>
	<463e11f90902261900o748940b6yf8410abda82524cc@mail.gmail.com>
	<069E94BE-B877-47C8-A723-703A7E3620B9@cs.toronto.edu>
Message-ID: <6CF9CBA8-B21A-44F2-BE77-218DBBC05648@cs.toronto.edu>


On 27-Feb-09, at 3:35 PM, David Warde-Farley wrote:
>
> a[[2,3,6],:,:][:,:,[3,2]] should do what you want.

Slightly more elegantly (I always forget about this syntax):

a[[2,3,6], ...][..., [3,2]]

David


From robert.kern at gmail.com  Sat Feb 28 01:58:49 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 28 Feb 2009 00:58:49 -0600
Subject: [Numpy-discussion] problem with assigning to recarrays
In-Reply-To: <2E4F7165-0A1D-47EB-9AF5-64093D99056B@slac.stanford.edu>
References: <F0AD3910-E447-46AC-9285-F8000A74CF4A@slac.stanford.edu>
	<3d375d730902271630w5f924425t9a88939273c32cce@mail.gmail.com>
	<2E4F7165-0A1D-47EB-9AF5-64093D99056B@slac.stanford.edu>
Message-ID: <3d375d730902272258k32be62a7j8120064ad1017de8@mail.gmail.com>

On Fri, Feb 27, 2009 at 19:06, Brian Gerke <bgerke at slac.stanford.edu> wrote:
>
> On Feb 27, 2009, at 4:30 PM, Robert Kern wrote:
>>>
>> r[where(r.field1 == 1.)] make a copy. There is no way for us to
>> construct a view onto the original memory for this circumstance given
>> numpy's memory model.
>
> Many thanks for the quick reply. ?I assume that this is true only for
> record arrays, not for ordinary arrays? ?Certainly I can make an
> assignment in this way with a normal array.

Well, you are doing two very different things. Let's back up a bit.

Python gives us two hooks to modify an object in-place with an
assignment: __setitem__ and __setattr__.

  x[<item>] = y   ==>  x.__setitem__(<item>, y)
  x.<attr>  = y   ==>  x.__setattr__('<attr>', y)

Now, we don't need to restrict ourselves to just variables for 'x'; we
can have any expression that evaluates to an object.

  (<expr>)[<item>] = y  ==> (<expr>).__setitem__(<item>, y)
  (<expr>).<attr>  = y  ==> (<expr>).__setattr__('<attr>', y)

The key here is that the (<expr>) on the LHS is evaluated just like
any expression appearing anywhere else in your code. The only special
in-place behavior is restricted to the *outermost* [<item>] or
.<attr>.

So when you do this:

  r[where(r.field1 == 1.)].field2 = 1.0

it translates to something like this:

  tmp = r.__getitem__(where(r.field1 == 1.0))  # Makes a copy!
  tmp.__setattr__('field2', 1.0)

Note that the first line is a __getitem__, not a __setitem__ which can
modify r in-place.

> Also, if it is truly impossible to change this behavior, or to have it
> raise an error--then are there any best-practice suggestions for how
> to remember and avoid running into this non-obvious behavior? ?If one
> thinks of record arrays as inheriting ?from numpy arrays, then this
> problem is certainly unexpected.

It's a natural consequence of the preceding rules. This a Python
thing, not a difference between numpy arrays and record arrays. Just
keep those rules in mind.

> Also, I've just found that the following syntax does do what is
> expected:
>
> (r.field2)[where(field1 == 1.)] = 1.
>
> It is at least a little aesthetically displeasing that the syntax
> works one way but not the other. ?Perhaps my best bet is to stick with
> this syntax and forget that the other exists? ?A less-than-satisfying
> solution, but workable.

If you drop the extraneous bits, it becomes a fair bit more readable:

  r.field2[r.field1 == 1] = 1

This is idiomatic; you'll see it all over the place where record
arrays are used. The reason that this form modifies r in-place is
because r.__getattr__('field2') is able to return a view rather than a
copy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Sat Feb 28 04:09:42 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sat, 28 Feb 2009 11:09:42 +0200
Subject: [Numpy-discussion] Bilateral filter
In-Reply-To: <2966A073-2846-4950-8A48-F170990503D8@yale.edu>
References: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il>
	<9457e7c80902130150h4dc8755eqf6097fc0d20b6522@mail.gmail.com>
	<710F2847B0018641891D9A216027636029C432@ex3.envision.co.il>
	<9457e7c80902141029w451160a2te6f7e6eb57db2d1e@mail.gmail.com>
	<2966A073-2846-4950-8A48-F170990503D8@yale.edu>
Message-ID: <9457e7c80902280109y304d8557s6dfd517e88ee2600@mail.gmail.com>

Hi Zachary

2009/2/27 Zachary Pincus <zachary.pincus at yale.edu>:
> I just grabbed the latest bilateral filter from St?fan's repository,
> but I can't get it to work! I'm using a recent numpy SVN and the
> latest release of cython...

Just to do a sanity check, did you do:

python setup.py build_ext -i

and then ran the example?  I've built this using the development
version of Cython, but your version sounds new enough.

Regards
St?fan


From pav at iki.fi  Sat Feb 28 07:11:44 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 28 Feb 2009 12:11:44 +0000 (UTC)
Subject: [Numpy-discussion] Tickets for closing
Message-ID: <gob9m0$st0$2@ger.gmane.org>

Hi,

Could someone with the proper Trac permissions close the following 
tickets:

	http://scipy.org/scipy/numpy/ticket/884
	http://scipy.org/scipy/numpy/ticket/902
		Fixed in trunk, when fix_float_format was merged back.

	http://scipy.org/scipy/numpy/ticket/955
		Fixed in trunk + tests.

	http://scipy.org/scipy/numpy/changeset/6283
		Addressed in trunk?

Also, the following could be checked/reviewed:

	http://scipy.org/scipy/numpy/ticket/975
		Possibly also fixed?

	http://scipy.org/scipy/numpy/ticket/629
		Does the fix_float_format branch address printing
		longdouble nans (which I guess was the last issue)?

		Could the currently commented-out test be uncommented,
		cf. at the end of numpy/core/tests/test_scalarmath.py?
		Is float32 still a problem?
		(The test passes for me.)

-- 
Pauli Virtanen



From cournape at gmail.com  Sat Feb 28 07:30:27 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 28 Feb 2009 21:30:27 +0900
Subject: [Numpy-discussion] Tickets for closing
In-Reply-To: <gob9m0$st0$2@ger.gmane.org>
References: <gob9m0$st0$2@ger.gmane.org>
Message-ID: <5b8d13220902280430tffec78dybc45837a719be404@mail.gmail.com>

On Sat, Feb 28, 2009 at 9:11 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Hi,
>
> Could someone with the proper Trac permissions close the following
> tickets:
>
> ? ? ? ?http://scipy.org/scipy/numpy/ticket/884
> ? ? ? ?http://scipy.org/scipy/numpy/ticket/902
> ? ? ? ? ? ? ? ?Fixed in trunk, when fix_float_format was merged back.
>
> ? ? ? ?http://scipy.org/scipy/numpy/ticket/955
> ? ? ? ? ? ? ? ?Fixed in trunk + tests.

Closed.

>
> ? ? ? ?http://scipy.org/scipy/numpy/changeset/6283
> ? ? ? ? ? ? ? ?Addressed in trunk?

I think there are issues with this on windows (removing files with
open handles fail), I will check and comment/close depending on the
result.

> Also, the following could be checked/reviewed:
>
> ? ? ? ?http://scipy.org/scipy/numpy/ticket/975
> ? ? ? ? ? ? ? ?Possibly also fixed?
>
> ? ? ? ?http://scipy.org/scipy/numpy/ticket/629
> ? ? ? ? ? ? ? ?Does the fix_float_format branch address printing
> ? ? ? ? ? ? ? ?longdouble nans (which I guess was the last issue)?

Yes it should fix this long-standing issue, but only in some cases I
believe (when printing is not done through our work in numpyos.c). I
will still check that the test works, at least.

David


From pav at iki.fi  Sat Feb 28 08:41:04 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 28 Feb 2009 13:41:04 +0000 (UTC)
Subject: [Numpy-discussion] Tickets for closing
References: <gob9m0$st0$2@ger.gmane.org>
	<5b8d13220902280430tffec78dybc45837a719be404@mail.gmail.com>
Message-ID: <gobetg$st0$3@ger.gmane.org>

Sat, 28 Feb 2009 21:30:27 +0900, David Cournapeau wrote:
[clip: #951]
> I think there are issues with this on windows (removing files with open
> handles fail), I will check and comment/close depending on the result.
[clip: #629]
> Yes it should fix this long-standing issue, but only in some cases I
> believe (when printing is not done through our work in numpyos.c). I
> will still check that the test works, at least.

Thanks!

-- 
Pauli Virtanen



From neilcrighton at gmail.com  Sat Feb 28 08:56:42 2009
From: neilcrighton at gmail.com (Neil)
Date: Sat, 28 Feb 2009 13:56:42 +0000 (UTC)
Subject: [Numpy-discussion] intersect1d and setmember1d
References: <90CBFFFE6273484B9579400AC950800502024765@ntsydexm01.pc.internal.macquarie.com>
	<f4f93d420902251648k4cc67d41lc858048db641b590@mail.gmail.com>
	<243385.2089.qm@web94910.mail.in2.yahoo.com>
Message-ID: <loom.20090228T132654-761@post.gmane.org>

mudit sharma <mudit_19a <at> yahoo.com> writes:

> intersect1d and setmember1d doesn't give expected results in case there are
duplicate values in either
> array becuase it works by sorting data and substracting previous value. Is
there an alternative in numpy
> to get indices of intersected values.
> 
> In [31]: p nonzero(setmember1d(v1.Id, v2.Id))[0]
> [ 0  1  2  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25  
                     <-------------- index 2 shouldn't be here look at the
> data below.
>  26 27 28 29]
> 
> In [32]: p v1.Id[:10]
> [ 232.  232.  233.  233.  234.  234.  235.  235.  237.  237.]
> 
> In [33]: p v2.Id[:10]
> [ 232.  232.  234.  234.  235.  235.  236.  236.  237.  237.]
> 


As far as I know there isn't an obvious way to get the functionality of
setmember1d working on non-unique inputs. However, I've needed this operation
quite a lot, so here's a function I wrote that does it. It's only a few times
slower than numpy's setmember1d. You're welcome to use it.

import numpy as np

def ismember(a1,a2):
    """ Test whether items from a2 are in a1.
    
    This does the same thing as np.setmember1d, but works on
    non-unique arrays.
    
    Only a few (2-4) times slower than np.setmember1d, and a lot
    faster than [i in a2 for i in a1].

    An example that np.setmember1d gets wrong: 
    
    >>> a1 = np.array([5,4,5,3,4,4,3,4,3,5,2,1,5,5])
    >>> a2 = [2,3,4]
    >>> mask = ismember(a1,a2)
    >>> a1[mask]
    array([4, 3, 4, 4, 3, 4, 3, 2])
    """
    a2 = set(a2)
    a1 = np.asarray(a1)
    ind = a1.argsort()
    a1 = a1[ind]
    mask  = []
    # need this bit because prev is not defined for first item
    item  = a1[0]
    if item in a2:
        mask.append(True)
        a2.remove(item)
    else:
        mask.append(False)
    prev = item
    # main loop
    for item in a1[1:]:
        if item == prev:
            mask.append(mask[-1])
        elif item in a2:
            mask.append(True)
            prev = item
            a2.remove(item)
        else:
            mask.append(False)
            prev = item
    # restore mask to original ordering of a1 and return
    mask = np.array(mask)
    return mask[ind.argsort()]





From pav at iki.fi  Sat Feb 28 09:08:38 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 28 Feb 2009 14:08:38 +0000 (UTC)
Subject: [Numpy-discussion] RFR: #1008 Loss of precision in (complex)
	arcsinh & arctanh
Message-ID: <gobgh6$st0$4@ger.gmane.org>


http://scipy.org/scipy/numpy/ticket/1008

http://codereview.appspot.com/22054

-- 
Pauli Virtanen



From stefan at sun.ac.za  Sat Feb 28 09:17:38 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sat, 28 Feb 2009 16:17:38 +0200
Subject: [Numpy-discussion] RFR: #1008 Loss of precision in (complex)
	arcsinh & arctanh
In-Reply-To: <gobgh6$st0$4@ger.gmane.org>
References: <gobgh6$st0$4@ger.gmane.org>
Message-ID: <9457e7c80902280617n43090245kfa72c410e0d4a1e9@mail.gmail.com>

Hey Pauli

2009/2/28 Pauli Virtanen <pav at iki.fi>:
>
> http://scipy.org/scipy/numpy/ticket/1008
> http://codereview.appspot.com/22054

Fantastic patch!  Please apply.

Cheers
St?fan


From cournape at gmail.com  Sat Feb 28 10:34:41 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 1 Mar 2009 00:34:41 +0900
Subject: [Numpy-discussion] RFR: #1008 Loss of precision in (complex)
	arcsinh & arctanh
In-Reply-To: <gobgh6$st0$4@ger.gmane.org>
References: <gobgh6$st0$4@ger.gmane.org>
Message-ID: <5b8d13220902280734i3baaeac3h1473535223b43@mail.gmail.com>

On Sat, Feb 28, 2009 at 11:08 PM, Pauli Virtanen <pav at iki.fi> wrote:
>
> http://scipy.org/scipy/numpy/ticket/1008
>
> http://codereview.appspot.com/22054

I added a few comments - the only significant one concerns types for
unit tests: I think it would be nice to test for float and long double
as well.

David


From jonathan.taylor at utoronto.ca  Sat Feb 28 12:27:07 2009
From: jonathan.taylor at utoronto.ca (Jonathan Taylor)
Date: Sat, 28 Feb 2009 12:27:07 -0500
Subject: [Numpy-discussion] Slicing/selection in multiple dimensions
	simultaneously
In-Reply-To: <6CF9CBA8-B21A-44F2-BE77-218DBBC05648@cs.toronto.edu>
References: <268febdf0709111511n3ca15d42o85d31831178d96a@mail.gmail.com>
	<46E71591.20802@gmail.com>
	<e4412d6b0709111542m355ad6a6w21784b5a2108c6c5@mail.gmail.com>
	<46E72116.8040408@enthought.com>
	<463e11f90902261900o748940b6yf8410abda82524cc@mail.gmail.com>
	<069E94BE-B877-47C8-A723-703A7E3620B9@cs.toronto.edu>
	<6CF9CBA8-B21A-44F2-BE77-218DBBC05648@cs.toronto.edu>
Message-ID: <463e11f90902280927k42a01ae5j7c0ed87ece03dca0@mail.gmail.com>

Hi Dave.

This does seem like the only way to write this nicely.  Unfortunately,
I think this may be wasteful memory wise (in contrast to what the
obvious matlab code would do) as it constructs an array with the whole
first index intact at first.  I think I will use it anyways though as
I find the other notations pretty awkward.

Jon.

On Fri, Feb 27, 2009 at 11:54 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>
> On 27-Feb-09, at 3:35 PM, David Warde-Farley wrote:
>>
>> a[[2,3,6],:,:][:,:,[3,2]] should do what you want.
>
> Slightly more elegantly (I always forget about this syntax):
>
> a[[2,3,6], ...][..., [3,2]]
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From cournape at gmail.com  Sat Feb 28 13:51:31 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 1 Mar 2009 03:51:31 +0900
Subject: [Numpy-discussion] Call for testing: full blas/lapack builds of
	numpy on windows 64 bits
In-Reply-To: <d05265cb0902271308h41688f6dxfa7b6e9e11f10c84@mail.gmail.com>
References: <49A82386.9030208@ar.media.kyoto-u.ac.jp>
	<d05265cb0902271115k533eddf5h167605701c22c446@mail.gmail.com>
	<5b8d13220902271133pe80530bj36200e42a1aeafd1@mail.gmail.com>
	<d05265cb0902271308h41688f6dxfa7b6e9e11f10c84@mail.gmail.com>
Message-ID: <5b8d13220902281051n31332145m2f23556e48f261ea@mail.gmail.com>

On Sat, Feb 28, 2009 at 6:08 AM, Nathan Bell <wnbell at gmail.com> wrote:
> On Fri, Feb 27, 2009 at 2:33 PM, David Cournapeau <cournape at gmail.com> wrote:
>>
>> Great, thanks. Do you have VS installed ? Did you install python for
>> all users (I would guess so, but I am not yet clear on all the details
>> on that matter).
>>
>
> I do not have VS installed. ?I just downloaded the official Python
> 2.6.1 msi installer and blindly clicked 'next' a few times. ?I then
> installed nose from the source .tar.gz.
>
> Python was installed to C:\Python26\, which I assume means all users.

Great, thank you very much for those informations. It looks like we
will be able to provide a 64 bits numpy binary for 1.3.0.

David


From nadavh at visionsense.com  Sat Feb 28 13:53:41 2009
From: nadavh at visionsense.com (Nadav Horesh)
Date: Sat, 28 Feb 2009 20:53:41 +0200
Subject: [Numpy-discussion] Bilateral filter
References: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il><9457e7c80902130150h4dc8755eqf6097fc0d20b6522@mail.gmail.com><710F2847B0018641891D9A216027636029C432@ex3.envision.co.il><9457e7c80902141029w451160a2te6f7e6eb57db2d1e@mail.gmail.com>
	<2966A073-2846-4950-8A48-F170990503D8@yale.edu>
Message-ID: <710F2847B0018641891D9A216027636029C450@ex3.envision.co.il>

I could not reproduce your error. I am using latest official releases of numpy and cython on a linux box (do you use Mac?).  I am attaching the package I have on my PC, for the small chance it would help.

  Nadav.


-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? Zachary Pincus
????: ? 27-??????-09 22:26
??: Discussion of Numerical Python
????: Re: [Numpy-discussion] Bilateral filter
 
Hi all,

I just grabbed the latest bilateral filter from St?fan's repository,  
but I can't get it to work! I'm using a recent numpy SVN and the  
latest release of cython...

In [10]: bl = bilateral.bilateral(image, 2, 150)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call  
last)

/Users/zpincus/Documents/Research/Slack Lab/Experimental Data/Big Data/ 
2009-2-17/<ipython console> in <module>()

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
packages/bilateral/bilateral.pyc in bilateral(mat, xy_sig, z_sig,  
filter_size, mode)
      36     '''
      37
---> 38     filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size)
      39     size = filter_fcn.xy_size
      40     return generic_filter(mat, filter_fcn.cfilter, size=size,  
mode=mode)

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
packages/bilateral/bilateral_base.so in  
bilateral.bilateral_base.Bilat_fcn.__init__ (bilateral/ 
bilateral_base.c:590)()

NameError: np

I know very little about the internals of cython, so I can't figure  
this error out... it seems to be a problem with the 'cimport numpy as  
np' line in the pyx file, but beyond that I'm lost.


Let me know if I can provide any additional information,

Zach




On Feb 14, 2009, at 1:29 PM, St?fan van der Walt wrote:

> Hi Nadav
>
> 2009/2/13 Nadav Horesh <nadavh at visionsense.com>:
>> I tried the installer and it did not copy bilateral.py. I tried to  
>> improve it and the result is in the attached file.  I hope it would  
>> pass the mail filter, if not contact me directly to the email  
>> address below.
>
> Thanks!  I applied your changes and modified the setup.py to support
> in-place building.  Again available here:
>
> http://github.com/stefanv/bilateral/tree/master
>
> Cheers
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion


-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 7031 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090228/8c694612/attachment.bin>

From cournape at gmail.com  Sat Feb 28 13:57:39 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 1 Mar 2009 03:57:39 +0900
Subject: [Numpy-discussion] Call for testing: full blas/lapack builds of
	numpy on windows 64 bits
In-Reply-To: <49A862D7.60009@gmail.com>
References: <49A82386.9030208@ar.media.kyoto-u.ac.jp>
	<d05265cb0902271115k533eddf5h167605701c22c446@mail.gmail.com>
	<5b8d13220902271133pe80530bj36200e42a1aeafd1@mail.gmail.com>
	<d05265cb0902271308h41688f6dxfa7b6e9e11f10c84@mail.gmail.com>
	<49A862D7.60009@gmail.com>
Message-ID: <5b8d13220902281057g4a53ccb1i8fdbf37cc511116e@mail.gmail.com>

On Sat, Feb 28, 2009 at 7:01 AM, Bruce Southey <bsouthey at gmail.com> wrote:

> Hi,
> After much frustration and disabling my antivirus software (McAfee), I
> was able to get the 'import numpy' to work and tests to pass as
> expected. Of course after turning it back on and restarting the IDLE gui
> a few times, the 'import numpy' crashes. This also happens on the
> command line.

Hm, let's assume this is a problem of the anti virus software, and not
from mingw :) One way to test this would be to see if a binary built
with VS 2008 would cause the same trouble.

>
> This is clearly related to the antivirus software because enabling or
> disabling the 'on-access scan' is usually sufficient to failure import
> failure or success, respectively. Perhaps some thing to do with MinGW on
> Vista?

Do you have this problem only with numpy ? I noticed those kind of
crashes (right at import time) on my server 2008, but I really could
not understand what was wrong - it crashes half of the time, and when
it does not, it works perfectly (I could run the test suite 10 times
without crash). Maybe some file-related failures and runtimes
incompatibilities. There are so many combinations on windows which can
influence this that I am perfectly fine with saying that Windows + AV
is not supported:)

David


From cournape at gmail.com  Sat Feb 28 14:00:10 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 1 Mar 2009 04:00:10 +0900
Subject: [Numpy-discussion] Call for testing: full blas/lapack builds of
	numpy on windows 64 bits
In-Reply-To: <a08e5f80902271346m5f46ae64q31a4fda26c6c4c50@mail.gmail.com>
References: <49A82386.9030208@ar.media.kyoto-u.ac.jp>
	<d05265cb0902271115k533eddf5h167605701c22c446@mail.gmail.com>
	<5b8d13220902271133pe80530bj36200e42a1aeafd1@mail.gmail.com>
	<d05265cb0902271308h41688f6dxfa7b6e9e11f10c84@mail.gmail.com>
	<a08e5f80902271346m5f46ae64q31a4fda26c6c4c50@mail.gmail.com>
Message-ID: <5b8d13220902281100k21f0f799yd82c4bc2b5c53471@mail.gmail.com>

On Sat, Feb 28, 2009 at 6:46 AM, Darren Dale <dsdale24 at gmail.com> wrote:

> VC++ 2005 redistributable (32 and 64 bit)
> VC++ 2008 express edition sp1
> VC++ 2008 redistributable x86
> MVS 2008 remote debugger light (x74)
> MS windows sdk for VS 2008 headers and libraries
> MS windows sdk for VS 2008 SP1 express tools for .Net
> MS windows sdk for VS 2008 SP1 express tools for win32
>
> Have you considered distributing the windows installer as an msi? I dont
> think you need to run those installers with admin privs on Vista.

The point was mainly to see if mingw could produce usable binaries.
Msi or not should not matter, normally (but we never know, this is
windows after all) - hopefully we will be able to produce MSI for the
release. Producing msi is a bit of a pain because of a stupid check
against development version which I don't know how to remove (if
distutils detects that the version is a dev version, it refuses to
build msi).

cheers,

David
>
> Darren
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From nadavh at visionsense.com  Sat Feb 28 14:19:41 2009
From: nadavh at visionsense.com (Nadav Horesh)
Date: Sat, 28 Feb 2009 21:19:41 +0200
Subject: [Numpy-discussion] Bilateral filter
References: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il><9457e7c80902130150h4dc8755eqf6097fc0d20b6522@mail.gmail.com><710F2847B0018641891D9A216027636029C432@ex3.envision.co.il><9457e7c80902141029w451160a2te6f7e6eb57db2d1e@mail.gmail.com>
	<2966A073-2846-4950-8A48-F170990503D8@yale.edu>
	<710F2847B0018641891D9A216027636029C450@ex3.envision.co.il>
Message-ID: <710F2847B0018641891D9A216027636029C453@ex3.envision.co.il>

A correction: The light distribution with two LEDs would be problematic.

  Nadav


-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? Nadav Horesh
????: ? 28-??????-09 20:53
??: Discussion of Numerical Python
????: RE: [Numpy-discussion] Bilateral filter
 
I could not reproduce your error. I am using latest official releases of numpy and cython on a linux box (do you use Mac?).  I am attaching the package I have on my PC, for the small chance it would help.

  Nadav.


-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? Zachary Pincus
????: ? 27-??????-09 22:26
??: Discussion of Numerical Python
????: Re: [Numpy-discussion] Bilateral filter
 
Hi all,

I just grabbed the latest bilateral filter from St?fan's repository,  
but I can't get it to work! I'm using a recent numpy SVN and the  
latest release of cython...

In [10]: bl = bilateral.bilateral(image, 2, 150)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call  
last)

/Users/zpincus/Documents/Research/Slack Lab/Experimental Data/Big Data/ 
2009-2-17/<ipython console> in <module>()

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
packages/bilateral/bilateral.pyc in bilateral(mat, xy_sig, z_sig,  
filter_size, mode)
      36     '''
      37
---> 38     filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size)
      39     size = filter_fcn.xy_size
      40     return generic_filter(mat, filter_fcn.cfilter, size=size,  
mode=mode)

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
packages/bilateral/bilateral_base.so in  
bilateral.bilateral_base.Bilat_fcn.__init__ (bilateral/ 
bilateral_base.c:590)()

NameError: np

I know very little about the internals of cython, so I can't figure  
this error out... it seems to be a problem with the 'cimport numpy as  
np' line in the pyx file, but beyond that I'm lost.


Let me know if I can provide any additional information,

Zach




On Feb 14, 2009, at 1:29 PM, St?fan van der Walt wrote:

> Hi Nadav
>
> 2009/2/13 Nadav Horesh <nadavh at visionsense.com>:
>> I tried the installer and it did not copy bilateral.py. I tried to  
>> improve it and the result is in the attached file.  I hope it would  
>> pass the mail filter, if not contact me directly to the email  
>> address below.
>
> Thanks!  I applied your changes and modified the setup.py to support
> in-place building.  Again available here:
>
> http://github.com/stefanv/bilateral/tree/master
>
> Cheers
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion





From nadavh at visionsense.com  Sat Feb 28 14:47:31 2009
From: nadavh at visionsense.com (Nadav Horesh)
Date: Sat, 28 Feb 2009 21:47:31 +0200
Subject: [Numpy-discussion] Bilateral filter
References: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il><9457e7c80902130150h4dc8755eqf6097fc0d20b6522@mail.gmail.com><710F2847B0018641891D9A216027636029C432@ex3.envision.co.il><9457e7c80902141029w451160a2te6f7e6eb57db2d1e@mail.gmail.com><2966A073-2846-4950-8A48-F170990503D8@yale.edu><710F2847B0018641891D9A216027636029C450@ex3.envision.co.il>
	<710F2847B0018641891D9A216027636029C453@ex3.envision.co.il>
Message-ID: <710F2847B0018641891D9A216027636029C454@ex3.envision.co.il>

Sorry, wrong addressing


-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? Nadav Horesh
????: ? 28-??????-09 21:19
??: Discussion of Numerical Python; Discussion of Numerical Python
????: Re: [Numpy-discussion] Bilateral filter
 
A correction: The light distribution with two LEDs would be problematic.

  Nadav


-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? Nadav Horesh
????: ? 28-??????-09 20:53
??: Discussion of Numerical Python
????: RE: [Numpy-discussion] Bilateral filter
 
I could not reproduce your error. I am using latest official releases of numpy and cython on a linux box (do you use Mac?).  I am attaching the package I have on my PC, for the small chance it would help.

  Nadav.


-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? Zachary Pincus
????: ? 27-??????-09 22:26
??: Discussion of Numerical Python
????: Re: [Numpy-discussion] Bilateral filter
 
Hi all,

I just grabbed the latest bilateral filter from St?fan's repository,  
but I can't get it to work! I'm using a recent numpy SVN and the  
latest release of cython...

In [10]: bl = bilateral.bilateral(image, 2, 150)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call  
last)

/Users/zpincus/Documents/Research/Slack Lab/Experimental Data/Big Data/ 
2009-2-17/<ipython console> in <module>()

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
packages/bilateral/bilateral.pyc in bilateral(mat, xy_sig, z_sig,  
filter_size, mode)
      36     '''
      37
---> 38     filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size)
      39     size = filter_fcn.xy_size
      40     return generic_filter(mat, filter_fcn.cfilter, size=size,  
mode=mode)

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
packages/bilateral/bilateral_base.so in  
bilateral.bilateral_base.Bilat_fcn.__init__ (bilateral/ 
bilateral_base.c:590)()

NameError: np

I know very little about the internals of cython, so I can't figure  
this error out... it seems to be a problem with the 'cimport numpy as  
np' line in the pyx file, but beyond that I'm lost.


Let me know if I can provide any additional information,

Zach




On Feb 14, 2009, at 1:29 PM, St?fan van der Walt wrote:

> Hi Nadav
>
> 2009/2/13 Nadav Horesh <nadavh at visionsense.com>:
>> I tried the installer and it did not copy bilateral.py. I tried to  
>> improve it and the result is in the attached file.  I hope it would  
>> pass the mail filter, if not contact me directly to the email  
>> address below.
>
> Thanks!  I applied your changes and modified the setup.py to support
> in-place building.  Again available here:
>
> http://github.com/stefanv/bilateral/tree/master
>
> Cheers
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From zachary.pincus at yale.edu  Sat Feb 28 17:18:17 2009
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Sat, 28 Feb 2009 17:18:17 -0500
Subject: [Numpy-discussion] Bilateral filter
In-Reply-To: <710F2847B0018641891D9A216027636029C450@ex3.envision.co.il>
References: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il><9457e7c80902130150h4dc8755eqf6097fc0d20b6522@mail.gmail.com><710F2847B0018641891D9A216027636029C432@ex3.envision.co.il><9457e7c80902141029w451160a2te6f7e6eb57db2d1e@mail.gmail.com>
	<2966A073-2846-4950-8A48-F170990503D8@yale.edu>
	<710F2847B0018641891D9A216027636029C450@ex3.envision.co.il>
Message-ID: <25CF9D14-91C8-4BDB-A824-7365EDBD6130@yale.edu>

Hi all,

So, I re-grabbed the latest bilateral code from the repository, and  
did the following:

[axlotl:~/Developer/Python/stefanv-bilateral] zpincus% python setup.py  
build_ext -i
[axlotl:~/Developer/Python/stefanv-bilateral] zpincus% ipython
In [1]: import numpy
In [2]: import bilateral
In [3]: bilateral.bilateral(numpy.ones((10,10)), 4, 4)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call  
last)

/Users/zpincus/Developer/Python/stefanv-bilateral/<ipython console> in  
<module>()

/Users/zpincus/Developer/Python/stefanv-bilateral/bilateral/ 
bilateral.pyc in bilateral(mat, xy_sig, z_sig, filter_size, mode)
      36     '''
      37
---> 38     filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size)
      39     size = filter_fcn.xy_size
      40     return generic_filter(mat, filter_fcn.cfilter, size=size,  
mode=mode)

/Users/zpincus/Developer/Python/stefanv-bilateral/bilateral/ 
bilateral_base.so in bilateral.bilateral_base.Bilat_fcn.__init__  
(bilateral/bilateral_base.c:590)()

NameError: np

In [4]: print numpy.version.version
1.3.0.dev6359
In [5]: import Cython.Compiler.Version
In [6]: print Cython.Compiler.Version.version
0.10.3

I'm using Python 2.5.2 on OS X 10.5.6. Oddly, when I run the cython  
tests, numpy_test fails to even compile because it can't find 'numpy/ 
arrayobject.h' -- I assume this is some problem with their test  
harness. However, I noticed that the numpy_test.pyx file does both:
'cimport numpy as np'
and
'import numpy as np'

so I added the second line to the bilateral_base.pyx file... re- 
running the above then gets a bit farther, but things choke again,  
with an "AttributeError: 'numpy.ndarray' object has no attribute  
'dimensions'", which comes from bilateral_base.c:1060, i.e.  
bilateral_base.pyx line 73.

This is perplexing. I wonder if the problem is using a dev numpy  
version, but a few-months-old cython. (Unfortunately, I don't have  
mercurial installed, so I can't easily grab the dev cython -- there's  
no simple way to download a mercurial repository over HTTP with wget  
or whatnot, which is annoying.)

Any thoughts? In the mean time, I guess I'll install mercurial so I  
can try the latest cython.

Zach


On Feb 28, 2009, at 1:53 PM, Nadav Horesh wrote:

> I could not reproduce your error. I am using latest official  
> releases of numpy and cython on a linux box (do you use Mac?).  I am  
> attaching the package I have on my PC, for the small chance it would  
> help.
>
>  Nadav.
>
>
> -----????? ??????-----
> ???: numpy-discussion-bounces at scipy.org ??? Zachary Pincus
> ????: ? 27-??????-09 22:26
> ??: Discussion of Numerical Python
> ????: Re: [Numpy-discussion] Bilateral filter
>
> Hi all,
>
> I just grabbed the latest bilateral filter from St?fan's repository,
> but I can't get it to work! I'm using a recent numpy SVN and the
> latest release of cython...
>
> In [10]: bl = bilateral.bilateral(image, 2, 150)
> ---------------------------------------------------------------------------
> NameError                                 Traceback (most recent call
> last)
>
> /Users/zpincus/Documents/Research/Slack Lab/Experimental Data/Big  
> Data/
> 2009-2-17/<ipython console> in <module>()
>
> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-
> packages/bilateral/bilateral.pyc in bilateral(mat, xy_sig, z_sig,
> filter_size, mode)
>      36     '''
>      37
> ---> 38     filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size)
>      39     size = filter_fcn.xy_size
>      40     return generic_filter(mat, filter_fcn.cfilter, size=size,
> mode=mode)
>
> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-
> packages/bilateral/bilateral_base.so in
> bilateral.bilateral_base.Bilat_fcn.__init__ (bilateral/
> bilateral_base.c:590)()
>
> NameError: np
>
> I know very little about the internals of cython, so I can't figure
> this error out... it seems to be a problem with the 'cimport numpy as
> np' line in the pyx file, but beyond that I'm lost.
>
>
> Let me know if I can provide any additional information,
>
> Zach
>
>
>
>
> On Feb 14, 2009, at 1:29 PM, St?fan van der Walt wrote:
>
>> Hi Nadav
>>
>> 2009/2/13 Nadav Horesh <nadavh at visionsense.com>:
>>> I tried the installer and it did not copy bilateral.py. I tried to
>>> improve it and the result is in the attached file.  I hope it would
>>> pass the mail filter, if not contact me directly to the email
>>> address below.
>>
>> Thanks!  I applied your changes and modified the setup.py to support
>> in-place building.  Again available here:
>>
>> http://github.com/stefanv/bilateral/tree/master
>>
>> Cheers
>> St?fan
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
> <winmail.dat>_______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From zachary.pincus at yale.edu  Sat Feb 28 17:37:25 2009
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Sat, 28 Feb 2009 17:37:25 -0500
Subject: [Numpy-discussion] Bilateral filter
In-Reply-To: <25CF9D14-91C8-4BDB-A824-7365EDBD6130@yale.edu>
References: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il><9457e7c80902130150h4dc8755eqf6097fc0d20b6522@mail.gmail.com><710F2847B0018641891D9A216027636029C432@ex3.envision.co.il><9457e7c80902141029w451160a2te6f7e6eb57db2d1e@mail.gmail.com>
	<2966A073-2846-4950-8A48-F170990503D8@yale.edu>
	<710F2847B0018641891D9A216027636029C450@ex3.envision.co.il>
	<25CF9D14-91C8-4BDB-A824-7365EDBD6130@yale.edu>
Message-ID: <AF44354D-E4CB-4768-922B-19ED3934A1B1@yale.edu>

Well, the latest cython doesn't help -- both errors still appear as  
below. (Also, the latest cython can't run the numpy tests either.) I'm  
befuddled.

Zach

On Feb 28, 2009, at 5:18 PM, Zachary Pincus wrote:

> Hi all,
>
> So, I re-grabbed the latest bilateral code from the repository, and
> did the following:
>
> [axlotl:~/Developer/Python/stefanv-bilateral] zpincus% python setup.py
> build_ext -i
> [axlotl:~/Developer/Python/stefanv-bilateral] zpincus% ipython
> In [1]: import numpy
> In [2]: import bilateral
> In [3]: bilateral.bilateral(numpy.ones((10,10)), 4, 4)
> ---------------------------------------------------------------------------
> NameError                                 Traceback (most recent call
> last)
>
> /Users/zpincus/Developer/Python/stefanv-bilateral/<ipython console> in
> <module>()
>
> /Users/zpincus/Developer/Python/stefanv-bilateral/bilateral/
> bilateral.pyc in bilateral(mat, xy_sig, z_sig, filter_size, mode)
>      36     '''
>      37
> ---> 38     filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size)
>      39     size = filter_fcn.xy_size
>      40     return generic_filter(mat, filter_fcn.cfilter, size=size,
> mode=mode)
>
> /Users/zpincus/Developer/Python/stefanv-bilateral/bilateral/
> bilateral_base.so in bilateral.bilateral_base.Bilat_fcn.__init__
> (bilateral/bilateral_base.c:590)()
>
> NameError: np
>
> In [4]: print numpy.version.version
> 1.3.0.dev6359
> In [5]: import Cython.Compiler.Version
> In [6]: print Cython.Compiler.Version.version
> 0.10.3
>
> I'm using Python 2.5.2 on OS X 10.5.6. Oddly, when I run the cython
> tests, numpy_test fails to even compile because it can't find 'numpy/
> arrayobject.h' -- I assume this is some problem with their test
> harness. However, I noticed that the numpy_test.pyx file does both:
> 'cimport numpy as np'
> and
> 'import numpy as np'
>
> so I added the second line to the bilateral_base.pyx file... re-
> running the above then gets a bit farther, but things choke again,
> with an "AttributeError: 'numpy.ndarray' object has no attribute
> 'dimensions'", which comes from bilateral_base.c:1060, i.e.
> bilateral_base.pyx line 73.
>
> This is perplexing. I wonder if the problem is using a dev numpy
> version, but a few-months-old cython. (Unfortunately, I don't have
> mercurial installed, so I can't easily grab the dev cython -- there's
> no simple way to download a mercurial repository over HTTP with wget
> or whatnot, which is annoying.)
>
> Any thoughts? In the mean time, I guess I'll install mercurial so I
> can try the latest cython.
>
> Zach
>
>
> On Feb 28, 2009, at 1:53 PM, Nadav Horesh wrote:
>
>> I could not reproduce your error. I am using latest official
>> releases of numpy and cython on a linux box (do you use Mac?).  I am
>> attaching the package I have on my PC, for the small chance it would
>> help.
>>
>> Nadav.
>>
>>
>> -----????? ??????-----
>> ???: numpy-discussion-bounces at scipy.org ??? Zachary Pincus
>> ????: ? 27-??????-09 22:26
>> ??: Discussion of Numerical Python
>> ????: Re: [Numpy-discussion] Bilateral filter
>>
>> Hi all,
>>
>> I just grabbed the latest bilateral filter from St?fan's repository,
>> but I can't get it to work! I'm using a recent numpy SVN and the
>> latest release of cython...
>>
>> In [10]: bl = bilateral.bilateral(image, 2, 150)
>> ---------------------------------------------------------------------------
>> NameError                                 Traceback (most recent call
>> last)
>>
>> /Users/zpincus/Documents/Research/Slack Lab/Experimental Data/Big
>> Data/
>> 2009-2-17/<ipython console> in <module>()
>>
>> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-
>> packages/bilateral/bilateral.pyc in bilateral(mat, xy_sig, z_sig,
>> filter_size, mode)
>>     36     '''
>>     37
>> ---> 38     filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size)
>>     39     size = filter_fcn.xy_size
>>     40     return generic_filter(mat, filter_fcn.cfilter, size=size,
>> mode=mode)
>>
>> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-
>> packages/bilateral/bilateral_base.so in
>> bilateral.bilateral_base.Bilat_fcn.__init__ (bilateral/
>> bilateral_base.c:590)()
>>
>> NameError: np
>>
>> I know very little about the internals of cython, so I can't figure
>> this error out... it seems to be a problem with the 'cimport numpy as
>> np' line in the pyx file, but beyond that I'm lost.
>>
>>
>> Let me know if I can provide any additional information,
>>
>> Zach
>>
>>
>>
>>
>> On Feb 14, 2009, at 1:29 PM, St?fan van der Walt wrote:
>>
>>> Hi Nadav
>>>
>>> 2009/2/13 Nadav Horesh <nadavh at visionsense.com>:
>>>> I tried the installer and it did not copy bilateral.py. I tried to
>>>> improve it and the result is in the attached file.  I hope it would
>>>> pass the mail filter, if not contact me directly to the email
>>>> address below.
>>>
>>> Thanks!  I applied your changes and modified the setup.py to support
>>> in-place building.  Again available here:
>>>
>>> http://github.com/stefanv/bilateral/tree/master
>>>
>>> Cheers
>>> St?fan
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Numpy-discussion at scipy.org
>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>> <winmail.dat>_______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From stefan at sun.ac.za  Sat Feb 28 18:15:04 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 1 Mar 2009 01:15:04 +0200
Subject: [Numpy-discussion] Bilateral filter
In-Reply-To: <AF44354D-E4CB-4768-922B-19ED3934A1B1@yale.edu>
References: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il>
	<9457e7c80902130150h4dc8755eqf6097fc0d20b6522@mail.gmail.com>
	<710F2847B0018641891D9A216027636029C432@ex3.envision.co.il>
	<9457e7c80902141029w451160a2te6f7e6eb57db2d1e@mail.gmail.com>
	<2966A073-2846-4950-8A48-F170990503D8@yale.edu>
	<710F2847B0018641891D9A216027636029C450@ex3.envision.co.il>
	<25CF9D14-91C8-4BDB-A824-7365EDBD6130@yale.edu>
	<AF44354D-E4CB-4768-922B-19ED3934A1B1@yale.edu>
Message-ID: <9457e7c80902281515n223156d9m6379b5dafe4a8352@mail.gmail.com>

Hey Zach

2009/3/1 Zachary Pincus <zachary.pincus at yale.edu>:
> Well, the latest cython doesn't help -- both errors still appear as
> below. (Also, the latest cython can't run the numpy tests either.) I'm
> befuddled.

That's pretty weird.  Did you remove the .so that was build as well as
any source files, before doing build_ext with the latest Cython?  Also
good to make sure that the latest Cython is, in fact, the one being
used.

Sorry, these are sanity checks, but I don't have anything better.

Cheers
St?fan


From nadavh at visionsense.com  Sat Feb 28 18:58:48 2009
From: nadavh at visionsense.com (Nadav Horesh)
Date: Sun, 1 Mar 2009 01:58:48 +0200
Subject: [Numpy-discussion] Bilateral filter
References: <710F2847B0018641891D9A216027636029C431@ex3.envision.co.il><9457e7c80902130150h4dc8755eqf6097fc0d20b6522@mail.gmail.com><710F2847B0018641891D9A216027636029C432@ex3.envision.co.il><9457e7c80902141029w451160a2te6f7e6eb57db2d1e@mail.gmail.com><2966A073-2846-4950-8A48-F170990503D8@yale.edu><710F2847B0018641891D9A216027636029C450@ex3.envision.co.il><25CF9D14-91C8-4BDB-A824-7365EDBD6130@yale.edu><AF44354D-E4CB-4768-922B-19ED3934A1B1@yale.edu>
	<9457e7c80902281515n223156d9m6379b5dafe4a8352@mail.gmail.com>
Message-ID: <710F2847B0018641891D9A216027636029C457@ex3.envision.co.il>

I tested again, and got no errors, with the following configuration:
cython 10.4
numpy 1.3.0dev6519 (the latest)
gcc 4.1.2
python 2.5.4
amd64 gentoo linux

  Nadav.

-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? St?fan van der Walt
????: ? 01-???-09 01:15
??: Discussion of Numerical Python
????: Re: [Numpy-discussion] Bilateral filter
 
Hey Zach

2009/3/1 Zachary Pincus <zachary.pincus at yale.edu>:
> Well, the latest cython doesn't help -- both errors still appear as
> below. (Also, the latest cython can't run the numpy tests either.) I'm
> befuddled.

That's pretty weird.  Did you remove the .so that was build as well as
any source files, before doing build_ext with the latest Cython?  Also
good to make sure that the latest Cython is, in fact, the one being
used.

Sorry, these are sanity checks, but I don't have anything better.

Cheers
St?fan
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 3652 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090301/33386a56/attachment.bin>

From bsouthey at gmail.com  Sat Feb 28 22:26:17 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Sat, 28 Feb 2009 21:26:17 -0600
Subject: [Numpy-discussion] Call for testing: full blas/lapack builds of
	numpy on windows 64 bits
In-Reply-To: <5b8d13220902281057g4a53ccb1i8fdbf37cc511116e@mail.gmail.com>
References: <49A82386.9030208@ar.media.kyoto-u.ac.jp>
	<d05265cb0902271115k533eddf5h167605701c22c446@mail.gmail.com>
	<5b8d13220902271133pe80530bj36200e42a1aeafd1@mail.gmail.com>
	<d05265cb0902271308h41688f6dxfa7b6e9e11f10c84@mail.gmail.com>
	<49A862D7.60009@gmail.com>
	<5b8d13220902281057g4a53ccb1i8fdbf37cc511116e@mail.gmail.com>
Message-ID: <bbcd77d00902281926u3b18eb8bg137619a3b8642749@mail.gmail.com>

On Sat, Feb 28, 2009 at 12:57 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Sat, Feb 28, 2009 at 7:01 AM, Bruce Southey <bsouthey at gmail.com> wrote:
>
>> Hi,
>> After much frustration and disabling my antivirus software (McAfee), I
>> was able to get the 'import numpy' to work and tests to pass as
>> expected. Of course after turning it back on and restarting the IDLE gui
>> a few times, the 'import numpy' crashes. This also happens on the
>> command line.
>
> Hm, let's assume this is a problem of the anti virus software, and not
> from mingw :) One way to test this would be to see if a binary built
> with VS 2008 would cause the same trouble.

Or just a Vista bug.

A possible option could be using wine64
(http://wiki.winehq.org/Wine64) . But is probably more work and if
even if it did work it may not be informative.

>>
>> This is clearly related to the antivirus software because enabling or
>> disabling the 'on-access scan' is usually sufficient to failure import
>> failure or success, respectively. Perhaps some thing to do with MinGW on
>> Vista?
>
> Do you have this problem only with numpy ? I noticed those kind of
> crashes (right at import time) on my server 2008, but I really could
> not understand what was wrong - it crashes half of the time, and when
> it does not, it works perfectly (I could run the test suite 10 times
> without crash). Maybe some file-related failures and runtimes
> incompatibilities. There are so many combinations on windows which can
> influence this that I am perfectly fine with saying that Windows + AV
> is not supported:)
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

I virtually avoid windows for work so I can not answer your question.
I knew about the issues from another person trying to compile code
using WinGW on another almost identical 64 bit Vista system. Even the
related thread seemed to solutions that only worked for some people
and I do not know if a solution was found in that case.

Anyhow, I do agree that having Python 2.6 support is more important
than running the anti-virus software.

Bruce