From fperez.net at gmail.com  Thu Feb  1 01:03:50 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 31 Jan 2007 23:03:50 -0700
Subject: [Numpy-discussion] [SciPy-user] What it the best way to install
	on a Mac?
In-Reply-To: <F662C4D6-AE39-4957-8D93-F86679052925@scu.edu>
References: <723eb6930701310617v42a4a098s5582ef6dff01e8d4@mail.gmail.com>
	<45C0C5F5.2000100@unc.edu>
	<db6b5ecc0701310947o3d9e8912lc72fd2fa300854be@mail.gmail.com>
	<20070131223800.GA20121@avicenna.cc.columbia.edu>
	<db6b5ecc0701311455m21462fbes1355532352f0421a@mail.gmail.com>
	<F662C4D6-AE39-4957-8D93-F86679052925@scu.edu>
Message-ID: <db6b5ecc0701312203p1cc4659asdde3004dd533d2e4@mail.gmail.com>

On 1/31/07, Sanjiv Das <srdas at scu.edu> wrote:
> OK - will give that a shot as well. Its a good suggestion!
> cheers

Great.  And it looks like the bulk of that work has already been done
for you, as I found out today thanks to the good and trusty Google
Alerts service:

http://mohinish.blogspot.com/2007/01/python-for-matlab-users-on-mac.html

Feel free to open up a page on the ipython wiki:

http://ipython.scipy.org/moin

Just make a login for yourself and type away!

Cheers,

f


From fperez.net at gmail.com  Thu Feb  1 01:05:51 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 31 Jan 2007 23:05:51 -0700
Subject: [Numpy-discussion] [SciPy-user] What it the best way to install
	on a Mac?
In-Reply-To: <db6b5ecc0701312203p1cc4659asdde3004dd533d2e4@mail.gmail.com>
References: <723eb6930701310617v42a4a098s5582ef6dff01e8d4@mail.gmail.com>
	<45C0C5F5.2000100@unc.edu>
	<db6b5ecc0701310947o3d9e8912lc72fd2fa300854be@mail.gmail.com>
	<20070131223800.GA20121@avicenna.cc.columbia.edu>
	<db6b5ecc0701311455m21462fbes1355532352f0421a@mail.gmail.com>
	<F662C4D6-AE39-4957-8D93-F86679052925@scu.edu>
	<db6b5ecc0701312203p1cc4659asdde3004dd533d2e4@mail.gmail.com>
Message-ID: <db6b5ecc0701312205i475da518tf3638733126dc4b7@mail.gmail.com>

On 1/31/07, Fernando Perez <fperez.net at gmail.com> wrote:
> On 1/31/07, Sanjiv Das <srdas at scu.edu> wrote:
> > OK - will give that a shot as well. Its a good suggestion!
> > cheers

And I should add: thanks!  Recent discussions on this list indicate
there's a real need for this information, so a complete and step by
step summary of this info in a purely tutorial manner may well help
many.

Having it in a publicly editable format will ensure it can stay up to
date as the problem evolves.

Cheers,

f


From zpincus at stanford.edu  Thu Feb  1 03:28:17 2007
From: zpincus at stanford.edu (Zachary Pincus)
Date: Thu, 1 Feb 2007 00:28:17 -0800
Subject: [Numpy-discussion] array copy-to-self and views
Message-ID: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>

Hello folks,

I recently was trying to write code to modify an array in-place (so  
as not to invalidate any references to that array) via the standard  
python idiom for lists, e.g.:

a[:] = numpy.flipud(a)

Now, flipud returns a view on 'a', so assigning that to 'a[:]'  
provides pretty strange results as the buffer that is being read (the  
view) is simultaneously modified. Here is an example:

In [2]: a = numpy.arange(10).reshape((5,2))
In [3]: a
Out[3]:
array([[0, 1],
        [2, 3],
        [4, 5],
        [6, 7],
        [8, 9]])
In [4]: numpy.flipud(a)
Out[4]:
array([[8, 9],
        [6, 7],
        [4, 5],
        [2, 3],
        [0, 1]])
In [5]: a[:] = numpy.flipud(a)
In [6]: a
Out[6]:
array([[8, 9],
        [6, 7],
        [4, 5],
        [6, 7],
        [8, 9]])

A question, then: Does this represent a bug? Or perhaps there is a  
better idiom for modifying an array in-place than 'a[:] = ...'? Or is  
incumbent on the user to ensure that any time an array is directly  
modified, that the modifying array is not a view of the original array?

Thanks for any thoughts,

Zach Pincus

Program in Biomedical Informatics and Department of Biochemistry
Stanford University School of Medicine



From peridot.faceted at gmail.com  Thu Feb  1 04:07:29 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 1 Feb 2007 04:07:29 -0500
Subject: [Numpy-discussion] array copy-to-self and views
In-Reply-To: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>
References: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>
Message-ID: <ce557a360702010107g1d1dec04q78e309cb3ef7c01f@mail.gmail.com>

On 01/02/07, Zachary Pincus <zpincus at stanford.edu> wrote:

> I recently was trying to write code to modify an array in-place (so
> as not to invalidate any references to that array) via the standard
> python idiom for lists, e.g.:

You can do this, but if your concern is invalidating references, you
might want to think about rearranging your application so you can just
return "new" arrays (that may share elements), if that is possible.

> a[:] = numpy.flipud(a)
>
> Now, flipud returns a view on 'a', so assigning that to 'a[:]'
> provides pretty strange results as the buffer that is being read (the
> view) is simultaneously modified. Here is an example:

> A question, then: Does this represent a bug? Or perhaps there is a
> better idiom for modifying an array in-place than 'a[:] = ...'? Or is
> incumbent on the user to ensure that any time an array is directly
> modified, that the modifying array is not a view of the original array?

It's the user's job to keep them separate. Sorry. If you're worried -
say if it's an array you don't have much control over (so it might
share elements without you knowing), you can either return a new
array, or if you must modify it in place, copy the right-hand side
before using it (a[:]=flipud(a).copy()).

It would in principle be possible for numpy to provide a function that
tells you if two arrays might share data (simply compare the pointer
to the malloc()ed storage and ignore strides and offset; a bit
conservative but probably Good Enough, though a bit more cleverness
should let one get the Right Answer efficiently).

Anne M. Archibald


From emsellem at obs.univ-lyon1.fr  Thu Feb  1 07:24:56 2007
From: emsellem at obs.univ-lyon1.fr (Eric Emsellem)
Date: Thu, 01 Feb 2007 13:24:56 +0100
Subject: [Numpy-discussion] problem with installation of numpy: undefined
	symbols
Message-ID: <45C1DC18.6010200@obs.univ-lyon1.fr>

Hi,

after trying to solve an installation problem with scipy, I had to
reinstall everything from scratch, and so I now turned back to numpy the
installation of which does not work for me (which may in fact explain
the pb I had with scipy).

To be clear on what I do:

- I install blas first, and create a libfblas.a which I copy under
/usr/local/lib
(using a g77 -fno-second-underscore -O2 -c *.f, ar r libfblas.a *.o,
ranlib libfblas.a)

- I then define BLAS environment variable accordingly
(/usr/local/lib/libfblas.a)
- I compile lapack-3.1.0, using "make lapacklib"
- I then use a precompiled ATLAS Linux version P4SSE2 (I tried compiling
it but it does have the same result)
- I copy all ATLAS .a and .h in /usr/local/lib/atlas, and define the
ATLAS variable accordingly
- I then merge the  *.o to have an extended liblapack.a by doing the usual:
         cd /usr/local/lib/atlas
         cp liblapack.a liblapack.a_ATLAS
         mkdir tmp
         cd tmp
         ar x ../liblapack.a_ATLAS
         cp
/soft/python/tar/Science_Packages/lapack-3.1.0/lapack_LINUX.a ../liblapack.a
         ar r ../liblapack.a *.o

- I finally get the svn version of numpy, and do the "python setup.py
install" (no site.cfg ! but environment variables defined as mentioned
above)

- I go into ipython, and try:


        import numpy

and I get :

exceptions.ImportError                               Traceback (most
recent call last)
................
---> 40     import linalg
................
ImportError:
/usr/local/lib/python2.4/site-packages/numpy/linalg/lapack_lite.so:
undefined symbol: ATL_cGetNB

I have tried many ways to avoid this problem but does not manage to
solve it.

Any help is welcome there. (and sorry for those who already saw my
numerous posts on scipy forum: I am trying to get to the heart of the pb...)

thanks in advance!
Eric


From robert.kern at gmail.com  Thu Feb  1 11:08:34 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 01 Feb 2007 10:08:34 -0600
Subject: [Numpy-discussion] problem with installation of numpy:
 undefined symbols
In-Reply-To: <45C1DC18.6010200@obs.univ-lyon1.fr>
References: <45C1DC18.6010200@obs.univ-lyon1.fr>
Message-ID: <45C21082.8060406@gmail.com>

Eric Emsellem wrote:

> - I finally get the svn version of numpy, and do the "python setup.py
> install" (no site.cfg ! but environment variables defined as mentioned
> above)

Show us the output of

  $ cd ~/src/numpy  # or whereever
  $ python setup.py config

Most likely, you are having the same problem you had with scipy. You will
probably need to make a site.cfg with the correct information about ATLAS.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Thu Feb  1 12:38:12 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 01 Feb 2007 09:38:12 -0800
Subject: [Numpy-discussion] array copy-to-self and views
In-Reply-To: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>
References: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>
Message-ID: <45C22584.4060904@noaa.gov>

Zachary Pincus wrote:
> Hello folks,
> 
> I recently was trying to write code to modify an array in-place (so  
> as not to invalidate any references to that array)

I'm not sure what this means exactly.

> via the standard  
> python idiom for lists, e.g.:
> 
> a[:] = numpy.flipud(a)
> 
> Now, flipud returns a view on 'a', so assigning that to 'a[:]'  
> provides pretty strange results as the buffer that is being read (the  
> view) is simultaneously modified.

yes, weird. So why not just:

a = numpy.flipud(a)

Since flipud returns a view, the new "a" will still be using the same 
data array. Does this satisfy your need above?

You've created a new numpy array object, but that was created by flipud 
anyway, so there is no performance loss.

It's too bad that to do this you need to know that flipud created a 
view, rather than a copy of the data, as if it were a copy, you would 
need to do the a[:] trick to make sure a kept the same data, but that's 
the price we pay for the flexibility and power of numpy -- the 
alternative is to have EVERYTHING create a copy, but there were be a 
substantial performance hit for that.

NOTE: the docstring doesn't make it clear that a view is created:

 >>> help(numpy.flipud)
Help on function flipud in module numpy.lib.twodim_base:

flipud(m)
     returns an array with the columns preserved and rows flipped in
     the up/down direction.  Works on the first dimension of m.

NOTE2: Maybe these kinds of functions should have an optional flag that 
specified whether you want a view or a copy -- I'd have expected a copy 
in this case!

QUESTION:
How do you tell if two arrays are views on the same data: is checking if 
they have the same .base reliable?

 >>> a = numpy.array((1,2,3,4))
 >>> b = a.view()
 >>> a.base is b.base
False

No, I guess not. Maybe .base should return self if it's the originator 
of the data.

Is there a reliable way? I usually just test by changing a value in one 
to see if it changes in the other, but that's one heck of kludge!

 >>> a.__array_interface__['data'][0] == b.__array_interface__['data'][0]
True

seems to work, but that's pretty ugly!

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From zpincus at stanford.edu  Thu Feb  1 13:52:13 2007
From: zpincus at stanford.edu (Zachary Pincus)
Date: Thu, 1 Feb 2007 10:52:13 -0800
Subject: [Numpy-discussion] array copy-to-self and views
In-Reply-To: <45C22584.4060904@noaa.gov>
References: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>
	<45C22584.4060904@noaa.gov>
Message-ID: <4A126763-2364-41AB-AD1C-ACBB5CE265FF@stanford.edu>

> Zachary Pincus wrote:
>> Hello folks,
>>
>> I recently was trying to write code to modify an array in-place (so
>> as not to invalidate any references to that array)
>
> I'm not sure what this means exactly.

Say one wants to keep two different variables referencing a single in- 
memory list, as so:
a = [1,2,3]
b = a
Now, if 'b' and 'a' go to live in different places (different class  
instances or whatever) but we want 'b' and 'a' to always refer to the  
same in-memory object, so that 'id(a) == id(b)', we need to make sure  
to not assign a brand new list to either one.

That is, if we do something like 'a = [i + 1 for i in a]' then 'id 
(a) != id(b)'. However, we can do 'a[:] = [i + 1 for i in a]' to  
modify a in-place. This is not super-common, but it's also not an  
uncommon python idiom.

I was in my email simply pointing out that na?vely translating that  
idiom to the numpy case can cause unexpected behavior in the case of  
views.

I think that this is is unquestionably a bug -- isn't the point of  
views that the user shouldn't need to care if a particular array  
object is a view or not? Given the lack of methods to query whether  
an array is a view, or what it might be a view on, this seems like a  
reasonable perspective... I mean, if certain operations produce  
completely different results when one of the operands is a view, that  
*seems* like a bug. It might not be worth fixing, but I can't see how  
that behavior would be considered a feature.

However, I do think there's a legitimate question about whether it  
would be worth fixing -- there could be a lot of complicated checks  
to catch these kind of corner cases.

>> via the standard
>> python idiom for lists, e.g.:
>>
>> a[:] = numpy.flipud(a)
>>
>> Now, flipud returns a view on 'a', so assigning that to 'a[:]'
>> provides pretty strange results as the buffer that is being read (the
>> view) is simultaneously modified.
>
> yes, weird. So why not just:
>
> a = numpy.flipud(a)
>
> Since flipud returns a view, the new "a" will still be using the same
> data array. Does this satisfy your need above?

Nope -- though 'a' and 'numpy.flipud(a)' share the same data, the  
actual ndarray instances are different. This means that any other  
references to the 'a' array (made via 'b = a' or whatever) now refer  
to the old 'a', not the flipped one.

The only other option for sharing arrays is to encapsulate them as  
attributes of *another* object, which itself won't change. That seems  
a bit clumsy.

> It's too bad that to do this you need to know that flipud created a
> view, rather than a copy of the data, as if it were a copy, you would
> need to do the a[:] trick to make sure a kept the same data, but  
> that's
> the price we pay for the flexibility and power of numpy -- the
> alternative is to have EVERYTHING create a copy, but there were be a
> substantial performance hit for that.

Well, Anne's email suggests another alternative -- each time a view  
is created, keep track of the original array from whence it came, and  
then only make a copy when collisions like the above would take place.

And actually, I suspect that views already need to keep a reference  
to their original array in order to keep that array from being  
deleted before the view is. But I don't know the guts of numpy well  
enough to say for sure.

> NOTE: the docstring doesn't make it clear that a view is created:
>
>>>> help(numpy.flipud)
> Help on function flipud in module numpy.lib.twodim_base:
>
> flipud(m)
>      returns an array with the columns preserved and rows flipped in
>      the up/down direction.  Works on the first dimension of m.
>
> NOTE2: Maybe these kinds of functions should have an optional flag  
> that
> specified whether you want a view or a copy -- I'd have expected a  
> copy
> in this case!

Well, it seems like in most cases one does not need to care whether  
one is looking at a view or an array. The only time that comes to  
mind is when you're attempting to modify the array in-place, e.g.
a[<something>] = <something else>

Even if the maybe-bug above were easily fixable (again, not sure  
about that), you might *still* want to be able to figure out if a  
were a view before such a modification. Whether this needs a runtime  
'is_view' method, or just consistent documentation about what returns  
a view, isn't clear to me. Certainly the latter couldn't hurt.

> QUESTION:
> How do you tell if two arrays are views on the same data: is  
> checking if
> they have the same .base reliable?
>
>>>> a = numpy.array((1,2,3,4))
>>>> b = a.view()
>>>> a.base is b.base
> False
>
> No, I guess not. Maybe .base should return self if it's the originator
> of the data.
>
> Is there a reliable way? I usually just test by changing a value in  
> one
> to see if it changes in the other, but that's one heck of kludge!
>
>>>> a.__array_interface__['data'][0] == b.__array_interface__['data'] 
>>>> [0]
> True
>
> seems to work, but that's pretty ugly!

Good question. As I mentioned above, I assume that this information  
is tracked internally to prevent the 'original' array data from being  
deleted before any views have; however I really don't know how it is  
exposed.

Zach

From tim.hochberg at ieee.org  Thu Feb  1 14:24:04 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Thu, 1 Feb 2007 12:24:04 -0700
Subject: [Numpy-discussion] array copy-to-self and views
In-Reply-To: <4A126763-2364-41AB-AD1C-ACBB5CE265FF@stanford.edu>
References: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>
	<45C22584.4060904@noaa.gov>
	<4A126763-2364-41AB-AD1C-ACBB5CE265FF@stanford.edu>
Message-ID: <e4412d6b0702011124m172dc55fsbde0925863219c2d@mail.gmail.com>

On 2/1/07, Zachary Pincus <zpincus at stanford.edu> wrote:

[CHOP]

I think that this is is unquestionably a bug


It's not a bug. It's a design decision. It has certain consequences. Many
good, some bad and some that just take some getting used to.


-- isn't the point of
> views that the user shouldn't need to care if a particular array
> object is a view or not?


As you state elsewhere, the issue isn't whether a given object is a view per
se, it's whether the objects that you are operating on refer to the same
block of memory. They could both be views, even of the same object, and as
long as they're disjoint, it's not a problem.

Given the lack of methods to query whether
> an array is a view, or what it might be a view on, this seems like a
> reasonable perspective... I mean, if certain operations produce
> completely different results when one of the operands is a view, that
> *seems* like a bug. It might not be worth fixing, but I can't see how
> that behavior would be considered a feature.


View semantics are a feature. A powerful and sometime dangerous feature.
Sometimes the consequences of these semantics can bite people, but that
doesn't make them a bug.

 [CHOP]

>
>
> Good question. As I mentioned above, I assume that this information
> is tracked internally to prevent the 'original' array data from being
> deleted before any views have; however I really don't know how it is
> exposed.


I believe that a reference is held to the original array, so the array
itself won't be deleted even if all of the references to it go away. The
details may be different, but that's the gist of it. Even ifyou could access
this, it wouldn't really tell you anything useful since two slices could
refer to pieces of the original chunk of data, yet still be disjoint.

If you wanted to be able to figure this out, probably the thing to do is
just to actually look at the block of data occupied by each array and see if
they overlap. I think you could even do this without resorting to C by using
the array interface.

However, I'd like to repeat what my doctor said as a kid when I complained
that "it hurts when I do this":
    "Don't do that!" -- Some Radom Doctor
In other words, I think you'd be better off restructuring your code so that
this isn't an issue. I've been using Numeric/numarray/numpy for over ten
years now this has never been a significant issue for me.


-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070201/5f5a92e7/attachment.html>

From oliphant at ee.byu.edu  Thu Feb  1 14:30:33 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu, 01 Feb 2007 12:30:33 -0700
Subject: [Numpy-discussion] array copy-to-self and views
In-Reply-To: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>
References: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>
Message-ID: <45C23FD9.6010108@ee.byu.edu>

Zachary Pincus wrote:

>Hello folks,
>
>I recently was trying to write code to modify an array in-place (so  
>as not to invalidate any references to that array) via the standard  
>python idiom for lists, e.g.:
>
>a[:] = numpy.flipud(a)
>
>Now, flipud returns a view on 'a', so assigning that to 'a[:]'  
>provides pretty strange results as the buffer that is being read (the  
>view) is simultaneously modified. Here is an example:
>  
>

This is a known feature of the "view" concept.   It has been present in 
Numeric from the beginning. Performing operations in-place using a view 
always gives "hard-to-predict" results.  It depends completely on how 
the algorithms are implemented.  

Knowing that numpy.flipud(a)  is just a different way to write 
a[::-1,...]  which works for any nested-sequence, helps you realize that 
if a is already an array, then it returns a reversed view, but when 
copied back into itself creates the results you obtained but might not 
have bee expecting.
 
You can understand the essence of what is happening with a simpler example:

a = arange(10)
a[:] = a[::-1]

What is a?

It is easy to see the answer when you realize that the code is doing the 
equivalent of

a[0] = a[9]
a[1] = a[8] 
a[2] = a[7] 
a[3] = a[6] 
a[4] = a[5]
a[5] = a[4]
a[6] = a[3]
a[7] = a[2]
a[8] = a[1]
a[9] = a[0]

Notice that the final 5 lines are completely redundant, so really all 
that is happening is

a[:5] = a[5:][::-1]

There was an explicit warning of the oddities of this construct in the 
original Numeric documentation.

Better documentation of the flipud function to indicate that it returns 
a view is definitely desireable.  In fact, all functions that return 
views should be clear about this in the docstring.

In addition, all users of "in-place" functionality of NumPy must be 
aware of the view concept and realize that you could be modifying the 
array you are using.

This came up before when somebody asked how to perform a "diff" in place 
and I was careful to make sure and not change the input array before it 
was used.

>A question, then: Does this represent a bug? Or perhaps there is a  
>better idiom for modifying an array in-place than 'a[:] = ...'? Or is  
>incumbent on the user to ensure that any time an array is directly  
>modified, that the modifying array is not a view of the original array?
>  
>
Yes, it is and has always been incumbent on the user to ensure that any 
time an array is directly modified in-place that the modifying array is 
not a "view" of the original array.

Good example...

-Travis




From Louis.Wicker at noaa.gov  Thu Feb  1 14:33:23 2007
From: Louis.Wicker at noaa.gov (Louis Wicker)
Date: Thu, 01 Feb 2007 13:33:23 -0600
Subject: [Numpy-discussion] large memory address space on Mac OS X (intel)
Message-ID: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>

Dear list:

I cannot seem to figure how to create arrays > 2 GB on a Mac Pro  
(using Intel chip and Tiger, 4.8).  I have hand compiled both Python  
2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB.  I also  
run out of space if I try and 3-6 several arrays of 1000 mb or so  
(the mem-alloc failure does not seem consistent, depends on whether I  
am creating them with a "numpy.ones()" call, or creating them on the  
fly by doing math with the other arrays "e.g., c  = 4.3*a + 3.1*b").

Is this a numpy issue, or a Python 2.5 issue for the Mac?  I have  
tried this on the SGI Altix, and this works fine.

If there is a compile flag to turn on 64 bit support in the Mac  
compile, I would be glad to find out about it.  Or do I have to wait  
for Leopard?

Thanks.

Lou Wicker

------------------------------------------------------------------------ 
----
| Dr. Louis J. Wicker
| NSSL/WRDD
| National Weather Center
| 120 David L. Boren Boulevard, Norman, OK 73072-7323
|
| E-mail:   Louis.Wicker at noaa.gov
| HTTP:  www.nssl.noaa.gov/~lwicker
| Phone:    (405) 325-6340
| Fax:        (405) 325-6780
|
| "Programming is not just creating strings of instructions
| for a computer to execute.  It's also 'literary' in that you
| are trying to communicate a program structure to
| other humans reading the code." - Paul Rubin
|
|"Real efficiency comes from elegant solutions, not optimized programs.
| Optimization is always just a few correctness-preserving  
transformations
| away." - Jonathan Sobel
------------------------------------------------------------------------ 
----
|
| "The contents  of this message are mine personally and
| do not reflect any position of  the Government or NOAA."
|
------------------------------------------------------------------------ 
----




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070201/8e427525/attachment.html>

From svetosch at gmx.net  Thu Feb  1 14:40:02 2007
From: svetosch at gmx.net (Sven Schreiber)
Date: Thu, 01 Feb 2007 20:40:02 +0100
Subject: [Numpy-discussion] Cutting 1.0.2 release
In-Reply-To: <45BE7E57.4000105@ee.byu.edu>
References: <45BE7E57.4000105@ee.byu.edu>
Message-ID: <45C24212.4060203@gmx.net>

Travis Oliphant schrieb:
> I think it's time for the 1.0.2 release of NumPy.
> 
> What outstanding issues need to be resolved before we do it?  
> 

Hi,
I just used real_if_close for the first time, and promptly discovered
that it turns matrix input into array output:

>>> import numpy as n
>>> n.__version__
'1.0.1'
>>> n.real_if_close(n.mat(1))
array([[1]])

Maybe it's something for 1.0.2, or should I file a ticket?

I could also do another round of systematic searches for other functions
that still behave like this, but I'm afraid not before 1.0.2 (if that
happens this weekend).

Thanks,
Sven


From oliphant at ee.byu.edu  Thu Feb  1 14:41:38 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu, 01 Feb 2007 12:41:38 -0700
Subject: [Numpy-discussion] large memory address space on Mac OS X
	(intel)
In-Reply-To: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>
References: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>
Message-ID: <45C24272.4010402@ee.byu.edu>

Louis Wicker wrote:

> Dear list:
>
> I cannot seem to figure how to create arrays > 2 GB on a Mac Pro 
> (using Intel chip and Tiger, 4.8).  I have hand compiled both Python 
> 2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB.  I also 
> run out of space if I try and 3-6 several arrays of 1000 mb or so (the 
> mem-alloc failure does not seem consistent, depends on whether I am 
> creating them with a "numpy.ones()" call, or creating them on the fly 
> by doing math with the other arrays "e.g., c  = 4.3*a + 3.1*b").
>
> Is this a numpy issue, or a Python 2.5 issue for the Mac?  I have 
> tried this on the SGI Altix, and this works fine.

It must be a malloc issue.  NumPy uses the system malloc to construct 
arrays.  It just reports errors back to you if it can't.

I don't think the Mac Pro uses a 64-bit chip, does it?

-Travis



From robert.kern at gmail.com  Thu Feb  1 14:48:08 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 01 Feb 2007 13:48:08 -0600
Subject: [Numpy-discussion] large memory address space on Mac OS
	X	(intel)
In-Reply-To: <45C24272.4010402@ee.byu.edu>
References: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>
	<45C24272.4010402@ee.byu.edu>
Message-ID: <45C243F8.3080306@gmail.com>

Travis Oliphant wrote:
> Louis Wicker wrote:
> 
>> Dear list:
>>
>> I cannot seem to figure how to create arrays > 2 GB on a Mac Pro 
>> (using Intel chip and Tiger, 4.8).  I have hand compiled both Python 
>> 2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB.  I also 
>> run out of space if I try and 3-6 several arrays of 1000 mb or so (the 
>> mem-alloc failure does not seem consistent, depends on whether I am 
>> creating them with a "numpy.ones()" call, or creating them on the fly 
>> by doing math with the other arrays "e.g., c  = 4.3*a + 3.1*b").
>>
>> Is this a numpy issue, or a Python 2.5 issue for the Mac?  I have 
>> tried this on the SGI Altix, and this works fine.
> 
> It must be a malloc issue.  NumPy uses the system malloc to construct 
> arrays.  It just reports errors back to you if it can't.
> 
> I don't think the Mac Pro uses a 64-bit chip, does it?

Intel Core 2 Duo's are 64-bit, but the OS and runtime libraries are not. None of
the Python distributions for OS X are compiled for 64-bit support, either.

When Leopard comes out, there will be better 64-bit support in the OS, and
Python will follow. Until then, Python on OS X is 32-bit.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From Louis.Wicker at noaa.gov  Thu Feb  1 14:48:16 2007
From: Louis.Wicker at noaa.gov (Louis Wicker)
Date: Thu, 01 Feb 2007 13:48:16 -0600
Subject: [Numpy-discussion] large memory address space on Mac OS
	X	(intel)
In-Reply-To: <45C24272.4010402@ee.byu.edu>
References: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>
	<45C24272.4010402@ee.byu.edu>
Message-ID: <AD2A4509-3EDA-4C7B-B5D2-B62DFA8231AE@noaa.gov>

Travis:

yes it does.  Its the Woodcrest server chip which supports 32 and 64  
bit operations.  For example the new Intel Fortran compiler can grab  
more than 2 GB of memory (its a beta10 version).  I think gcc 4.x can  
as well.

However, Tiger (OS X 10.4.x) is not completely 64 bit compliant -  
Leopard is supposed to be pretty darn close.

Is there a numpy flag I could try for compilation....

Lou


On Feb 1, 2007, at 1:41 PM, Travis Oliphant wrote:

> Louis Wicker wrote:
>
>> Dear list:
>>
>> I cannot seem to figure how to create arrays > 2 GB on a Mac Pro
>> (using Intel chip and Tiger, 4.8).  I have hand compiled both Python
>> 2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB.  I also
>> run out of space if I try and 3-6 several arrays of 1000 mb or so  
>> (the
>> mem-alloc failure does not seem consistent, depends on whether I am
>> creating them with a "numpy.ones()" call, or creating them on the fly
>> by doing math with the other arrays "e.g., c  = 4.3*a + 3.1*b").
>>
>> Is this a numpy issue, or a Python 2.5 issue for the Mac?  I have
>> tried this on the SGI Altix, and this works fine.
>
> It must be a malloc issue.  NumPy uses the system malloc to construct
> arrays.  It just reports errors back to you if it can't.
>
> I don't think the Mac Pro uses a 64-bit chip, does it?
>
> -Travis
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

------------------------------------------------------------------------ 
----
| Dr. Louis J. Wicker
| NSSL/WRDD
| National Weather Center
| 120 David L. Boren Boulevard, Norman, OK 73072-7323
|
| E-mail:   Louis.Wicker at noaa.gov
| HTTP:  www.nssl.noaa.gov/~lwicker
| Phone:    (405) 325-6340
| Fax:        (405) 325-6780
|
| "Programming is not just creating strings of instructions
| for a computer to execute.  It's also 'literary' in that you
| are trying to communicate a program structure to
| other humans reading the code." - Paul Rubin
|
|"Real efficiency comes from elegant solutions, not optimized programs.
| Optimization is always just a few correctness-preserving  
transformations
| away." - Jonathan Sobel
------------------------------------------------------------------------ 
----
|
| "The contents  of this message are mine personally and
| do not reflect any position of  the Government or NOAA."
|
------------------------------------------------------------------------ 
----




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070201/0ec8c935/attachment.html>

From Louis.Wicker at noaa.gov  Thu Feb  1 14:50:04 2007
From: Louis.Wicker at noaa.gov (Louis Wicker)
Date: Thu, 01 Feb 2007 13:50:04 -0600
Subject: [Numpy-discussion] large memory address space on Mac OS
	X	(intel)
In-Reply-To: <45C24272.4010402@ee.byu.edu>
References: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>
	<45C24272.4010402@ee.byu.edu>
Message-ID: <665E5E7D-1F12-4A52-8FB8-CAC043041EF0@noaa.gov>

Travis:

quick follow up:  Mac Pro's currently have the dual-core 5100 Xeon  
(two processors, two cores each), the 5300 Xeon's (quad-core) are  
coming in a few weeks, we think.

Lou

On Feb 1, 2007, at 1:41 PM, Travis Oliphant wrote:

> Louis Wicker wrote:
>
>> Dear list:
>>
>> I cannot seem to figure how to create arrays > 2 GB on a Mac Pro
>> (using Intel chip and Tiger, 4.8).  I have hand compiled both Python
>> 2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB.  I also
>> run out of space if I try and 3-6 several arrays of 1000 mb or so  
>> (the
>> mem-alloc failure does not seem consistent, depends on whether I am
>> creating them with a "numpy.ones()" call, or creating them on the fly
>> by doing math with the other arrays "e.g., c  = 4.3*a + 3.1*b").
>>
>> Is this a numpy issue, or a Python 2.5 issue for the Mac?  I have
>> tried this on the SGI Altix, and this works fine.
>
> It must be a malloc issue.  NumPy uses the system malloc to construct
> arrays.  It just reports errors back to you if it can't.
>
> I don't think the Mac Pro uses a 64-bit chip, does it?
>
> -Travis
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

------------------------------------------------------------------------ 
----
| Dr. Louis J. Wicker
| NSSL/WRDD
| National Weather Center
| 120 David L. Boren Boulevard, Norman, OK 73072-7323
|
| E-mail:   Louis.Wicker at noaa.gov
| HTTP:  www.nssl.noaa.gov/~lwicker
| Phone:    (405) 325-6340
| Fax:        (405) 325-6780
|
| "Programming is not just creating strings of instructions
| for a computer to execute.  It's also 'literary' in that you
| are trying to communicate a program structure to
| other humans reading the code." - Paul Rubin
|
|"Real efficiency comes from elegant solutions, not optimized programs.
| Optimization is always just a few correctness-preserving  
transformations
| away." - Jonathan Sobel
------------------------------------------------------------------------ 
----
|
| "The contents  of this message are mine personally and
| do not reflect any position of  the Government or NOAA."
|
------------------------------------------------------------------------ 
----




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070201/8377b585/attachment.html>

From Louis.Wicker at noaa.gov  Thu Feb  1 14:50:59 2007
From: Louis.Wicker at noaa.gov (Louis Wicker)
Date: Thu, 01 Feb 2007 13:50:59 -0600
Subject: [Numpy-discussion] large memory address space on Mac
	OS	X	(intel)
In-Reply-To: <45C243F8.3080306@gmail.com>
References: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>
	<45C24272.4010402@ee.byu.edu> <45C243F8.3080306@gmail.com>
Message-ID: <9A3986E1-FE7E-490A-B0A1-E2F0FAAC25C3@noaa.gov>

Thanks Robert

thats kinda what I thought.  Since Leopard is not far off, then by  
summer things will be fine, I hope...

L

On Feb 1, 2007, at 1:48 PM, Robert Kern wrote:

> Travis Oliphant wrote:
>> Louis Wicker wrote:
>>
>>> Dear list:
>>>
>>> I cannot seem to figure how to create arrays > 2 GB on a Mac Pro
>>> (using Intel chip and Tiger, 4.8).  I have hand compiled both Python
>>> 2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB.  I  
>>> also
>>> run out of space if I try and 3-6 several arrays of 1000 mb or so  
>>> (the
>>> mem-alloc failure does not seem consistent, depends on whether I am
>>> creating them with a "numpy.ones()" call, or creating them on the  
>>> fly
>>> by doing math with the other arrays "e.g., c  = 4.3*a + 3.1*b").
>>>
>>> Is this a numpy issue, or a Python 2.5 issue for the Mac?  I have
>>> tried this on the SGI Altix, and this works fine.
>>
>> It must be a malloc issue.  NumPy uses the system malloc to construct
>> arrays.  It just reports errors back to you if it can't.
>>
>> I don't think the Mac Pro uses a 64-bit chip, does it?
>
> Intel Core 2 Duo's are 64-bit, but the OS and runtime libraries are  
> not. None of
> the Python distributions for OS X are compiled for 64-bit support,  
> either.
>
> When Leopard comes out, there will be better 64-bit support in the  
> OS, and
> Python will follow. Until then, Python on OS X is 32-bit.
>
> -- 
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a  
> harmless enigma
>  that is made terrible by our own mad attempt to interpret it as  
> though it had
>  an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

------------------------------------------------------------------------ 
----
| Dr. Louis J. Wicker
| NSSL/WRDD
| National Weather Center
| 120 David L. Boren Boulevard, Norman, OK 73072-7323
|
| E-mail:   Louis.Wicker at noaa.gov
| HTTP:  www.nssl.noaa.gov/~lwicker
| Phone:    (405) 325-6340
| Fax:        (405) 325-6780
|
| "Programming is not just creating strings of instructions
| for a computer to execute.  It's also 'literary' in that you
| are trying to communicate a program structure to
| other humans reading the code." - Paul Rubin
|
|"Real efficiency comes from elegant solutions, not optimized programs.
| Optimization is always just a few correctness-preserving  
transformations
| away." - Jonathan Sobel
------------------------------------------------------------------------ 
----
|
| "The contents  of this message are mine personally and
| do not reflect any position of  the Government or NOAA."
|
------------------------------------------------------------------------ 
----




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070201/1a67dac7/attachment.html>

From oliphant at ee.byu.edu  Thu Feb  1 14:55:55 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu, 01 Feb 2007 12:55:55 -0700
Subject: [Numpy-discussion] large memory address space on Mac
	OS	X	(intel)
In-Reply-To: <AD2A4509-3EDA-4C7B-B5D2-B62DFA8231AE@noaa.gov>
References: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>	<45C24272.4010402@ee.byu.edu>
	<AD2A4509-3EDA-4C7B-B5D2-B62DFA8231AE@noaa.gov>
Message-ID: <45C245CB.2080502@ee.byu.edu>

Louis Wicker wrote:

> Travis:
>
> yes it does.  Its the Woodcrest server chip 
> <http://www.intel.com/business/xeon/?cid=cim:ggl%7Cxeon_us_woodcrest%7Ck6913%7Cs> which 
> supports 32 and 64 bit operations.  For example the new Intel Fortran 
> compiler can grab more than 2 GB of memory (its a beta10 version).  I 
> think gcc 4.x can as well.
>
Nice.  I didn't know this.

> However, Tiger (OS X 10.4.x) is not completely 64 bit compliant - 
> Leopard is supposed to be pretty darn close.  
>
> Is there a numpy flag I could try for compilation....

It's entirely compiler and system dependent.  NumPy just uses the system 
malloc.  If you can compile it so that the system malloc supports 64-bit 
then O.K. (but you will probably run into trouble unless Python is also 
compiled as a 64-bit application).   From Robert's answer, I guess it is 
impossible under Tiger to compile with 64-bit support.

-Travis



From zpincus at stanford.edu  Thu Feb  1 15:01:00 2007
From: zpincus at stanford.edu (Zachary Pincus)
Date: Thu, 1 Feb 2007 12:01:00 -0800
Subject: [Numpy-discussion] array copy-to-self and views
In-Reply-To: <45C23FD9.6010108@ee.byu.edu>
References: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>
	<45C23FD9.6010108@ee.byu.edu>
Message-ID: <9A5766D8-8C3F-4C40-A54C-A286F264ECE7@stanford.edu>

>> A question, then: Does this represent a bug? Or perhaps there is a
>> better idiom for modifying an array in-place than 'a[:] = ...'? Or is
>> incumbent on the user to ensure that any time an array is directly
>> modified, that the modifying array is not a view of the original  
>> array?
>>
>>
> Yes, it is and has always been incumbent on the user to ensure that  
> any
> time an array is directly modified in-place that the modifying  
> array is
> not a "view" of the original array.

Fair enough. Now, how does a user ensure this -- say someone like me,  
who has been using numpy (et alia) for a couple of years, but clearly  
not long enough to have an 'intuitive' feel for every time something  
might be a view (a feeling that must seem quite natural to long-time  
numpy users, who may have forgotten precisely how long it takes to  
develop that level of intuition)?

Documentation of what returns views helps, for sure. Would any other  
'training' mechanisms help? Say a function that (despite Tim's pretty  
reasonable 'don't do that' warning) will return true when two arrays  
have overlapping memory? Or an 'inplace_modify' function that takes  
the time to make that check?

Perhaps I'm the first to have views bite me in this precise way.  
However, if there are common failure-modes with views, I hope it's  
not too unreasonable to ask about ways that those common problems  
might be addressed. (Other than just saying "train for ten years, and  
you too will have numpy-fu, my son.") Giving newbies tools to deal  
with common problems with admittedly "dangerous" constructs might be  
useful.

Zach


From seb.haase at gmx.net  Thu Feb  1 15:01:27 2007
From: seb.haase at gmx.net (Sebastian Haase)
Date: Thu, 1 Feb 2007 12:01:27 -0800
Subject: [Numpy-discussion] large memory address space on Mac OS X
	(intel)
In-Reply-To: <45C245CB.2080502@ee.byu.edu>
References: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>
	<45C24272.4010402@ee.byu.edu>
	<AD2A4509-3EDA-4C7B-B5D2-B62DFA8231AE@noaa.gov>
	<45C245CB.2080502@ee.byu.edu>
Message-ID: <bc657ead0702011201m653b7c63wc3f7bcdb91d75a38@mail.gmail.com>

Here is a small c program that we used more than a year ago to confirm
that tiger is really doing a 64-bit malloc (on G5).

<code: test64.c >
#include <stdlib.h>
#include <stdio.h>
int main()  {  size_t n;  void *p;  double gb;
  for(gb=10;gb>.3;gb-=.5) {
        n= 1024L * 1024L * 1024L * gb;
        p = malloc( n );
        printf("%12lu %4.1lfGb %p\n",n,n/1024./1024./1024.,p);
        free(p); }  return 0;  }
</code>

Hope this helps anyone.
Sebastian


On 2/1/07, Travis Oliphant <oliphant at ee.byu.edu> wrote:
> Louis Wicker wrote:
>
> > Travis:
> >
> > yes it does.  Its the Woodcrest server chip
> > <http://www.intel.com/business/xeon/?cid=cim:ggl%7Cxeon_us_woodcrest%7Ck6913%7Cs> which
> > supports 32 and 64 bit operations.  For example the new Intel Fortran
> > compiler can grab more than 2 GB of memory (its a beta10 version).  I
> > think gcc 4.x can as well.
> >
> Nice.  I didn't know this.
>
> > However, Tiger (OS X 10.4.x) is not completely 64 bit compliant -
> > Leopard is supposed to be pretty darn close.
> >
> > Is there a numpy flag I could try for compilation....
>
> It's entirely compiler and system dependent.  NumPy just uses the system
> malloc.  If you can compile it so that the system malloc supports 64-bit
> then O.K. (but you will probably run into trouble unless Python is also
> compiled as a 64-bit application).   From Robert's answer, I guess it is
> impossible under Tiger to compile with 64-bit support.
>
> -Travis
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From haase at msg.ucsf.edu  Thu Feb  1 15:01:48 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Thu, 1 Feb 2007 12:01:48 -0800
Subject: [Numpy-discussion] large memory address space on Mac OS X
	(intel)
In-Reply-To: <45C245CB.2080502@ee.byu.edu>
References: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>
	<45C24272.4010402@ee.byu.edu>
	<AD2A4509-3EDA-4C7B-B5D2-B62DFA8231AE@noaa.gov>
	<45C245CB.2080502@ee.byu.edu>
Message-ID: <bc657ead0702011201m3860ef0dw93afe36e9a7da1b9@mail.gmail.com>

Here is a small c program that we used more than a year ago to confirm
that tiger is really doing a 64-bit malloc (on G5).

<code: test64.c >
#include <stdlib.h>
#include <stdio.h>
int main()  {  size_t n;  void *p;  double gb;
 for(gb=10;gb>.3;gb-=.5) {
       n= 1024L * 1024L * 1024L * gb;
       p = malloc( n );
       printf("%12lu %4.1lfGb %p\n",n,n/1024./1024./1024.,p);
       free(p); }  return 0;  }
</code>

Hope this helps anyone.
Sebastian


On 2/1/07, Travis Oliphant <oliphant at ee.byu.edu> wrote:
> Louis Wicker wrote:
>
> > Travis:
> >
> > yes it does.  Its the Woodcrest server chip
> > <http://www.intel.com/business/xeon/?cid=cim:ggl%7Cxeon_us_woodcrest%7Ck6913%7Cs> which
> > supports 32 and 64 bit operations.  For example the new Intel Fortran
> > compiler can grab more than 2 GB of memory (its a beta10 version).  I
> > think gcc 4.x can as well.
> >
> Nice.  I didn't know this.
>
> > However, Tiger (OS X 10.4.x) is not completely 64 bit compliant -
> > Leopard is supposed to be pretty darn close.
> >
> > Is there a numpy flag I could try for compilation....
>
> It's entirely compiler and system dependent.  NumPy just uses the system
> malloc.  If you can compile it so that the system malloc supports 64-bit
> then O.K. (but you will probably run into trouble unless Python is also
> compiled as a 64-bit application).   From Robert's answer, I guess it is
> impossible under Tiger to compile with 64-bit support.
>
> -Travis
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From Chris.Barker at noaa.gov  Thu Feb  1 15:12:34 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 01 Feb 2007 12:12:34 -0800
Subject: [Numpy-discussion] array copy-to-self and views
In-Reply-To: <4A126763-2364-41AB-AD1C-ACBB5CE265FF@stanford.edu>
References: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>
	<45C22584.4060904@noaa.gov>
	<4A126763-2364-41AB-AD1C-ACBB5CE265FF@stanford.edu>
Message-ID: <45C249B2.6070804@noaa.gov>

Zachary Pincus wrote:
>>> I recently was trying to write code to modify an array in-place (so
>>> as not to invalidate any references to that array)
>> I'm not sure what this means exactly.
> 
> Say one wants to keep two different variables referencing a single in- 
> memory list, as so:
> a = [1,2,3]
> b = a
> Now, if 'b' and 'a' go to live in different places (different class  
> instances or whatever) but we want 'b' and 'a' to always refer to the  
> same in-memory object, so that 'id(a) == id(b)', we need to make sure  
> to not assign a brand new list to either one.

OK, got it, but numpy arrays are not quite the same as lists, there is 
the additional complication that two different array objects can share 
the same data:

 >>> b = a[:]
 >>> a = N.ones((5,))
 >>> b = a[:]
 >>> a is b
False
 >>> a[2] = 5
 >>> a
array([ 1.,  1.,  5.,  1.,  1.])
 >>> b
array([ 1.,  1.,  5.,  1.,  1.])

This is very useful, but can be tricky. In a way, it's like a nested list:
 >>> a = [[1,2,3,4]]
 >>> b = [a[0]]
 >>> a is b
False
 >>> a[0][2] = 5
 >>> a
[[1, 2, 5, 4]]
 >>> b
[[1, 2, 5, 4]]

hey! changing a changed b too!

So key is that in your case, it probably doesn't matter if a and b are 
the same object, as long as they share the same data, and having 
multiple arrays sharing the same data is a common idiom in numpy.

> That is, if we do something like 'a = [i + 1 for i in a]' then 'id 
> (a) != id(b)'. However, we can do 'a[:] = [i + 1 for i in a]' to  
> modify a in-place.

Ah, but at Travis pointed out, the difference is not in assignment or 
anything like that, but in the fact that a list comprehension produces a 
copy, which is analogous to :

flipud(a).copy

In numpy, you DO need to be aware of when you are getting copies, and 
when you are getting views, and what the consequences are.

So really, the only "bug" here is in the docs -- they should make it 
clear whether a function returns a copy or a view.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From oliphant at ee.byu.edu  Thu Feb  1 15:15:45 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu, 01 Feb 2007 13:15:45 -0700
Subject: [Numpy-discussion] array copy-to-self and views
In-Reply-To: <9A5766D8-8C3F-4C40-A54C-A286F264ECE7@stanford.edu>
References: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>	<45C23FD9.6010108@ee.byu.edu>
	<9A5766D8-8C3F-4C40-A54C-A286F264ECE7@stanford.edu>
Message-ID: <45C24A71.6070100@ee.byu.edu>

Zachary Pincus wrote:

>>>A question, then: Does this represent a bug? Or perhaps there is a
>>>better idiom for modifying an array in-place than 'a[:] = ...'? Or is
>>>incumbent on the user to ensure that any time an array is directly
>>>modified, that the modifying array is not a view of the original  
>>>array?
>>>
>>>
>>>      
>>>
>>Yes, it is and has always been incumbent on the user to ensure that  
>>any
>>time an array is directly modified in-place that the modifying  
>>array is
>>not a "view" of the original array.
>>    
>>
>
>Fair enough. Now, how does a user ensure this -- say someone like me,  
>who has been using numpy (et alia) for a couple of years, but clearly  
>not long enough to have an 'intuitive' feel for every time something  
>might be a view (a feeling that must seem quite natural to long-time  
>numpy users, who may have forgotten precisely how long it takes to  
>develop that level of intuition)?
>  
>
Basically, red-flags go off when you do in-place modification of any 
kind and you make sure you don't have an inappropriate view.  That 
pretty much describes my "intuition."  Views arise from "slicing" 
notation.  The flipud returning a view is a bit obscure and should be 
documented better.  

>Documentation of what returns views helps, for sure. Would any other  
>'training' mechanisms help? Say a function that (despite Tim's pretty  
>reasonable 'don't do that' warning) will return true when two arrays  
>have overlapping memory? Or an 'inplace_modify' function that takes  
>the time to make that check?
>  
>
I thought I had written a function that would see if two input arrays 
have over-lapping memory, but maybe not.  It's not hard for a contiguous 
chunk of memory, but for two views it's a harder function to write.  

It's probably a good idea to have such a thing, however.

-Travis




From Chris.Barker at noaa.gov  Thu Feb  1 15:39:09 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 01 Feb 2007 12:39:09 -0800
Subject: [Numpy-discussion] array copy-to-self and views
In-Reply-To: <9A5766D8-8C3F-4C40-A54C-A286F264ECE7@stanford.edu>
References: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>
	<45C23FD9.6010108@ee.byu.edu>
	<9A5766D8-8C3F-4C40-A54C-A286F264ECE7@stanford.edu>
Message-ID: <45C24FED.5050300@noaa.gov>

Zachary Pincus wrote:
> Say a function that (despite Tim's pretty  
> reasonable 'don't do that' warning) will return true when two arrays  
> have overlapping memory?

I think it would be useful, even if it's not robust. I'd still like to 
know if a given two arrays COULD share data.

I suppose to really be robust, what I'd really want to know is if a 
given array shares data with ANY other array, i.e. could changing this 
mess something up? -- but I'm pretty sure that is next to impossible

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From oliphant at ee.byu.edu  Thu Feb  1 15:46:44 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu, 01 Feb 2007 13:46:44 -0700
Subject: [Numpy-discussion] array copy-to-self and views
In-Reply-To: <45C24FED.5050300@noaa.gov>
References: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>	<45C23FD9.6010108@ee.byu.edu>	<9A5766D8-8C3F-4C40-A54C-A286F264ECE7@stanford.edu>
	<45C24FED.5050300@noaa.gov>
Message-ID: <45C251B4.1030404@ee.byu.edu>

Christopher Barker wrote:

>Zachary Pincus wrote:
>  
>
>>Say a function that (despite Tim's pretty  
>>reasonable 'don't do that' warning) will return true when two arrays  
>>have overlapping memory?
>>    
>>
>
>I think it would be useful, even if it's not robust. I'd still like to 
>know if a given two arrays COULD share data.
>
>I suppose to really be robust, what I'd really want to know is if a 
>given array shares data with ANY other array, i.e. could changing this 
>mess something up? -- but I'm pretty sure that is next to impossible
>
>  
>
Yeah, we don't keep track of who has a reference to a particular array.  
They only way to get that information would be to walk through all the 
Objects defined and see if any of them share memory with me.

You can sometimes get away with it by looking at the reference count of 
the object.  But, the reference count is used in more ways than that and 
so it's a very conservative check.   In the array interface I'm 
proposing for inclusion into Python, an object that shares memory could 
define a "call-back" function that (if defined) would be called when the 
view to the memory was released.  That way objects could store 
information regarding how many "views" they have extant.

-Travis





From tim.hochberg at ieee.org  Thu Feb  1 15:54:23 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Thu, 1 Feb 2007 13:54:23 -0700
Subject: [Numpy-discussion] array copy-to-self and views
In-Reply-To: <45C24FED.5050300@noaa.gov>
References: <66C0943A-1019-4B59-B5B6-F0D13EFF8BA3@stanford.edu>
	<45C23FD9.6010108@ee.byu.edu>
	<9A5766D8-8C3F-4C40-A54C-A286F264ECE7@stanford.edu>
	<45C24FED.5050300@noaa.gov>
Message-ID: <e4412d6b0702011254hc5f8aa9gb323250b1686e199@mail.gmail.com>

On 2/1/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
>
> Zachary Pincus wrote:
> > Say a function that (despite Tim's pretty
> > reasonable 'don't do that' warning) will return true when two arrays
> > have overlapping memory?
>
> I think it would be useful, even if it's not robust. I'd still like to
> know if a given two arrays COULD share data.
>
> I suppose to really be robust, what I'd really want to know is if a
> given array shares data with ANY other array, i.e. could changing this
> mess something up? -- but I'm pretty sure that is next to impossible


It's not totally impossible in theory -- languages like Haskell and Clean
(which I'm playing with now) manage to use arrays that get updated without
copying, while still maintaining the illusion that everything is constant
and thus you can't mess up any other arrays. While it's fun to play with and
Clean is allegedly pretty fast, it takes quite a bit of work to wrap ones
head around.

In a language like Python I expect that it would be pretty hard to come up
with something useful. Most of the checks would probably be too conservative
and thus not useful

-tim










-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070201/86ff3e0b/attachment.html>

From Louis.Wicker at noaa.gov  Thu Feb  1 16:02:32 2007
From: Louis.Wicker at noaa.gov (Louis Wicker)
Date: Thu, 01 Feb 2007 15:02:32 -0600
Subject: [Numpy-discussion] large memory address space on Mac OS
	X	(intel)
In-Reply-To: <bc657ead0702011201m3860ef0dw93afe36e9a7da1b9@mail.gmail.com>
References: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>
	<45C24272.4010402@ee.byu.edu>
	<AD2A4509-3EDA-4C7B-B5D2-B62DFA8231AE@noaa.gov>
	<45C245CB.2080502@ee.byu.edu>
	<bc657ead0702011201m3860ef0dw93afe36e9a7da1b9@mail.gmail.com>
Message-ID: <B8841F62-17FC-4B01-8E4D-EFA52BEFAECF@noaa.gov>

Sebastian:

that code helps a lot.  A standard gcc (no flags) of that code  
breaks, but if you compile it with gcc -m64, you can address large  
memory spaces.

So I will try and compile numpy with -m64....

Lou

On Feb 1, 2007, at 2:01 PM, Sebastian Haase wrote:

> #include <stdlib.h>
> #include <stdio.h>
> int main()  {  size_t n;  void *p;  double gb;
>  for(gb=10;gb>.3;gb-=.5) {
>        n= 1024L * 1024L * 1024L * gb;
>        p = malloc( n );
>        printf("%12lu %4.1lfGb %p\n",n,n/1024./1024./1024.,p);
>        free(p); }  return 0;  }

------------------------------------------------------------------------ 
----
| Dr. Louis J. Wicker
| NSSL/WRDD
| National Weather Center
| 120 David L. Boren Boulevard, Norman, OK 73072-7323
|
| E-mail:   Louis.Wicker at noaa.gov
| HTTP:  www.nssl.noaa.gov/~lwicker
| Phone:    (405) 325-6340
| Fax:        (405) 325-6780
|
| "Programming is not just creating strings of instructions
| for a computer to execute.  It's also 'literary' in that you
| are trying to communicate a program structure to
| other humans reading the code." - Paul Rubin
|
|"Real efficiency comes from elegant solutions, not optimized programs.
| Optimization is always just a few correctness-preserving  
transformations
| away." - Jonathan Sobel
------------------------------------------------------------------------ 
----
|
| "The contents  of this message are mine personally and
| do not reflect any position of  the Government or NOAA."
|
------------------------------------------------------------------------ 
----




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070201/dcf681d5/attachment.html>

From robert.kern at gmail.com  Thu Feb  1 16:11:29 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 01 Feb 2007 15:11:29 -0600
Subject: [Numpy-discussion] large memory address space on Mac
	OS	X	(intel)
In-Reply-To: <B8841F62-17FC-4B01-8E4D-EFA52BEFAECF@noaa.gov>
References: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>	<45C24272.4010402@ee.byu.edu>	<AD2A4509-3EDA-4C7B-B5D2-B62DFA8231AE@noaa.gov>	<45C245CB.2080502@ee.byu.edu>	<bc657ead0702011201m3860ef0dw93afe36e9a7da1b9@mail.gmail.com>
	<B8841F62-17FC-4B01-8E4D-EFA52BEFAECF@noaa.gov>
Message-ID: <45C25781.6060507@gmail.com>

Louis Wicker wrote:
> Sebastian:
> 
> that code helps a lot.  A standard gcc (no flags) of that code breaks,
> but if you compile it with gcc -m64, you can address large memory spaces.
> 
> So I will try and compile numpy with -m64....

It won't work. Your Python is not compiled as a 64-bit program. The whole stack
down to the runtime libraries needs to be built as a 64-bit program. That's easy
enough to do with a single-file C program, but in Tiger the 64-bit runtime
provides very few services, not enough to build Python.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From Louis.Wicker at noaa.gov  Thu Feb  1 16:18:09 2007
From: Louis.Wicker at noaa.gov (Louis Wicker)
Date: Thu, 01 Feb 2007 15:18:09 -0600
Subject: [Numpy-discussion] large memory address space on
	Mac	OS	X	(intel)
In-Reply-To: <45C25781.6060507@gmail.com>
References: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>
	<45C24272.4010402@ee.byu.edu>
	<AD2A4509-3EDA-4C7B-B5D2-B62DFA8231AE@noaa.gov>
	<45C245CB.2080502@ee.byu.edu>
	<bc657ead0702011201m3860ef0dw93afe36e9a7da1b9@mail.gmail.com>
	<B8841F62-17FC-4B01-8E4D-EFA52BEFAECF@noaa.gov>
	<45C25781.6060507@gmail.com>
Message-ID: <15555BFC-908D-48FA-B281-467267BD930F@noaa.gov>

Robert:

thanks - I appreciate the advice, and hopefully a) Leopard will get  
here in a few months, and b) that will fix this.

cheers!

Lou Wicker

On Feb 1, 2007, at 3:11 PM, Robert Kern wrote:

> Louis Wicker wrote:
>> Sebastian:
>>
>> that code helps a lot.  A standard gcc (no flags) of that code  
>> breaks,
>> but if you compile it with gcc -m64, you can address large memory  
>> spaces.
>>
>> So I will try and compile numpy with -m64....
>
> It won't work. Your Python is not compiled as a 64-bit program. The  
> whole stack
> down to the runtime libraries needs to be built as a 64-bit  
> program. That's easy
> enough to do with a single-file C program, but in Tiger the 64-bit  
> runtime
> provides very few services, not enough to build Python.
>
> -- 
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a  
> harmless enigma
>  that is made terrible by our own mad attempt to interpret it as  
> though it had
>  an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

------------------------------------------------------------------------ 
----
| Dr. Louis J. Wicker
| NSSL/WRDD
| National Weather Center
| 120 David L. Boren Boulevard, Norman, OK 73072-7323
|
| E-mail:   Louis.Wicker at noaa.gov
| HTTP:  www.nssl.noaa.gov/~lwicker
| Phone:    (405) 325-6340
| Fax:        (405) 325-6780
|
| "Programming is not just creating strings of instructions
| for a computer to execute.  It's also 'literary' in that you
| are trying to communicate a program structure to
| other humans reading the code." - Paul Rubin
|
|"Real efficiency comes from elegant solutions, not optimized programs.
| Optimization is always just a few correctness-preserving  
transformations
| away." - Jonathan Sobel
------------------------------------------------------------------------ 
----
|
| "The contents  of this message are mine personally and
| do not reflect any position of  the Government or NOAA."
|
------------------------------------------------------------------------ 
----




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070201/6d7a71aa/attachment.html>

From Chris.Barker at noaa.gov  Thu Feb  1 16:58:45 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 01 Feb 2007 13:58:45 -0800
Subject: [Numpy-discussion] SciPy '07 ???
Message-ID: <45C26295.4060707@noaa.gov>

Hi,

Does anyone know if there will be a SciPy '07 conference, and if so, when?

I'd really like to try to get there this year, but need to start 
planning my summer schedule.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From oliphant at ee.byu.edu  Thu Feb  1 18:48:56 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu, 01 Feb 2007 16:48:56 -0700
Subject: [Numpy-discussion] classmethods for ndarray
Message-ID: <45C27C68.7010306@ee.byu.edu>

What is the attitude of this group about the ndarray growing some class 
methods?

I'm thinking that we should have several.  For example all the fromXXX 
functions should probably be classmethods

ndarray.frombuffer
ndarray.fromfile

etc.



-Travis


From pgmdevlist at gmail.com  Thu Feb  1 19:13:07 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 1 Feb 2007 19:13:07 -0500
Subject: [Numpy-discussion] classmethods for ndarray
In-Reply-To: <45C27C68.7010306@ee.byu.edu>
References: <45C27C68.7010306@ee.byu.edu>
Message-ID: <200702011913.07062.pgmdevlist@gmail.com>

On Thursday 01 February 2007 18:48:56 Travis Oliphant wrote:
> What is the attitude of this group about the ndarray growing some class
> methods?

> ndarray.frombuffer
> ndarray.fromfile

Sounds great. But what would really make my semester is to have 
ndarray.__new__ accept optional keywords (as **whatever) on top of the 
shape/buffer/order... That'd be a big help for subclassing.
On a side note, I came to the fact that Santa Claus doesn't really live in the 
North Pole, so I would understand if it wasn't feasible...


From robert.kern at gmail.com  Thu Feb  1 19:16:23 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 01 Feb 2007 18:16:23 -0600
Subject: [Numpy-discussion] classmethods for ndarray
In-Reply-To: <45C27C68.7010306@ee.byu.edu>
References: <45C27C68.7010306@ee.byu.edu>
Message-ID: <45C282D7.50901@gmail.com>

Travis Oliphant wrote:
> What is the attitude of this group about the ndarray growing some class 
> methods?

Works for me.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Thu Feb  1 19:23:23 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 01 Feb 2007 16:23:23 -0800
Subject: [Numpy-discussion] classmethods for ndarray
In-Reply-To: <45C27C68.7010306@ee.byu.edu>
References: <45C27C68.7010306@ee.byu.edu>
Message-ID: <45C2847B.7080504@noaa.gov>

Travis Oliphant wrote:
> I'm thinking that we should have several.  For example all the fromXXX 
> functions should probably be classmethods
> 
> ndarray.frombuffer
> ndarray.fromfile

would they still be accessible in their functional form in the numpy 
namespace?

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From haase at msg.ucsf.edu  Thu Feb  1 19:24:22 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Thu, 1 Feb 2007 16:24:22 -0800
Subject: [Numpy-discussion] classmethods for ndarray
In-Reply-To: <45C282D7.50901@gmail.com>
References: <45C27C68.7010306@ee.byu.edu> <45C282D7.50901@gmail.com>
Message-ID: <bc657ead0702011624j3207c54fr6168dcb4dc06a34b@mail.gmail.com>

Travis,
Could you explain what a possible downside of this would be !?
It seems that if you don't need to refer to a specific "self" object
that a class-method is what it should - is this not always right !?

-Sebastian



On 2/1/07, Robert Kern <robert.kern at gmail.com> wrote:
> Travis Oliphant wrote:
> > What is the attitude of this group about the ndarray growing some class
> > methods?
>
> Works for me.
>
> --
> Robert Kern


From russel at appliedminds.net  Thu Feb  1 19:46:12 2007
From: russel at appliedminds.net (Russel Howe)
Date: Thu, 1 Feb 2007 16:46:12 -0800
Subject: [Numpy-discussion] Complex arange
Message-ID: <CBA4634B-A589-4BE3-97C3-D166D4EA4AC7@appliedminds.net>

Should this work?

Python 2.4.3 (#1, Dec 27 2006, 21:18:13)
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import numpy as N
 >>> N.__version__
'1.0.2.dev3531'
 >>> N.arange(1j, 5j)
array([], dtype=complex128)
 >>> N.arange(1j, 5j, 1j)
array([], dtype=complex128)

Currently, the real direction is determined to be of zero length  
(multiarraymodule.c _calc_length), and the length of the array is the  
minimal length.  I can understand the first one not working (default  
step is 1, it takes 0 of those to cover this range), but the second  
seems like a bug.



From oliphant at ee.byu.edu  Thu Feb  1 19:50:31 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu, 01 Feb 2007 17:50:31 -0700
Subject: [Numpy-discussion] classmethods for ndarray
In-Reply-To: <bc657ead0702011624j3207c54fr6168dcb4dc06a34b@mail.gmail.com>
References: <45C27C68.7010306@ee.byu.edu> <45C282D7.50901@gmail.com>
	<bc657ead0702011624j3207c54fr6168dcb4dc06a34b@mail.gmail.com>
Message-ID: <45C28AD7.4060305@ee.byu.edu>

Sebastian Haase wrote:

>Travis,
>Could you explain what a possible downside of this would be !?
>It seems that if you don't need to refer to a specific "self" object
>that a class-method is what it should - is this not always right !?
>
>  
>
I don't understand the last point.   Classmethods would get inherited by 
sub-classes by default, as far as I know.

I can't think of any downsides.  I have to understand how class-methods 
are actually implemented, though before I could comment on speed 
implications of class methods.

-Travis



From oliphant at ee.byu.edu  Thu Feb  1 19:51:22 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu, 01 Feb 2007 17:51:22 -0700
Subject: [Numpy-discussion] classmethods for ndarray
In-Reply-To: <45C2847B.7080504@noaa.gov>
References: <45C27C68.7010306@ee.byu.edu> <45C2847B.7080504@noaa.gov>
Message-ID: <45C28B0A.6000001@ee.byu.edu>

Christopher Barker wrote:

>Travis Oliphant wrote:
>  
>
>>I'm thinking that we should have several.  For example all the fromXXX 
>>functions should probably be classmethods
>>
>>ndarray.frombuffer
>>ndarray.fromfile
>>    
>>
>
>would they still be accessible in their functional form in the numpy 
>namespace?
>
>  
>
Yes, until a major revision at which point they could (if deemed useful) 
be removed after a deprecation warning period.

-Travis




From Chris.Barker at noaa.gov  Thu Feb  1 19:58:17 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 01 Feb 2007 16:58:17 -0800
Subject: [Numpy-discussion] classmethods for ndarray
In-Reply-To: <bc657ead0702011624j3207c54fr6168dcb4dc06a34b@mail.gmail.com>
References: <45C27C68.7010306@ee.byu.edu> <45C282D7.50901@gmail.com>
	<bc657ead0702011624j3207c54fr6168dcb4dc06a34b@mail.gmail.com>
Message-ID: <45C28CA9.7060704@noaa.gov>

Sebastian Haase wrote:

> Could you explain what a possible downside of this would be !?
> It seems that if you don't need to refer to a specific "self" object
> that a class-method is what it should - is this not always right !?

Well, what these really are are alternate constructors. I don't think 
I've seen class methods used that way, but then I haven't seen them used 
much at all.

Sometimes I have wished for an overloaded constructor, i.e.:

array(SomeBuffer)

results in the same thing as

frombuffer(SomeBuffer)


but Python doesn't really "do" overloaded methods, and there are some 
times when there wouldn't be only one way the input could be interpreted.

That all being the case, it seems to make some sense to put these in as 
class methods, but :

a = numpy.ndarray.fromfile(MyFile)

does feel a bit awkward.

Wx Python handles this by having a few constructors:

wx.EmptyBitmap()
wx.BitmapFromImage()
wx.BitmapFromBuffer()
etc...

but that's kind of clunky too.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From oliphant at ee.byu.edu  Thu Feb  1 20:00:25 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu, 01 Feb 2007 18:00:25 -0700
Subject: [Numpy-discussion] New may_share_memory function
Message-ID: <45C28D29.4060902@ee.byu.edu>


In SVN there is a new function  may_share_memory(a,b) which will return 
True if the memory foot-print of the two arrays over-lap.

 >>> may_share_memory(a, flipud(a))
True

This is based on another utility function byte_bounds that returns the 
byte-boundaries of any object exporting the Python side of the array 
interface.

Perhaps these utilities will help (I know they can be used to make the 
who function a bit more intelligent about how many bytes are being used).

-Travis





From robert.kern at gmail.com  Thu Feb  1 20:01:19 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 01 Feb 2007 19:01:19 -0600
Subject: [Numpy-discussion] Complex arange
In-Reply-To: <CBA4634B-A589-4BE3-97C3-D166D4EA4AC7@appliedminds.net>
References: <CBA4634B-A589-4BE3-97C3-D166D4EA4AC7@appliedminds.net>
Message-ID: <45C28D5F.70109@gmail.com>

Russel Howe wrote:

(It's good to see so many Rudds seeing sense and using Python and numpy.  ;-))

> Should this work?
> 
> Python 2.4.3 (#1, Dec 27 2006, 21:18:13)
> [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import numpy as N
>  >>> N.__version__
> '1.0.2.dev3531'
>  >>> N.arange(1j, 5j)
> array([], dtype=complex128)
>  >>> N.arange(1j, 5j, 1j)
> array([], dtype=complex128)
> 
> Currently, the real direction is determined to be of zero length  
> (multiarraymodule.c _calc_length), and the length of the array is the  
> minimal length.  I can understand the first one not working (default  
> step is 1, it takes 0 of those to cover this range), but the second  
> seems like a bug.

arange() is pretty much only defined for real numbers. For general z0, z1, and
dz, there is no guarantee that (z1 - z0)/dz is a real number as it needs to be.
dz may point in a different direction than (z1-z0). For example, what should
arange(1j, 5j, 1) do? Numeric raises an exception here, and I thing numpy
should, too.

Of course, linspace() is generally preferred for floating point types regardless
of whether complex or real. It's difficult to predetermine whether the stop
value will be included or not. And since you give it a count instead of a step
size, we can guarantee that the step does lie along the vector (z1-z0).

In [8]: linspace(1j, 5j, 5)
Out[8]: array([ 0.+1.j,  0.+2.j,  0.+3.j,  0.+4.j,  0.+5.j])

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Thu Feb  1 20:02:37 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 01 Feb 2007 17:02:37 -0800
Subject: [Numpy-discussion] New may_share_memory function
In-Reply-To: <45C28D29.4060902@ee.byu.edu>
References: <45C28D29.4060902@ee.byu.edu>
Message-ID: <45C28DAD.7070201@noaa.gov>

thanks Travis,

Now I just need to remember it's there when I need it!

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From robert.kern at gmail.com  Thu Feb  1 20:06:03 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 01 Feb 2007 19:06:03 -0600
Subject: [Numpy-discussion] classmethods for ndarray
In-Reply-To: <45C28CA9.7060704@noaa.gov>
References: <45C27C68.7010306@ee.byu.edu>
	<45C282D7.50901@gmail.com>	<bc657ead0702011624j3207c54fr6168dcb4dc06a34b@mail.gmail.com>
	<45C28CA9.7060704@noaa.gov>
Message-ID: <45C28E7B.9050405@gmail.com>

Christopher Barker wrote:
> Sebastian Haase wrote:
> 
>> Could you explain what a possible downside of this would be !?
>> It seems that if you don't need to refer to a specific "self" object
>> that a class-method is what it should - is this not always right !?
> 
> Well, what these really are are alternate constructors. I don't think 
> I've seen class methods used that way, but then I haven't seen them used 
> much at all.

Alternate constructors is probably the primary use case for class methods that
I've seen. It's certainly the most frequent reason I've made them.

> Sometimes I have wished for an overloaded constructor, i.e.:
> 
> array(SomeBuffer)
> 
> results in the same thing as
> 
> frombuffer(SomeBuffer)
> 
> 
> but Python doesn't really "do" overloaded methods, and there are some 
> times when there wouldn't be only one way the input could be interpreted.

Well, array() is already very, very overloaded. That's why it's difficult to use
sometimes. BTW, you might want to check out the module simplegeneric for a good
way to implement certain kinds of overloading. Just not numpy.array(), please ;-).

  http://cheeseshop.python.org/pypi/simplegeneric/

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From tim.hochberg at ieee.org  Thu Feb  1 20:20:55 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Thu, 1 Feb 2007 18:20:55 -0700
Subject: [Numpy-discussion] Complex arange
In-Reply-To: <45C28D5F.70109@gmail.com>
References: <CBA4634B-A589-4BE3-97C3-D166D4EA4AC7@appliedminds.net>
	<45C28D5F.70109@gmail.com>
Message-ID: <e4412d6b0702011720u106d5b84r147c1f9eb91c2c63@mail.gmail.com>

On 2/1/07, Robert Kern <robert.kern at gmail.com> wrote:
>
> Russel Howe wrote:
>
> (It's good to see so many Rudds seeing sense and using Python and
> numpy.  ;-))


rudds! Here? Dear me.


-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070201/4f3ad6cb/attachment.html>

From russel at appliedminds.net  Thu Feb  1 20:34:38 2007
From: russel at appliedminds.net (Russel Howe)
Date: Thu, 1 Feb 2007 17:34:38 -0800
Subject: [Numpy-discussion] Complex arange
In-Reply-To: <45C28D5F.70109@gmail.com>
References: <CBA4634B-A589-4BE3-97C3-D166D4EA4AC7@appliedminds.net>
	<45C28D5F.70109@gmail.com>
Message-ID: <C90CC77A-4E3F-4E51-A063-BA2931C205FF@appliedminds.net>


> arange(1j, 5j, 1) do? Numeric raises an exception here, and I thing  
> numpy
> should, too.
>

The same as arange(1, 5, 1j) - an empty array since it takes 0 of the  
step to cross the distance.  But something like
arange(1j, 5j, 1j) seems fine.  As does arange(1j, 3+5j, 2+1j) which  
should give [ 1j, 2+2j ].  The idea is to walk by step up to the edge  
of the box.  I seem to recall a discussion of why this was a bad idea  
a while ago on this list, but I can't find it...

The exception is a good answer too, but it should probably happen for  
all complex arguments, since most seem to return an empty array now.

Russel
they're all fine hovses.


From robert.kern at gmail.com  Thu Feb  1 20:47:33 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 01 Feb 2007 19:47:33 -0600
Subject: [Numpy-discussion] Complex arange
In-Reply-To: <C90CC77A-4E3F-4E51-A063-BA2931C205FF@appliedminds.net>
References: <CBA4634B-A589-4BE3-97C3-D166D4EA4AC7@appliedminds.net>	<45C28D5F.70109@gmail.com>
	<C90CC77A-4E3F-4E51-A063-BA2931C205FF@appliedminds.net>
Message-ID: <45C29835.70207@gmail.com>

Russel Howe wrote:
>> arange(1j, 5j, 1) do? Numeric raises an exception here, and I thing  
>> numpy
>> should, too.
> 
> The same as arange(1, 5, 1j) - an empty array since it takes 0 of the  
> step to cross the distance. 

I'm not sure that's really the answer. I think it's simply not defined. No
number of steps (which is different than 0 steps) along the imaginary axis will
take 1+0j to 5+0j.

> But something like
> arange(1j, 5j, 1j) seems fine.  As does arange(1j, 3+5j, 2+1j) which  
> should give [ 1j, 2+2j ].  The idea is to walk by step up to the edge  
> of the box.  I seem to recall a discussion of why this was a bad idea  
> a while ago on this list, but I can't find it...

Box? *Aaah*! You're looking at z1 placing distinct upper(lower) bounds on the
real and imaginary parts rather than specifying a point target. That's
a...unique perspective.  ;-)

But then, I'm of the opinion that arange() should be reserved for integers, and
the other use cases are better served by linspace() instead.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From aisaac at american.edu  Thu Feb  1 22:23:27 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Thu, 1 Feb 2007 22:23:27 -0500
Subject: [Numpy-discussion] [SciPy-user] Release 0.6.1 of pyaudio,
	renamed pyaudiolab
In-Reply-To: <45C0A076.2090805@ar.media.kyoto-u.ac.jp>
References: <45C0A076.2090805@ar.media.kyoto-u.ac.jp>
Message-ID: <Mahogany-0.67.0-1096-20070201-222327.00@american.edu>

On Wed, 31 Jan 2007, David Cournapeau apparently wrote: 
> With pyaudiolab, you should be able to read and write most 
> common audio files from and to numpy arrays. The 
> underlying IO operations are done using libsndfile from 
> Erik Castro Lopo (http://www.mega-nerd.com/libsndfile/) 

I think it is worth mentioning (on this list) that 
pyaudiolab uses the SciPy license and libsndfile is LGPL.

Cheers,
Alan Isaac






From david at ar.media.kyoto-u.ac.jp  Thu Feb  1 22:50:42 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 02 Feb 2007 12:50:42 +0900
Subject: [Numpy-discussion] [SciPy-user] Release 0.6.1 of pyaudio,
	renamed pyaudiolab
In-Reply-To: <Mahogany-0.67.0-1096-20070201-222327.00@american.edu>
References: <45C0A076.2090805@ar.media.kyoto-u.ac.jp>
	<Mahogany-0.67.0-1096-20070201-222327.00@american.edu>
Message-ID: <45C2B512.9070003@ar.media.kyoto-u.ac.jp>

Alan G Isaac wrote:
> On Wed, 31 Jan 2007, David Cournapeau apparently wrote: 
>> With pyaudiolab, you should be able to read and write most 
>> common audio files from and to numpy arrays. The 
>> underlying IO operations are done using libsndfile from 
>> Erik Castro Lopo (http://www.mega-nerd.com/libsndfile/) 
>
> I think it is worth mentioning (on this list) that 
> pyaudiolab uses the SciPy license and libsndfile is LGPL.
Indeed, I forgot to mention this fact in the announcement. It is 
mentioned somewhere in the source, but it should be done better. It is 
the only reason that pyaudiolab is not part of scipy.

Your post made me realize that I actually didn't look at how applying 
the license correctly, which is not good at all (that's the first 
project I started from scratch). I will change that.

David


From charlesr.harris at gmail.com  Thu Feb  1 23:07:50 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 1 Feb 2007 21:07:50 -0700
Subject: [Numpy-discussion] Different results from repeated calculation
In-Reply-To: <f4f93d420701280740h67e5f4afl94463af5b170b2a4@mail.gmail.com>
References: <f4f93d420701271225l18260e45x3ccc93f47ba982f0@mail.gmail.com>
	<20070127210641.GA12685@zaphod.lagged.za.net>
	<e06186140701271411j43a41783oaddfae80a8884a97@mail.gmail.com>
	<20070127223756.GC5742@mentat.za.net>
	<e06186140701271500u61ca8e41s9112f24698661962@mail.gmail.com>
	<db6b5ecc0701271511m5908638t8f8e0f2d0b0f5a81@mail.gmail.com>
	<45BBE3C7.3000207@gmail.com>
	<db6b5ecc0701271552j5d296d22r214447933c140297@mail.gmail.com>
	<f4f93d420701271623k28738164q85980647938bb390@mail.gmail.com>
	<f4f93d420701280740h67e5f4afl94463af5b170b2a4@mail.gmail.com>
Message-ID: <e06186140702012007w48735a70m42dba0d8f457377e@mail.gmail.com>

On 1/28/07, Keith Goodman <kwgoodman at gmail.com> wrote:
>
> On 1/27/07, Keith Goodman <kwgoodman at gmail.com> wrote:
> > On 1/27/07, Fernando Perez <fperez.net at gmail.com> wrote:
> > > It's definitely looking like something SMP related: on my laptop, with
> > > everything other than the hardware being identical (Linux distro,
> > > kernel, numpy build, etc), I can't make it fail no matter how I muck
> > > with it.  I always get '0 differences'.
> > >
> > > The desktop is a dual-core AMD Athlon as indicated before, the laptop
> > > is an oldie Pentium III.  They both run the same SMP-aware Ubuntu i686
> > > kernel, since Ubuntu now ships a unified kernel, though obviously on
> > > the laptop the SMP code isn't active.
> >
> > After installing a kernel that is not smp aware, I still have the same
> problem.
>
> The problem goes away if I remove atlas (atlas3-sse2 for me). But that
> just introduces another problem: slowness.


This problem may be related to this bug:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=279294

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070201/a8ef91ca/attachment.html>

From kwgoodman at gmail.com  Thu Feb  1 23:46:12 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 1 Feb 2007 20:46:12 -0800
Subject: [Numpy-discussion] Different results from repeated calculation
In-Reply-To: <e06186140702012007w48735a70m42dba0d8f457377e@mail.gmail.com>
References: <f4f93d420701271225l18260e45x3ccc93f47ba982f0@mail.gmail.com>
	<e06186140701271411j43a41783oaddfae80a8884a97@mail.gmail.com>
	<20070127223756.GC5742@mentat.za.net>
	<e06186140701271500u61ca8e41s9112f24698661962@mail.gmail.com>
	<db6b5ecc0701271511m5908638t8f8e0f2d0b0f5a81@mail.gmail.com>
	<45BBE3C7.3000207@gmail.com>
	<db6b5ecc0701271552j5d296d22r214447933c140297@mail.gmail.com>
	<f4f93d420701271623k28738164q85980647938bb390@mail.gmail.com>
	<f4f93d420701280740h67e5f4afl94463af5b170b2a4@mail.gmail.com>
	<e06186140702012007w48735a70m42dba0d8f457377e@mail.gmail.com>
Message-ID: <f4f93d420702012046v7849a452y72d61b72e2d44c1f@mail.gmail.com>

On 2/1/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
> This problem may be related to this bug:
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=279294

It says it is fixed in libc6 2.3.5. I'm on 2.3.6. But do you think it
is something similar?

A port to Octave of the test script works fine on the same system.


From robert.kern at gmail.com  Thu Feb  1 23:51:14 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 01 Feb 2007 22:51:14 -0600
Subject: [Numpy-discussion] Different results from repeated calculation
In-Reply-To: <f4f93d420702012046v7849a452y72d61b72e2d44c1f@mail.gmail.com>
References: <f4f93d420701271225l18260e45x3ccc93f47ba982f0@mail.gmail.com>	<e06186140701271411j43a41783oaddfae80a8884a97@mail.gmail.com>	<20070127223756.GC5742@mentat.za.net>	<e06186140701271500u61ca8e41s9112f24698661962@mail.gmail.com>	<db6b5ecc0701271511m5908638t8f8e0f2d0b0f5a81@mail.gmail.com>	<45BBE3C7.3000207@gmail.com>	<db6b5ecc0701271552j5d296d22r214447933c140297@mail.gmail.com>	<f4f93d420701271623k28738164q85980647938bb390@mail.gmail.com>	<f4f93d420701280740h67e5f4afl94463af5b170b2a4@mail.gmail.com>	<e06186140702012007w48735a70m42dba0d8f457377e@mail.gmail.com>
	<f4f93d420702012046v7849a452y72d61b72e2d44c1f@mail.gmail.com>
Message-ID: <45C2C342.8050906@gmail.com>

Keith Goodman wrote:
> A port to Octave of the test script works fine on the same system.

Are you sure that your Octave port uses ATLAS to do the matrix product? Could
you post your port?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Thu Feb  1 23:56:30 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 1 Feb 2007 21:56:30 -0700
Subject: [Numpy-discussion] Different results from repeated calculation
In-Reply-To: <f4f93d420702012046v7849a452y72d61b72e2d44c1f@mail.gmail.com>
References: <f4f93d420701271225l18260e45x3ccc93f47ba982f0@mail.gmail.com>
	<20070127223756.GC5742@mentat.za.net>
	<e06186140701271500u61ca8e41s9112f24698661962@mail.gmail.com>
	<db6b5ecc0701271511m5908638t8f8e0f2d0b0f5a81@mail.gmail.com>
	<45BBE3C7.3000207@gmail.com>
	<db6b5ecc0701271552j5d296d22r214447933c140297@mail.gmail.com>
	<f4f93d420701271623k28738164q85980647938bb390@mail.gmail.com>
	<f4f93d420701280740h67e5f4afl94463af5b170b2a4@mail.gmail.com>
	<e06186140702012007w48735a70m42dba0d8f457377e@mail.gmail.com>
	<f4f93d420702012046v7849a452y72d61b72e2d44c1f@mail.gmail.com>
Message-ID: <e06186140702012056r6b352418ocf80d09219d3f5d5@mail.gmail.com>

On 2/1/07, Keith Goodman <kwgoodman at gmail.com> wrote:
>
> On 2/1/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
> > This problem may be related to this bug:
> > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=279294
>
> It says it is fixed in libc6 2.3.5. I'm on 2.3.6. But do you think it
> is something similar?


I do, I am suspicious that the roundoff mode flag is changing state. But
these sort of bugs are notoriously hard to track down. You did good
isolating it to atlas and sse.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070201/0e11e65c/attachment.html>

From kwgoodman at gmail.com  Fri Feb  2 00:02:59 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 1 Feb 2007 21:02:59 -0800
Subject: [Numpy-discussion] Different results from repeated calculation
In-Reply-To: <45C2C342.8050906@gmail.com>
References: <f4f93d420701271225l18260e45x3ccc93f47ba982f0@mail.gmail.com>
	<e06186140701271500u61ca8e41s9112f24698661962@mail.gmail.com>
	<db6b5ecc0701271511m5908638t8f8e0f2d0b0f5a81@mail.gmail.com>
	<45BBE3C7.3000207@gmail.com>
	<db6b5ecc0701271552j5d296d22r214447933c140297@mail.gmail.com>
	<f4f93d420701271623k28738164q85980647938bb390@mail.gmail.com>
	<f4f93d420701280740h67e5f4afl94463af5b170b2a4@mail.gmail.com>
	<e06186140702012007w48735a70m42dba0d8f457377e@mail.gmail.com>
	<f4f93d420702012046v7849a452y72d61b72e2d44c1f@mail.gmail.com>
	<45C2C342.8050906@gmail.com>
Message-ID: <f4f93d420702012102x7b02e495md8b6699eb4867ae2@mail.gmail.com>

On 2/1/07, Robert Kern <robert.kern at gmail.com> wrote:
> Keith Goodman wrote:
> > A port to Octave of the test script works fine on the same system.
>
> Are you sure that your Octave port uses ATLAS to do the matrix product? Could
> you post your port?

Here's the port. Yes, Octave uses atlas for matrix multiplication.
Maybe the problem is a race condition and due to timing the outcome is
always the same in Octave...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: repeat.m
Type: text/x-objcsrc
Size: 835 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070201/6a2dce54/attachment.bin>

From fperez.net at gmail.com  Fri Feb  2 03:41:57 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 2 Feb 2007 01:41:57 -0700
Subject: [Numpy-discussion] classmethods for ndarray
In-Reply-To: <45C28E7B.9050405@gmail.com>
References: <45C27C68.7010306@ee.byu.edu> <45C282D7.50901@gmail.com>
	<bc657ead0702011624j3207c54fr6168dcb4dc06a34b@mail.gmail.com>
	<45C28CA9.7060704@noaa.gov> <45C28E7B.9050405@gmail.com>
Message-ID: <db6b5ecc0702020041od45f18ib33a37f0b4878cf7@mail.gmail.com>

On 2/1/07, Robert Kern <robert.kern at gmail.com> wrote:
> Christopher Barker wrote:
> > Sebastian Haase wrote:
> >
> >> Could you explain what a possible downside of this would be !?
> >> It seems that if you don't need to refer to a specific "self" object
> >> that a class-method is what it should - is this not always right !?
> >
> > Well, what these really are are alternate constructors. I don't think
> > I've seen class methods used that way, but then I haven't seen them used
> > much at all.
>
> Alternate constructors is probably the primary use case for class methods that
> I've seen. It's certainly the most frequent reason I've made them.

Same here (not in any publicly released code) and I happen to find
them handy in that role.  Absent truly overloaded constructors ? la
C++, they appear to be an acceptable compromise to me.

Cheers,

f


From david.douard at logilab.fr  Fri Feb  2 04:56:12 2007
From: david.douard at logilab.fr (David Douard)
Date: Fri, 2 Feb 2007 10:56:12 +0100
Subject: [Numpy-discussion] large memory address space on Mac OS X
	(intel)
In-Reply-To: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>
References: <4E1D1DFE-1FDE-43CF-97AC-F72EFC3D3E3C@noaa.gov>
Message-ID: <20070202095612.GB5353@crater.logilab.fr>

On Thu, Feb 01, 2007 at 01:33:23PM -0600, Louis Wicker wrote:
> Dear list:
Hi,

may I suggest you to read this?

http://orange.blender.org/blog/stupid-memory-problems

It worth a read.

David


> 
> I cannot seem to figure how to create arrays > 2 GB on a Mac Pro  
> (using Intel chip and Tiger, 4.8).  I have hand compiled both Python  
> 2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB.  I also  
> run out of space if I try and 3-6 several arrays of 1000 mb or so  
> (the mem-alloc failure does not seem consistent, depends on whether I  
> am creating them with a "numpy.ones()" call, or creating them on the  
> fly by doing math with the other arrays "e.g., c  = 4.3*a + 3.1*b").
> 
> Is this a numpy issue, or a Python 2.5 issue for the Mac?  I have  
> tried this on the SGI Altix, and this works fine.
> 
> If there is a compile flag to turn on 64 bit support in the Mac  
> compile, I would be glad to find out about it.  Or do I have to wait  
> for Leopard?
> 
> Thanks.
> 
> Lou Wicker
> 
> ------------------------------------------------------------------------ 
> ----
> | Dr. Louis J. Wicker
> | NSSL/WRDD
> | National Weather Center
> | 120 David L. Boren Boulevard, Norman, OK 73072-7323
> |
> | E-mail:   Louis.Wicker at noaa.gov
> | HTTP:  www.nssl.noaa.gov/~lwicker
> | Phone:    (405) 325-6340
> | Fax:        (405) 325-6780
> |
> | "Programming is not just creating strings of instructions
> | for a computer to execute.  It's also 'literary' in that you
> | are trying to communicate a program structure to
> | other humans reading the code." - Paul Rubin
> |
> |"Real efficiency comes from elegant solutions, not optimized programs.
> | Optimization is always just a few correctness-preserving  
> transformations
> | away." - Jonathan Sobel
> ------------------------------------------------------------------------ 
> ----
> |
> | "The contents  of this message are mine personally and
> | do not reflect any position of  the Government or NOAA."
> |
> ------------------------------------------------------------------------ 
> ----
> 
> 
> 
> 

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion


-- 
David Douard                             LOGILAB, Paris (France)
Formations Python, Zope, Plone, Debian : http://www.logilab.fr/formations
D?veloppement logiciel sur mesure :      http://www.logilab.fr/services
Informatique scientifique :              http://www.logilab.fr/science
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070202/7e29f5d6/attachment.sig>

From bblais at bryant.edu  Fri Feb  2 05:43:48 2007
From: bblais at bryant.edu (Brian Blais)
Date: Fri, 02 Feb 2007 05:43:48 -0500
Subject: [Numpy-discussion] classmethods for ndarray
In-Reply-To: <45C28AD7.4060305@ee.byu.edu>
References: <45C27C68.7010306@ee.byu.edu>
	<45C282D7.50901@gmail.com>	<bc657ead0702011624j3207c54fr6168dcb4dc06a34b@mail.gmail.com>
	<45C28AD7.4060305@ee.byu.edu>
Message-ID: <45C315E4.3040407@bryant.edu>

Travis Oliphant wrote:
> Sebastian Haase wrote:
> 
>> Travis,
>> Could you explain what a possible downside of this would be !?
> 
> I can't think of any downsides.  I have to understand how class-methods 
> are actually implemented, though before I could comment on speed 
> implications of class methods.

Would this break previously saved pickles, so you couldn't load them?  I ran into
that problem last year when there was a change to numpy.  Is that something that 
would happen here?


			bb


-- 
-----------------

              bblais at bryant.edu
              http://web.bryant.edu/~bblais



From faltet at carabos.com  Fri Feb  2 07:57:53 2007
From: faltet at carabos.com (Francesc Altet)
Date: Fri, 2 Feb 2007 13:57:53 +0100
Subject: [Numpy-discussion] Native byteorder representation
Message-ID: <200702021357.55549.faltet@carabos.com>

Hi,

We have been bitten by a small glitch related with the representation of 
native byteorders. Here is an example exposing the problem:

>>> numpy.dtype('<i4').byteorder
'='
>>> numpy.dtype('>i4').newbyteorder('little').byteorder
'<'

[the example was run on a little endian machine]

We thought that native byteorder were represented always by a '=', and this is 
true when you create the type from scratch. But, if you create a dtype with a 
different byteorder and then switch to a native one (in this case, 'little') 
the representation of the byteorder changes to '<' instead of '='.

We can live with this, but IMO it would be better if the final representation 
of native byteorders could always be made to read '='.

Thanks,

-- 
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"


From bsouthey at gmail.com  Fri Feb  2 09:14:02 2007
From: bsouthey at gmail.com (Bruce Southey)
Date: Fri, 2 Feb 2007 08:14:02 -0600
Subject: [Numpy-discussion] Different results from repeated calculation
In-Reply-To: <e06186140702012056r6b352418ocf80d09219d3f5d5@mail.gmail.com>
References: <f4f93d420701271225l18260e45x3ccc93f47ba982f0@mail.gmail.com>
	<e06186140701271500u61ca8e41s9112f24698661962@mail.gmail.com>
	<db6b5ecc0701271511m5908638t8f8e0f2d0b0f5a81@mail.gmail.com>
	<45BBE3C7.3000207@gmail.com>
	<db6b5ecc0701271552j5d296d22r214447933c140297@mail.gmail.com>
	<f4f93d420701271623k28738164q85980647938bb390@mail.gmail.com>
	<f4f93d420701280740h67e5f4afl94463af5b170b2a4@mail.gmail.com>
	<e06186140702012007w48735a70m42dba0d8f457377e@mail.gmail.com>
	<f4f93d420702012046v7849a452y72d61b72e2d44c1f@mail.gmail.com>
	<e06186140702012056r6b352418ocf80d09219d3f5d5@mail.gmail.com>
Message-ID: <bbcd77d00702020614y5dfa1aeve904c579e27ab91@mail.gmail.com>

Hi,
I am curious why  I do not see any mention of the compilers and
versions that were used in this thread. Having just finally managed to
get SciPY installed from scratch (but not with atlas), I could see
that using different compliers or versions or options especially
compiling done at different times could be a factor.

Bruce

On 2/1/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
>
>
> On 2/1/07, Keith Goodman <kwgoodman at gmail.com> wrote:
> > On 2/1/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
> > > This problem may be related to this bug:
> > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=279294
> >
> > It says it is fixed in libc6 2.3.5. I'm on 2.3.6. But do you think it
> > is something similar?
>
> I do, I am suspicious that the roundoff mode flag is changing state. But
> these sort of bugs are notoriously hard to track down. You did good
> isolating it to atlas and sse.
>
> Chuck
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From kwgoodman at gmail.com  Fri Feb  2 09:21:17 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Fri, 2 Feb 2007 06:21:17 -0800
Subject: [Numpy-discussion] Different results from repeated calculation
In-Reply-To: <bbcd77d00702020614y5dfa1aeve904c579e27ab91@mail.gmail.com>
References: <f4f93d420701271225l18260e45x3ccc93f47ba982f0@mail.gmail.com>
	<db6b5ecc0701271511m5908638t8f8e0f2d0b0f5a81@mail.gmail.com>
	<45BBE3C7.3000207@gmail.com>
	<db6b5ecc0701271552j5d296d22r214447933c140297@mail.gmail.com>
	<f4f93d420701271623k28738164q85980647938bb390@mail.gmail.com>
	<f4f93d420701280740h67e5f4afl94463af5b170b2a4@mail.gmail.com>
	<e06186140702012007w48735a70m42dba0d8f457377e@mail.gmail.com>
	<f4f93d420702012046v7849a452y72d61b72e2d44c1f@mail.gmail.com>
	<e06186140702012056r6b352418ocf80d09219d3f5d5@mail.gmail.com>
	<bbcd77d00702020614y5dfa1aeve904c579e27ab91@mail.gmail.com>
Message-ID: <f4f93d420702020621x186d4b43h9f8a3b3a4c5683bb@mail.gmail.com>

On 2/2/07, Bruce Southey <bsouthey at gmail.com> wrote:
> I am curious why  I do not see any mention of the compilers and
> versions that were used in this thread. Having just finally managed to
> get SciPY installed from scratch (but not with atlas), I could see
> that using different compliers or versions or options especially
> compiling done at different times could be a factor.

Yeah, good point. I installed 1.0.1 from binary from Debian sid. Maybe
a chart of which configurations have the problem and which don't would
help.

If the problem is ATLAS I don't understand why test1 passes. Could the
loading of the values be the problem and not the multiplication
itself?


From sturla at molden.no  Fri Feb  2 11:05:09 2007
From: sturla at molden.no (Sturla Molden)
Date: Fri, 2 Feb 2007 17:05:09 +0100 (CET)
Subject: [Numpy-discussion] Complex arange
In-Reply-To: <45C29835.70207@gmail.com>
References: <CBA4634B-A589-4BE3-97C3-D166D4EA4AC7@appliedminds.net>
	<45C28D5F.70109@gmail.com>
	<C90CC77A-4E3F-4E51-A063-BA2931C205FF@appliedminds.net>
	<45C29835.70207@gmail.com>
Message-ID: <1365.129.240.194.142.1170432309.squirrel@webmail.uio.no>


I think arange for complex numbers should work like meshgrid,
with the real and imaginary axis replacing the x and y axis. That would
mean something like this:

def complex_arange(start,end,stride):

   def iscomplex(x):
      if ((type(x)==complex) or (type(x)==complex64)
         or (type(x)==complex128)):
         return True
      else:
         return False

   if iscomplex(start) or iscomplex(end) or iscomplex(stride):
      start = complex(start)
      end = complex(end)
      stride = complex(stride)
      ar = arange(start.real, end.real, stride.real)
      ai = arange(start.imag ,end.imag, stride.imag)
      rr,ri = meshgrid(ar,ai)
      tmp = rr + 1j*ri
      if tmp.shape[0] == 1 or tmp.shape[1] == 1: tmp = tmp.flatten()
      return tmp
   else:
      return arange(start,end,stride)


I think this is a reasonable extension of arange to complex numbers. Here
complex_arange(1j, 5j, 1) throws a ZeroDivisionError as the stride for the
imaginary part i 0. Observe that complex_arange(1j, 5j, 1j) throws an
exception as well, as the extent of the real part is arange(0,0,0). But
complex_arange(0+1j,1+5j,1+1j) does exist, and so does
complex_arange(0+1j,0+5j,1+1j). But in the case of
complex_arange(0+1j,0+5j,1+1j) the return value is an empty array, as the
extent along the real axis is 0.

Regards,

Sturla Molden




From oliphant at ee.byu.edu  Fri Feb  2 12:22:12 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri, 02 Feb 2007 10:22:12 -0700
Subject: [Numpy-discussion] Native byteorder representation
In-Reply-To: <200702021357.55549.faltet@carabos.com>
References: <200702021357.55549.faltet@carabos.com>
Message-ID: <45C37344.3000504@ee.byu.edu>

Francesc Altet wrote:

>Hi,
>
>We have been bitten by a small glitch related with the representation of 
>native byteorders. Here is an example exposing the problem:
>
>  
>
>>>>numpy.dtype('<i4').byteorder
>>>>        
>>>>
>'='
>  
>
>>>>numpy.dtype('>i4').newbyteorder('little').byteorder
>>>>        
>>>>
>'<'
>
>  
>
This is somewhat inconsistent.  But, I'm not sure it's worth changing.  

In the second case, you request a "little" byteorder data-type.  Keeping 
this as '<' seems O.K.

One could instead ask why the first example did not report a byte-order 
of "<" when that's what was explicitly asked for.

-Travis



From faltet at carabos.com  Fri Feb  2 13:11:11 2007
From: faltet at carabos.com (Francesc Altet)
Date: Fri, 02 Feb 2007 19:11:11 +0100
Subject: [Numpy-discussion] Native byteorder representation
In-Reply-To: <45C37344.3000504@ee.byu.edu>
References: <200702021357.55549.faltet@carabos.com>
	<45C37344.3000504@ee.byu.edu>
Message-ID: <1170439871.2880.4.camel@localhost.localdomain>

El dv 02 de 02 del 2007 a les 10:22 -0700, en/na Travis Oliphant va
escriure:
> Francesc Altet wrote:
> 
> >Hi,
> >
> >We have been bitten by a small glitch related with the representation of 
> >native byteorders. Here is an example exposing the problem:
> >
> >  
> >
> >>>>numpy.dtype('<i4').byteorder
> >>>>        
> >>>>
> >'='
> >  
> >
> >>>>numpy.dtype('>i4').newbyteorder('little').byteorder
> >>>>        
> >>>>
> >'<'
> >
> >  
> >
> This is somewhat inconsistent.  But, I'm not sure it's worth changing.  
> 
> In the second case, you request a "little" byteorder data-type.  Keeping 
> this as '<' seems O.K.
> 
> One could instead ask why the first example did not report a byte-order 
> of "<" when that's what was explicitly asked for.

Well, just because of the same reason that

numpy.dtype('<i4').byteorder

returns a '=' instead of a '<' (the latter being explicitely set in the
constructor).

I think that returning a '=' whenever the byteorder is the same than the
underlying machine is desirable because the user can quickly see whether
her data is in native order or not.

Cheers,

-- 
Francesc Altet    |  Be careful about using the following code --
Carabos Coop. V.  |  I've only proven that it works, 
www.carabos.com   |  I haven't tested it. -- Donald Knuth



From faltet at carabos.com  Fri Feb  2 13:42:24 2007
From: faltet at carabos.com (Francesc Altet)
Date: Fri, 02 Feb 2007 19:42:24 +0100
Subject: [Numpy-discussion] Native byteorder representation
In-Reply-To: <1170439871.2880.4.camel@localhost.localdomain>
References: <200702021357.55549.faltet@carabos.com>
	<45C37344.3000504@ee.byu.edu>
	<1170439871.2880.4.camel@localhost.localdomain>
Message-ID: <1170441744.2880.9.camel@localhost.localdomain>

El dv 02 de 02 del 2007 a les 19:11 +0100, en/na Francesc Altet va
escriure:
> El dv 02 de 02 del 2007 a les 10:22 -0700, en/na Travis Oliphant va
> escriure:
> > Francesc Altet wrote:
> > 
> > >Hi,
> > >
> > >We have been bitten by a small glitch related with the representation of 
> > >native byteorders. Here is an example exposing the problem:
> > >
> > >  
> > >
> > >>>>numpy.dtype('<i4').byteorder
> > >>>>        
> > >>>>
> > >'='
> > >  
> > >
> > >>>>numpy.dtype('>i4').newbyteorder('little').byteorder
> > >>>>        
> > >>>>
> > >'<'
> > >
> > >  
> > >
> > This is somewhat inconsistent.  But, I'm not sure it's worth changing.  
> > 
> > In the second case, you request a "little" byteorder data-type.  Keeping 
> > this as '<' seems O.K.
> > 
> > One could instead ask why the first example did not report a byte-order 
> > of "<" when that's what was explicitly asked for.
> 
> Well, just because of the same reason that
> 
> numpy.dtype('<i4').byteorder
> 
> returns a '=' instead of a '<' (the latter being explicitely set in the
> constructor).

Ops. I was confused about what you was saying here, sorry. Forget this.

> 
> I think that returning a '=' whenever the byteorder is the same than the
> underlying machine is desirable because the user can quickly see whether
> her data is in native order or not.

I think that this is the only reason I can argue.

-- 
Francesc Altet    |  Be careful about using the following code --
Carabos Coop. V.  |  I've only proven that it works, 
www.carabos.com   |  I haven't tested it. -- Donald Knuth



From stefan at sun.ac.za  Fri Feb  2 17:15:13 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Sat, 3 Feb 2007 00:15:13 +0200
Subject: [Numpy-discussion] classmethods for ndarray
In-Reply-To: <45C28B0A.6000001@ee.byu.edu>
References: <45C27C68.7010306@ee.byu.edu> <45C2847B.7080504@noaa.gov>
	<45C28B0A.6000001@ee.byu.edu>
Message-ID: <20070202221513.GA6439@mentat.za.net>

On Thu, Feb 01, 2007 at 05:51:22PM -0700, Travis Oliphant wrote:
> Christopher Barker wrote:
> 
> >Travis Oliphant wrote:
> >  
> >
> >>I'm thinking that we should have several.  For example all the fromXXX 
> >>functions should probably be classmethods
> >>
> >>ndarray.frombuffer
> >>ndarray.fromfile
> >>    
> >>
> >
> >would they still be accessible in their functional form in the numpy 
> >namespace?
> >
> >  
> >
> Yes, until a major revision at which point they could (if deemed useful) 
> be removed after a deprecation warning period.

That would be a happy day.  I'd love to see the numpy namespace go on
a diet...

Cheers
St?fan


From dalcinl at gmail.com  Fri Feb  2 20:28:08 2007
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 2 Feb 2007 22:28:08 -0300
Subject: [Numpy-discussion] Requests for NumPy Ports?
In-Reply-To: <45C0D5E7.4@ee.byu.edu>
References: <45BFE7FB.6080000@ee.byu.edu>
	<f4f93d420701310910i1ed9ae9s20c7e08b1ab38df4@mail.gmail.com>
	<45C0CF7A.6040807@ee.byu.edu> <45C0D3DF.1070606@noaa.gov>
	<45C0D5E7.4@ee.byu.edu>
Message-ID: <e7ba66e40702021728m51e4489di3ab5f567eebae048@mail.gmail.com>

On 1/31/07, Travis Oliphant <oliphant at ee.byu.edu> wrote:
> To me this is so obvious that I don't understand the resistance in the
> Python community to the concept.

Indeed

Travis, I was not reading this for a some time ago. Can you point me
your last proposal? I remember reading about extending the builting
buffer interface slots.

Just an idea, as previous step to convince Py-Dev people: perhaps it
is possible to develop an C extension, independent of numpy,
implementing all what you proposed about dtype objects and defining a
good C-API and Py-API to deal with all this, and next perhaps market
it to people like gtk-python developers...

In any case, you are contributing to a community standard which can be
used once ready and next adopted by core python, like happened with
ctype...

What do you think?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594


From mattknox_ca at hotmail.com  Fri Feb  2 20:35:02 2007
From: mattknox_ca at hotmail.com (Matt Knox)
Date: Sat, 3 Feb 2007 01:35:02 +0000 (UTC)
Subject: [Numpy-discussion] classmethods for ndarray
References: <45C27C68.7010306@ee.byu.edu>
	<45C282D7.50901@gmail.com>	<bc657ead0702011624j3207c54fr6168dcb4dc06a34b@mail.gmail.com>
	<45C28AD7.4060305@ee.byu.edu> <45C315E4.3040407@bryant.edu>
Message-ID: <loom.20070203T022859-314@post.gmane.org>

> Would this break previously saved pickles, so you couldn't load them?  I ran 
into
> that problem last year when there was a change to numpy.  Is that something 
that 
> would happen here?
> 
> 			bb
> 

Regardless of whether or not this change will "break pickles", it seems highly 
likely to me that other future changes will prevent you from loading arrays 
pickled with older versions of numpy and python. That is one of several reasons 
why I generally think relying on direct pickling for storage is not a good 
idea. Another reason is that loading data becomes an all or nothing 
proposition, you can't read half of an array off the disk for example (not 
easily anyway). If you need to store numpy arrays directly, pytables 
(www.pytables.org) is the best solution that I am aware of, and it will 
(mostly) save you from worrying about these kinds of issues in the future.

- Matt



From robert.kern at gmail.com  Fri Feb  2 20:40:03 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 02 Feb 2007 19:40:03 -0600
Subject: [Numpy-discussion] classmethods for ndarray
In-Reply-To: <45C315E4.3040407@bryant.edu>
References: <45C27C68.7010306@ee.byu.edu>	<45C282D7.50901@gmail.com>	<bc657ead0702011624j3207c54fr6168dcb4dc06a34b@mail.gmail.com>	<45C28AD7.4060305@ee.byu.edu>
	<45C315E4.3040407@bryant.edu>
Message-ID: <45C3E7F3.7050709@gmail.com>

Brian Blais wrote:
> Would this break previously saved pickles, so you couldn't load them?  I ran into
> that problem last year when there was a change to numpy.  Is that something that 
> would happen here?

No.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From mail at stevesimmons.com  Sat Feb  3 20:22:37 2007
From: mail at stevesimmons.com (Stephen Simmons)
Date: Sun, 04 Feb 2007 12:22:37 +1100
Subject: [Numpy-discussion] array.sum() slower than expected along some
	array axes?
In-Reply-To: <1170439871.2880.4.camel@localhost.localdomain>
References: <200702021357.55549.faltet@carabos.com>	<45C37344.3000504@ee.byu.edu>
	<1170439871.2880.4.camel@localhost.localdomain>
Message-ID: <45C5355D.1020904@stevesimmons.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070204/299963c7/attachment.html>

From kwgoodman at gmail.com  Sat Feb  3 20:47:08 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Sat, 3 Feb 2007 17:47:08 -0800
Subject: [Numpy-discussion] array.sum() slower than expected along some
	array axes?
In-Reply-To: <45C5355D.1020904@stevesimmons.com>
References: <200702021357.55549.faltet@carabos.com>
	<45C37344.3000504@ee.byu.edu>
	<1170439871.2880.4.camel@localhost.localdomain>
	<45C5355D.1020904@stevesimmons.com>
Message-ID: <f4f93d420702031747u4b3620d7x9d5e17ce47645426@mail.gmail.com>

On 2/3/07, Stephen Simmons <mail at stevesimmons.com> wrote:
>  Does anyone know why there is an order of magnitude difference
>  in the speed of numpy's array.sum() function depending on the axis
>  of the matrix summed?
>
>  To see this, import numpy and create a big array with two rows:
>     >>> import numpy
>     >>> a = numpy.ones([2,1000000], 'f4')
>
>  Then using ipython's timeit function:
>                                                    Time (ms)
>     sum(a)                                           20
>     a.sum()                                           9
>     a.sum(axis=1)                                     9
>     a.sum(axis=0)                                   159
>     numpy.dot(numpy.ones(a.shape[0], a.dtype), a)    15
>
>  This last one using a dot product is functionally equivalent
>  to a.sum(axis=0), suggesting that the slowdown is due to how
>  indexing is implemented in array.sum().

I don't know how much time this would account for, but a.sum(0) has to
create a much larger array than a.sum(1) does.


From robert.kern at gmail.com  Sat Feb  3 20:52:17 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 03 Feb 2007 19:52:17 -0600
Subject: [Numpy-discussion] array.sum() slower than expected along some
 array axes?
In-Reply-To: <f4f93d420702031747u4b3620d7x9d5e17ce47645426@mail.gmail.com>
References: <200702021357.55549.faltet@carabos.com>	<45C37344.3000504@ee.byu.edu>	<1170439871.2880.4.camel@localhost.localdomain>	<45C5355D.1020904@stevesimmons.com>
	<f4f93d420702031747u4b3620d7x9d5e17ce47645426@mail.gmail.com>
Message-ID: <45C53C51.3020907@gmail.com>

Keith Goodman wrote:
> On 2/3/07, Stephen Simmons <mail at stevesimmons.com> wrote:
>>  Does anyone know why there is an order of magnitude difference
>>  in the speed of numpy's array.sum() function depending on the axis
>>  of the matrix summed?
>>
>>  To see this, import numpy and create a big array with two rows:
>>     >>> import numpy
>>     >>> a = numpy.ones([2,1000000], 'f4')
>>
>>  Then using ipython's timeit function:
>>                                                    Time (ms)
>>     sum(a)                                           20
>>     a.sum()                                           9
>>     a.sum(axis=1)                                     9
>>     a.sum(axis=0)                                   159
>>     numpy.dot(numpy.ones(a.shape[0], a.dtype), a)    15
>>
>>  This last one using a dot product is functionally equivalent
>>  to a.sum(axis=0), suggesting that the slowdown is due to how
>>  indexing is implemented in array.sum().
> 
> I don't know how much time this would account for, but a.sum(0) has to
> create a much larger array than a.sum(1) does.

However, so does sum(a) and numpy.dot().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From kwgoodman at gmail.com  Sat Feb  3 21:12:39 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Sat, 3 Feb 2007 18:12:39 -0800
Subject: [Numpy-discussion] array.sum() slower than expected along some
	array axes?
In-Reply-To: <45C53C51.3020907@gmail.com>
References: <200702021357.55549.faltet@carabos.com>
	<45C37344.3000504@ee.byu.edu>
	<1170439871.2880.4.camel@localhost.localdomain>
	<45C5355D.1020904@stevesimmons.com>
	<f4f93d420702031747u4b3620d7x9d5e17ce47645426@mail.gmail.com>
	<45C53C51.3020907@gmail.com>
Message-ID: <f4f93d420702031812k63c337e6y78d3b336c0df0768@mail.gmail.com>

On 2/3/07, Robert Kern <robert.kern at gmail.com> wrote:
> Keith Goodman wrote:
> > On 2/3/07, Stephen Simmons <mail at stevesimmons.com> wrote:
> >>  Does anyone know why there is an order of magnitude difference
> >>  in the speed of numpy's array.sum() function depending on the axis
> >>  of the matrix summed?
> >>
> >>  To see this, import numpy and create a big array with two rows:
> >>     >>> import numpy
> >>     >>> a = numpy.ones([2,1000000], 'f4')
> >>
> >>  Then using ipython's timeit function:
> >>                                                    Time (ms)
> >>     sum(a)                                           20
> >>     a.sum()                                           9
> >>     a.sum(axis=1)                                     9
> >>     a.sum(axis=0)                                   159
> >>     numpy.dot(numpy.ones(a.shape[0], a.dtype), a)    15
> >>
> >>  This last one using a dot product is functionally equivalent
> >>  to a.sum(axis=0), suggesting that the slowdown is due to how
> >>  indexing is implemented in array.sum().
> >
> > I don't know how much time this would account for, but a.sum(0) has to
> > create a much larger array than a.sum(1) does.
>
> However, so does sum(a) and numpy.dot().

The speed difference across axis 0 and 1 is also seen in Octave and
Matlab (but it is more like a factor of 5). But in those languages
axis=0 is much faster. And numpy, if I remember, stores arrays in the
opposite way as Octave (by row or column, I forget).

So a lot of the speed difference could be in how the array is stored.

http://velveeta.che.wisc.edu/octave/lists/help-octave/2005/2195
http://velveeta.che.wisc.edu/octave/lists/help-octave/2005/1912
http://velveeta.che.wisc.edu/octave/lists/help-octave/2005/1897


From charlesr.harris at gmail.com  Sat Feb  3 21:19:55 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 3 Feb 2007 19:19:55 -0700
Subject: [Numpy-discussion] array.sum() slower than expected along some
	array axes?
In-Reply-To: <45C5355D.1020904@stevesimmons.com>
References: <200702021357.55549.faltet@carabos.com>
	<45C37344.3000504@ee.byu.edu>
	<1170439871.2880.4.camel@localhost.localdomain>
	<45C5355D.1020904@stevesimmons.com>
Message-ID: <e06186140702031819w3a6946b5jcd5213dfbf9d116c@mail.gmail.com>

On 2/3/07, Stephen Simmons <mail at stevesimmons.com> wrote:
>
>  Hi,
>
> Does anyone know why there is an order of magnitude difference
> in the speed of numpy's array.sum() function depending on the axis
> of the matrix summed?
>
> To see this, import numpy and create a big array with two rows:
>    >>> import numpy
>    >>> a = numpy.ones([2,1000000], 'f4')
>
> Then using ipython's timeit function:
>                                                   Time (ms)
>    sum(a)                                           20
>    a.sum()                                           9
>    a.sum(axis=1)                                     9
>    a.sum(axis=0)                                   159
>    numpy.dot(numpy.ones(a.shape[0], a.dtype), a)    15
>
> This last one using a dot product is functionally equivalent
> to a.sum(axis=0), suggesting that the slowdown is due to how
> indexing is implemented in array.sum().
>

In this case it is expected. There are inner and outer loops, in the slow
case the inner loop with its extra code is called 1000000 times, in the fast
case, twice. On the other hand, note this:

In [10]: timeit a[0,:] + a[1,:]
100 loops, best of 3: 19.7 ms per loop


Which has only one loop. Caching could also be a problem, but in this case
it is dominated by loop overhead.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070203/7d2ad20b/attachment.html>

From charlesr.harris at gmail.com  Sat Feb  3 21:28:19 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 3 Feb 2007 19:28:19 -0700
Subject: [Numpy-discussion] array.sum() slower than expected along some
	array axes?
In-Reply-To: <e06186140702031819w3a6946b5jcd5213dfbf9d116c@mail.gmail.com>
References: <200702021357.55549.faltet@carabos.com>
	<45C37344.3000504@ee.byu.edu>
	<1170439871.2880.4.camel@localhost.localdomain>
	<45C5355D.1020904@stevesimmons.com>
	<e06186140702031819w3a6946b5jcd5213dfbf9d116c@mail.gmail.com>
Message-ID: <e06186140702031828n42c2876fn98e989fe54c2c27b@mail.gmail.com>

On 2/3/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
>
>
>
> On 2/3/07, Stephen Simmons <mail at stevesimmons.com> wrote:
> >
> >  Hi,
> >
> > Does anyone know why there is an order of magnitude difference
> > in the speed of numpy's array.sum() function depending on the axis
> > of the matrix summed?
> >
> > To see this, import numpy and create a big array with two rows:
> >    >>> import numpy
> >    >>> a = numpy.ones([2,1000000], 'f4')
> >
> > Then using ipython's timeit function:
> >                                                   Time (ms)
> >    sum(a)                                           20
> >    a.sum()                                           9
> >    a.sum(axis=1)                                     9
> >    a.sum(axis=0)                                   159
> >    numpy.dot(numpy.ones(a.shape[0], a.dtype), a)    15
> >
> > This last one using a dot product is functionally equivalent
> > to a.sum(axis=0), suggesting that the slowdown is due to how
> > indexing is implemented in array.sum().
> >
>
> In this case it is expected. There are inner and outer loops, in the slow
> case the inner loop with its extra code is called 1000000 times, in the fast
> case, twice. On the other hand, note this:
>
> In [10]: timeit a[0,:] + a[1,:]
> 100 loops, best of 3: 19.7 ms per loop
>
>
> Which has only one loop. Caching could also be a problem, but in this case
> it is dominated by loop overhead.
>

PS, I think this indicate that the code would run faster in this case if it
accumulated along the last axis, one at a time for each leading index. I
suspect that the current implementation accumulates down the first axis,
then repeats for each of the last indices. This shows that rearranging the
way the accumulation is done could be a big gain, especially if the largest
axis is chosen.

Chuck

Chuck
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070203/25ca667b/attachment.html>

From mail at stevesimmons.com  Sat Feb  3 21:29:45 2007
From: mail at stevesimmons.com (Stephen Simmons)
Date: Sun, 04 Feb 2007 13:29:45 +1100
Subject: [Numpy-discussion] array.sum() slower than expected along some
 array axes?
In-Reply-To: <e06186140702031819w3a6946b5jcd5213dfbf9d116c@mail.gmail.com>
References: <200702021357.55549.faltet@carabos.com>	<45C37344.3000504@ee.byu.edu>	<1170439871.2880.4.camel@localhost.localdomain>	<45C5355D.1020904@stevesimmons.com>
	<e06186140702031819w3a6946b5jcd5213dfbf9d116c@mail.gmail.com>
Message-ID: <45C54519.4070109@stevesimmons.com>

Charles R Harris wrote:
>
>
> On 2/3/07, *Stephen Simmons* <mail at stevesimmons.com 
> <mailto:mail at stevesimmons.com>> wrote:
>
>     Hi,
>
>     Does anyone know why there is an order of magnitude difference
>     in the speed of numpy's array.sum() function depending on the axis
>     of the matrix summed?
>
>     To see this, import numpy and create a big array with two rows:
>        >>> import numpy
>        >>> a = numpy.ones([2,1000000], 'f4')
>
>     Then using ipython's timeit function:
>                                                       Time (ms)
>        sum(a)                                           20
>        a.sum()                                           9     
>        a.sum(axis=1)                                     9
>        a.sum(axis=0)                                   159
>        numpy.dot(numpy.ones(a.shape[0], a.dtype), a)    15
>
>     This last one using a dot product is functionally equivalent
>     to a.sum(axis=0), suggesting that the slowdown is due to how
>     indexing is implemented in array.sum().
>
>
> In this case it is expected. There are inner and outer loops, in the 
> slow case the inner loop with its extra code is called 1000000 times, 
> in the fast case, twice. On the other hand, note this:
>
> In [10]: timeit a[0,:] + a[1,:]
> 100 loops, best of 3: 19.7 ms per loop
>
>
> Which has only one loop. Caching could also be a problem, but in this 
> case it is dominated by loop overhead.
>
> Chuck
I agree that summing along the longer axis is most probably slower 
because it makes more passes through the inner loop.

The question though is whether all of the inner loop's overhead is 
necessary.
My counterexample using numpy.dot() suggests there's considerable scope 
for improvement, at least for certain common cases.


From robert.kern at gmail.com  Sat Feb  3 21:34:38 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 03 Feb 2007 20:34:38 -0600
Subject: [Numpy-discussion] array.sum() slower than expected along some
 array axes?
In-Reply-To: <45C54519.4070109@stevesimmons.com>
References: <200702021357.55549.faltet@carabos.com>	<45C37344.3000504@ee.byu.edu>	<1170439871.2880.4.camel@localhost.localdomain>	<45C5355D.1020904@stevesimmons.com>	<e06186140702031819w3a6946b5jcd5213dfbf9d116c@mail.gmail.com>
	<45C54519.4070109@stevesimmons.com>
Message-ID: <45C5463E.9030000@gmail.com>

Stephen Simmons wrote:

> The question though is whether all of the inner loop's overhead is 
> necessary.
> My counterexample using numpy.dot() suggests there's considerable scope 
> for improvement, at least for certain common cases.

Well, yes. You most likely have an ATLAS-accelerated dot(). The ATLAS put a lot
of work into making matrix products really fast. However, they did so at a cost:
different architectures use different code. That's not really something we can
do in the core of numpy without making numpy as difficult to build as ATLAS is.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From provos at citi.umich.edu  Sun Feb  4 13:18:00 2007
From: provos at citi.umich.edu (Niels Provos)
Date: Sun, 4 Feb 2007 10:18:00 -0800
Subject: [Numpy-discussion] python eats memory like the cookie monster eats
	cookies
Message-ID: <850f7cbe0702041018s27f64d44n4f0f77245654eefb@mail.gmail.com>

Good morning,

not sure if I got the right list, but I hope that somebody here will
be able to shed some light on a Python-related memory problem.  The
following code eats over >2GB of memory and fails with MemoyError
after just a few iterations.

def ZeroPadData(A, shape):
    a = Numeric.zeros(shape, 'w')
    a.savespace()

    for y in xrange(A.shape[0]):
        for x in xrange(A.shape[1]):
            a[y, x] = A[y, x]

    return a

def EatMemoryLikeTheCookieMonster(limit=10):
    A = Numeric.ones([1998, 3022])

    count = 0
    a = A
    while count < limit:
        print count
        count += 1

        a = ZeroPadData(a, [2048, 4096])

        b = fft2(a)
        b = ifft2(b)

        a = b[:1998,:3022].real

EatMemoryLikeTheCookieMonster()

This is for Python 2.4.3 on Mac OS X 10.4.8 (intel) using SciPy 0.5.2.

If anyone could enlighten me about what I am doing wrong, I would very
much appreciate it.

Thank you,
 Niels.


From haase at msg.ucsf.edu  Sun Feb  4 14:37:49 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Sun, 4 Feb 2007 11:37:49 -0800
Subject: [Numpy-discussion] array.sum() slower than expected along some
	array axes?
In-Reply-To: <45C5463E.9030000@gmail.com>
References: <200702021357.55549.faltet@carabos.com>
	<45C37344.3000504@ee.byu.edu>
	<1170439871.2880.4.camel@localhost.localdomain>
	<45C5355D.1020904@stevesimmons.com>
	<e06186140702031819w3a6946b5jcd5213dfbf9d116c@mail.gmail.com>
	<45C54519.4070109@stevesimmons.com> <45C5463E.9030000@gmail.com>
Message-ID: <bc657ead0702041137p1350527dwbe3daacf8c2bd0a5@mail.gmail.com>

On 2/3/07, Robert Kern <robert.kern at gmail.com> wrote:
> Stephen Simmons wrote:
>
> > The question though is whether all of the inner loop's overhead is
> > necessary.
> > My counterexample using numpy.dot() suggests there's considerable scope
> > for improvement, at least for certain common cases.
>
> Well, yes. You most likely have an ATLAS-accelerated dot(). The ATLAS put a lot
> of work into making matrix products really fast. However, they did so at a cost:
> different architectures use different code. That's not really something we can
> do in the core of numpy without making numpy as difficult to build as ATLAS is.
>
Maybe this argument could be inverted:
maybe numpy could check if ATLAS is installed and automatically switch to the
numpy.dot(numpy.ones(a.shape[0], a.dtype), a)
variant that Stephen suggested.

Of course -- as I see it -- the numpy.ones(...)  part requires lots of
extra memory. Maybe there are other downsides ... !?
-Sebastian


From kwgoodman at gmail.com  Sun Feb  4 15:00:26 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Sun, 4 Feb 2007 12:00:26 -0800
Subject: [Numpy-discussion] array.sum() slower than expected along some
	array axes?
In-Reply-To: <bc657ead0702041137p1350527dwbe3daacf8c2bd0a5@mail.gmail.com>
References: <200702021357.55549.faltet@carabos.com>
	<45C37344.3000504@ee.byu.edu>
	<1170439871.2880.4.camel@localhost.localdomain>
	<45C5355D.1020904@stevesimmons.com>
	<e06186140702031819w3a6946b5jcd5213dfbf9d116c@mail.gmail.com>
	<45C54519.4070109@stevesimmons.com> <45C5463E.9030000@gmail.com>
	<bc657ead0702041137p1350527dwbe3daacf8c2bd0a5@mail.gmail.com>
Message-ID: <f4f93d420702041200w505d3194oc625491842a21af@mail.gmail.com>

On 2/4/07, Sebastian Haase <haase at msg.ucsf.edu> wrote:
> On 2/3/07, Robert Kern <robert.kern at gmail.com> wrote:
> > Stephen Simmons wrote:
> >
> > > The question though is whether all of the inner loop's overhead is
> > > necessary.
> > > My counterexample using numpy.dot() suggests there's considerable scope
> > > for improvement, at least for certain common cases.
> >
> > Well, yes. You most likely have an ATLAS-accelerated dot(). The ATLAS put a lot
> > of work into making matrix products really fast. However, they did so at a cost:
> > different architectures use different code. That's not really something we can
> > do in the core of numpy without making numpy as difficult to build as ATLAS is.
> >
> Maybe this argument could be inverted:
> maybe numpy could check if ATLAS is installed and automatically switch to the
> numpy.dot(numpy.ones(a.shape[0], a.dtype), a)
> variant that Stephen suggested.
>
> Of course -- as I see it -- the numpy.ones(...)  part requires lots of
> extra memory. Maybe there are other downsides ... !?

I use multiplication instead of sum in heavily used loops. I'm often
able to predefine the ones outside the loop.

In Octave I made my own sum functions---separate ones for axis 0 and
1---that use multiplication. Maybe it is better to make a new function
rather than complicate the existing one.


From robert.kern at gmail.com  Sun Feb  4 15:20:52 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 04 Feb 2007 14:20:52 -0600
Subject: [Numpy-discussion] python eats memory like the cookie monster
 eats	cookies
In-Reply-To: <850f7cbe0702041018s27f64d44n4f0f77245654eefb@mail.gmail.com>
References: <850f7cbe0702041018s27f64d44n4f0f77245654eefb@mail.gmail.com>
Message-ID: <45C64024.3000305@gmail.com>

Niels Provos wrote:
> Good morning,
> 
> not sure if I got the right list, but I hope that somebody here will
> be able to shed some light on a Python-related memory problem.  The
> following code eats over >2GB of memory and fails with MemoyError
> after just a few iterations.
> 
> def ZeroPadData(A, shape):
>     a = Numeric.zeros(shape, 'w')
>     a.savespace()
> 
>     for y in xrange(A.shape[0]):
>         for x in xrange(A.shape[1]):
>             a[y, x] = A[y, x]
> 
>     return a
> 
> def EatMemoryLikeTheCookieMonster(limit=10):
>     A = Numeric.ones([1998, 3022])
> 
>     count = 0
>     a = A
>     while count < limit:
>         print count
>         count += 1
> 
>         a = ZeroPadData(a, [2048, 4096])
> 
>         b = fft2(a)
>         b = ifft2(b)
> 
>         a = b[:1998,:3022].real
> 
> EatMemoryLikeTheCookieMonster()
> 
> This is for Python 2.4.3 on Mac OS X 10.4.8 (intel) using SciPy 0.5.2.

Could you also post a complete example? Why are you using Numeric? scipy 0.5.2
requires numpy, not Numeric. Where are the fft2() and ifft2() functions coming
from, scipy.fftpack or numpy?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From provos at citi.umich.edu  Sun Feb  4 16:33:05 2007
From: provos at citi.umich.edu (Niels Provos)
Date: Sun, 4 Feb 2007 13:33:05 -0800
Subject: [Numpy-discussion] python eats memory like the cookie monster
	eats cookies
In-Reply-To: <45C64024.3000305@gmail.com>
References: <850f7cbe0702041018s27f64d44n4f0f77245654eefb@mail.gmail.com>
	<45C64024.3000305@gmail.com>
Message-ID: <850f7cbe0702041333j28812749j693737291b9bbc3e@mail.gmail.com>

The missing imports are

import Numeric # for zeros and ones
from scipy.fftpack import fft2,ifft2

Curiously, replacing Numeric.zeros with scipy.zeros makes the problem
go away.  Why?

Thank you,
 Niels.

On 2/4/07, Robert Kern <robert.kern at gmail.com> wrote:
> Niels Provos wrote:
> > Good morning,
> >
> > not sure if I got the right list, but I hope that somebody here will
> > be able to shed some light on a Python-related memory problem.  The
> > following code eats over >2GB of memory and fails with MemoyError
> > after just a few iterations.
> >
> > def ZeroPadData(A, shape):
> >     a = Numeric.zeros(shape, 'w')
> >     a.savespace()
> >
> >     for y in xrange(A.shape[0]):
> >         for x in xrange(A.shape[1]):
> >             a[y, x] = A[y, x]
> >
> >     return a
> >
> > def EatMemoryLikeTheCookieMonster(limit=10):
> >     A = Numeric.ones([1998, 3022])
> >
> >     count = 0
> >     a = A
> >     while count < limit:
> >         print count
> >         count += 1
> >
> >         a = ZeroPadData(a, [2048, 4096])
> >
> >         b = fft2(a)
> >         b = ifft2(b)
> >
> >         a = b[:1998,:3022].real
> >
> > EatMemoryLikeTheCookieMonster()
> >
> > This is for Python 2.4.3 on Mac OS X 10.4.8 (intel) using SciPy 0.5.2.
>
> Could you also post a complete example? Why are you using Numeric? scipy 0.5.2
> requires numpy, not Numeric. Where are the fft2() and ifft2() functions coming
> from, scipy.fftpack or numpy?
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless enigma
>  that is made terrible by our own mad attempt to interpret it as though it had
>  an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From robert.kern at gmail.com  Sun Feb  4 16:34:04 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 04 Feb 2007 15:34:04 -0600
Subject: [Numpy-discussion] python eats memory like the cookie monster
 eats cookies
In-Reply-To: <850f7cbe0702041333j28812749j693737291b9bbc3e@mail.gmail.com>
References: <850f7cbe0702041018s27f64d44n4f0f77245654eefb@mail.gmail.com>	<45C64024.3000305@gmail.com>
	<850f7cbe0702041333j28812749j693737291b9bbc3e@mail.gmail.com>
Message-ID: <45C6514C.7020800@gmail.com>

Niels Provos wrote:
> The missing imports are
> 
> import Numeric # for zeros and ones
> from scipy.fftpack import fft2,ifft2
> 
> Curiously, replacing Numeric.zeros with scipy.zeros makes the problem
> go away.  Why?

Possibly a bug in Numeric.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From gnata at obs.univ-lyon1.fr  Sun Feb  4 17:52:46 2007
From: gnata at obs.univ-lyon1.fr (Xavier Gnata)
Date: Sun, 04 Feb 2007 23:52:46 +0100
Subject: [Numpy-discussion] numpy.tests() FAILED (errors=3)
Message-ID: <45C663BE.1030100@obs.univ-lyon1.fr>

Hi,

I'm using numpy svn and running numpy.test().

The error log is the following :

======================================================================
ERROR: check_complex_bad (numpy.lib.tests.test_type_check.test_nan_to_num)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/usr/lib/python2.4/site-packages/numpy/lib/tests/test_type_check.py",
line 245, in check_complex_bad
    vals = nan_to_num(v)
  File "/usr/lib/python2.4/site-packages/numpy/lib/type_check.py", line
132, in nan_to_num
    are_inf = isposinf(y)
  File "/usr/lib/python2.4/site-packages/numpy/lib/ufunclike.py", line
33, in isposinf
    umath.logical_and(isinf(x), ~signbit(x), y)
TypeError: function not supported for these types, and can't coerce
safely to supported types

======================================================================
ERROR: check_complex_bad2 (numpy.lib.tests.test_type_check.test_nan_to_num)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/usr/lib/python2.4/site-packages/numpy/lib/tests/test_type_check.py",
line 253, in check_complex_bad2
    vals = nan_to_num(v)
  File "/usr/lib/python2.4/site-packages/numpy/lib/type_check.py", line
132, in nan_to_num
    are_inf = isposinf(y)
  File "/usr/lib/python2.4/site-packages/numpy/lib/ufunclike.py", line
33, in isposinf
    umath.logical_and(isinf(x), ~signbit(x), y)
TypeError: function not supported for these types, and can't coerce
safely to supported types

======================================================================
ERROR: check_complex_good (numpy.lib.tests.test_type_check.test_nan_to_num)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/usr/lib/python2.4/site-packages/numpy/lib/tests/test_type_check.py",
line 238, in check_complex_good
    vals = nan_to_num(1+1j)
  File "/usr/lib/python2.4/site-packages/numpy/lib/type_check.py", line
132, in nan_to_num
    are_inf = isposinf(y)
  File "/usr/lib/python2.4/site-packages/numpy/lib/ufunclike.py", line
33, in isposinf
    umath.logical_and(isinf(x), ~signbit(x), y)
TypeError: function not supported for these types, and can't coerce
safely to supported types

----------------------------------------------------------------------

It should be a side effect of one patch because I can read things like "
# !! This is actually (unexpectedly) zero" in
/usr/lib/python2.4/site-packages/numpy/lib/tests/test_type_check.py

Xavier.

-- 
############################################
Xavier Gnata
CRAL - Observatoire de Lyon
9, avenue Charles Andr?
69561 Saint Genis Laval cedex
Phone: +33 4 78 86 85 28
Fax: +33 4 78 86 83 86
E-mail: gnata at obs.univ-lyon1.fr
############################################ 



From haase at msg.ucsf.edu  Sun Feb  4 19:36:04 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Sun, 4 Feb 2007 16:36:04 -0800
Subject: [Numpy-discussion] python eats memory like the cookie monster
	eats cookies
In-Reply-To: <45C6514C.7020800@gmail.com>
References: <850f7cbe0702041018s27f64d44n4f0f77245654eefb@mail.gmail.com>
	<45C64024.3000305@gmail.com>
	<850f7cbe0702041333j28812749j693737291b9bbc3e@mail.gmail.com>
	<45C6514C.7020800@gmail.com>
Message-ID: <bc657ead0702041636yd53a9c4x984ea4b0468e8565@mail.gmail.com>

On 2/4/07, Robert Kern <robert.kern at gmail.com> wrote:
> Niels Provos wrote:
> > The missing imports are
> >
> > import Numeric # for zeros and ones
> > from scipy.fftpack import fft2,ifft2
> >
> > Curiously, replacing Numeric.zeros with scipy.zeros makes the problem
> > go away.  Why?
>
> Possibly a bug in Numeric.
>
> --
> Robert Kern

Is there *any* support for old Numeric on this list !?

Maybe it should be officially stated that the one way to go is
numpy
and that problems with Numeric ( or numarray ) can only be noticed but
will likely not get fixed....

-Sebastian


From robert.kern at gmail.com  Sun Feb  4 19:39:17 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 04 Feb 2007 18:39:17 -0600
Subject: [Numpy-discussion] python eats memory like the cookie monster
 eats cookies
In-Reply-To: <bc657ead0702041636yd53a9c4x984ea4b0468e8565@mail.gmail.com>
References: <850f7cbe0702041018s27f64d44n4f0f77245654eefb@mail.gmail.com>	<45C64024.3000305@gmail.com>	<850f7cbe0702041333j28812749j693737291b9bbc3e@mail.gmail.com>	<45C6514C.7020800@gmail.com>
	<bc657ead0702041636yd53a9c4x984ea4b0468e8565@mail.gmail.com>
Message-ID: <45C67CB5.1000303@gmail.com>

Sebastian Haase wrote:
> Is there *any* support for old Numeric on this list !?

Not unless if you are offering some.

> Maybe it should be officially stated that the one way to go is
> numpy
> and that problems with Numeric ( or numarray ) can only be noticed but
> will likely not get fixed....

That's pretty much what we've been officially stating for some time now.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From kwgoodman at gmail.com  Sun Feb  4 20:06:34 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Sun, 4 Feb 2007 17:06:34 -0800
Subject: [Numpy-discussion] numpy.matlib.abs
Message-ID: <f4f93d420702041706t2a695f55h8c6a6d5980e70b96@mail.gmail.com>

There's a numpy.abs but no numpy.matlib.abs.

>> import numpy as N
>> import numpy.matlib as M
>>
>> N.abs?
Type:           ufunc
Base Class:     <type 'numpy.ufunc'>
String Form:    <ufunc 'absolute'>
Namespace:      Interactive
Docstring:
    y = absolute(x) takes |x| elementwise.

>> M.abs?
Object `M.abs` not found.


From robert.kern at gmail.com  Sun Feb  4 20:12:44 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 04 Feb 2007 19:12:44 -0600
Subject: [Numpy-discussion] numpy.matlib.abs
In-Reply-To: <f4f93d420702041706t2a695f55h8c6a6d5980e70b96@mail.gmail.com>
References: <f4f93d420702041706t2a695f55h8c6a6d5980e70b96@mail.gmail.com>
Message-ID: <45C6848C.4060105@gmail.com>

Keith Goodman wrote:
> There's a numpy.abs but no numpy.matlib.abs.
> 
>>> import numpy as N
>>> import numpy.matlib as M
>>>
>>> N.abs?
> Type:           ufunc
> Base Class:     <type 'numpy.ufunc'>
> String Form:    <ufunc 'absolute'>
> Namespace:      Interactive
> Docstring:
>     y = absolute(x) takes |x| elementwise.
> 
>>> M.abs?
> Object `M.abs` not found.

numpy.abs() is not exported via "from numpy import *", which is where
numpy.matlib gets all of its non-overridden functions from. It is not exported
because it conflicts with the builtin abs().

Of course, absolute() is preferred for the same reason, and
numpy.matlib.absolute() does exist.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From jeremit0 at gmail.com  Sun Feb  4 20:22:44 2007
From: jeremit0 at gmail.com (Jeremy Conlin)
Date: Sun, 4 Feb 2007 20:22:44 -0500
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
Message-ID: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>

I have subclassed the numpy.ndarray object, but need some help setting
some attributes. I have read http://scipy.org/Subclasses but it
doesn't provide the answer I am looking for.

I create an instance of the class in my __new__ method as:

import numpy
class MyClass(numpy.ndarray):
  __new__(self,?):
    # Some stuff here
    H, edges = numpy.histogramdd(?)
    return H


This sets H as the instance of my object.  I would also like to have
edges be an attribute of MyClass.  I can't do:

self.edges = edges

because the object hasn't been instantiated yet.  Can someone show me
how I can also keep the information from the variable edges?
Thanks,
Jeremy


From kwgoodman at gmail.com  Sun Feb  4 20:28:48 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Sun, 4 Feb 2007 17:28:48 -0800
Subject: [Numpy-discussion] numpy.matlib.abs
In-Reply-To: <45C6848C.4060105@gmail.com>
References: <f4f93d420702041706t2a695f55h8c6a6d5980e70b96@mail.gmail.com>
	<45C6848C.4060105@gmail.com>
Message-ID: <f4f93d420702041728t61c37c7ak40b01c0a5e74d574@mail.gmail.com>

On 2/4/07, Robert Kern <robert.kern at gmail.com> wrote:
> Keith Goodman wrote:
> > There's a numpy.abs but no numpy.matlib.abs.
> >
> >>> import numpy as N
> >>> import numpy.matlib as M
> >>>
> >>> N.abs?
> > Type:           ufunc
> > Base Class:     <type 'numpy.ufunc'>
> > String Form:    <ufunc 'absolute'>
> > Namespace:      Interactive
> > Docstring:
> >     y = absolute(x) takes |x| elementwise.
> >
> >>> M.abs?
> > Object `M.abs` not found.
>
> numpy.abs() is not exported via "from numpy import *", which is where
> numpy.matlib gets all of its non-overridden functions from. It is not exported
> because it conflicts with the builtin abs().
>
> Of course, absolute() is preferred for the same reason, and
> numpy.matlib.absolute() does exist.

Could numpy.matlib get the same functions as numpy? Would that have to
be done with a manually maintained import list?

I always use "import numpy.matlib as M" and then search for function
names in ipython (M.a[TAB]). I didn't realize that some functions are
missing.


From pgmdevlist at gmail.com  Sun Feb  4 20:33:01 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 4 Feb 2007 20:33:01 -0500
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>
Message-ID: <200702042033.01767.pgmdevlist@gmail.com>

On Sunday 04 February 2007 20:22:44 Jeremy Conlin wrote:
> I have subclassed the numpy.ndarray object, but need some help setting
> some attributes. I have read http://scipy.org/Subclasses but it
> doesn't provide the answer I am looking for.

Actually, yes: 
In the example given in http://scipy.org/Subclasses an attribute 'info' is 
defined from a class-generic one '__defaultinfo'. Just do the same thing with 
your 'edges'

def __new__(cls,...)
     ...
    (H, edges) = numpy.histogramdd(..)
    cls.__defaultedges = edges

def __array_finalize__(self, obj):
    if  not hasattr(self, 'edges'):
        self.edges = self.__defaultedges

That should the trick. 


From tim.leslie at gmail.com  Mon Feb  5 00:32:43 2007
From: tim.leslie at gmail.com (Tim Leslie)
Date: Mon, 5 Feb 2007 16:32:43 +1100
Subject: [Numpy-discussion] r3530 breaks nan_to_num for complex64 arrays
In-Reply-To: <d32b45350702041755n1960c3d2v6befbe6a9b6648ee@mail.gmail.com>
References: <d32b45350702041755n1960c3d2v6befbe6a9b6648ee@mail.gmail.com>
Message-ID: <d32b45350702042132w27a62397me92fa6c2ed23f993@mail.gmail.com>

On 2/5/07, Tim Leslie <tim.leslie at gmail.com> wrote:
> Hi All,
>
> As of svn revision 3530 N.nan_to_num no longer works for arrays of
> complex64. The actual error is raised in the signbit function, but I'm
> not sure why this is failing. If someone has a quick fix for this
> that'd be great, if not I'll lodge a full bug report when I get back
> from lunch :-).

Just noticed that this has already been reported.

http://projects.scipy.org/scipy/numpy/ticket/443

Cheers,

Tim

>
> Cheers,
>
> Tim
>
> In [1]: import numpy as N
>
> In [2]: a = N.ones((10, 10, 10), N.complex64)
>
> In [3]: N.nan_to_num(a)
> ---------------------------------------------------------------------------
> exceptions.TypeError                                 Traceback (most
> recent call last)
>
> /home/timl/<ipython console>
>
> /usr/lib/python2.4/site-packages/numpy/lib/type_check.py in nan_to_num(x)
>     130         else:
>     131             scalar = False
> --> 132         are_inf = isposinf(y)
>     133         are_neg_inf = isneginf(y)
>     134         are_nan = isnan(y)
>
> /usr/lib/python2.4/site-packages/numpy/lib/ufunclike.py in isposinf(x, y)
>      31     if y is None:
>      32         y = empty(x.shape, dtype=nx.bool_)
> ---> 33     umath.logical_and(isinf(x), ~signbit(x), y)
>      34     return y
>      35
>
> TypeError: function not supported for these types, and can't coerce
> safely to supported types
>


From robert.kern at gmail.com  Mon Feb  5 01:48:19 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 05 Feb 2007 00:48:19 -0600
Subject: [Numpy-discussion] python eats memory like the cookie monster
 eats	cookies
In-Reply-To: <850f7cbe0702041018s27f64d44n4f0f77245654eefb@mail.gmail.com>
References: <850f7cbe0702041018s27f64d44n4f0f77245654eefb@mail.gmail.com>
Message-ID: <45C6D333.7050808@gmail.com>

Niels Provos wrote:
> Good morning,
> 
> not sure if I got the right list, but I hope that somebody here will
> be able to shed some light on a Python-related memory problem.  The
> following code eats over >2GB of memory and fails with MemoyError
> after just a few iterations.

Here is a minimal example that demonstrates the problem. It appears that
assigning a numpy scalar to an element in an n-D Numeric array (where n is
strictly > 1) leaks memory. Neither the data-types of the Numeric array nor the
numpy scalar type seem to matter.


import Numeric
import numpy
a = Numeric.zeros([1, 1])
while True:
    a[0, 0] = numpy.int32(0)


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From klemm at phys.ethz.ch  Mon Feb  5 05:20:47 2007
From: klemm at phys.ethz.ch (Hanno Klemm)
Date: Mon, 05 Feb 2007 11:20:47 +0100
Subject: [Numpy-discussion] FFT definition
Message-ID: <twig.1170670847.18450@phys.ethz.ch>


Hi there,

I have a question regarding the definitions surrounding FFTs. The help
to numpy.fft.fft says:

>>> help(N.fft.fft)
Help on function fft in module numpy.fft.fftpack:

fft(a, n=None, axis=-1)
    fft(a, n=None, axis=-1)
    
    Will return the n point discrete Fourier transform of a. n
defaults to the
    length of a. If n is larger than a, then a will be zero-padded to
make up
    the difference. If n is smaller than a, the first n items in a will be
    used.
    
    The packing of the result is "standard": If A = fft(a, n), then A[0]
    contains the zero-frequency term, A[1:n/2+1] contains the
    positive-frequency terms, and A[n/2+1:] contains the
negative-frequency
    terms, in order of decreasingly negative frequency. So for an 8-point
    transform, the frequencies of the result are [ 0, 1, 2, 3, 4, -3,
-2, -1].
    
    This is most efficient for n a power of two. This also stores a
cache of
    working memory for different sizes of fft's, so you could
theoretically
    run into memory problems if you call this too many times with too many
    different n's.

>>> 

However, the help to numpy.fft.helper.fftfreq says:

>>> help(N.fft.helper.fftfreq)
Help on function fftfreq in module numpy.fft.helper:

fftfreq(n, d=1.0)
    fftfreq(n, d=1.0) -> f
    
    DFT sample frequencies
    
    The returned float array contains the frequency bins in
    cycles/unit (with zero at the start) given a window length n and a
    sample spacing d:
    
      f = [0,1,...,n/2-1,-n/2,...,-1]/(d*n)         if n is even
      f = [0,1,...,(n-1)/2,-(n-1)/2,...,-1]/(d*n)   if n is odd

>>> 

So one claims, that the packing goes from [0,1,...,n/2,-n/2+1,..,-1]
(fft) and the other one claims the frequencies go from
[0,1,...,n/2-1,-n/2,...-1] 

Is this inconsistent or am I missing something here?

Hanno

-- 
Hanno Klemm
klemm at phys.ethz.ch




From focke at slac.stanford.edu  Mon Feb  5 10:27:08 2007
From: focke at slac.stanford.edu (Warren Focke)
Date: Mon, 5 Feb 2007 07:27:08 -0800 (PST)
Subject: [Numpy-discussion] FFT definition
In-Reply-To: <twig.1170670847.18450@phys.ethz.ch>
References: <twig.1170670847.18450@phys.ethz.ch>
Message-ID: <Pine.LNX.4.58.0702050713560.23117@circinus.slac.stanford.edu>

The frequencies produced by the two recipies are not the same.  But the
DFT is periodic in both frequency and time.  So whether you think that the
number in bin in n/2 should correspond to frequency n/2 or -n/2, it's the
same number.

w

On Mon, 5 Feb 2007, Hanno Klemm wrote:

>
> Hi there,
>
> I have a question regarding the definitions surrounding FFTs. The help
> to numpy.fft.fft says:
>
> >>> help(N.fft.fft)
> Help on function fft in module numpy.fft.fftpack:
>
> fft(a, n=None, axis=-1)
>     fft(a, n=None, axis=-1)
>
>     Will return the n point discrete Fourier transform of a. n
> defaults to the
>     length of a. If n is larger than a, then a will be zero-padded to
> make up
>     the difference. If n is smaller than a, the first n items in a will be
>     used.
>
>     The packing of the result is "standard": If A = fft(a, n), then A[0]
>     contains the zero-frequency term, A[1:n/2+1] contains the
>     positive-frequency terms, and A[n/2+1:] contains the
> negative-frequency
>     terms, in order of decreasingly negative frequency. So for an 8-point
>     transform, the frequencies of the result are [ 0, 1, 2, 3, 4, -3,
> -2, -1].
>
>     This is most efficient for n a power of two. This also stores a
> cache of
>     working memory for different sizes of fft's, so you could
> theoretically
>     run into memory problems if you call this too many times with too many
>     different n's.
>
> >>>
>
> However, the help to numpy.fft.helper.fftfreq says:
>
> >>> help(N.fft.helper.fftfreq)
> Help on function fftfreq in module numpy.fft.helper:
>
> fftfreq(n, d=1.0)
>     fftfreq(n, d=1.0) -> f
>
>     DFT sample frequencies
>
>     The returned float array contains the frequency bins in
>     cycles/unit (with zero at the start) given a window length n and a
>     sample spacing d:
>
>       f = [0,1,...,n/2-1,-n/2,...,-1]/(d*n)         if n is even
>       f = [0,1,...,(n-1)/2,-(n-1)/2,...,-1]/(d*n)   if n is odd
>
> >>>
>
> So one claims, that the packing goes from [0,1,...,n/2,-n/2+1,..,-1]
> (fft) and the other one claims the frequencies go from
> [0,1,...,n/2-1,-n/2,...-1]
>
> Is this inconsistent or am I missing something here?
>
> Hanno
>
> --
> Hanno Klemm
> klemm at phys.ethz.ch
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From jeremit0 at gmail.com  Mon Feb  5 11:32:22 2007
From: jeremit0 at gmail.com (Jeremy Conlin)
Date: Mon, 5 Feb 2007 11:32:22 -0500
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <200702042033.01767.pgmdevlist@gmail.com>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>
	<200702042033.01767.pgmdevlist@gmail.com>
Message-ID: <3db594f70702050832y586069e5rb6186984b904f62@mail.gmail.com>

On 2/4/07, Pierre GM <pgmdevlist at gmail.com> wrote:
> On Sunday 04 February 2007 20:22:44 Jeremy Conlin wrote:
> > I have subclassed the numpy.ndarray object, but need some help setting
> > some attributes. I have read http://scipy.org/Subclasses but it
> > doesn't provide the answer I am looking for.
>
> Actually, yes:
> In the example given in http://scipy.org/Subclasses an attribute 'info' is
> defined from a class-generic one '__defaultinfo'. Just do the same thing with
> your 'edges'
>
> def __new__(cls,...)
>      ...
>     (H, edges) = numpy.histogramdd(..)
>     cls.__defaultedges = edges
>
> def __array_finalize__(self, obj):
>     if  not hasattr(self, 'edges'):
>         self.edges = self.__defaultedges
>
> That should the trick.

Thanks for clarifying that.  I didn't understand what the
__array_finalize__ did.

Jeremy


From kwgoodman at gmail.com  Mon Feb  5 11:45:39 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Mon, 5 Feb 2007 08:45:39 -0800
Subject: [Numpy-discussion] Memory leak in argsort?
Message-ID: <f4f93d420702050845r31f9a4f4ua92ab68607df30db@mail.gmail.com>

This eats up memory quickly on my system.

import numpy.matlib as M

def memleak():
    a = M.randn(500, 1)
    while True:
        a = a.argsort(0)


From faltet at carabos.com  Mon Feb  5 12:04:35 2007
From: faltet at carabos.com (Francesc Altet)
Date: Mon, 05 Feb 2007 18:04:35 +0100
Subject: [Numpy-discussion] Memory leak in argsort?
In-Reply-To: <f4f93d420702050845r31f9a4f4ua92ab68607df30db@mail.gmail.com>
References: <f4f93d420702050845r31f9a4f4ua92ab68607df30db@mail.gmail.com>
Message-ID: <1170695075.2543.12.camel@localhost.localdomain>

El dl 05 de 02 del 2007 a les 08:45 -0800, en/na Keith Goodman va
escriure:
> This eats up memory quickly on my system.
> 
> import numpy.matlib as M
> 
> def memleak():
>     a = M.randn(500, 1)
>     while True:
>         a = a.argsort(0)

Yeah, the guilty in this case is argsort():

http://projects.scipy.org/scipy/numpy/ticket/394

Travis fixed this in trunk (but forgot to close the ticket ;)

-- 
Francesc Altet    |  Be careful about using the following code --
Carabos Coop. V.  |  I've only proven that it works, 
www.carabos.com   |  I haven't tested it. -- Donald Knuth



From tim.hochberg at ieee.org  Mon Feb  5 12:09:04 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Mon, 5 Feb 2007 10:09:04 -0700
Subject: [Numpy-discussion] FFT definition
In-Reply-To: <twig.1170670847.18450@phys.ethz.ch>
References: <twig.1170670847.18450@phys.ethz.ch>
Message-ID: <e4412d6b0702050909i2878fefaj579eb7bf3980a13b@mail.gmail.com>

On 2/5/07, Hanno Klemm <klemm at phys.ethz.ch> wrote:
[numpy.fft[

    The packing of the result is "standard": If A = fft(a, n), then A[0]
>     contains the zero-frequency term, A[1:n/2+1] contains the
>     positive-frequency terms, and A[n/2+1:] contains the
> negative-frequency
>     terms, in order of decreasingly negative frequency. So for an 8-point
>     transform, the frequencies of the result are [ 0, 1, 2, 3, 4, -3,
> -2, -1].


[scipy.fft]


      f = [0,1,...,n/2-1,-n/2,...,-1]/(d*n)         if n is even
>       f = [0,1,...,(n-1)/2,-(n-1)/2,...,-1]/(d*n)   if n is odd
>
> >>>
>
> So one claims, that the packing goes from [0,1,...,n/2,-n/2+1,..,-1]
> (fft) and the other one claims the frequencies go from
> [0,1,...,n/2-1,-n/2,...-1]
>
> Is this inconsistent or am I missing something here?


Both, I think.

In the even case, the frequency at n/2 is shared by both the positive
frequencies, so for that case things are consistent if not terribly clear.
For the odd case, this is not true, and the scipy docs look correct in this
case, while the numpy docs appear to assign an extra frequency to the
positive branch. Of course that's not the one you were complaining about
;-).

To be super pedantic, the discrete Fourier transform is periodic, so all of
the frequencies can be regarded as positive or negative. That's not
generally useful, since the assumptions that go into the DFT that make it
periodic don't usually apply to the signal that you are sampling. Then again
the results of DFTs are typicallly either small or silly in the vicinity of
N//2.


//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070205/01cc0e33/attachment.html>

From kwgoodman at gmail.com  Mon Feb  5 12:20:39 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Mon, 5 Feb 2007 09:20:39 -0800
Subject: [Numpy-discussion] Memory leak in argsort?
In-Reply-To: <1170695075.2543.12.camel@localhost.localdomain>
References: <f4f93d420702050845r31f9a4f4ua92ab68607df30db@mail.gmail.com>
	<1170695075.2543.12.camel@localhost.localdomain>
Message-ID: <f4f93d420702050920g764b7e7dy617799aa7e9a8432@mail.gmail.com>

On 2/5/07, Francesc Altet <faltet at carabos.com> wrote:
> El dl 05 de 02 del 2007 a les 08:45 -0800, en/na Keith Goodman va
> escriure:
> > This eats up memory quickly on my system.
> >
> > import numpy.matlib as M
> >
> > def memleak():
> >     a = M.randn(500, 1)
> >     while True:
> >         a = a.argsort(0)
>
> Yeah, the guilty in this case is argsort():
>
> http://projects.scipy.org/scipy/numpy/ticket/394

The first page of trac search results for argsort shows the ticket.
Now I know to check there first.

Will the latest numpy from svn work with matplotlib 0.87.7?


From pgmdevlist at gmail.com  Mon Feb  5 12:25:00 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 5 Feb 2007 12:25:00 -0500
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <3db594f70702050832y586069e5rb6186984b904f62@mail.gmail.com>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>
	<200702042033.01767.pgmdevlist@gmail.com>
	<3db594f70702050832y586069e5rb6186984b904f62@mail.gmail.com>
Message-ID: <200702051225.00780.pgmdevlist@gmail.com>

On Monday 05 February 2007 11:32:22 Jeremy Conlin wrote:
> Thanks for clarifying that.  I didn't understand what the
> __array_finalize__ did.

That means I should clarify some points on the wiki, then. 
A good exercise is to put some temporary comments in your code in __new__ and 
__array_finalize__, to show when these methods are called and how (that's how 
I learned)

Thinking about it, the example you gave can't work. Your __new__ method 
returns H, viz, a pure ndarray. There won't be any call to __array_finalize__ 
in that case, which is not what you want. Force the call by accessing a view 
of your array:

class myhistog(N.ndarray):
    def __new__(self, iniarray, inibin):
        (H,edges) = N.histogramdd(iniarray,inibin)
        self._defedges = edges
        return H.view(self)

Now, you do return a 'myhistog' class, not a pure 'ndarray', and 
__array_finalize__ is called.

    def __array_finalize__(self, obj):
        print "__array_finalize__ got %s as %s" % (obj, type(obj))
        if not hasattr(self, 'edges'):
            self.edges = self._defedges
	myhistog._defedges = None

Note the last line: you reset the class default to None (if this is what you 
want). Otherwise, new 'myhistog' objects wil inherit the previous edges.



From robert.kern at gmail.com  Mon Feb  5 12:26:16 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 05 Feb 2007 11:26:16 -0600
Subject: [Numpy-discussion] Memory leak in argsort?
In-Reply-To: <f4f93d420702050920g764b7e7dy617799aa7e9a8432@mail.gmail.com>
References: <f4f93d420702050845r31f9a4f4ua92ab68607df30db@mail.gmail.com>	<1170695075.2543.12.camel@localhost.localdomain>
	<f4f93d420702050920g764b7e7dy617799aa7e9a8432@mail.gmail.com>
Message-ID: <45C768B8.1030405@gmail.com>

Keith Goodman wrote:
> Will the latest numpy from svn work with matplotlib 0.87.7?

It should. We are committed to backwards compatibility both at the Python level
and the C binary level.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From focke at slac.stanford.edu  Mon Feb  5 12:53:04 2007
From: focke at slac.stanford.edu (Warren Focke)
Date: Mon, 5 Feb 2007 09:53:04 -0800 (PST)
Subject: [Numpy-discussion] FFT definition
In-Reply-To: <e4412d6b0702050909i2878fefaj579eb7bf3980a13b@mail.gmail.com>
References: <twig.1170670847.18450@phys.ethz.ch>
	<e4412d6b0702050909i2878fefaj579eb7bf3980a13b@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0702050939450.23613@circinus.slac.stanford.edu>



On Mon, 5 Feb 2007, Timothy Hochberg wrote:

> On 2/5/07, Hanno Klemm <klemm at phys.ethz.ch> wrote:
> [numpy.fft[
>
>     The packing of the result is "standard": If A = fft(a, n), then A[0]
> >     contains the zero-frequency term, A[1:n/2+1] contains the
> >     positive-frequency terms, and A[n/2+1:] contains the
> > negative-frequency
> >     terms, in order of decreasingly negative frequency. So for an 8-point
> >     transform, the frequencies of the result are [ 0, 1, 2, 3, 4, -3,
> > -2, -1].
>
>
> [scipy.fft]
>
>
>       f = [0,1,...,n/2-1,-n/2,...,-1]/(d*n)         if n is even
> >       f = [0,1,...,(n-1)/2,-(n-1)/2,...,-1]/(d*n)   if n is odd
> >
> > >>>
> >
> > So one claims, that the packing goes from [0,1,...,n/2,-n/2+1,..,-1]
> > (fft) and the other one claims the frequencies go from
> > [0,1,...,n/2-1,-n/2,...-1]
> >
> > Is this inconsistent or am I missing something here?
>
>
> Both, I think.
>
> In the even case, the frequency at n/2 is shared by both the positive
> frequencies, so for that case things are consistent if not terribly clear.
> For the odd case, this is not true, and the scipy docs look correct in this
> case, while the numpy docs appear to assign an extra frequency to the
> positive branch. Of course that's not the one you were complaining about
> ;-).

Extra frequency where?

(numpy 1.0, debian sarge)
>>> n=9
>>> A=arange(n)

numpy docs:
>>> A[1:n/2+1]
array([1, 2, 3, 4])
>>> A[n/2+1:]
array([5, 6, 7, 8])

scipy docs:
>>> (n-1)/2
4
>>> -(n-1)/2
-4

Note that in the odd-n case, there is no Nyquist term.  If
F = fft(f), len(f) == 9
then F[-4] != F[4] (F[5] == F[-4] by periodicty in frequency)

w

>
> To be super pedantic, the discrete Fourier transform is periodic, so all of
> the frequencies can be regarded as positive or negative. That's not
> generally useful, since the assumptions that go into the DFT that make it
> periodic don't usually apply to the signal that you are sampling. Then again
> the results of DFTs are typicallly either small or silly in the vicinity of
> N//2.
>
>
> //=][=\\
>
> tim.hochberg at ieee.org
>


From tim.hochberg at ieee.org  Mon Feb  5 13:13:29 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Mon, 5 Feb 2007 11:13:29 -0700
Subject: [Numpy-discussion] FFT definition
In-Reply-To: <Pine.LNX.4.58.0702050939450.23613@circinus.slac.stanford.edu>
References: <twig.1170670847.18450@phys.ethz.ch>
	<e4412d6b0702050909i2878fefaj579eb7bf3980a13b@mail.gmail.com>
	<Pine.LNX.4.58.0702050939450.23613@circinus.slac.stanford.edu>
Message-ID: <e4412d6b0702051013p793457c2q46afc44b6fa82516@mail.gmail.com>

On 2/5/07, Warren Focke <focke at slac.stanford.edu> wrote:
>
>
>
> On Mon, 5 Feb 2007, Timothy Hochberg wrote:
>
> > On 2/5/07, Hanno Klemm <klemm at phys.ethz.ch> wrote:
> > [numpy.fft[
> >
> >     The packing of the result is "standard": If A = fft(a, n), then A[0]
> > >     contains the zero-frequency term, A[1:n/2+1] contains the
> > >     positive-frequency terms, and A[n/2+1:] contains the
> > > negative-frequency
> > >     terms, in order of decreasingly negative frequency. So for an
> 8-point
> > >     transform, the frequencies of the result are [ 0, 1, 2, 3, 4, -3,
> > > -2, -1].
> >
> >
> > [scipy.fft]
> >
> >
> >       f = [0,1,...,n/2-1,-n/2,...,-1]/(d*n)         if n is even
> > >       f = [0,1,...,(n-1)/2,-(n-1)/2,...,-1]/(d*n)   if n is odd
> > >
> > > >>>
> > >
> > > So one claims, that the packing goes from [0,1,...,n/2,-n/2+1,..,-1]
> > > (fft) and the other one claims the frequencies go from
> > > [0,1,...,n/2-1,-n/2,...-1]
> > >
> > > Is this inconsistent or am I missing something here?
> >
> >
> > Both, I think.
> >
> > In the even case, the frequency at n/2 is shared by both the positive
> > frequencies, so for that case things are consistent if not terribly
> clear.
> > For the odd case, this is not true, and the scipy docs look correct in
> this
> > case, while the numpy docs appear to assign an extra frequency to the
> > positive branch. Of course that's not the one you were complaining about
> > ;-).
>
> Extra frequency where?
>
> (numpy 1.0, debian sarge)
> >>> n=9
> >>> A=arange(n)
>
> numpy docs:
> >>> A[1:n/2+1]
> array([1, 2, 3, 4])
> >>> A[n/2+1:]
> array([5, 6, 7, 8])
>
> scipy docs:
> >>> (n-1)/2
> 4
> >>> -(n-1)/2
> -4
>
> Note that in the odd-n case, there is no Nyquist term.  If
> F = fft(f), len(f) == 9
> then F[-4] != F[4] (F[5] == F[-4] by periodicty in frequency)



Ooops, you're right I worked this through on paper and just blew it. I
suppose I'd look less silly had I actually checked the results using the
interpreter.


-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070205/a9e1e709/attachment.html>

From jeremit0 at gmail.com  Mon Feb  5 13:13:50 2007
From: jeremit0 at gmail.com (Jeremy Conlin)
Date: Mon, 5 Feb 2007 13:13:50 -0500
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <200702051225.00780.pgmdevlist@gmail.com>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>
	<200702042033.01767.pgmdevlist@gmail.com>
	<3db594f70702050832y586069e5rb6186984b904f62@mail.gmail.com>
	<200702051225.00780.pgmdevlist@gmail.com>
Message-ID: <3db594f70702051013x181cdc81x6a21dd471657d105@mail.gmail.com>

On 2/5/07, Pierre GM <pgmdevlist at gmail.com> wrote:
> On Monday 05 February 2007 11:32:22 Jeremy Conlin wrote:
> > Thanks for clarifying that.  I didn't understand what the
> > __array_finalize__ did.
>
> That means I should clarify some points on the wiki, then.
> A good exercise is to put some temporary comments in your code in __new__ and
> __array_finalize__, to show when these methods are called and how (that's how
> I learned)
>
> Thinking about it, the example you gave can't work. Your __new__ method
> returns H, viz, a pure ndarray. There won't be any call to __array_finalize__
> in that case, which is not what you want. Force the call by accessing a view
> of your array:
>
> class myhistog(N.ndarray):
>     def __new__(self, iniarray, inibin):
>         (H,edges) = N.histogramdd(iniarray,inibin)
>         self._defedges = edges
>         return H.view(self)
>
> Now, you do return a 'myhistog' class, not a pure 'ndarray', and
> __array_finalize__ is called.
>
>     def __array_finalize__(self, obj):
>         print "__array_finalize__ got %s as %s" % (obj, type(obj))
>         if not hasattr(self, 'edges'):
>             self.edges = self._defedges
>         myhistog._defedges = None
>
> Note the last line: you reset the class default to None (if this is what you
> want). Otherwise, new 'myhistog' objects wil inherit the previous edges.


Excellent now it does what I want!  But it raises more questions.
What exactly is a "view" of H?

Thanks again,
Jeremy


From millman at berkeley.edu  Tue Feb  6 01:14:15 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 5 Feb 2007 22:14:15 -0800
Subject: [Numpy-discussion] Fwd: [Rpy] status of numpy version of rpy
In-Reply-To: <0D56E923-C9FE-479C-8552-370BCD0F3EAD@rochester.edu>
References: <c7009a550702030003i1dd21ac6r70e7d848705b40c4@mail.gmail.com>
	<0D56E923-C9FE-479C-8552-370BCD0F3EAD@rochester.edu>
Message-ID: <c7009a550702052214v68776617y7b735e36e728498@mail.gmail.com>

Hey,

After reading Travis' email about getting more projects ported to
Numpy, I sent an email to the RPy developer's asking when they would
commit Travis' patch.  Anyway, George Warnes replied that he would
apply it this week (see the forwarded email below).

I would encourage anyone who is using a project that hasn't converted
to Numpy to send the developer's an email letting them know that you
want them to move the project to Numpy as soon as possible.

Thanks to all the Numpy developers, and in particular to Travis, for
working so hard to make Numpy so great.

Jarrod

---------- Forwarded message ----------
From: Gregory. R. Warnes <greg at warnes.net>
Date: Feb 5, 2007 7:33 AM
Subject: Re: [Rpy] status of numpy version of rpy
To: "RPy help,  support and design discussion list"
<rpy-list at lists.sourceforge.net>


Hi Jarrod,

The Numpy patch should be applied sometime this week.

-Greg

On Feb 3, 2007, at 3:03 AM, Jarrod Millman wrote:

> Hello,
>
> Thanks for all the great work.  I am involved with the Neuroimaging in
> Python project, which uses numpy.  Numpy is very stable and much
> better than Numeric.  The problem is that some of us are also using
> RPy for our data processing.  So we are using Travis' numpy patch for
> RPy.  It seems to work perfectly.  This process is acceptable for our
> developers, but creates additional installation headaches for our
> regular users.
>
> Do you have a timeline for when you will apply the numpy patch?  Is
> there anything you need help with before applying the patch?
>
> Thanks,
>
> --
> Jarrod Millman
> Computational Infrastructure for Research Labs
> 10 Giannini Hall, UC Berkeley
> phone: 510.643.4014
> http://cirl.berkeley.edu/
>
> ----------------------------------------------------------------------
> ---
> Using Tomcat but need to do more? Need to support web services,
> security?
> Get stuff done quickly with pre-integrated technology to make your
> job easier.
> Download IBM WebSphere Application Server v.1.0.1 based on Apache
> Geronimo
> http://sel.as-us.falkag.net/sel?
> cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> rpy-list mailing list
> rpy-list at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/rpy-list


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
rpy-list mailing list
rpy-list at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rpy-list


-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From millman at berkeley.edu  Tue Feb  6 01:41:18 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 5 Feb 2007 22:41:18 -0800
Subject: [Numpy-discussion] NumPy compatible version of ScientificPython
Message-ID: <c7009a550702052241j34760fc9n1f49392e531842df@mail.gmail.com>

The current development release of ScientificPython (2.7.3) supports
NumPy.  The release notes are here:
http://sourcesup.cru.fr/frs/shownotes.php?release_id=634

I updated the "Porting to NumPy" wiki page:
http://www.scipy.org/Porting_to_NumPy

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From pjssilva at ime.usp.br  Tue Feb  6 05:30:26 2007
From: pjssilva at ime.usp.br (Paulo J. S. Silva)
Date: Tue, 06 Feb 2007 08:30:26 -0200
Subject: [Numpy-discussion] numpy.matlib.abs
In-Reply-To: <f4f93d420702041728t61c37c7ak40b01c0a5e74d574@mail.gmail.com>
References: <f4f93d420702041706t2a695f55h8c6a6d5980e70b96@mail.gmail.com>
	<45C6848C.4060105@gmail.com>
	<f4f93d420702041728t61c37c7ak40b01c0a5e74d574@mail.gmail.com>
Message-ID: <1170757826.21666.8.camel@localhost.localdomain>

Em Dom, 2007-02-04 ?s 17:28 -0800, Keith Goodman escreveu:

> Could numpy.matlib get the same functions as numpy? Would that have to
> be done with a manually maintained import list?
> I always use "import numpy.matlib as M" and then search for function
> names in ipython (M.a[TAB]). I didn't realize that some functions are
> missing.

As the list knows, I am trying to build a special module that can
convert any other module to behave nicely with matrices. I have special
interest in using it as an interface to scipy modules that may return
arrays when given a matrix. This effort let me to learn some tricks
about modules imports in Python.

I believe that if you add the following code to the end of matlib.py
file it will behave just like you want without any manual intervention:

--- Start Python code ---

import inspect
import matlib as M
for i in dir(N):
    attribute = getattr(N, i)
    if type(attribute) is N.ufunc or inspect.isroutine(attribute):
        try:
            getattr(M, i)
        except AttributeError:
            setattr(M, i, attribute)

--- End Python code ---

Here is an ipython session:

--- ipython session ---

In [1]:import numpy.matlib as M

In [2]:M.abs
Out[2]:<ufunc 'absolute'>

--- End of ipython sesssion ---

By the way, there were only four functions that are missing without this
code: abs, max, min, and round. You can see this by adding a "print i"
in the except block above.

If the list thinks this code is useful, I am donating it to numpy.

Best,

Paulo



From sturla at molden.no  Tue Feb  6 07:06:37 2007
From: sturla at molden.no (Sturla Molden)
Date: Tue, 6 Feb 2007 13:06:37 +0100 (CET)
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <200702042033.01767.pgmdevlist@gmail.com>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>
	<200702042033.01767.pgmdevlist@gmail.com>
Message-ID: <2558.129.240.194.142.1170763597.squirrel@webmail.uio.no>


> def __new__(cls,...)
>      ...
>     (H, edges) = numpy.histogramdd(..)
>     cls.__defaultedges = edges
>
> def __array_finalize__(self, obj):
>     if  not hasattr(self, 'edges'):
>         self.edges = self.__defaultedges

So in order to get an instance attribute, one has to temporarily define it
as a class attribute? What happens if there is a thread switch between
__new__ and __array_finalize__? This design is not thread safe and can
produce strange race conditions.

IMHO, the preferred way to set an instance attribute is to use __init__
method, which is the 'Pythonic' way to do it.

Sturla Molden






From stefan at sun.ac.za  Tue Feb  6 07:56:01 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Tue, 6 Feb 2007 14:56:01 +0200
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <2558.129.240.194.142.1170763597.squirrel@webmail.uio.no>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>
	<200702042033.01767.pgmdevlist@gmail.com>
	<2558.129.240.194.142.1170763597.squirrel@webmail.uio.no>
Message-ID: <20070206125600.GN6274@mentat.za.net>

On Tue, Feb 06, 2007 at 01:06:37PM +0100, Sturla Molden wrote:
> 
> > def __new__(cls,...)
> >      ...
> >     (H, edges) = numpy.histogramdd(..)
> >     cls.__defaultedges = edges
> >
> > def __array_finalize__(self, obj):
> >     if  not hasattr(self, 'edges'):
> >         self.edges = self.__defaultedges
> 
> IMHO, the preferred way to set an instance attribute is to use __init__
> method, which is the 'Pythonic' way to do it.

I don't pretend to know all the inner workings of subclassing, but I
don't think that would work, given the following output:

In [1]: import numpy as N

In [2]: import numpy as N

In [3]: 

In [3]: class MyArray(N.ndarray):
   ...:     def __new__(cls,data):
   ...:         return N.asarray(data).view(cls)
   ...:     
   ...:     def __init__(self,obj):
   ...:         print "This is where __init__ is called"
   ...:         
   ...:     def __array_finalize__(self,obj):
   ...:         print "This is where __array_finalize__ is called"
   ...: 

In [4]: x = MyArray(3)
This is where __array_finalize__ is called
This is where __init__ is called

In [5]: y = N.array([1,2,3])

In [6]: x+y
This is where __array_finalize__ is called
Out[6]: MyArray([4, 5, 6])


Regards
St?fan


From jeremit0 at gmail.com  Tue Feb  6 08:14:13 2007
From: jeremit0 at gmail.com (Jeremy Conlin)
Date: Tue, 6 Feb 2007 08:14:13 -0500
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <2558.129.240.194.142.1170763597.squirrel@webmail.uio.no>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>
	<200702042033.01767.pgmdevlist@gmail.com>
	<2558.129.240.194.142.1170763597.squirrel@webmail.uio.no>
Message-ID: <3db594f70702060514k6c78dcc5lbfa216ff0521450f@mail.gmail.com>

On 2/6/07, Sturla Molden <sturla at molden.no> wrote:
>
> > def __new__(cls,...)
> >      ...
> >     (H, edges) = numpy.histogramdd(..)
> >     cls.__defaultedges = edges
> >
> > def __array_finalize__(self, obj):
> >     if  not hasattr(self, 'edges'):
> >         self.edges = self.__defaultedges
>
> So in order to get an instance attribute, one has to temporarily define it
> as a class attribute? What happens if there is a thread switch between
> __new__ and __array_finalize__? This design is not thread safe and can
> produce strange race conditions.
>
> IMHO, the preferred way to set an instance attribute is to use __init__
> method, which is the 'Pythonic' way to do it.
>
> Sturla Molden
>
Yes using __init__ to set an instance attribute is the Pythonic way to
do this.  However, I calculate/create the data in __new__.  The data
is unavailable to __init__.

Jeremy


From sturla at molden.no  Tue Feb  6 09:01:37 2007
From: sturla at molden.no (Sturla Molden)
Date: Tue, 6 Feb 2007 15:01:37 +0100 (CET)
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <3db594f70702060514k6c78dcc5lbfa216ff0521450f@mail.gmail.com>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>
	<200702042033.01767.pgmdevlist@gmail.com>
	<2558.129.240.194.142.1170763597.squirrel@webmail.uio.no>
	<3db594f70702060514k6c78dcc5lbfa216ff0521450f@mail.gmail.com>
Message-ID: <2655.129.240.194.142.1170770497.squirrel@webmail.uio.no>


> Yes using __init__ to set an instance attribute is the Pythonic way to
> do this.  However, I calculate/create the data in __new__.  The data
> is unavailable to __init__.

The signatures of __new__ and __init__ is:

def __new__(cls, *args, **kwds)
def __init__(self, *args, **kwds)

If __new__ has access to the data, __init__ has access to the data as
well. But in order for __init__ to be called, it must return an instance
of cls. Otherwise, Python leaves the class as returned by __new__.

But it remains that the subclassing example is not thread safe. The only
way to make it thread safe would be if __new__ sets a global lock and
__array_finalize_
releases it. I think NumPy can get away with this because it holds the GIL
inside its C extension, but when you subclass ndarray in Python, the GIL
is released.







From sturla at molden.no  Tue Feb  6 09:17:39 2007
From: sturla at molden.no (Sturla Molden)
Date: Tue, 6 Feb 2007 15:17:39 +0100 (CET)
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <20070206125600.GN6274@mentat.za.net>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>
	<200702042033.01767.pgmdevlist@gmail.com>
	<2558.129.240.194.142.1170763597.squirrel@webmail.uio.no>
	<20070206125600.GN6274@mentat.za.net>
Message-ID: <2663.129.240.194.142.1170771459.squirrel@webmail.uio.no>


> I don't pretend to know all the inner workings of subclassing, but I
> don't think that would work, given the following output:

> In [6]: x+y
> This is where __array_finalize__ is called
> Out[6]: MyArray([4, 5, 6])

Why is not __new__ called for the return value of x + y? Does it call
__new__ for ndarray instead of MyArray?







From kwgoodman at gmail.com  Tue Feb  6 09:43:48 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Tue, 6 Feb 2007 06:43:48 -0800
Subject: [Numpy-discussion] numpy.matlib.abs
In-Reply-To: <1170757826.21666.8.camel@localhost.localdomain>
References: <f4f93d420702041706t2a695f55h8c6a6d5980e70b96@mail.gmail.com>
	<45C6848C.4060105@gmail.com>
	<f4f93d420702041728t61c37c7ak40b01c0a5e74d574@mail.gmail.com>
	<1170757826.21666.8.camel@localhost.localdomain>
Message-ID: <f4f93d420702060643r22a646b4p6f813176824e0577@mail.gmail.com>

On 2/6/07, Paulo J. S. Silva <pjssilva at ime.usp.br> wrote:
> Em Dom, 2007-02-04 ?s 17:28 -0800, Keith Goodman escreveu:
>
> > Could numpy.matlib get the same functions as numpy? Would that have to
> > be done with a manually maintained import list?
> > I always use "import numpy.matlib as M" and then search for function
> > names in ipython (M.a[TAB]). I didn't realize that some functions are
> > missing.
>
> As the list knows, I am trying to build a special module that can
> convert any other module to behave nicely with matrices. I have special
> interest in using it as an interface to scipy modules that may return
> arrays when given a matrix. This effort let me to learn some tricks
> about modules imports in Python.
>
> I believe that if you add the following code to the end of matlib.py
> file it will behave just like you want without any manual intervention:
>
> --- Start Python code ---
>
> import inspect
> import matlib as M
> for i in dir(N):
>     attribute = getattr(N, i)
>     if type(attribute) is N.ufunc or inspect.isroutine(attribute):
>         try:
>             getattr(M, i)
>         except AttributeError:
>             setattr(M, i, attribute)
>
> --- End Python code ---
>
> Here is an ipython session:
>
> --- ipython session ---
>
> In [1]:import numpy.matlib as M
>
> In [2]:M.abs
> Out[2]:<ufunc 'absolute'>
>
> --- End of ipython sesssion ---
>
> By the way, there were only four functions that are missing without this
> code: abs, max, min, and round. You can see this by adding a "print i"
> in the except block above.
>
> If the list thinks this code is useful, I am donating it to numpy.

That is great.

I could think of a few uses for abs, max, min, and round. So I would
like to see them imported.

BTW, why can I do

x.max()
x.min()
x.round()

but not

x.abs()?


From haase at msg.ucsf.edu  Tue Feb  6 17:29:38 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue, 6 Feb 2007 14:29:38 -0800
Subject: [Numpy-discussion] memmap on 64bit Linux for > 2 GB files
Message-ID: <bc657ead0702061429g3b537b81o8fbd03e1522d66e4@mail.gmail.com>

Hi,
I finally tried to do the test, to memmap a large file
filesize: 2.8G

a memmap call gives this error:

{{{
>>> N.memmap('20050622-1648-Y_DEMO-1')
Traceback (most recent call last):
  File "<input>", line 1, in ?
  File "/jws30/haase/PrLinN64/numpy/core/memmap.py", line 67, in __new__
    mm = mmap.mmap(fid.fileno(), bytes, access=acc)
OverflowError: memory mapped size is too large (limited by C int)
}}}

I'm using a recent numpy on a 64bit Linux (debian etch, kernel:
2.6.16-2-em64t-p4-smp)
{{{
>>> N.__version__
'1.0.2.dev3509'
>>> N.int0
<type 'numpy.int64'>
}}}

Is this supposed to work ?

Thanks,
Sebastian


From robert.kern at gmail.com  Tue Feb  6 17:35:04 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 06 Feb 2007 16:35:04 -0600
Subject: [Numpy-discussion] memmap on 64bit Linux for > 2 GB files
In-Reply-To: <bc657ead0702061429g3b537b81o8fbd03e1522d66e4@mail.gmail.com>
References: <bc657ead0702061429g3b537b81o8fbd03e1522d66e4@mail.gmail.com>
Message-ID: <45C90298.4060705@gmail.com>

Sebastian Haase wrote:
> Hi,
> I finally tried to do the test, to memmap a large file
> filesize: 2.8G
> 
> a memmap call gives this error:
> 
> {{{
>>>> N.memmap('20050622-1648-Y_DEMO-1')
> Traceback (most recent call last):
>   File "<input>", line 1, in ?
>   File "/jws30/haase/PrLinN64/numpy/core/memmap.py", line 67, in __new__
>     mm = mmap.mmap(fid.fileno(), bytes, access=acc)
> OverflowError: memory mapped size is too large (limited by C int)
> }}}
> 
> I'm using a recent numpy on a 64bit Linux (debian etch, kernel:
> 2.6.16-2-em64t-p4-smp)
> {{{
>>>> N.__version__
> '1.0.2.dev3509'
>>>> N.int0
> <type 'numpy.int64'>
> }}}
> 
> Is this supposed to work ?

You need Python 2.5 for it to work.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From seb.haase at gmx.net  Tue Feb  6 17:44:26 2007
From: seb.haase at gmx.net (Sebastian Haase)
Date: Tue, 6 Feb 2007 14:44:26 -0800
Subject: [Numpy-discussion] memmap on 64bit Linux for > 2 GB files
In-Reply-To: <45C90298.4060705@gmail.com>
References: <bc657ead0702061429g3b537b81o8fbd03e1522d66e4@mail.gmail.com>
	<45C90298.4060705@gmail.com>
Message-ID: <bc657ead0702061444v6225b2a6l1b9e74571bbe11ed@mail.gmail.com>

Of course !
Now I remember why I didn't test it yet...

Thanks,
-Sebastian

On 2/6/07, Robert Kern <robert.kern at gmail.com> wrote:
> Sebastian Haase wrote:
> > Hi,
> > I finally tried to do the test, to memmap a large file
> > filesize: 2.8G
> >
> > a memmap call gives this error:
> >
> > {{{
> >>>> N.memmap('20050622-1648-Y_DEMO-1')
> > Traceback (most recent call last):
> >   File "<input>", line 1, in ?
> >   File "/jws30/haase/PrLinN64/numpy/core/memmap.py", line 67, in __new__
> >     mm = mmap.mmap(fid.fileno(), bytes, access=acc)
> > OverflowError: memory mapped size is too large (limited by C int)
> > }}}
> >
> > I'm using a recent numpy on a 64bit Linux (debian etch, kernel:
> > 2.6.16-2-em64t-p4-smp)
> > {{{
> >>>> N.__version__
> > '1.0.2.dev3509'
> >>>> N.int0
> > <type 'numpy.int64'>
> > }}}
> >
> > Is this supposed to work ?
>
> You need Python 2.5 for it to work.
>


From tom.denniston at alum.dartmouth.org  Tue Feb  6 19:09:20 2007
From: tom.denniston at alum.dartmouth.org (Tom Denniston)
Date: Tue, 6 Feb 2007 18:09:20 -0600
Subject: [Numpy-discussion] bug in numpy.equal?
Message-ID: <d6f2d3dd0702061609i2b494496j72a3e01c7410e2df@mail.gmail.com>

The behavior below seems strange to me.  The string array is type S3
yet it says that comparison with 'abc' is not implemented.  The ==
operator seems to work though.  Is there a subtlty I am missing or is
it simply a bug?


In [1]: import numpy


In [2]: numpy.equal(numpy.array(['abc', 'def']), 'abc')

Out[2]: NotImplemented



In [3]: numpy.array(['abc', 'def']) == 'abc'

Out[3]: array([ True, False], dtype=bool)



In [4]: numpy.equal?

Type:           ufunc

Base Class:     <type 'numpy.ufunc'>

String Form:    <ufunc 'equal'>

Namespace:      Interactive

Docstring:

    y = equal(x1,x2) returns elementwise x1 == x2 in a bool array


From gsteele at qualcomm.com  Tue Feb  6 19:28:26 2007
From: gsteele at qualcomm.com (Steele, Greg)
Date: Tue, 6 Feb 2007 16:28:26 -0800
Subject: [Numpy-discussion] importing multiarraymodule.c in Python embedding
Message-ID: <4D7539987C6B3340ADA6BC7AE609F73606FC8F@NAEX14.na.qualcomm.com>

I have run into an interesting issue with multiarraymodule.c with
regards to embedding Python in a C application on Windows XP. In the
application, the following abbreviated sequence was executed

 

1) Py_Initialize

2) numpy is imported

3) Py_Finalize

4) Py_Initialize

5) attempt to import numpy results in an error.

 

I have tracked this down to two things

1) Python does not call the FreeLibrary windows API on C extension
modules when shutting down

2) multiarraymodule.c has a static variable _multiarray_module_loaded
which, when set, bypasses the call to Py_InitModule

 

The error at step 5 above occurs because the multiarraymodule DLL is
loaded at step 2, the _multiarray_module_loaded variable is set at step
2, and Py_InitModule is bypassed at step 5. Since Py_InitModule is not
called, multiarraymodule is not placed in the module dictionary (i.e.
sys.modules) and an error occurs in importdl.h which checks if the
module is there.

 

The obvious solution is to not execute the sequence above.

 

This issue does seem to highlight a weakness in the implementation of
multiarraymodule.c. Could someone comment on the need for static
variable _multiarray_module_loaded? Is there a more robust way to
achieve the goal?

 

Thanks.

 

Greg

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070206/d8ba40f8/attachment.html>

From mattknox_ca at hotmail.com  Tue Feb  6 19:54:32 2007
From: mattknox_ca at hotmail.com (Matt Knox)
Date: Tue, 6 Feb 2007 19:54:32 -0500
Subject: [Numpy-discussion] thread safe subclasses of ndarray
Message-ID: <BAY118-W6FF8C87F0823BC22566B68C9E0@phx.gbl>


Sturla Molden brought up a point in an earlier thread ("Please help with subclassing numpy.ndarray") that I think is worth highlighting. A "common" approach to subclassing ndarray in python (http://www.scipy.org/Subclasses) results in code that is not thread safe. This approach involves setting some class level variables inside the __new__ method, and then retrieving those values inside the __array_finalize__ method.
 
A simple work around that Sturla alluded to is to explictly set a lock inside the __new__ method, and later release it in the __array_finalize__ method. However, this would cause problems if someone invoked the __new__ method in such a way that __array_finalize__ was not called afterwards (which I think is possible, isn't it?).
 
The two subclasses of ndarray in the core numpy distribution (that I am aware of) , chararray and matrix, don't run into this problem because they can get all the relevant info they need by directly inspecting the ndarray that comes out of the __new__ method, and so they don't need to share any extra info between __new__ and __array_finalize__
 
So, I am wondering if perhaps one of the experts on this list would be able to shed some light on a good way to create thread safe subclasses of ndarray.
 
Thanks,
 
- Matt Knox
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070206/24174a87/attachment.html>

From ckkart at hoc.net  Wed Feb  7 05:35:14 2007
From: ckkart at hoc.net (Christian)
Date: Wed, 7 Feb 2007 10:35:14 +0000 (UTC)
Subject: [Numpy-discussion] force column vector
Message-ID: <loom.20070207T113019-604@post.gmane.org>

Hi,

when creating an ndarray from a list, how can I force the result to be
2d *and* a column vector? So in case I pass a nested list, there will be no
modification of the shape and when I pass a simple list, it will be 
converted to a 2d column vector. I can only think of a solution using 'if'
clauses but I suppose there is a more elegant way.

Thanks, Christian





From bardeau at iram.fr  Wed Feb  7 07:51:31 2007
From: bardeau at iram.fr (Sebastien Bardeau)
Date: Wed, 07 Feb 2007 13:51:31 +0100
Subject: [Numpy-discussion] importing multiarraymodule.c in Python
	embedding
In-Reply-To: <4D7539987C6B3340ADA6BC7AE609F73606FC8F@NAEX14.na.qualcomm.com>
References: <4D7539987C6B3340ADA6BC7AE609F73606FC8F@NAEX14.na.qualcomm.com>
Message-ID: <45C9CB53.2020905@iram.fr>

Hi,

it seems that I have the same trouble, but using Python under Linux:

> I have run into an interesting issue with multiarraymodule.c with 
> regards to embedding Python in a C application on Windows XP. In the 
> application, the following abbreviated sequence was executed
>
>  
>
> 1) Py_Initialize
>
> 2) numpy is imported
>
> 3) Py_Finalize
>
> 4) Py_Initialize
>
> 5) attempt to import numpy results in an error.
>

As an example I wrote a small C program that embeds Python. The steps 
are the same as above:
1) Python is initialized (Py_Initialize)
2) Numpy is "manually" imported (import numpy) under an interactive loop 
(PyRun_InteractiveLoop)
3) Python is finalized (Py_Finalize)
4) Python is initialized again (Py_Initialize)
5) Attempt to import numpy now fails:

 >>> import numpy
Traceback (most recent call last):
  File "<STDIN>", line 1, in <module>
  File "/usr/lib/python2.5/site-packages/numpy/__init__.py", line 36, in 
<module>
    import core
  File "/usr/lib/python2.5/site-packages/numpy/core/__init__.py", line 
5, in <module>
    import multiarray
SystemError: dynamic module not initialized properly

I'm using Numpy 1.0 and Python 2.5 under a Fedora Core 6:
Python 2.5 (r25:51908, Nov 15 2006, 14:24:03)
[GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2

I have not searched further why this happens. I embed Python in a larger 
project and let the choice to the user to start a Python session if he 
needs it. Numpy is automatically imported right after Python 
initialization (because it is needed in my context), so that if user 
starts a session, ends it, and starts a new one, the initialization 
automatically fails at the second time. The solution at the present I 
found is  preventing Python to be finalized (Py_Finalize).

I hope this can help. Thanks.

Sebastien



From meesters at uni-mainz.de  Wed Feb  7 08:00:52 2007
From: meesters at uni-mainz.de (Christian Meesters)
Date: Wed, 7 Feb 2007 14:00:52 +0100
Subject: [Numpy-discussion] getting indices for array positions
Message-ID: <200702071400.53315.meesters@uni-mainz.de>

Hi

This questions might seem stupid, but I didn't get a clever solution myself, 
or found one in the archives, the cookbook, etc. . If I overlooked something, 
please give a pointer.

Well, if I have an 1D array like
[ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5]
,a scalar like 0.122 and want to retrieve the index postion of the closest 
value of the scalar in the array: Is there any fast method to get this?
Right now I've implemented the following method:

def _get_value_index(value, a):
    mindiff = 1e20
    index = 0
    
    for intensity, temp_index in zip(a, xrange(a.shape[0])):
        diff = abs(intensity - value)
        #closer to given value?
        if diff <= mindiff:
            mindiff = diff
            index = temp_index
    return index

It works, but is akward and takes too much time (I've no benchmark), if the 
array is long and the method called often within a different function. But it 
should help clarify the problem.

TIA
Christian


From cimrman3 at ntc.zcu.cz  Wed Feb  7 08:08:42 2007
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Wed, 07 Feb 2007 14:08:42 +0100
Subject: [Numpy-discussion] getting indices for array positions
In-Reply-To: <200702071400.53315.meesters@uni-mainz.de>
References: <200702071400.53315.meesters@uni-mainz.de>
Message-ID: <45C9CF5A.7070208@ntc.zcu.cz>

Christian Meesters wrote:
> Hi
> 
> This questions might seem stupid, but I didn't get a clever solution myself, 
> or found one in the archives, the cookbook, etc. . If I overlooked something, 
> please give a pointer.
> 
> Well, if I have an 1D array like
> [ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5]
> ,a scalar like 0.122 and want to retrieve the index postion of the closest 
> value of the scalar in the array: Is there any fast method to get this?

Try searchsorted.

r.



From meesters at uni-mainz.de  Wed Feb  7 08:20:03 2007
From: meesters at uni-mainz.de (Christian Meesters)
Date: Wed, 7 Feb 2007 14:20:03 +0100
Subject: [Numpy-discussion] getting indices for array positions
In-Reply-To: <45C9CF5A.7070208@ntc.zcu.cz>
References: <200702071400.53315.meesters@uni-mainz.de>
	<45C9CF5A.7070208@ntc.zcu.cz>
Message-ID: <200702071420.04008.meesters@uni-mainz.de>

> Try searchsorted.
Thanks, but that doesn't work. Sorry, if my question wasn't clear.

To illustrate the requirement:
For instance:
>>> a
array([ 0. ,  0.1,  0.2,  0.3,  0.4])
>>> # should be 1
...
>>> a.searchsorted(0.11)
2
>>> # should be 2
...
>>> a.searchsorted(0.16)
2

I could correct for one index position, of course, but I still have the 
requirement to get the index of the item with the closest value to the key. 
Since searchsorted returns the index of the first item in a that is >= or > 
the key, it can't make the distinction between 0.1 and 0.2 as I would like to 
have.

Hope this clarifies my question.
 Christian


From cimrman3 at ntc.zcu.cz  Wed Feb  7 08:32:08 2007
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Wed, 07 Feb 2007 14:32:08 +0100
Subject: [Numpy-discussion] getting indices for array positions
In-Reply-To: <200702071420.04008.meesters@uni-mainz.de>
References: <200702071400.53315.meesters@uni-mainz.de>	<45C9CF5A.7070208@ntc.zcu.cz>
	<200702071420.04008.meesters@uni-mainz.de>
Message-ID: <45C9D4D8.3050602@ntc.zcu.cz>

Christian Meesters wrote:
>> Try searchsorted.
> Thanks, but that doesn't work. Sorry, if my question wasn't clear.
> 
> To illustrate the requirement:
> For instance:
>>>> a
> array([ 0. ,  0.1,  0.2,  0.3,  0.4])
>>>> # should be 1
> ...
>>>> a.searchsorted(0.11)
> 2
>>>> # should be 2
> ...
>>>> a.searchsorted(0.16)
> 2
> 
> I could correct for one index position, of course, but I still have the 
> requirement to get the index of the item with the closest value to the key. 
> Since searchsorted returns the index of the first item in a that is >= or > 
> the key, it can't make the distinction between 0.1 and 0.2 as I would like to 
> have.

I see. But it gives you the index (say 'ii') of the first item that is
grater than your scalar - then you just have to compare your scalar with
a[ii] and a[ii-1] and choose whichever is closer, no?

r.


From stefan at sun.ac.za  Wed Feb  7 09:03:50 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Wed, 7 Feb 2007 16:03:50 +0200
Subject: [Numpy-discussion] force column vector
In-Reply-To: <loom.20070207T113019-604@post.gmane.org>
References: <loom.20070207T113019-604@post.gmane.org>
Message-ID: <20070207140350.GB6274@mentat.za.net>

On Wed, Feb 07, 2007 at 10:35:14AM +0000, Christian wrote:
> Hi,
> 
> when creating an ndarray from a list, how can I force the result to be
> 2d *and* a column vector? So in case I pass a nested list, there will be no
> modification of the shape and when I pass a simple list, it will be 
> converted to a 2d column vector. I can only think of a solution using 'if'
> clauses but I suppose there is a more elegant way.

One way is to sub-class ndarray:

import numpy as N

class ColumnVectorArray(N.ndarray):
    def __new__(cls,data):
        data = N.asarray(data).view(cls)
        if len(data.shape) == 1:
            data.shape = (-1,1)
        return data

x = ColumnVectorArray([[1,2],[3,4],[5,6]])
print 'x ='
print x
print

y = ColumnVectorArray([1,2,3])
print 'y ='
print y
print

print 'x+y ='
print x+y

which yields:


x =
[[1 2]
 [3 4]
 [5 6]]

y =
[[1]
 [2]
 [3]]

x+y =
[[2 3]
 [5 6]
 [8 9]]


Cheers
St?fan


From stefan at sun.ac.za  Wed Feb  7 09:08:27 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Wed, 7 Feb 2007 16:08:27 +0200
Subject: [Numpy-discussion] getting indices for array positions
In-Reply-To: <200702071400.53315.meesters@uni-mainz.de>
References: <200702071400.53315.meesters@uni-mainz.de>
Message-ID: <20070207140827.GC6274@mentat.za.net>

On Wed, Feb 07, 2007 at 02:00:52PM +0100, Christian Meesters wrote:
> This questions might seem stupid, but I didn't get a clever solution myself, 
> or found one in the archives, the cookbook, etc. . If I overlooked something, 
> please give a pointer.
> 
> Well, if I have an 1D array like
> [ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5]
> ,a scalar like 0.122 and want to retrieve the index postion of the closest 
> value of the scalar in the array: Is there any fast method to get
> this?

If I understand correctly:

data = N.array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5])
diff = N.abs(data - val)
print N.argmin(diff)

Regards
St?fan


From joris at ster.kuleuven.be  Wed Feb  7 09:11:53 2007
From: joris at ster.kuleuven.be (Joris De Ridder)
Date: Wed, 7 Feb 2007 15:11:53 +0100
Subject: [Numpy-discussion] confused with apply_along_axis()
Message-ID: <200702071511.53362.joris@ster.kuleuven.be>

Hi,

I'm confused by the output of apply_along_axis() in the following very simple 
example:

In [93]: a = arange(12.).reshape(2,2,3)
In [95]: a
Out[95]:
array([[[  0.,   1.,   2.],
        [  3.,   4.,   5.]],
       [[  6.,   7.,   8.],
        [  9.,  10.,  11.]]])
In [96]: def myfunc(b):
   ....:     print "slice:", b, "  middle value:", b[1]
   ....:     return b[1]
   ....:
In [97]: apply_along_axis(myfunc,2,a)
slice: [ 0.  1.  2.]   middle value: 1.0
slice: [ 3.  4.  5.]   middle value: 4.0
slice: [ 6.  7.  8.]   middle value: 7.0
slice: [  9.  10.  11.]   middle value: 10.0
Out[97]:
array([[  7.,   7.],
       [ 10.,  10.]])


I expected as output
array([[  1.,   4.],
       [ 7.,  10.]])

Why is this not the case? How exactly does apply_along_axis() use the output 
of myfunc() to produce its result?

Cheers,
J.

P.S. I'm working with Python 2.5 & Numpy 1.0

Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm



From stefan at sun.ac.za  Wed Feb  7 09:22:53 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Wed, 7 Feb 2007 16:22:53 +0200
Subject: [Numpy-discussion] confused with apply_along_axis()
In-Reply-To: <200702071511.53362.joris@ster.kuleuven.be>
References: <200702071511.53362.joris@ster.kuleuven.be>
Message-ID: <20070207142253.GD6274@mentat.za.net>

Hi Joris

On Wed, Feb 07, 2007 at 03:11:53PM +0100, Joris De Ridder wrote:
> I expected as output
> array([[  1.,   4.],
>        [ 7.,  10.]])

That is the answer I get with numpy 1.0.2.dev3537 under Python 2.4.

Cheers
St?fan


From svetosch at gmx.net  Wed Feb  7 10:11:01 2007
From: svetosch at gmx.net (Sven Schreiber)
Date: Wed, 07 Feb 2007 16:11:01 +0100
Subject: [Numpy-discussion] force column vector
In-Reply-To: <loom.20070207T113019-604@post.gmane.org>
References: <loom.20070207T113019-604@post.gmane.org>
Message-ID: <45C9EC05.8040607@gmx.net>

Christian schrieb:
> Hi,
> 
> when creating an ndarray from a list, how can I force the result to be
> 2d *and* a column vector? So in case I pass a nested list, there will be no
> modification of the shape and when I pass a simple list, it will be 
> converted to a 2d column vector. I can only think of a solution using 'if'
> clauses but I suppose there is a more elegant way.
> 

I'd be interested if you find a way without a single 'if'; I can't think
of any, sorry. (which doesn't mean much, however... :-)

Maybe your best bet is to make sure the lists are always nested at the
source, if you have any control over that particular source.

good luck,
sven


From joris at ster.kuleuven.be  Wed Feb  7 10:19:41 2007
From: joris at ster.kuleuven.be (Joris De Ridder)
Date: Wed, 7 Feb 2007 16:19:41 +0100
Subject: [Numpy-discussion] confused with apply_along_axis()
In-Reply-To: <20070207142253.GD6274@mentat.za.net>
References: <200702071511.53362.joris@ster.kuleuven.be>
	<20070207142253.GD6274@mentat.za.net>
Message-ID: <200702071619.41532.joris@ster.kuleuven.be>


On Wednesday 07 February 2007 15:22, Stefan van der Walt wrote:

>On Wed, Feb 07, 2007 at 03:11:53PM +0100, Joris De Ridder wrote:
>> I expected as output
>> array([[  1.,   4.],
>>        [ 7.,  10.]])
>
>That is the answer I get with numpy 1.0.2.dev3537 under Python 2.4.

Python 2.5 + numpy 1.0.2.dev3540 also solved the problem for me. 

Thanks,
Joris

Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm



From Chris.Barker at noaa.gov  Wed Feb  7 11:32:36 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 07 Feb 2007 08:32:36 -0800
Subject: [Numpy-discussion] force column vector
In-Reply-To: <loom.20070207T113019-604@post.gmane.org>
References: <loom.20070207T113019-604@post.gmane.org>
Message-ID: <45C9FF24.7000005@noaa.gov>

Christian wrote:
> when creating an ndarray from a list, how can I force the result to be
> 2d *and* a column vector? So in case I pass a nested list, there will be no
> modification of the shape and when I pass a simple list, it will be 
> converted to a 2d column vector.

I'm not sure I understand the specification of the problem. I would 
think that the definition of a column vector is that it's shape is:

(-1,1)

which makes it easy:

def MakeColumn(input):
     a = asarray(input)
     a.shape = (-1,1)
     return a

however, if you want: MakeColumn([[1,2],[3,4],[5,6]])

to return:
array([[1, 2],
        [3, 4],
        [5, 6]])

that's not what I would call a column vector, and if that's what you 
want, then what would you want:

MakeColumn([[1,2,3,4],[5,6,7,8]])

to return?

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From oliphant at ee.byu.edu  Wed Feb  7 12:04:14 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed, 07 Feb 2007 10:04:14 -0700
Subject: [Numpy-discussion] bug in numpy.equal?
In-Reply-To: <d6f2d3dd0702061609i2b494496j72a3e01c7410e2df@mail.gmail.com>
References: <d6f2d3dd0702061609i2b494496j72a3e01c7410e2df@mail.gmail.com>
Message-ID: <45CA068E.9040909@ee.byu.edu>

Tom Denniston wrote:

>The behavior below seems strange to me.  The string array is type S3
>yet it says that comparison with 'abc' is not implemented.  The ==
>operator seems to work though.  Is there a subtlty I am missing or is
>it simply a bug?
>
>  
>

No bug.  Ufuncs do not work with variable-sized arrays.  However, we 
have implemented the equal operator using a different approach.  

-Travis



From kwgoodman at gmail.com  Wed Feb  7 12:15:38 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 7 Feb 2007 09:15:38 -0800
Subject: [Numpy-discussion] NaN, min, and max
Message-ID: <f4f93d420702070915y1f68bf6bvadbe3ab27c3dfc8c@mail.gmail.com>

I keep running into this problem:

>> import numpy.matlib as M
>> x = M.rand(3,3)
>> x[1,1] = M.nan
>> x

matrix([[ 0.94425407,  0.02216611,  0.999475  ],
        [ 0.40444129,         nan,  0.23264341],
        [ 0.24202372,  0.05344269,  0.37967564]])
>> x.max()
0.379675636032   <---- Wrong (for me)
>> x[1,1] = 0
>> x.max()
0.999474999444  <----- Beautiful! Look at all the tripple digits!

How do I add nanmax as a method of the matrix class? How would I
replace max with nanmax? Would I have to do that in every module of my
package if I want to access numpy.matlib as M instead of
packagename.M?


From fperez.net at gmail.com  Wed Feb  7 14:16:41 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 7 Feb 2007 12:16:41 -0700
Subject: [Numpy-discussion] numpy.linalg.qr bug on 64-bit platforms
Message-ID: <db6b5ecc0702071116s637fc919u26498361606db21d@mail.gmail.com>

Hi all,

I recently got a report of a bug triggered only on 64-bit hardware,
and on a machine (in case it's relevant) that runs python 2.5.  This
is with current numpy SVN which I just rebuilt a moment ago to
triple-check:

In [3]: a = numpy.array([[1.0,2],[3,4]])

In [4]: numpy.linalg.qr(a)
 ** On entry to DGEQRF parameter number  2 had an illegal value
sage[~]>  # dumped back at system prompt


In case anyone has ideas, I added some more initial detail and guesses
as to where the problem may be coming from here:

http://projects.scipy.org/scipy/numpy/ticket/446


Unfortunately I don't know this part of the code well enough to come
up with a quick fix myself.

Cheers,

f


From tom.denniston at alum.dartmouth.org  Wed Feb  7 14:16:53 2007
From: tom.denniston at alum.dartmouth.org (Tom Denniston)
Date: Wed, 7 Feb 2007 13:16:53 -0600
Subject: [Numpy-discussion] bug in numpy.equal?
In-Reply-To: <45CA068E.9040909@ee.byu.edu>
References: <d6f2d3dd0702061609i2b494496j72a3e01c7410e2df@mail.gmail.com>
	<45CA068E.9040909@ee.byu.edu>
Message-ID: <d6f2d3dd0702071116m52ab185do4b2e434a7f9bfb73@mail.gmail.com>

got it.  thanks.

On 2/7/07, Travis Oliphant <oliphant at ee.byu.edu> wrote:
> Tom Denniston wrote:
>
> >The behavior below seems strange to me.  The string array is type S3
> >yet it says that comparison with 'abc' is not implemented.  The ==
> >operator seems to work though.  Is there a subtlty I am missing or is
> >it simply a bug?
> >
> >
> >
>
> No bug.  Ufuncs do not work with variable-sized arrays.  However, we
> have implemented the equal operator using a different approach.
>
> -Travis
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From oliphant at ee.byu.edu  Wed Feb  7 16:36:39 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed, 07 Feb 2007 14:36:39 -0700
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <2558.129.240.194.142.1170763597.squirrel@webmail.uio.no>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>	<200702042033.01767.pgmdevlist@gmail.com>
	<2558.129.240.194.142.1170763597.squirrel@webmail.uio.no>
Message-ID: <45CA4667.1000402@ee.byu.edu>

Sturla Molden wrote:

>>def __new__(cls,...)
>>     ...
>>    (H, edges) = numpy.histogramdd(..)
>>    cls.__defaultedges = edges
>>
>>def __array_finalize__(self, obj):
>>    if  not hasattr(self, 'edges'):
>>        self.edges = self.__defaultedges
>>    
>>
>
>So in order to get an instance attribute, one has to temporarily define it
>as a class attribute? 
>

No, you don't *have* to do it this way for all instance attributes.  

In this example, the user was trying to keep the edges computed during 
the __new__ method as an attribute.   What are the possibilities?

1) Use the __new__ method to create the object in full and then store 
the edges in some kind of global (or class global) variable. 

This solution because it uses global variables has all of the thread 
problems global variables bring.

2) Create a "dummy" arrayobject in the __new__ method and fill it in 
(i.e. using setstate or resize) during the __init__ method where the 
instance attribute is actually set.

The __array_finalize__ method is intended for "passing-on" attributes to 
sub-classes from parent classes during operations where __new__ and 
__init__ are not called (but a new instance is still created). It was 
not intended to be used in all circumstances.


-Travis



From svetosch at gmx.net  Wed Feb  7 17:07:11 2007
From: svetosch at gmx.net (Sven Schreiber)
Date: Wed, 07 Feb 2007 23:07:11 +0100
Subject: [Numpy-discussion] force column vector
In-Reply-To: <45C9FF24.7000005@noaa.gov>
References: <loom.20070207T113019-604@post.gmane.org>
	<45C9FF24.7000005@noaa.gov>
Message-ID: <45CA4D8F.5040807@gmx.net>

Christopher Barker schrieb:
> Christian wrote:
>> when creating an ndarray from a list, how can I force the result to be
>> 2d *and* a column vector? So in case I pass a nested list, there will be no
>> modification of the shape and when I pass a simple list, it will be 
>> converted to a 2d column vector.
> 
> I'm not sure I understand the specification of the problem. I would 
> think that the definition of a column vector is that it's shape is:
> 
> (-1,1)
> 

So I think what's needed is:

b = array(yourlist)
b.reshape(b.shape[0], -1)

Now it seems I finally understood this business with the -1 in the
shapes... (well it's trivial if you have the book :-)

-sven


From kwgoodman at gmail.com  Wed Feb  7 17:41:21 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 7 Feb 2007 14:41:21 -0800
Subject: [Numpy-discussion] force column vector
In-Reply-To: <45CA4D8F.5040807@gmx.net>
References: <loom.20070207T113019-604@post.gmane.org>
	<45C9FF24.7000005@noaa.gov> <45CA4D8F.5040807@gmx.net>
Message-ID: <f4f93d420702071441j3caa4932h63431f50f3e49bbb@mail.gmail.com>

On 2/7/07, Sven Schreiber <svetosch at gmx.net> wrote:
> Christopher Barker schrieb:
> > Christian wrote:
> >> when creating an ndarray from a list, how can I force the result to be
> >> 2d *and* a column vector? So in case I pass a nested list, there will be no
> >> modification of the shape and when I pass a simple list, it will be
> >> converted to a 2d column vector.
> >
> > I'm not sure I understand the specification of the problem. I would
> > think that the definition of a column vector is that it's shape is:
> >
> > (-1,1)
> >
>
> So I think what's needed is:
>
> b = array(yourlist)
> b.reshape(b.shape[0], -1)
>
> Now it seems I finally understood this business with the -1 in the
> shapes... (well it's trivial if you have the book :-)


I'd like to know what the -1 means. But first I'm trying to figure out
why there are two reshapes? Do they behave identically? The doc
strings make it look like they might not.

>> x = M.rand(3,3)
>> x.reshape?
Type:           builtin_function_or_method
Base Class:     <type 'builtin_function_or_method'>
String Form:    <built-in method reshape of matrix object at 0xb4b41df4>
Namespace:      Interactive
Docstring:
    a.reshape(d1, d2, ..., dn, order='c')

    Return a new array from this one.  The new array must have the same number
    of elements as self.  Also always returns a view or raises a ValueError if
    that is impossible.;

>> M.reshape?
Type:           function
Base Class:     <type 'function'>
String Form:    <function reshape at 0xb776541c>
Namespace:      Interactive
File:           /usr/local/lib/python2.4/site-packages/numpy/core/fromnumeric.py
Definition:     M.reshape(a, newshape, order='C')
Docstring:
    Return an array that uses the data of the given array, but with a new
    shape.

    :Parameters:
      - `a` : array
      - `newshape` : shape tuple or int
        The new shape should be compatible with the original shape. If an
        integer, then the result will be a 1D array of that length.
      - `order` : 'C' or 'FORTRAN', optional (default='C')
        Whether the array data should be viewed as in C (row-major) order or
        FORTRAN (column-major) order.

    :Returns:
      - `reshaped_array` : array
        This will be a new view object if possible; otherwise, it will return
        a copy.

    :See also:
      numpy.ndarray.reshape() is the equivalent method.


From robert.kern at gmail.com  Wed Feb  7 17:52:16 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 07 Feb 2007 16:52:16 -0600
Subject: [Numpy-discussion] force column vector
In-Reply-To: <f4f93d420702071441j3caa4932h63431f50f3e49bbb@mail.gmail.com>
References: <loom.20070207T113019-604@post.gmane.org>	<45C9FF24.7000005@noaa.gov>
	<45CA4D8F.5040807@gmx.net>
	<f4f93d420702071441j3caa4932h63431f50f3e49bbb@mail.gmail.com>
Message-ID: <45CA5820.5060100@gmail.com>

Keith Goodman wrote:
> I'd like to know what the -1 means.

It means "fill in with whatever is necessary to make the size correct given the
other specified dimensions".

> But first I'm trying to figure out
> why there are two reshapes?

reshape() used to be simply a function, not a method.

> Do they behave identically? The doc
> strings make it look like they might not.

The docstrings are different only because I updated the one and not, yet, the other.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From tom.denniston at alum.dartmouth.org  Wed Feb  7 18:16:07 2007
From: tom.denniston at alum.dartmouth.org (Tom Denniston)
Date: Wed, 7 Feb 2007 17:16:07 -0600
Subject: [Numpy-discussion] Bug in numpy user-define types mechanism causes
	segfault on multiple calls to a ufunc
Message-ID: <d6f2d3dd0702071516h47dec131yed6a55d950bf57bf@mail.gmail.com>

I am trying to register a custom type to numpy.  When I do so it works
and the ufuncs work but then when I invoke any ufunc twice the second
time my python interpretter segfaults.  I think i know what the
problem is.  In the select_types method in ufuncobject.c in
numpy/core/src/ numpy gets a reference to the key for the loop via a
call to PyInt_FromLong:
key = PyInt_FromLong((long) userdef);
Then it gets the actual loop via a call to PyDict_GetItem:
obj = PyDict_GetItem(self->userloops, key);
It later proceeds to do a decref on key:
Py_DECREF(key);
and later a decref on obj
Py_DECREF(obj);

None of this code actually runs unless you are doing an operation on a
user defined type because it is all in the block with an if statement
if (userdef > 0).

The Py_DECREF on key is correct because it returns a new reference per
python c api doc:

PyObject* PyInt_FromLong( long ival)

Return value: New reference.
Create a new integer object with a value of ival.

However the Py_DECREF on the obj looks incorrect to me because the
PyDict_Getitem returns a borrowed reference and the numpy code doesn't
increment the reference:

PyObject* PyDict_GetItem( PyObject *p, PyObject *key)

Return value: Borrowed reference.
Return the object from dictionary p which has a key key. Return NULL
if the key key is not present, but without setting an exception.


So what seems to happen is the last reference gets decremented and the
garbage collector frees up obj (the ufunc loop) on the first time
through.  The second time the through the ufunc loop is garbage memory
and it segfaults.

If I comment out the obj DECREF it works.  I think one needs to either
do that or add an INCREF right after retrieving key.  I think either
will work but the multithreading implications are different.  I don't
think it matters give that (I believe) numpy doesn't release the GIL
but I thought someone on this list would be a better judge than I
(maybe Travis) of what the correct fix should be.

Am I correct in my analysis?

In either case it should be a one or two line fix.

--Tom


From robert.kern at gmail.com  Wed Feb  7 18:25:40 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 07 Feb 2007 17:25:40 -0600
Subject: [Numpy-discussion] SciPy '07 ???
In-Reply-To: <45C26295.4060707@noaa.gov>
References: <45C26295.4060707@noaa.gov>
Message-ID: <45CA5FF4.3070304@gmail.com>

Christopher Barker wrote:
> Hi,
> 
> Does anyone know if there will be a SciPy '07 conference, and if so, when?

Yes, there will be one. We are currently speaking with the venue (Caltech in
Pasadena, CA) to set the dates. Expect the conference to be either late August
or perhaps earlyish in September.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From oliphant at ee.byu.edu  Wed Feb  7 18:29:31 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed, 07 Feb 2007 16:29:31 -0700
Subject: [Numpy-discussion] Bug in numpy user-define types mechanism
 causes segfault on multiple calls to a ufunc
In-Reply-To: <d6f2d3dd0702071516h47dec131yed6a55d950bf57bf@mail.gmail.com>
References: <d6f2d3dd0702071516h47dec131yed6a55d950bf57bf@mail.gmail.com>
Message-ID: <45CA60DB.1070703@ee.byu.edu>

Tom Denniston wrote:

>I am trying to register a custom type to numpy.  When I do so it works
>and the ufuncs work but then when I invoke any ufunc twice the second
>time my python interpretter segfaults.  I think i know what the
>problem is.  In the select_types method in ufuncobject.c in
>numpy/core/src/ numpy gets a reference to the key for the loop via a
>  
>
Thank you very much for your review of this less-used code.

>If I comment out the obj DECREF it works.  I think one needs to either
>do that or add an INCREF right after retrieving key.  I think either
>will work but the multithreading implications are different.  I don't
>think it matters give that (I believe) numpy doesn't release the GIL
>but I thought someone on this list would be a better judge than I
>(maybe Travis) of what the correct fix should be.
>
>Am I correct in my analysis?
>  
>

Your analysis seems spot on.  I think removing the  is the right course 
of action.   I've done it in SVN.



-Travis



From reggie at merfinllc.com  Wed Feb  7 18:32:16 2007
From: reggie at merfinllc.com (Reggie Dugard)
Date: Wed, 07 Feb 2007 15:32:16 -0800
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <45CA4667.1000402@ee.byu.edu>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>
	<200702042033.01767.pgmdevlist@gmail.com>
	<2558.129.240.194.142.1170763597.squirrel@webmail.uio.no>
	<45CA4667.1000402@ee.byu.edu>
Message-ID: <1170891136.28353.75.camel@fox>

On Wed, 2007-02-07 at 14:36 -0700, Travis Oliphant wrote:
> Sturla Molden wrote:
> 
> >>def __new__(cls,...)
> >>     ...
> >>    (H, edges) = numpy.histogramdd(..)
> >>    cls.__defaultedges = edges
> >>
> >>def __array_finalize__(self, obj):
> >>    if  not hasattr(self, 'edges'):
> >>        self.edges = self.__defaultedges
> >>    
> >>
> >
> >So in order to get an instance attribute, one has to temporarily define it
> >as a class attribute? 
> >
> 
> No, you don't *have* to do it this way for all instance attributes.  
> 
> In this example, the user was trying to keep the edges computed during 
> the __new__ method as an attribute.   What are the possibilities?
> 
> 1) Use the __new__ method to create the object in full and then store 
> the edges in some kind of global (or class global) variable. 
> 
> This solution because it uses global variables has all of the thread 
> problems global variables bring.
> 
> 2) Create a "dummy" arrayobject in the __new__ method and fill it in 
> (i.e. using setstate or resize) during the __init__ method where the 
> instance attribute is actually set.
> 
I'm probably missing something obvious here, but why can't you just
attach the attribute to the actual object in the __new__ method before
returning it.  For example:

class MyClass(numpy.ndarray):
    def __new__(self, ...):
        # Some stuff here
        H, edges = numpy.histogramdd(...)
        result = H.view(MyClass)
        result.edges = edges
        return result

    def __array_finalize__(self, obj):
        self.edges = getattr(obj, 'edges', [])

If you could show me the error of my ways, it would help me in *my*
attempt to subclass ndarray.

> The __array_finalize__ method is intended for "passing-on" attributes to 
> sub-classes from parent classes during operations where __new__ and 
> __init__ are not called (but a new instance is still created). It was 
> not intended to be used in all circumstances.

Thanks,

-Reggie



From tom.denniston at alum.dartmouth.org  Wed Feb  7 18:38:41 2007
From: tom.denniston at alum.dartmouth.org (Tom Denniston)
Date: Wed, 7 Feb 2007 17:38:41 -0600
Subject: [Numpy-discussion] Bug in numpy user-define types mechanism
	causes segfault on multiple calls to a ufunc
In-Reply-To: <45CA60DB.1070703@ee.byu.edu>
References: <d6f2d3dd0702071516h47dec131yed6a55d950bf57bf@mail.gmail.com>
	<45CA60DB.1070703@ee.byu.edu>
Message-ID: <d6f2d3dd0702071538s7d67ed74h694c2eb584df3cd1@mail.gmail.com>

Many thanks, Travis.  I'll test the new version tonight.

--Tom

On 2/7/07, Travis Oliphant <oliphant at ee.byu.edu> wrote:
> Tom Denniston wrote:
>
> >I am trying to register a custom type to numpy.  When I do so it works
> >and the ufuncs work but then when I invoke any ufunc twice the second
> >time my python interpretter segfaults.  I think i know what the
> >problem is.  In the select_types method in ufuncobject.c in
> >numpy/core/src/ numpy gets a reference to the key for the loop via a
> >
> >
> Thank you very much for your review of this less-used code.
>
> >If I comment out the obj DECREF it works.  I think one needs to either
> >do that or add an INCREF right after retrieving key.  I think either
> >will work but the multithreading implications are different.  I don't
> >think it matters give that (I believe) numpy doesn't release the GIL
> >but I thought someone on this list would be a better judge than I
> >(maybe Travis) of what the correct fix should be.
> >
> >Am I correct in my analysis?
> >
> >
>
> Your analysis seems spot on.  I think removing the  is the right course
> of action.   I've done it in SVN.
>
>
>
> -Travis
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From oliphant at ee.byu.edu  Wed Feb  7 18:41:47 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed, 07 Feb 2007 16:41:47 -0700
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <1170891136.28353.75.camel@fox>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>	<200702042033.01767.pgmdevlist@gmail.com>	<2558.129.240.194.142.1170763597.squirrel@webmail.uio.no>	<45CA4667.1000402@ee.byu.edu>
	<1170891136.28353.75.camel@fox>
Message-ID: <45CA63BB.9050707@ee.byu.edu>

Reggie Dugard wrote:

>On Wed, 2007-02-07 at 14:36 -0700, Travis Oliphant wrote:
>  
>
>>Sturla Molden wrote:
>>
>>    
>>
>>>>def __new__(cls,...)
>>>>    ...
>>>>   (H, edges) = numpy.histogramdd(..)
>>>>   cls.__defaultedges = edges
>>>>
>>>>def __array_finalize__(self, obj):
>>>>   if  not hasattr(self, 'edges'):
>>>>       self.edges = self.__defaultedges
>>>>   
>>>>
>>>>        
>>>>
>>>So in order to get an instance attribute, one has to temporarily define it
>>>as a class attribute? 
>>>
>>>      
>>>
>>No, you don't *have* to do it this way for all instance attributes.  
>>
>>In this example, the user was trying to keep the edges computed during 
>>the __new__ method as an attribute.   What are the possibilities?
>>
>>1) Use the __new__ method to create the object in full and then store 
>>the edges in some kind of global (or class global) variable. 
>>
>>This solution because it uses global variables has all of the thread 
>>problems global variables bring.
>>
>>2) Create a "dummy" arrayobject in the __new__ method and fill it in 
>>(i.e. using setstate or resize) during the __init__ method where the 
>>instance attribute is actually set.
>>
>>    
>>
>I'm probably missing something obvious here, but why can't you just
>attach the attribute to the actual object in the __new__ method before
>returning it.  For example:
>  
>
Good point.  I guess I thought the OP had tried that already.  It turns 
out it works fine, too.

The __array_finalize__ is useful if you want the attribute to be carried 
around when arrays are created automatically internally (after math 
operations for example).


-Travis



From sturla at molden.no  Wed Feb  7 18:50:11 2007
From: sturla at molden.no (Sturla Molden)
Date: Thu, 8 Feb 2007 00:50:11 +0100 (CET)
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <45CA63BB.9050707@ee.byu.edu>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>
	<200702042033.01767.pgmdevlist@gmail.com>
	<2558.129.240.194.142.1170763597.squirrel@webmail.uio.no>
	<45CA4667.1000402@ee.byu.edu> <1170891136.28353.75.camel@fox>
	<45CA63BB.9050707@ee.byu.edu>
Message-ID: <4790.89.8.42.137.1170892211.squirrel@webmail.uio.no>


> Good point.  I guess I thought the OP had tried that already.  It turns
> out it works fine, too.
>
> The __array_finalize__ is useful if you want the attribute to be carried
> around when arrays are created automatically internally (after math
> operations for example).

I too may be missing something here.

Will using __array_finalize__ this way be thread safe or not?


Sturla Molden




From Chris.Barker at noaa.gov  Wed Feb  7 18:59:48 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 07 Feb 2007 15:59:48 -0800
Subject: [Numpy-discussion] SciPy '07 ???
In-Reply-To: <45CA5FF4.3070304@gmail.com>
References: <45C26295.4060707@noaa.gov> <45CA5FF4.3070304@gmail.com>
Message-ID: <45CA67F4.9050603@noaa.gov>

Robert Kern wrote:
>> Does anyone know if there will be a SciPy '07 conference, and if so, when?
> 
> Yes, there will be one. We are currently speaking with the venue (Caltech in
> Pasadena, CA) to set the dates. Expect the conference to be either late August
> or perhaps earlyish in September.

Nice to know. The sooner we get dates, the better, my wife needs to 
schedule out summer vacation now!

I've missed it for various reasons for the last 3 years, I hope I can do 
this one!

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From oliphant at ee.byu.edu  Wed Feb  7 19:07:55 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed, 07 Feb 2007 17:07:55 -0700
Subject: [Numpy-discussion] Please help with subclassing numpy.ndarray
In-Reply-To: <4790.89.8.42.137.1170892211.squirrel@webmail.uio.no>
References: <3db594f70702041722m36d3de67t2119b83ceca981a6@mail.gmail.com>	<200702042033.01767.pgmdevlist@gmail.com>	<2558.129.240.194.142.1170763597.squirrel@webmail.uio.no>	<45CA4667.1000402@ee.byu.edu>
	<1170891136.28353.75.camel@fox>	<45CA63BB.9050707@ee.byu.edu>
	<4790.89.8.42.137.1170892211.squirrel@webmail.uio.no>
Message-ID: <45CA69DB.2020304@ee.byu.edu>

Sturla Molden wrote:

>>Good point.  I guess I thought the OP had tried that already.  It turns
>>out it works fine, too.
>>
>>The __array_finalize__ is useful if you want the attribute to be carried
>>around when arrays are created automatically internally (after math
>>operations for example).
>>    
>>
>
>I too may be missing something here.
>
>Will using __array_finalize__ this way be thread safe or not?
>  
>
Yes because __array_finalize__ is called while NumPy owns the GIL.   It 
is called in one place during array creation (in the C routine that all 
array-creation routines call).

-Travis



From ckkart at hoc.net  Wed Feb  7 20:01:31 2007
From: ckkart at hoc.net (Christian)
Date: Thu, 8 Feb 2007 01:01:31 +0000 (UTC)
Subject: [Numpy-discussion] force column vector
References: <loom.20070207T113019-604@post.gmane.org>
	<45C9FF24.7000005@noaa.gov>
Message-ID: <loom.20070208T015249-957@post.gmane.org>

Christopher Barker <Chris.Barker <at> noaa.gov> writes:

> I'm not sure I understand the specification of the problem. I would 
> think that the definition of a column vector is that it's shape is:
> 
> (-1,1)

I was not aware of that possibility althoug I own the book I - shame on me.
Thank you (and all others) for pointing that out.

What I want is a 2d array regardless of whether the input is a simple or 
nested list. However if it is a simple list I want the result to be a column
vector rather than a row vector, which happens by default.
Like Sven I'm wondering if there is solution without using any 'if'.
However using the subclassed array class which was proposed by Stefan is pretty
elegant.

Thanks to everybody, Christian





From ckkart at hoc.net  Wed Feb  7 20:03:02 2007
From: ckkart at hoc.net (Christian)
Date: Thu, 8 Feb 2007 01:03:02 +0000 (UTC)
Subject: [Numpy-discussion] getting indices for array positions
References: <200702071400.53315.meesters@uni-mainz.de>
	<45C9CF5A.7070208@ntc.zcu.cz>
	<200702071420.04008.meesters@uni-mainz.de>
Message-ID: <loom.20070208T020222-560@post.gmane.org>

Christian Meesters <meesters <at> uni-mainz.de> writes:

> Since searchsorted returns the index of the first item in a that is >= or > 
> the key, it can't make the distinction between 0.1 and 0.2 as I would like to 

Then how about a.searchsorted(val+0.5) 

Christian





From ckkart at hoc.net  Wed Feb  7 20:06:01 2007
From: ckkart at hoc.net (Christian)
Date: Thu, 8 Feb 2007 01:06:01 +0000 (UTC)
Subject: [Numpy-discussion] force column vector
References: <loom.20070207T113019-604@post.gmane.org>
	<45C9FF24.7000005@noaa.gov> <45CA4D8F.5040807@gmx.net>
Message-ID: <loom.20070208T020520-149@post.gmane.org>

Sven Schreiber <svetosch <at> gmx.net> writes:
> So I think what's needed is:
> 
> b = array(yourlist)
> b.reshape(b.shape[0], -1)

Yes! That is it.
Thanks, Christian




From pgmdevlist at gmail.com  Wed Feb  7 22:00:28 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 7 Feb 2007 22:00:28 -0500
Subject: [Numpy-discussion] Comparing x and x.view
Message-ID: <200702072200.30228.pgmdevlist@gmail.com>

All,

I want to compare whether two arrays point to the same data.
I've been using 'is' so far, but I'm wondering whether it's the right 
approach.

If x is a plain ndarray,  `x is x`, and `x is not x.view()`. I understand the 
second one (I think so...), `x` and `x.view `are two different Python 
objects. However, `x.__array_interface__ == x.view().__array_interface__`, 
which means that the underlying memory hasn't been modified at all, right ? 
In other terms, the data hasn't been copied, it's just being accessed 
slightly differently.

So, when I'm using `x is y` to test whether some data has been copied, I 
should in fact compare the __array_interface__s, shouldn't I ?

Sorry for the poor phrasing, and thanks a lot for your forthcoming inputs.

P.


From robert.kern at gmail.com  Wed Feb  7 22:38:30 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 07 Feb 2007 21:38:30 -0600
Subject: [Numpy-discussion] Comparing x and x.view
In-Reply-To: <200702072200.30228.pgmdevlist@gmail.com>
References: <200702072200.30228.pgmdevlist@gmail.com>
Message-ID: <45CA9B36.3020408@gmail.com>

Pierre GM wrote:
> All,
> 
> I want to compare whether two arrays point to the same data.
> I've been using 'is' so far, but I'm wondering whether it's the right 
> approach.

It isn't. Your analysis is correct.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From pgmdevlist at gmail.com  Wed Feb  7 22:52:14 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 7 Feb 2007 22:52:14 -0500
Subject: [Numpy-discussion] Comparing x and x.view
In-Reply-To: <45CA9B36.3020408@gmail.com>
References: <200702072200.30228.pgmdevlist@gmail.com>
	<45CA9B36.3020408@gmail.com>
Message-ID: <200702072252.15238.pgmdevlist@gmail.com>

On Wednesday 07 February 2007 22:38:30 Robert Kern wrote:
> Pierre GM wrote:
> > All,
> >
> > I want to compare whether two arrays point to the same data.
> > I've been using 'is' so far, but I'm wondering whether it's the right
> > approach.
>
> It isn't. Your analysis is correct.

So, there's no real point in using the Python 'id' function ? Do we need a 
shortcut to __array_interface__['data'] as id number ?


From oliphant at ee.byu.edu  Wed Feb  7 23:09:16 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed, 07 Feb 2007 21:09:16 -0700
Subject: [Numpy-discussion] Comparing x and x.view
In-Reply-To: <200702072252.15238.pgmdevlist@gmail.com>
References: <200702072200.30228.pgmdevlist@gmail.com>	<45CA9B36.3020408@gmail.com>
	<200702072252.15238.pgmdevlist@gmail.com>
Message-ID: <45CAA26C.5020509@ee.byu.edu>

Pierre GM wrote:
> On Wednesday 07 February 2007 22:38:30 Robert Kern wrote:
>   
>> Pierre GM wrote:
>>     
>>> All,
>>>
>>> I want to compare whether two arrays point to the same data.
>>> I've been using 'is' so far, but I'm wondering whether it's the right
>>> approach.
>>>       
>> It isn't. Your analysis is correct.
>>     
>
> So, there's no real point in using the Python 'id' function ? Do we need a 
> shortcut to __array_interface__['data'] as id number ?
>   
You have it (sort of a short-cut).

.ctypes.data

This works even if ctypes is not installed.

-Travis



From robert.kern at gmail.com  Wed Feb  7 23:05:46 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 07 Feb 2007 22:05:46 -0600
Subject: [Numpy-discussion] Comparing x and x.view
In-Reply-To: <200702072252.15238.pgmdevlist@gmail.com>
References: <200702072200.30228.pgmdevlist@gmail.com>	<45CA9B36.3020408@gmail.com>
	<200702072252.15238.pgmdevlist@gmail.com>
Message-ID: <45CAA19A.1010205@gmail.com>

Pierre GM wrote:
> On Wednesday 07 February 2007 22:38:30 Robert Kern wrote:
>> Pierre GM wrote:
>>> All,
>>>
>>> I want to compare whether two arrays point to the same data.
>>> I've been using 'is' so far, but I'm wondering whether it's the right
>>> approach.
>> It isn't. Your analysis is correct.
> 
> So, there's no real point in using the Python 'id' function ?

Not for this purpose, no.

> Do we need a 
> shortcut to __array_interface__['data'] as id number ?

I'm not sure that would be useful. Different arrays may share the same starting
pointer but have different strides, and arrays may also have different starting
pointers but share some overlapping data.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Wed Feb  7 23:46:59 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 7 Feb 2007 21:46:59 -0700
Subject: [Numpy-discussion] force column vector
In-Reply-To: <loom.20070208T020520-149@post.gmane.org>
References: <loom.20070207T113019-604@post.gmane.org>
	<45C9FF24.7000005@noaa.gov> <45CA4D8F.5040807@gmx.net>
	<loom.20070208T020520-149@post.gmane.org>
Message-ID: <e06186140702072046x4da117a1je52f13903a07231@mail.gmail.com>

On 2/7/07, Christian <ckkart at hoc.net> wrote:
>
> Sven Schreiber <svetosch <at> gmx.net> writes:
> > So I think what's needed is:
> >
> > b = array(yourlist)
> > b.reshape(b.shape[0], -1)


Row vectors are easy to get.

In [1]: asmatrix([1,2,3,4])
Out[1]: matrix([[1, 2, 3, 4]])

And nested lists work, but you will be stuck with using matrices.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070207/9f23f572/attachment.html>

From ckkart at hoc.net  Thu Feb  8 00:21:08 2007
From: ckkart at hoc.net (Christian)
Date: Thu, 8 Feb 2007 05:21:08 +0000 (UTC)
Subject: [Numpy-discussion] NaN, min, and max
References: <f4f93d420702070915y1f68bf6bvadbe3ab27c3dfc8c@mail.gmail.com>
Message-ID: <loom.20070208T062025-536@post.gmane.org>

Keith Goodman <kwgoodman <at> gmail.com> writes:

> matrix([[ 0.94425407,  0.02216611,  0.999475  ],
>         [ 0.40444129,         nan,  0.23264341],
>         [ 0.24202372,  0.05344269,  0.37967564]])
> >> x.max()
> 0.379675636032   <---- Wrong (for me)
> >> x[1,1] = 0
> >> x.max()
> 0.999474999444  <----- Beautiful! Look at all the tripple digits!

Works as expected with python2.4/numpy1.0

Christian





From meesters at uni-mainz.de  Thu Feb  8 07:54:57 2007
From: meesters at uni-mainz.de (Christian Meesters)
Date: Thu, 8 Feb 2007 13:54:57 +0100
Subject: [Numpy-discussion] getting indices for array positions
In-Reply-To: <loom.20070208T020222-560@post.gmane.org>
References: <200702071400.53315.meesters@uni-mainz.de>
	<200702071420.04008.meesters@uni-mainz.de>
	<loom.20070208T020222-560@post.gmane.org>
Message-ID: <200702081354.57887.meesters@uni-mainz.de>

Hi

Thanks for all your suggestions.

Christian


From Chris.Barker at noaa.gov  Thu Feb  8 13:43:55 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 08 Feb 2007 10:43:55 -0800
Subject: [Numpy-discussion] Comparing x and x.view
In-Reply-To: <200702072200.30228.pgmdevlist@gmail.com>
References: <200702072200.30228.pgmdevlist@gmail.com>
Message-ID: <45CB6F6B.7040204@noaa.gov>

Pierre GM wrote:
> I want to compare whether two arrays point to the same data.

Travis just posted a note about a couple utility functions that may help:

Travis Oliphant wrote:
> In SVN there is a new function  may_share_memory(a,b) which will return 
> True if the memory foot-print of the two arrays over-lap.
> 
>  >>> may_share_memory(a, flipud(a))
> True
> 
> This is based on another utility function byte_bounds that returns the 
> byte-boundaries of any object exporting the Python side of the array 
> interface.
> 
> Perhaps these utilities will help (I know they can be used to make the 
> who function a bit more intelligent about how many bytes are being used).
> 
> -Travis



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From mclay at l2sg.com  Thu Feb  8 13:42:42 2007
From: mclay at l2sg.com (Michael McLay)
Date: Thu, 8 Feb 2007 13:42:42 -0500
Subject: [Numpy-discussion] Latest Array-Interface PEP
In-Reply-To: <459D9D9C.30501@ee.byu.edu>
References: <459D6964.30701@ee.byu.edu> <459D9590.80202@noaa.gov>
	<459D9D9C.30501@ee.byu.edu>
Message-ID: <200702081342.42565.mclay@l2sg.com>

On Thursday 04 January 2007 19:36, Travis Oliphant wrote:
> Christopher Barker wrote:
> > eople like:
> >
> > wxPython -- Robin Dunn
> > PIL      -- Fredrik Lundh
> > PyOpenGL -- Who?
> > PyObjC -- would it be useful there? (Ronald Oussoren)
> > MatplotLib (but maybe it's already married to numpy...)
> > PyGtk ?
>
> It's a good start, but their is also
>
> PyMedia, PyVoxel, any video-library interface writers, any audo-library
> interface writers.
>
> Anybody who wants to wrap/write code that does some kind of manipulation
> on a chunk of data of a specific data-format.
>
> There are so many people who would use it that I don't feel qualified to
> speak for them all.

Two more places to look for projects that may be interested:

  SQL wrappers, such as Psycopg2, and the Python DB API 2.0 community
  QuantLib (see the message below from the enthought-dev mailing list.)
 
On Saturday 03 February 2007 00:23, Prabhu Ramachandran wrote:
> >>>>> "Joseph" == Joseph Wang <joe at gnacademy.org> writes:
>     Joseph> 3) I'd like to make the impedance mismatch between things
>     Joseph> like QuantLib arrays and numpy arrays as seemless as
>     Joseph> possible.  Any words of wisdom on how to do this?
>
> I don't know anything about QuantLib arrays so will shoot in the dark
> here.  If your arrays can take or provide a block of contiguous memory
> that is interpreted according to a particular data type then I think
> it is easily possible to get these two array types talking to each
> other.  However doing this right and optimally will take a bit of
> effort (I've done this for tvtk which lets VTK and numpy arrays talk
> to each other seamlessly).  I think there is a fair bit of
> documentation on the scipy wiki on the various ways to do this right.
> The scipy-user list is a good place to ask for pointers since the
> folks who actually develop numpy and do all sorts of things with numpy
> arrays are on that list and will be happy to answer questions.



From pgmdevlist at gmail.com  Thu Feb  8 14:51:12 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 8 Feb 2007 14:51:12 -0500
Subject: [Numpy-discussion] Comparing x and x.view
In-Reply-To: <45CAA26C.5020509@ee.byu.edu>
References: <200702072200.30228.pgmdevlist@gmail.com>
	<200702072252.15238.pgmdevlist@gmail.com>
	<45CAA26C.5020509@ee.byu.edu>
Message-ID: <200702081451.14506.pgmdevlist@gmail.com>

On Wednesday 07 February 2007 23:09:16 Travis Oliphant wrote:
> > So, there's no real point in using the Python 'id' function ? Do we need
> > a shortcut to __array_interface__['data'] as id number ?
>
> You have it (sort of a short-cut).
>
> .ctypes.data

OK, great, thanks a lot

On Wednesday 07 February 2007 23:05:46 Robert Kern wrote:
> > Do we need a
> > shortcut to __array_interface__['data'] as id number ?
>
> I'm not sure that would be useful. Different arrays may share the same
> starting pointer but have different strides, and arrays may also have
> different starting pointers but share some overlapping data.

So, in the first case, I can still compare the __array_interfaces__, right ? 
If they're equal, then the two arrays are strictly equivalent. In the second 
case, even if the arrays share some overlapping data, they're intrinsically 
different, so a test on whether the data has been copied/moved around would 
fail anyway.

Well, thank y'all again.


From humufr at yahoo.fr  Thu Feb  8 16:21:09 2007
From: humufr at yahoo.fr (humufr at yahoo.fr)
Date: Thu, 8 Feb 2007 16:21:09 -0500
Subject: [Numpy-discussion] bug or feature?
Message-ID: <200702081621.09495.humufr@yahoo.fr>

I have a big problem with numpy, numarray and Numeric (all version)

If I'm using the script at the bottom, I obtain these results:

var1 before function is  [3 4 5]
var2 before function is  1
var1 after function must be [3 4 5] is  [ 9 12 15]     <------ problem
var2 after function must be 1 is  1
var3 must be the [9 12 15] is  [ 9 12 15]
var4 must be the 'toto' is  toto


I'm very surprised by the line noted. I always thinking that the input 
variable didn't change the variable itself outside the function. It's the 
comportement for var2 but var1 is changed and it's a big problem (at least 
for me). The only object in python with this behavior are the numeric object 
(Numeric, numarray or numpy), with list or other kind of object I have the 
expected result (the var1 before to go inside the function)

I can't keep the input variable if I'm not doing a copy before to call a 
function...

Is it normal and so do I have to do a copy of the input data each time I'm 
calling a function? 

Thanks


Nicolas

#!/usr/bin/env python
import numpy
print "numpy version ", numpy.__version__
def test(var1,var2):
        #print "var1 input function is",var1
        #print "var2 input function is",var2
        var1 *=3
        var2 = 'toto'
        return var1,var2

var1=numpy.array([3,4,5])
var2=1
print "var1 before function is ",var1
print "var2 before function is ",var2
var3,var4=test(var1,var2)
print "var1 after function must be [3 4 5] is ",var1
print "var2 after function must be 1 is ", var2
print "var3 must be the [9 12 15] is ", var3
print "var4 must be the 'toto' is ", var4


From pgmdevlist at gmail.com  Thu Feb  8 16:42:34 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 8 Feb 2007 16:42:34 -0500
Subject: [Numpy-discussion] bug or feature?
In-Reply-To: <200702081621.09495.humufr@yahoo.fr>
References: <200702081621.09495.humufr@yahoo.fr>
Message-ID: <200702081642.35801.pgmdevlist@gmail.com>

On Thursday 08 February 2007 16:21:09 humufr at yahoo.fr wrote:

>
> I'm very surprised by the line noted. I always thinking that the input
> variable didn't change the variable itself outside the function. 

Except that in this particular case, you explicitly change the input array 
itself by using an inplace operator.

> I can't keep the input variable if I'm not doing a copy before to call a
> function...

what about just using var1 = var1*3 ? That way, the initial array is left 
intact, and var1 inside your function is multiplied by 3.


From oliphant at ee.byu.edu  Thu Feb  8 17:01:36 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu, 08 Feb 2007 15:01:36 -0700
Subject: [Numpy-discussion] bug or feature?
In-Reply-To: <200702081621.09495.humufr@yahoo.fr>
References: <200702081621.09495.humufr@yahoo.fr>
Message-ID: <45CB9DC0.3090206@ee.byu.edu>

humufr at yahoo.fr wrote:

>I have a big problem with numpy, numarray and Numeric (all version)
>
>If I'm using the script at the bottom, I obtain these results:
>
>var1 before function is  [3 4 5]
>var2 before function is  1
>var1 after function must be [3 4 5] is  [ 9 12 15]     <------ problem
>var2 after function must be 1 is  1
>var3 must be the [9 12 15] is  [ 9 12 15]
>var4 must be the 'toto' is  toto
>
>
>I'm very surprised by the line noted. I always thinking that the input 
>variable didn't change the variable itself outside the function.
>
To save yourself confusion, you need to understand the difference 
between mutable and immutable types.  Mutable types can be changed 
inside of a function call.

You also need to understand that = is a "name-binding operation only"  
it does not change objects.

>Is it normal and so do I have to do a copy of the input data each time I'm 
>calling a function? 
>  
>
Yes, it's very normal, if your function does an "in-place" operation on 
a mutable type.

Consider the following code:

def test(var1, var2):
      var1[0] *= 3  # this accesses the 0'th element of var1 and alters it.
      var2 = 'toto'  # this makes a new object and names it with var2
                            # whatever was passed in is gone
      return var1, var2

test([1,2,3],[1,2])

will return

[3,2,3], 'toto'



-Travis



From humufr at yahoo.fr  Thu Feb  8 17:15:08 2007
From: humufr at yahoo.fr (humufr at yahoo.fr)
Date: Thu, 8 Feb 2007 17:15:08 -0500
Subject: [Numpy-discussion] bug or feature?
In-Reply-To: <45CB9DC0.3090206@ee.byu.edu>
References: <200702081621.09495.humufr@yahoo.fr> <45CB9DC0.3090206@ee.byu.edu>
Message-ID: <200702081715.08431.humufr@yahoo.fr>

Thank you to both of you for this explanatin. I'm coming from the fortran 
world and so I never had to deal with this before...

Sorry to have polluate the list for a stupid things

Thanks again that clarify the problem 

Nicolas



Le Thursday 08 February 2007 17:01:36 Travis Oliphant, vous avez ?crit?:
> humufr at yahoo.fr wrote:
> >I have a big problem with numpy, numarray and Numeric (all version)
> >
> >If I'm using the script at the bottom, I obtain these results:
> >
> >var1 before function is  [3 4 5]
> >var2 before function is  1
> >var1 after function must be [3 4 5] is  [ 9 12 15]     <------ problem
> >var2 after function must be 1 is  1
> >var3 must be the [9 12 15] is  [ 9 12 15]
> >var4 must be the 'toto' is  toto
> >
> >
> >I'm very surprised by the line noted. I always thinking that the input
> >variable didn't change the variable itself outside the function.
>
> To save yourself confusion, you need to understand the difference
> between mutable and immutable types.  Mutable types can be changed
> inside of a function call.
>
> You also need to understand that = is a "name-binding operation only"
> it does not change objects.
>
> >Is it normal and so do I have to do a copy of the input data each time I'm
> >calling a function?
>
> Yes, it's very normal, if your function does an "in-place" operation on
> a mutable type.
>
> Consider the following code:
>
> def test(var1, var2):
>       var1[0] *= 3  # this accesses the 0'th element of var1 and alters it.
>       var2 = 'toto'  # this makes a new object and names it with var2
>                             # whatever was passed in is gone
>       return var1, var2
>
> test([1,2,3],[1,2])
>
> will return
>
> [3,2,3], 'toto'
>
>
>
> -Travis
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion




From robert.kern at gmail.com  Thu Feb  8 17:28:39 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 08 Feb 2007 14:28:39 -0800
Subject: [Numpy-discussion] bug or feature?
In-Reply-To: <200702081715.08431.humufr@yahoo.fr>
References: <200702081621.09495.humufr@yahoo.fr> <45CB9DC0.3090206@ee.byu.edu>
	<200702081715.08431.humufr@yahoo.fr>
Message-ID: <45CBA417.1020301@gmail.com>

humufr at yahoo.fr wrote:
> Thank you to both of you for this explanatin. I'm coming from the fortran 
> world and so I never had to deal with this before...
> 
> Sorry to have polluate the list for a stupid things

No need to apologize. All programming systems take some getting-used-to at first
before they become natural.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From drife at ucar.edu  Fri Feb  9 09:31:47 2007
From: drife at ucar.edu (Daran L. Rife)
Date: Fri, 9 Feb 2007 07:31:47 -0700 (MST)
Subject: [Numpy-discussion] Request for porting pycdf to NumPy
Message-ID: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>

Hi Travis,

If you're still offering NumPy "patches" to third party
packages that rely upon Numeric, I would really like for
pycdf to be ported to NumPy. This would allow me to
completely transition to NumPy.

Thanks very much for considering my request.


Daran Rife



From oliphant at ee.byu.edu  Fri Feb  9 12:15:20 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri, 09 Feb 2007 10:15:20 -0700
Subject: [Numpy-discussion] Request for porting pycdf to NumPy
In-Reply-To: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>
References: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>
Message-ID: <45CCAC28.3060006@ee.byu.edu>

Daran L. Rife wrote:

>Hi Travis,
>
>If you're still offering NumPy "patches" to third party
>packages that rely upon Numeric, I would really like for
>pycdf to be ported to NumPy. This would allow me to
>completely transition to NumPy.
>
>Thanks very much for considering my request.
>  
>

Is pycdf the same as pynetcdf?  Where is it located?

There is a netcdf in the sandbox of SciPy that works with NumPy.   We 
should get it moved over into the main SciPy tree.

Thanks,

-Travis



From otto at tronarp.se  Sat Feb  3 11:39:11 2007
From: otto at tronarp.se (Otto Tronarp)
Date: Sat, 03 Feb 2007 17:39:11 +0100
Subject: [Numpy-discussion] numarray argmax problem
Message-ID: <20070203173911.ax2xhqvu8sk84g44@mathcore.kicks-ass.org>

Hi,

I have a problem with numarray.argmax, the following code

import numarray as N
import sys
c = N.zeros((10, ), N.Float64)
while 1:
     print 'B: ', sys.getrefcount(None)
     l = N.argmax(c)
     print 'A: ', sys.getrefcount(None)
     print

Dies with:
Fatal Python error: deallocating None
Aborted

I'm using numarray 1.5.2

Any chance that there is an easy fix for this? I know I should  
consider switching to numpy, but I don't have the time right now to do  
that.

Otto



From a.schmolck at gmx.net  Wed Feb  7 08:08:01 2007
From: a.schmolck at gmx.net (Alexander Schmolck)
Date: 07 Feb 2007 13:08:01 +0000
Subject: [Numpy-discussion] force column vector
In-Reply-To: <loom.20070207T113019-604@post.gmane.org>
References: <loom.20070207T113019-604@post.gmane.org>
Message-ID: <yfsabzqt8su.fsf@oc.ex.ac.uk>

Christian <ckkart at hoc.net> writes:

> Hi,
> 
> when creating an ndarray from a list, how can I force the result to be
> 2d *and* a column vector? So in case I pass a nested list, there will be no
> modification of the shape and when I pass a simple list, it will be 
> converted to a 2d column vector. I can only think of a solution using 'if'
> clauses but I suppose there is a more elegant way.

This will always return a column vector, but I'm not sure from you description
if that's what you want (do you want [[1,2,3]] etc. to come out as a row-vector?)

    In [8]: ravel(array([1,2,3]))[:,newaxis]
    Out[8]: 
    array([[1],
           [2],
           [3]])

'as


From robert.kern at gmail.com  Fri Feb  9 12:17:41 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 09 Feb 2007 09:17:41 -0800
Subject: [Numpy-discussion] Request for porting pycdf to NumPy
In-Reply-To: <45CCAC28.3060006@ee.byu.edu>
References: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>
	<45CCAC28.3060006@ee.byu.edu>
Message-ID: <45CCACB5.2030206@gmail.com>

Travis Oliphant wrote:

> Is pycdf the same as pynetcdf?  Where is it located?

I presume it's this one:

  http://pysclint.sourceforge.net/pycdf/

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Fri Feb  9 12:24:03 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 09 Feb 2007 09:24:03 -0800
Subject: [Numpy-discussion] bug or feature?
In-Reply-To: <200702081621.09495.humufr@yahoo.fr>
References: <200702081621.09495.humufr@yahoo.fr>
Message-ID: <45CCAE33.5040701@noaa.gov>

Your question has been answered, but I think a few comments are in order:

1) Read this, everyone new to Python should:
http://python.net/crew/mwh/hacks/objectthink.html

2) > The only object in python with this behavior are the numeric object
(Numeric, numarray or numpy)

Not the case (see the above reference). Num* arrays are different than 
the standard objects in that slicing returns views on data of the array, 
  but all mutable types will behave the same way in your code:

 >>> def test(l):
...    l *= 3
...
 >>> print l
[3, 3, 3]
 >>> test(l)
 >>> print l
[3, 3, 3, 3, 3, 3, 3, 3, 3]

As Travis said, in some languages, the question is "copy or reference?" 
(and Fortran is always reference, IIRC), in Python, the question is 
"mutable or immutable?". Immutable objects can not be effected when 
passed into function, mutable objects can. Using the same test function 
above:

 >>> i = 5
 >>> test(5)
 >>> i
5

integers are immutable, so i was not changed (more accurately, the 
object bound to i was not changed).

Compounding the confusion, is this:

the *=, etc operators mean: "mutate the object in place if possible, 
otherwise return a new object". This is confusing as that means that 
some objects (ndarrays, list), will get changed by a function like that, 
and others (ints, floats, tuples) will not.

Personally, I would be happier if +=, etc. meant "mutate the object in 
place if possible, otherwise raise an exception", but then you couldn't do

i = 0
i += 1

which is probably the most common use.

I think the mistake arose from trying to solve two totally different 
problems at once: numpy users and the like wanted a clean and compact 
way to mutate in place. Lots of others (particularly those coming from 
the C family of languages) wanted a quick and easy notation for 
"increment this value". Since python numbers are not mutable, these 
CAN'T be the same thing, so using the same notation for both causes 
confusion.

Of course, you can now use numpy rank zero arrays almost like mutable 
numbers:

 >>> s = N.array(5)
 >>> test(s)
 >>> s
array(15)

By the way, there are times when I think mutable scalars would be handy. 
rank-zero arrays almost fit the bill, but I can't see a way to directly 
re-set their value, and they don't seem to take precedence in operations 
with other numbers:

 >>> s = N.array(5)
 >>> type(s)
<type 'numpy.ndarray'>

 >>> s2 = 4 * s
 >>> type(s2)
<type 'numpy.int32'>
# so I got a numpy scalar back instead of a rank-zero array

However:
 >>> s = N.array((5,6))
 >>> type(s)
<type 'numpy.ndarray'>

 >>> s2 = 4 * s
 >>> type(s2)
<type 'numpy.ndarray'>

This stayed an array, as, of course, it would have to! I now the whole 
"how the heck should a rank-zero array" behave? has been discussed a 
lot, but I still wonder if this is how it should be.

Maybe having a whole new object that explicitly defined as a mutable 
scalar would be the way to do it. It would probably have less overhead 
than a rank-zero array as well.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From Chris.Barker at noaa.gov  Fri Feb  9 12:33:24 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 09 Feb 2007 09:33:24 -0800
Subject: [Numpy-discussion] Request for porting pycdf to NumPy
In-Reply-To: <45CCACB5.2030206@gmail.com>
References: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>
	<45CCAC28.3060006@ee.byu.edu> <45CCACB5.2030206@gmail.com>
Message-ID: <45CCB064.4010706@noaa.gov>

hmmm, this page from unidata:

http://www.unidata.ucar.edu/software/netcdf/software.html#Python

Indicates 4-6 different Python netcfd interfaces!

Has anyone done a review of these to see where effort would best be put 
into upgrading?

I've used Konrad Hinsen's package a little bit with success, and I think 
he's upgraded all of Scientific Python to support numpy now, but I know 
nothing of the others.

I think it would be ideal if Unidata could be persuaded to adopt an 
maintain one as the official version to save this extra work an 
confusion. Maybe there is some hope of this for netcdf4.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From jmiller at stsci.edu  Fri Feb  9 13:02:08 2007
From: jmiller at stsci.edu (Todd Miller)
Date: Fri, 09 Feb 2007 13:02:08 -0500
Subject: [Numpy-discussion] numarray argmax problem
In-Reply-To: <20070203173911.ax2xhqvu8sk84g44@mathcore.kicks-ass.org>
References: <20070203173911.ax2xhqvu8sk84g44@mathcore.kicks-ass.org>
Message-ID: <45CCB720.70705@stsci.edu>

Otto Tronarp wrote:
> Hi,
>
> I have a problem with numarray.argmax, the following code
>
> import numarray as N
> import sys
> c = N.zeros((10, ), N.Float64)
> while 1:
>      print 'B: ', sys.getrefcount(None)
>      l = N.argmax(c)
>      print 'A: ', sys.getrefcount(None)
>      print
>
> Dies with:
> Fatal Python error: deallocating None
> Aborted
>
> I'm using numarray 1.5.2
>
> Any chance that there is an easy fix for this? I know I should  
>   
If you're willing to check out numarray and build it from source 
yourself,  this problem is fixed in numarray CVS here:

http://sourceforge.net/cvs/?group_id=1369

You can check it out like this:

cvs -d:pserver:anonymous at numpy.cvs.sourceforge.net:/cvsroot/numpy login
 
cvs -z3 -d:pserver:anonymous at numpy.cvs.sourceforge.net:/cvsroot/numpy co 
-P /numarray
/

Regards,
Todd
> consider switching to numpy, but I don't have the time right now to do  
> that.
>
> Otto
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From wfspotz at sandia.gov  Fri Feb  9 13:12:42 2007
From: wfspotz at sandia.gov (Bill Spotz)
Date: Fri, 9 Feb 2007 11:12:42 -0700
Subject: [Numpy-discussion] Request for porting pycdf to NumPy
In-Reply-To: <45CCB064.4010706@noaa.gov>
References: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>
	<45CCAC28.3060006@ee.byu.edu> <45CCACB5.2030206@gmail.com>
	<45CCB064.4010706@noaa.gov>
Message-ID: <981E18ED-8CE6-49F7-A583-EF57B8F97163@sandia.gov>

I believe the person most qualified to answer that question would be  
Mary Haley, (haley at ucar.edu).

On Feb 9, 2007, at 10:33 AM, Christopher Barker wrote:

> Has anyone done a review of these to see where effort would best be  
> put
> into upgrading?

** Bill Spotz                                              **
** Sandia National Laboratories  Voice: (505)845-0170      **
** P.O. Box 5800                 Fax:   (505)284-5451      **
** Albuquerque, NM 87185-0370    Email: wfspotz at sandia.gov **





From drife at ucar.edu  Fri Feb  9 13:30:45 2007
From: drife at ucar.edu (Daran Rife)
Date: Fri, 9 Feb 2007 11:30:45 -0700
Subject: [Numpy-discussion] Request for porting pycdf to NumPy
Message-ID: <D01E33D1-8DF4-47D3-9C42-8638D61FDC37@ucar.edu>

Yep, I was referring to: http://pysclint.sourceforge.net/pycdf/

Regarding the issue of deciding which netCDF interface to adopt
as the "standard", although it is unlikely we'll ever get consensus
on this, I have tried several of the netCDF interfaces, including
the one in Scientific, and have found that pycdf is the "cleanest";
it is very easy to use and understand.


Daran

--

Travis Oliphant wrote:


> Is pycdf the same as pynetcdf?  Where is it located?
>

I presume it's this one:

   http://pysclint.sourceforge.net/pycdf/



From efiring at hawaii.edu  Fri Feb  9 14:19:37 2007
From: efiring at hawaii.edu (Eric Firing)
Date: Fri, 09 Feb 2007 09:19:37 -1000
Subject: [Numpy-discussion] Request for porting pycdf to NumPy
In-Reply-To: <D01E33D1-8DF4-47D3-9C42-8638D61FDC37@ucar.edu>
References: <D01E33D1-8DF4-47D3-9C42-8638D61FDC37@ucar.edu>
Message-ID: <45CCC949.9050408@hawaii.edu>

I have been using Jeff Whitaker's netcdf4 interface with good results.

I could not find the web page for it on a NOAA site--I think NOAA is 
reorganizing--but a search turned it up here.  Maybe Jeff can provide a 
better link.

http://netcdf4-python.googlecode.com/svn/trunk/docs/netCDF4-module.html

Eric

Daran Rife wrote:
> Yep, I was referring to: http://pysclint.sourceforge.net/pycdf/
> 
> Regarding the issue of deciding which netCDF interface to adopt
> as the "standard", although it is unlikely we'll ever get consensus
> on this, I have tried several of the netCDF interfaces, including
> the one in Scientific, and have found that pycdf is the "cleanest";
> it is very easy to use and understand.
> 
> 
> Daran
> 
> --
> 
> Travis Oliphant wrote:
> 
> 
>> Is pycdf the same as pynetcdf?  Where is it located?
>>
> 
> I presume it's this one:
> 
>    http://pysclint.sourceforge.net/pycdf/
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From mithrandir42 at web.de  Fri Feb  9 14:22:49 2007
From: mithrandir42 at web.de (N. Volbers)
Date: Fri, 09 Feb 2007 20:22:49 +0100
Subject: [Numpy-discussion] Request for porting pycdf to NumPy
In-Reply-To: <D01E33D1-8DF4-47D3-9C42-8638D61FDC37@ucar.edu>
References: <D01E33D1-8DF4-47D3-9C42-8638D61FDC37@ucar.edu>
Message-ID: <45CCCA09.80603@web.de>

Daran Rife schrieb:
> Yep, I was referring to: http://pysclint.sourceforge.net/pycdf/
>
> Regarding the issue of deciding which netCDF interface to adopt
> as the "standard", although it is unlikely we'll ever get consensus
> on this, I have tried several of the netCDF interfaces, including
> the one in Scientific, and have found that pycdf is the "cleanest";
> it is very easy to use and understand.
>
>   
+1 for this one for the same reason. I have tried two or three different 
ones and pycdf was by far the easiest to use.

Niklas Volbers.


From jswhit at fastmail.fm  Fri Feb  9 14:31:31 2007
From: jswhit at fastmail.fm (Jeff Whitaker)
Date: Fri, 09 Feb 2007 12:31:31 -0700
Subject: [Numpy-discussion] Request for porting pycdf to NumPy
In-Reply-To: <45CCC949.9050408@hawaii.edu>
References: <D01E33D1-8DF4-47D3-9C42-8638D61FDC37@ucar.edu>
	<45CCC949.9050408@hawaii.edu>
Message-ID: <45CCCC13.8000708@fastmail.fm>

Eric Firing wrote:
> I have been using Jeff Whitaker's netcdf4 interface with good results.
>
> I could not find the web page for it on a NOAA site--I think NOAA is 
> reorganizing--but a search turned it up here.  Maybe Jeff can provide 
> a better link.
>
> http://netcdf4-python.googlecode.com/svn/trunk/docs/netCDF4-module.html
>
> Eric
>
Eric:

Yep, that's a link to the docs from the google code homepage

http://code.google.com/p/netcdf4-python/

AFAIK, this is the only one of the python interfaces that supports both 
the netcdf version 3 API and the new (still in alpha) version 4 API, 
which is built on top of HDF5.

As far as getting unidata to support  or bless an 'official' python 
interface, that's not going to happen.  They barely have enough staff to 
support the C, fortran and Java interfaces.

-Jeff

-- 
Jeffrey S. Whitaker         Phone  : (303)497-6313
Meteorologist               FAX    : (303)497-6449
NOAA/OAR/PSD  R/PSD1        Email  : Jeffrey.S.Whitaker at noaa.gov
325 Broadway                Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web    : http://tinyurl.com/5telg



From hetland at tamu.edu  Fri Feb  9 14:35:27 2007
From: hetland at tamu.edu (Rob Hetland)
Date: Fri, 9 Feb 2007 13:35:27 -0600
Subject: [Numpy-discussion] Request for porting pycdf to NumPy
In-Reply-To: <45CCC949.9050408@hawaii.edu>
References: <D01E33D1-8DF4-47D3-9C42-8638D61FDC37@ucar.edu>
	<45CCC949.9050408@hawaii.edu>
Message-ID: <1E07F547-E441-44C2-B43F-C5596DD301A4@tamu.edu>



+1 for Whitaker's package.

The new location of the package is:

http://code.google.com/p/netcdf4-python/

I use netcdf for everything I do, and so I checked out all of the  
packages in detail.  All follow a similar API, although there are  
very minor differences that will catch you if you are not careful  
(like writing a single character).

Although this is (austensibly) made for netcdf4, Whitaker's package  
works well both reading and writing netcdf3 files.  It also has  
support for stitching together multiple files.  It also has support  
for compression of numpy4 data (as well as all of the other cool  
netcdf4 features).  The only disadvantage is that netcdf4 is a bit  
harder to install than netcdf3 -- primarily because the user must  
first install hdf.  However, recent versions work fine on all the  
platforms I use (Mac and Redhat).

Clearly this is the way to go.

-Rob


On Feb 9, 2007, at 1:19 PM, Eric Firing wrote:

> I have been using Jeff Whitaker's netcdf4 interface with good results.
>
> I could not find the web page for it on a NOAA site--I think NOAA is
> reorganizing--but a search turned it up here.  Maybe Jeff can  
> provide a
> better link.
>
> http://netcdf4-python.googlecode.com/svn/trunk/docs/netCDF4- 
> module.html
>
> Eric
>
> Daran Rife wrote:
>> Yep, I was referring to: http://pysclint.sourceforge.net/pycdf/
>>
>> Regarding the issue of deciding which netCDF interface to adopt
>> as the "standard", although it is unlikely we'll ever get consensus
>> on this, I have tried several of the netCDF interfaces, including
>> the one in Scientific, and have found that pycdf is the "cleanest";
>> it is very easy to use and understand.
>>
>>
>> Daran
>>
>> --
>>
>> Travis Oliphant wrote:
>>
>>
>>> Is pycdf the same as pynetcdf?  Where is it located?
>>>
>>
>> I presume it's this one:
>>
>>    http://pysclint.sourceforge.net/pycdf/
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

----
Rob Hetland, Associate Professor
Dept. of Oceanography, Texas A&M University
http://pong.tamu.edu/~rob
phone: 979-458-0096, fax: 979-845-6331




From christian at marquardt.sc  Fri Feb  9 16:00:27 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Fri, 9 Feb 2007 22:00:27 +0100 (CET)
Subject: [Numpy-discussion] Request for porting pycdf to NumPy
In-Reply-To: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>
References: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>
Message-ID: <21280.84.167.122.11.1171054827.squirrel@webmail.marquardt.sc>

Dear list,

attached is a patch for the original pycdf-0.6.2-rc1 distribution as
available through sourceforge - and a little shell script illustrating
how to install it. After applyng the patch, it should be configured for
numpy. Note that the "installation" script uses an environment variable
PREFIX to define where the pacjkage is installed; it also assumes that
the netcdf libraries are installed in $PREFIX/lib.

The orginal author already supported both numeric and numarray, so I just
added a new subdirectory for numpy - which is simply the numeric version
slightly changed. The patch is only that large because it replicates much
of already existing code...

I have been using this "port" for many weeks now without any problems or
difficulties. I hope it's useful for others as well;-)

  Christian.



On Fri, February 9, 2007 15:31, Daran L. Rife wrote:
> Hi Travis,
>
> If you're still offering NumPy "patches" to third party
> packages that rely upon Numeric, I would really like for
> pycdf to be ported to NumPy. This would allow me to
> completely transition to NumPy.
>
> Thanks very much for considering my request.
>
>
> Daran Rife
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pycdf-0.6.2-rc1-numpy.patch.gz
Type: application/x-gzip
Size: 27143 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070209/435ca031/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build_pycdf-0.6-2-rc1-linux.sh
Type: application/x-shellscript
Size: 585 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070209/435ca031/attachment-0001.bin>

From christian at marquardt.sc  Fri Feb  9 16:14:34 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Fri, 9 Feb 2007 22:14:34 +0100 (CET)
Subject: [Numpy-discussion] pyhdf / was: Request for porting pycdf to NumPy
In-Reply-To: <21280.84.167.122.11.1171054827.squirrel@webmail.marquardt.sc>
References: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>
	<21280.84.167.122.11.1171054827.squirrel@webmail.marquardt.sc>
Message-ID: <4069.84.167.122.11.1171055674.squirrel@webmail.marquardt.sc>

As we are at it,

Andre Gosselin (the guy who wrote pycdf) also wrote an interface to HDF4
(not 5) named pyhdf. I'm using that with numpy as well (patch attached),
but I haven't tested it much - little more than just running the examples,
really (which appear to be ok). Maybe it's useful...

In pyhdf, the author didn't support different array interfaces; so the
attached patch just modifes the existing source code and moves it over
to numpy. I've also attached a "install script" which makes use of the
particular lacotion of the HDF4 libraries (in $PREFIX/lib) and header
files (in $PREFIX/include/hdf), so it it almost certainly needs to be
adapted to the location of the actual HDF libraries and headers.

Regards,

  Christian.


On Fri, February 9, 2007 22:00, Christian Marquardt wrote:
> Dear list,
>
> attached is a patch for the original pycdf-0.6.2-rc1 distribution as
> available through sourceforge - and a little shell script illustrating
> how to install it. After applyng the patch, it should be configured for
> numpy. Note that the "installation" script uses an environment variable
> PREFIX to define where the pacjkage is installed; it also assumes that
> the netcdf libraries are installed in $PREFIX/lib.
>
> The orginal author already supported both numeric and numarray, so I just
> added a new subdirectory for numpy - which is simply the numeric version
> slightly changed. The patch is only that large because it replicates much
> of already existing code...
>
> I have been using this "port" for many weeks now without any problems or
> difficulties. I hope it's useful for others as well;-)
>
>   Christian.
>
>
>
> On Fri, February 9, 2007 15:31, Daran L. Rife wrote:
>> Hi Travis,
>>
>> If you're still offering NumPy "patches" to third party
>> packages that rely upon Numeric, I would really like for
>> pycdf to be ported to NumPy. This would allow me to
>> completely transition to NumPy.
>>
>> Thanks very much for considering my request.
>>
>>
>> Daran Rife
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pyhdf-0.7-3-numpy.patch
Type: text/x-patch
Size: 12890 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070209/ecd37781/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build_pyhdf-0.7-3-numpy-linux.sh
Type: application/x-shellscript
Size: 604 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070209/ecd37781/attachment-0001.bin>

From oliphant at ee.byu.edu  Fri Feb  9 16:16:57 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri, 09 Feb 2007 14:16:57 -0700
Subject: [Numpy-discussion] pyhdf / was: Request for porting pycdf to
 NumPy
In-Reply-To: <4069.84.167.122.11.1171055674.squirrel@webmail.marquardt.sc>
References: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>	<21280.84.167.122.11.1171054827.squirrel@webmail.marquardt.sc>
	<4069.84.167.122.11.1171055674.squirrel@webmail.marquardt.sc>
Message-ID: <45CCE4C9.5020504@ee.byu.edu>

Christian Marquardt wrote:

>As we are at it,
>
>Andre Gosselin (the guy who wrote pycdf) also wrote an interface to HDF4
>(not 5) named pyhdf. I'm using that with numpy as well (patch attached),
>but I haven't tested it much - little more than just running the examples,
>really (which appear to be ok). Maybe it's useful...
>
>In pyhdf, the author didn't support different array interfaces; so the
>attached patch just modifes the existing source code and moves it over
>to numpy. I've also attached a "install script" which makes use of the
>particular lacotion of the HDF4 libraries (in $PREFIX/lib) and header
>files (in $PREFIX/include/hdf), so it it almost certainly needs to be
>adapted to the location of the actual HDF libraries and headers.
>
>Regards,
>  
>

Great job.   Thanks so much.  Would you mind adding these patches to the 
scipy web page.

Or, perhaps we should place them in the numpy svn directory somewhere.

-Travis



From Chris.Barker at noaa.gov  Fri Feb  9 16:28:15 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 09 Feb 2007 13:28:15 -0800
Subject: [Numpy-discussion] pyhdf / was: Request for porting pycdf to
	NumPy
In-Reply-To: <4069.84.167.122.11.1171055674.squirrel@webmail.marquardt.sc>
References: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>
	<21280.84.167.122.11.1171054827.squirrel@webmail.marquardt.sc>
	<4069.84.167.122.11.1171055674.squirrel@webmail.marquardt.sc>
Message-ID: <45CCE76F.70301@noaa.gov>

> Andre Gosselin (the guy who wrote pycdf) also wrote an interface to HDF4
> (not 5) named pyhdf. 

Is he still maintaining these packages? Have you submitted the patches 
to him?

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From evanmason at gmail.com  Fri Feb  9 17:22:41 2007
From: evanmason at gmail.com (Evan Mason)
Date: Fri, 9 Feb 2007 14:22:41 -0800
Subject: [Numpy-discussion] averaging array, with fill values
Message-ID: <4254ff910702091422j78fff464i1c80fe0c20555399@mail.gmail.com>

Hi, I want to get the mean over an array with 3 dimensions.  The array
has, say, dimensions 2x3x4 and I want the result to be 3x4; so the
averaging is to be done over the 1st dimension.  Also, there are some
fill values, and I want these to be excluded from the calculation.  I
know I can use the ma module to locate the fill values, but after that
I am unsure how to exclude them, and how to do the averaging.

In [150]: x = rand(2,3,4)

In [151]: x[0,2,1]=999 # fill value

In [152]: x[1,2,3]=999

In [153]: x[0,1,3]=999

In [154]: x
Out[154]:
array([[[  3.01880915e-02,   5.77085271e-02,   7.59176038e-01,
          4.15271486e-01],
       [  5.48643693e-01,   3.84995126e-01 ,   5.01683678e-01,
          9.99000000e+02],
       [  5.72779004e-01,   9.99000000e+02,   4.18143934e-01,
          5.84781674e-01]],

      [[  8.90443158e-01,   3.76986788e-01,   8.81270409e-01,
          5.19094405e-01],
       [  8.12944573e-01,   3.89858156e-01,   5.99219891e-01,
          9.99000000e+02],
       [  2.31215256e-01,   5.93222883e-01,   2.45004093e-01,
          9.18647954e-01]]])

In [155]:

Thanks in advance for any help.  -Evan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070209/5a3f4836/attachment.html>

From christian at marquardt.sc  Fri Feb  9 17:24:56 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Fri, 9 Feb 2007 23:24:56 +0100 (CET)
Subject: [Numpy-discussion] pyhdf / was: Request for porting pycdf to
 NumPy
In-Reply-To: <45CCE76F.70301@noaa.gov>
References: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>
	<21280.84.167.122.11.1171054827.squirrel@webmail.marquardt.sc>
	<4069.84.167.122.11.1171055674.squirrel@webmail.marquardt.sc>
	<45CCE76F.70301@noaa.gov>
Message-ID: <27130.84.167.122.11.1171059896.squirrel@webmail.marquardt.sc>

On Fri, February 9, 2007 22:28, Christopher Barker wrote:
>> Andre Gosselin (the guy who wrote pycdf) also wrote an interface to HDF4
>> (not 5) named pyhdf.
>
> Is he still maintaining these packages? Have you submitted the patches
> to him?

a) Don't know; the last releases of pycdf and pyhdf are from February 2001
and July 2005, respectively.

b) Yes, but just half an hour ago, after I had seen the request for pycdf
here.

I would actually prefer if Andre would apply the patches in some way in
his distribution. The C wrappers for both pycdf and pyhdf are created with
swig,
but the original interface description files are not included in the
distribution. So I patched the generated wrapper code instead of the
original files. Or rather let Travis' alter_codeN do the job;-))

  Chris.


>
> -Chris
>
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>




From christian at marquardt.sc  Fri Feb  9 17:27:05 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Fri, 9 Feb 2007 23:27:05 +0100 (CET)
Subject: [Numpy-discussion] pyhdf / was: Request for porting pycdf to
 NumPy
In-Reply-To: <27130.84.167.122.11.1171059896.squirrel@webmail.marquardt.sc>
References: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>
	<21280.84.167.122.11.1171054827.squirrel@webmail.marquardt.sc>
	<4069.84.167.122.11.1171055674.squirrel@webmail.marquardt.sc>
	<45CCE76F.70301@noaa.gov>
	<27130.84.167.122.11.1171059896.squirrel@webmail.marquardt.sc>
Message-ID: <23364.84.167.122.11.1171060025.squirrel@webmail.marquardt.sc>


Oops!

> a) Don't know; the last releases of pycdf and pyhdf are from February 2001
                                                                        ^^^^
pycdf is from 2006, of course. Sorry!

  Chris.




From nicolas.grilly at garden-paris.com  Fri Feb  9 22:11:45 2007
From: nicolas.grilly at garden-paris.com (Nicolas Grilly)
Date: Sat, 10 Feb 2007 04:11:45 +0100
Subject: [Numpy-discussion] Function numpy.get_include() has a side effet
Message-ID: <554cfd4f0702091911w662b09e5t1abf3e32ba0685d5@mail.gmail.com>

Hello,

I have a package with a distutils script. This script calls
numpy.get_include(), in order to get numpy's header files location.
But the function seems to have a side effect by silently patching
distutils. Some classes and functions of standard library distutils
are replaced by numpy's ones.

For example, before calling numpy.get_include(), the function
new_compiler comes from standard library:

>>> from distutils.ccompiler import new_compiler
>>> new_compiler.__module__
'distutils.ccompiler'

But after calling numpy.get_include(), the function comes from numpy:

>>> import numpy
>>> numpy.get_include()
>>> from distutils.ccompiler import new_compiler
>>> new_compiler.__module__
'numpy.distutils.ccompiler'

This is because the code of numpy.get_include() imports
numpy.distutils, and this triggers the patching mechanism.

This is a problem because the behavior of distutils is different
before and after the call to numpy.get_include(), and it breaks my
distutils script.

Is it possible to remove this side effect?

Thanks,

Nicolas Grilly


From kwgoodman at gmail.com  Sat Feb 10 11:25:44 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Sat, 10 Feb 2007 08:25:44 -0800
Subject: [Numpy-discussion] fill_value in masked arrays
Message-ID: <f4f93d420702100825n3c038fm1368ed10d9d76554@mail.gmail.com>

The doc strings for MaskedArray and ma.argsort do not seem consistent.

The doc string for MaskedArray says "The fill_value is not used for
computation within this module." But the doc string for ma.argsort
says "Treating masked values as if they have the value fill_value,
return sort indices for sorting along given axis."

I'm trying to argsort matrices that have missing values.


From matthew.brett at gmail.com  Sun Feb 11 09:30:17 2007
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sun, 11 Feb 2007 14:30:17 +0000
Subject: [Numpy-discussion] Fwd: Numpy x Matlab: some synthetic benchmarks
In-Reply-To: <1137584647.19613.17.camel@localhost.localdomain>
References: <1137584647.19613.17.camel@localhost.localdomain>
Message-ID: <1e2af89e0702110630q7e274603j6ce53061ae6502d0@mail.gmail.com>

Hi,

I don't know if people remember this thread, but my memory was that
there might be some interest in including numpy / matlab benchmark
code somewhere in the distribution, to check on performance and allow
matlab people to do a direct comparison.  Is this still of interest?
If so, what should be the next step?

Thanks a lot,

Matthew

---------- Forwarded message ----------
From: Paulo Jose da Silva e Silva <pjssilva at ime.usp.br>
Date: Jan 18, 2006 11:44 AM
Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks
To: numpy-discussion <numpy-discussion at lists.sourceforge.net>


Hello,

Travis asked me to benchmark numpy versus matlab in some basic linear
algebra operations. Here are the resuts for matrices/vectors of
dimensions 5, 50 and 500:

Operation       x'*y    x*y'    A*x     A*B     A'*x    Half    2in2

Dimension 5
Array           0.94    0.7     0.22    0.28    1.12    0.98    1.1
Matrix          7.06    1.57    0.66    0.79    1.6     3.11    4.56
Matlab          1.88    0.44    0.41    0.35    0.37    1.2     0.98

Dimension 50
Array           9.74    3.09    0.56    18.12   13.93   4.2     4.33
Matrix          81.99   3.81    1.04    19.13   14.58   6.3     7.88
Matlab          16.98   1.94    1.07    17.86   0.73    1.57    1.77

Dimension 500
Array           1.2     8.97    2.03    166.59  20.34   3.99    4.31
Matrix          17.95   9.09    2.07    166.62  20.67   4.11    4.45
Matlab          2.09    6.07    2.17    169.45  2.1     2.56    3.06

Obs: The operation Half is actually A*x using only the lower half of the
matrix and vector. The operation 2in2 is A*x using only the even
indexes.

Of course there are many repetitions of the same operation: 100000 for
dim 5 and 50 and 1000 for dim 500. The inner product is number of
repetitions is multiplied by dimension (it is very fast).

The software is

numpy svn version 1926
Matlab 6.5.0.180913a Release 13 (Jun 18 2002)

Both softwares are using the *same* BLAS and LAPACK (ATLAS for sse).

As you can see, numpy array looks very competitive. The matrix class in
numpy has too much overhead for small dimension though. This overhead is
very small for medium size arrays. Looking at the results above
(specially the small dimensions ones, for higher dimensions the main
computations are being performed by the same BLAS) I believe we can say:

1) Numpy array is faster on usual operations but outerproduct (I believe
the reason is that the dot function uses the regular matrix
multiplication to compute outer-products, instead of using a special
function. This can "easily" changes). In particular numpy was faster in
matrix times vector operations, which is the most usual in numerical
linear algebra.

2) Any operation that involves transpose suffers a very big penalty in
numpy. Compare A'*x and A*x, it is 10 times slower. In contrast Matlab
deals with transpose quite well. Travis is already aware of this and it
can be probably solved.

3) When using subarrays, numpy is a slower. The difference seems
acceptable. Travis, can this be improved?

Best,

Paulo

Obs: Latter on (in a couple of days) I may present less synthetic
benchmarks (a QR factorization and a Modified Cholesky).



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion


From pgmdevlist at gmail.com  Sun Feb 11 13:14:37 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 11 Feb 2007 13:14:37 -0500
Subject: [Numpy-discussion] fill_value in masked arrays
In-Reply-To: <f4f93d420702100825n3c038fm1368ed10d9d76554@mail.gmail.com>
References: <f4f93d420702100825n3c038fm1368ed10d9d76554@mail.gmail.com>
Message-ID: <200702111314.38064.pgmdevlist@gmail.com>

On Saturday 10 February 2007 11:25:44 Keith Goodman wrote:
> The doc strings for MaskedArray and ma.argsort do not seem consistent.
numpy.core.ma ? Or the other implementation in the SVN ?

> The doc string for MaskedArray says "The fill_value is not used for
> computation within this module." But the doc string for ma.argsort
> says "Treating masked values as if they have the value fill_value,

The fill_value is used to convert a MaskedArray to a regular ndarray. When 
sorting an array with missing data, the fill_value is used to find the end at 
which the missing data should be: for example, when sorting a 1D float array, 
a fill_value of -inf will put the missing data on the left (beginning, small 
indices), a fill_value of +inf will put the missing data on the right (end, 
large indices). 
Other than that, the fill_value is not used in computations: for sum, it is 
temporarily set to 0, for prod, to 1...

> I'm trying to argsort matrices that have missing values.

You have an argsort method and an argsort function that behaves similarly. The 
output is an array of indices. Depending on the fill_value you chose, the 
indices corresponding to the missing data wil be either at the beginning or 
the end.

example:
>>>x=array([0,2,4,3,1],mask=[0,0,1,1,0])
>>>x
masked_array(data = [0 2 -- -- 1],
      mask = [False False  True  True False],
      fill_value=999999)

>>>x.argsort(fill_value=-999999)
array([2, 3, 0, 4, 1])
>>>x.argsort(fill_value=+999999)
array([0, 4, 1, 2, 3])

Now, if you want only the indices that are not masked, you can use something 
like that:
>>>a=x.argsort(fill_value=+99999)
>>>a[~x._mask[a]]
array([0,4,1])



From david at ar.media.kyoto-u.ac.jp  Mon Feb 12 00:02:08 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 12 Feb 2007 14:02:08 +0900
Subject: [Numpy-discussion] Possible solution to binary distributionproblems
	for numpy on linux?
Message-ID: <45CFF4D0.5080401@ar.media.kyoto-u.ac.jp>

Hi there,

    I came across an interesting post on Miguel De Icaza's blog this 
week-end:

    http://tirania.org/blog/archive/2007/Jan-26.html

    The build system used in opensuse is open sourced for a few weeks. 
The basic idea is that it provides a build farm to build packages for 
most major distributions, automatically, with automatic dependency 
tracking for rebuilding a package when its dependencies changed, etc... [1]

    My questions are:
       - does it seem interesting to numpy developers ? My impression is 
that binary distribution of numpy is a big problem for many linux users, 
and that is entry barrier for many users (I may be wrong, that's just an 
impression from the ML).
       - the registration requires agreement from the open build 
system's team for now. I would be interesting in trying this out, but I 
didn't want to "proclaim" myself as a numpy developer without consent 
from the numpy dev team.

    cheers,

    David

    [1]I have not studied throughly, but the idea is:
    - you submit the sources of your package +  a description file
    - you upload it to the build system
    - the build systeme consists in a build farm to build binary 
packages automatically for many distribution (including opensuse, Suse, 
fedora, ubuntu and debian; the biggest distribution in term of 
marketshare which are not there are slackware + gentoo, but I guess 
users of those distribution would know enough to compile packages 
themselves).
  
    Besides the build farm, some advantages are:
    - automatic rebuilding when one of the dependency changed (let's say 
the fortran compiler changed in debian -> numpy which depends on it 
would be rebuilt automatically)
    - a system for mirroring.

    The system is still in beta, and requires registration for trying it 
as a developer.


From eric at enthought.com  Mon Feb 12 00:19:52 2007
From: eric at enthought.com (eric jones)
Date: Sun, 11 Feb 2007 23:19:52 -0600
Subject: [Numpy-discussion] pickling ufuncs?
Message-ID: <45CFF8F8.40206@enthought.com>

I recently noticed that we can't pickle ufuncs (like sin, ...).  Is 
there any technical reason this doesn't work, or is it in the category 
of 'just needs to be done...'

FYI, I noticed that it didn't work on the old Numeric either.

thanks,
eric

Python 2.4.3 - Enthought Edition 1.0.0 (#69, Aug  2 2006, 12:09:59) [MSC 
v.1310
32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
 >>> import pickle
 >>> import numpy
 >>> import math
 >>> pickle.dumps(numpy.arange)
'cnumpy.core.multiarray\narange\np0\n.'
 >>> pickle.dumps(math.sin)
'cmath\nsin\np0\n.'
 >>> pickle.dumps(numpy.sin)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "C:\Python24\lib\pickle.py", line 1386, in dumps
    Pickler(file, protocol, bin).dump(obj)
  File "C:\Python24\lib\pickle.py", line 231, in dump
    self.save(obj)
  File "C:\Python24\lib\pickle.py", line 313, in save
    rv = reduce(self.proto)
  File "C:\Python24\lib\copy_reg.py", line 69, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle ufunc objects
 >>>


From gleban at gmail.com  Mon Feb 12 03:51:54 2007
From: gleban at gmail.com (Gregor Leban)
Date: Mon, 12 Feb 2007 09:51:54 +0100
Subject: [Numpy-discussion] using a vector of values in masked array's
	filled() function
Message-ID: <b2b547200702120051t6ac1d48cua3255a0da52542b6@mail.gmail.com>

Hi,

I'm converting my Numeric code to numpy and I ran into a problem.

In Numeric the MA.filled() function was able to accept a vector of fill
values and it used the first value in the vector to fill the missing values
in the first row of the matrix, the second value for the second row, etc.
The filled() function in numpy now throws an exception "ValueError: array is
not broadcastable to correct shape" if I give it a vector of fill values.

Is there any other way or function in numpy that would perform the thing as
Numeric did? I know that I could write a loop but I often have data with a
huge number of rows and I would prefer some optimized way of doing this.

Best regards,
Gregor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070212/bbd14ec0/attachment.html>

From robert.kern at gmail.com  Mon Feb 12 09:01:25 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 12 Feb 2007 08:01:25 -0600
Subject: [Numpy-discussion] pickling ufuncs?
In-Reply-To: <45CFF8F8.40206@enthought.com>
References: <45CFF8F8.40206@enthought.com>
Message-ID: <45D07335.7080805@gmail.com>

eric jones wrote:
> I recently noticed that we can't pickle ufuncs (like sin, ...).  Is 
> there any technical reason this doesn't work, or is it in the category 
> of 'just needs to be done...'

We talked about this about two weeks ago:

  http://projects.scipy.org/pipermail/numpy-discussion/2007-January/025778.html

In short, there is a technical reason that it doesn't work in general, there is
a workaround that will work for all of the ufuncs that are exposed in modules
(but not those created on the fly by scipy.vectorize(), for example), and that
workaround is in the category of "just needs to be done."

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From eike.welk at gmx.net  Mon Feb 12 12:29:40 2007
From: eike.welk at gmx.net (Eike Welk)
Date: Mon, 12 Feb 2007 18:29:40 +0100
Subject: [Numpy-discussion] Possible solution to binary
 distributionproblems for numpy on linux?
In-Reply-To: <45CFF4D0.5080401@ar.media.kyoto-u.ac.jp>
References: <45CFF4D0.5080401@ar.media.kyoto-u.ac.jp>
Message-ID: <200702121829.41208.eike.welk@gmx.net>

Hello David!

You may get in contact with Werner Hoch. He currently creates NumPy, 
SciPy, and Matplotlib packages for Opensuse.

werner dot ho (at the server) gmx dot de

The packages are here:
http://repos.opensuse.org/science/

Regards
Eike.



From gosselina at dfo-mpo.gc.ca  Mon Feb 12 12:45:26 2007
From: gosselina at dfo-mpo.gc.ca (Andre Gosselin)
Date: Mon, 12 Feb 2007 12:45:26 -0500
Subject: [Numpy-discussion] New release of pycdf package ported to NumPy
In-Reply-To: <1171247042.4560.6.camel@maboule>
References: <1171247042.4560.6.camel@maboule>
Message-ID: <1171302326.4803.4.camel@maboule>

Hi everybody.

I have just released version 0.6-3 of pycdf, which finally supports the
NumPy array package (along Numeric and numarray).

Here is the announcement.
==================================
Project "pysclint" ('pysclint') has released the new version of package
'pycdf'.
You can download it from SourceForge.net by following this link:
<https://sourceforge.net/project/showfiles.php?group_id=143145&release_id=485773>

or browse Release Notes and ChangeLog by visiting this link:
<https://sourceforge.net/project/shownotes.php?release_id=485773> 
==================================

I was pleased to follow a previous thread about pycdf, and see the number of
people who are pleased with it. 

To remove any doubt, I will still actively maintain pycdf.

Best regards,

Andr? Gosselin
gosselina at dfo-mpo.gc.ca




From kwgoodman at gmail.com  Mon Feb 12 12:45:42 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Mon, 12 Feb 2007 09:45:42 -0800
Subject: [Numpy-discussion] Possible solution to binary
	distributionproblems for numpy on linux?
In-Reply-To: <45CFF4D0.5080401@ar.media.kyoto-u.ac.jp>
References: <45CFF4D0.5080401@ar.media.kyoto-u.ac.jp>
Message-ID: <f4f93d420702120945k59cd887h2f68d27c8c051a0b@mail.gmail.com>

On 2/11/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> My impression is that binary distribution of numpy is a big problem for many
> linux users, and that is entry barrier for many users (I may be wrong, that's just
> an impression from the ML).

Do all of the major GNU/Linux distributions have recent versions of NumPy?

Debian Etch is at NumPy 1.0.1


From gosselina at dfo-mpo.gc.ca  Mon Feb 12 16:01:54 2007
From: gosselina at dfo-mpo.gc.ca (Andre Gosselin)
Date: Mon, 12 Feb 2007 16:01:54 -0500
Subject: [Numpy-discussion]  New release of pycdf package ported to NumPy
In-Reply-To: 1171247042.4560.6.camel@maboule
Message-ID: <1171314114.6259.4.camel@maboule>

A small error slipped into the pycdf-0.6-3 release. Attribute deletion
through del(), and dimension length inquiry through len() were missing
in that release. 

A new pycdf-0.6.3b fixes those problems. I have withdrawn pycdf-0.6.3
from Sourceforge.net . Those people who have already downloaded this
release can safely continue to use it,if they do not mind missing the
del() and len() features.

Reagrds,
Andre Gosselin



From evanmason at gmail.com  Mon Feb 12 18:55:00 2007
From: evanmason at gmail.com (Evan Mason)
Date: Mon, 12 Feb 2007 15:55:00 -0800
Subject: [Numpy-discussion] concatenate and different numbers of dimensions
Message-ID: <4254ff910702121555m7867752fic5d6565666188d09@mail.gmail.com>

Hi, I am trying to use concatenate to join together a 40x50x45 array and a
50x45 array.  The shape of the resulting array should be 41x50x45.

In [132]: tempLevel.shape
Out[132]: (40, 50, 45)

In [133]: temp.shape
Out[133]: (50, 45)

I've tried various ways to do this using concatenate but always get the
following:

In [142]: concatenate((tempLevel, temp), axis=0)
---------------------------------------------------------------------------
exceptions.ValueError                                Traceback (most recent
call last)

/home/emason/python/tools/<ipython console>

ValueError: arrays must have same number of dimensions


How do I do this with concatenate, or is there another way to do it?

Many thanks, Evan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070212/dec5663d/attachment.html>

From torgil.svensson at gmail.com  Mon Feb 12 19:02:15 2007
From: torgil.svensson at gmail.com (Torgil Svensson)
Date: Tue, 13 Feb 2007 01:02:15 +0100
Subject: [Numpy-discussion] concatenate and different numbers of
	dimensions
In-Reply-To: <4254ff910702121555m7867752fic5d6565666188d09@mail.gmail.com>
References: <4254ff910702121555m7867752fic5d6565666188d09@mail.gmail.com>
Message-ID: <e7bda7770702121602re3ab42am8a126ce0e4bd098f@mail.gmail.com>

As the error-message say, they must match number of dimensions, try:
concatenate((tempLevel, temp[newaxis,...]), axis=0)

On 2/13/07, Evan Mason <evanmason at gmail.com> wrote:
> Hi, I am trying to use concatenate to join together a 40x50x45 array and a
> 50x45 array.  The shape of the resulting array should be 41x50x45.
>
> In [132]: tempLevel.shape
> Out[132]: (40, 50, 45)
>
> In [133]: temp.shape
> Out[133]: (50, 45)
>
> I've tried various ways to do this using concatenate but always get the
> following:
>
> In [142]: concatenate((tempLevel, temp), axis=0)
> ---------------------------------------------------------------------------
> exceptions.ValueError                                Traceback (most recent
> call last)
>
> /home/emason/python/tools/<ipython console>
>
> ValueError: arrays must have same number of dimensions
>
>
>  How do I do this with concatenate, or is there another way to do it?
>
> Many thanks, Evan
>
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From david at ar.media.kyoto-u.ac.jp  Mon Feb 12 20:48:44 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 13 Feb 2007 10:48:44 +0900
Subject: [Numpy-discussion] Possible solution to
 binary	distributionproblems for numpy on linux?
In-Reply-To: <f4f93d420702120945k59cd887h2f68d27c8c051a0b@mail.gmail.com>
References: <45CFF4D0.5080401@ar.media.kyoto-u.ac.jp>
	<f4f93d420702120945k59cd887h2f68d27c8c051a0b@mail.gmail.com>
Message-ID: <45D118FC.7020403@ar.media.kyoto-u.ac.jp>

Keith Goodman wrote:
> On 2/11/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
>> My impression is that binary distribution of numpy is a big problem for many
>> linux users, and that is entry barrier for many users (I may be wrong, that's just
>> an impression from the ML).
>
> Do all of the major GNU/Linux distributions have recent versions of NumPy?
>
> Debian Etch is at NumPy 1.0.1
I think debian has numpy now (I am not using debian on workstation 
anymore, so I am not really following), but what about new versions of 
numpy/scipy ? If I want to give some of my code to people in my lab who 
do not use the same distribution than me, can I give a 10 minutes 
instructions set to have everything ?

If you read the wiki for linux, you have all kind of information 
distribution specific, which demands some knowledge about compiler and 
co, etc... If opensuse build system works as I think it does, this would 
be kind of helpful. Particularly the rebuilding of packages when one of 
the dependency changed.

cheers,

David


From konrad.hinsen at laposte.net  Tue Feb 13 08:34:20 2007
From: konrad.hinsen at laposte.net (Konrad Hinsen)
Date: Tue, 13 Feb 2007 14:34:20 +0100
Subject: [Numpy-discussion] Request for porting pycdf to NumPy
In-Reply-To: <45CCB064.4010706@noaa.gov>
References: <36897.24.56.171.189.1171031507.squirrel@imap.rap.ucar.edu>
	<45CCAC28.3060006@ee.byu.edu> <45CCACB5.2030206@gmail.com>
	<45CCB064.4010706@noaa.gov>
Message-ID: <E290E06A-3505-4A64-967A-B14B6EFE48A2@laposte.net>

On Feb 9, 2007, at 18:33, Christopher Barker wrote:

> hmmm, this page from unidata:
>
> http://www.unidata.ucar.edu/software/netcdf/software.html#Python
>
> Indicates 4-6 different Python netcfd interfaces!

...

> I think it would be ideal if Unidata could be persuaded to adopt an
> maintain one as the official version to save this extra work an
> confusion. Maybe there is some hope of this for netcdf4.

It would certainly be nice to have an "official" Python interface  
maintained by somebody else, but I don't expect this to happen any  
time soon. Moreover, the existing interfaces have different  
characteristics that are partially incompatible but each appeal to  
some users.

My own netCDF interface (part of ScientificPython) was designed to  
satisfy two major criteria:

1) Be as Pythonic as possible by implementing common object  
interfaces: arrays, dictionaries, and object attributes. netCDF  
variables are seen as "arrays on disk".

2) Provide netCDF access from Python as well as from Python extension  
modules written in C (or nowadays Pyrex).

However, I can understand the needs of those with different  
priorities, e.g. those who want an API that resembles the C API for  
netCDF. Fortunately, there is something for everybody.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Mol?culaire, CNRS Orl?ans
Synchrotron Soleil - Division Exp?riences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: hinsen at cnrs-orleans.fr
---------------------------------------------------------------------




From mjanikas at esri.com  Tue Feb 13 14:42:35 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 13 Feb 2007 11:42:35 -0800
Subject: [Numpy-discussion] fromstring, tostring slow?
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFB0@hardwire.esri.com>

Hello all,

 

I am finding that directly packing numpy arrays into binary using the
tostring and fromstring methods do not provide a speed improvement over
writing the same arrays to ascii files.  Obviously, the size of the
resulting files is far smaller, but I was hoping to get an improvement
in the speed of writing.  I got that speed improvement using the struct
module directly, or by using generic python arrays.  Let me further
describe my methodological issue as it may directly relate to any
solution you might have.

 

My output file is heterogeneous.  Each line is either an array of
integers or floats.  Each record is made up of three entries.

They serve as a sparse representation of a large matrix.

 

1)       row, n (both integers)

2)       array of integers of length n, representing columns

3)       array of floats of length n, representing values

 

 

Here, "n" is not constant across the records, so many of the database
structures I have looked at do not apply.  Any suggestions would be
greatly appreciated.

 

 

Mark Janikas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070213/5ffc3ff4/attachment.html>

From stefan at sun.ac.za  Tue Feb 13 15:03:26 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Tue, 13 Feb 2007 22:03:26 +0200
Subject: [Numpy-discussion] fromstring, tostring slow?
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFB0@hardwire.esri.com>
References: <627102C921CD9745B070C3B10CB8199B010EBFB0@hardwire.esri.com>
Message-ID: <20070213200326.GN6150@mentat.za.net>

On Tue, Feb 13, 2007 at 11:42:35AM -0800, Mark Janikas wrote:
> I am finding that directly packing numpy arrays into binary using the tostring
> and fromstring methods do not provide a speed improvement over writing the same
> arrays to ascii files.  Obviously, the size of the resulting files is far
> smaller, but I was hoping to get an improvement in the speed of writing.  I got
> that speed improvement using the struct module directly, or by using generic
> python arrays.  Let me further describe my methodological issue as it may
> directly relate to any solution you might have.

Hi Mark

Can you post a benchmark code snippet to demonstrate your results?
Here, using 1.0.2.dev3545, I see:

In [26]: x = N.random.random(100)

In [27]: timeit f = file('/tmp/blah.dat','w'); f.write(str(x))
100 loops, best of 3: 1.77 ms per loop

In [28]: timeit f = file('/tmp/blah','w'); x.tofile(f)
10000 loops, best of 3: 100 ?s per loop

(I see the same results for heterogeneous arrays)

Cheers
St?fan


From mjanikas at esri.com  Tue Feb 13 15:36:01 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 13 Feb 2007 12:36:01 -0800
Subject: [Numpy-discussion] fromstring, tostring slow?
In-Reply-To: <20070213200326.GN6150@mentat.za.net>
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFB1@hardwire.esri.com>

Good call Stefan,

I decoupled the timing from the application (duh!) and got better results:

from numpy import *
import numpy.random as RAND
import time as TIME

x = RAND.random(1000)
xl = x.tolist()

t1 = TIME.clock()
xStringOut = [ str(i) for i in xl ]
xStringOut = " ".join(xStringOut)
f = file('blah.dat','w'); f.write(xStringOut)
t2 = TIME.clock()
total = t2 - t1
t1 = TIME.clock()
f = file('blah.bwt','wb')
xBinaryOut = x.tostring()
f.write(xBinaryOut)
t2 = TIME.clock()
total1 = t2 - t1

>>> total
0.00661
>>> total1
0.00229

Printing x directly to a string took REALLY long: f.write(str(x)) = 0.0258

The problem therefore, must be in the way I am appending values to the empty arrays.  I am currently using the append method: 

myArray = append(myArray, newValue)

Or would it be faster to concat or use a list append then convert?

But to be more sure, Ill have to profile it.  It seems a bit odd in that there are far less loops and conversions in my current implementation for the binary, yet it is still running slower. 


-----Original Message-----
From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Stefan van der Walt
Sent: Tuesday, February 13, 2007 12:03 PM
To: numpy-discussion at scipy.org
Subject: Re: [Numpy-discussion] fromstring, tostring slow?

On Tue, Feb 13, 2007 at 11:42:35AM -0800, Mark Janikas wrote:
> I am finding that directly packing numpy arrays into binary using the tostring
> and fromstring methods do not provide a speed improvement over writing the same
> arrays to ascii files.  Obviously, the size of the resulting files is far
> smaller, but I was hoping to get an improvement in the speed of writing.  I got
> that speed improvement using the struct module directly, or by using generic
> python arrays.  Let me further describe my methodological issue as it may
> directly relate to any solution you might have.

Hi Mark

Can you post a benchmark code snippet to demonstrate your results?
Here, using 1.0.2.dev3545, I see:

In [26]: x = N.random.random(100)

In [27]: timeit f = file('/tmp/blah.dat','w'); f.write(str(x))
100 loops, best of 3: 1.77 ms per loop

In [28]: timeit f = file('/tmp/blah','w'); x.tofile(f)
10000 loops, best of 3: 100 ?s per loop

(I see the same results for heterogeneous arrays)

Cheers
St?fan
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From charlesr.harris at gmail.com  Tue Feb 13 15:44:28 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 13 Feb 2007 13:44:28 -0700
Subject: [Numpy-discussion] fromstring, tostring slow?
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFB1@hardwire.esri.com>
References: <20070213200326.GN6150@mentat.za.net>
	<627102C921CD9745B070C3B10CB8199B010EBFB1@hardwire.esri.com>
Message-ID: <e06186140702131244r26fa985ew7b0e4d43ead37dbf@mail.gmail.com>

On 2/13/07, Mark Janikas <mjanikas at esri.com> wrote:
>
> Good call Stefan,
>
> I decoupled the timing from the application (duh!) and got better results:
>
> from numpy import *
> import numpy.random as RAND
> import time as TIME
>
> x = RAND.random(1000)
> xl = x.tolist()
>
> t1 = TIME.clock()
> xStringOut = [ str(i) for i in xl ]
> xStringOut = " ".join(xStringOut)
> f = file('blah.dat','w'); f.write(xStringOut)
> t2 = TIME.clock()
> total = t2 - t1
> t1 = TIME.clock()
> f = file('blah.bwt','wb')
> xBinaryOut = x.tostring()
> f.write(xBinaryOut)
> t2 = TIME.clock()
> total1 = t2 - t1
>
> >>> total
> 0.00661
> >>> total1
> 0.00229
>
> Printing x directly to a string took REALLY long: f.write(str(x)) = 0.0258
>
> The problem therefore, must be in the way I am appending values to the
> empty arrays.  I am currently using the append method:
>
> myArray = append(myArray, newValue)
>
> Or would it be faster to concat or use a list append then convert?


I am going to guess that a list would be faster for appending. Concat and, I
suspect, append make new arrays for each use, rather like string
concatenation in Python. A list, on the other hand, is no doubt optimized
for adding new values. Another option might be using PyTables with
extensible arrays. In any case, a bit of timing should show the way if the
performance is that crucial to your application.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070213/4ecd32cc/attachment.html>

From pgmdevlist at gmail.com  Tue Feb 13 16:10:09 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 13 Feb 2007 16:10:09 -0500
Subject: [Numpy-discussion] Docstring formatting
Message-ID: <200702131610.09779.pgmdevlist@gmail.com>

Dear All,
Did we come to some kind of agreement about the formatting of docstring ? In 
any case, where could I find the 'official' format that Travis recommended ?
Thanks a lot in advance
P.


From mjanikas at esri.com  Tue Feb 13 16:24:09 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 13 Feb 2007 13:24:09 -0800
Subject: [Numpy-discussion] fromstring, tostring slow?
In-Reply-To: <e06186140702131244r26fa985ew7b0e4d43ead37dbf@mail.gmail.com>
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFB2@hardwire.esri.com>

Yup.  It was faster to:

 

 Use lists for the append, then transform into an array, then transform
into a binary string

 

Rather than

 

Create empty arrays and use its append method, then transform into a
binary string.

 

The last question on the output when then be to test the speed of using
generic Python arrays, which have append methods as well.  Then, there
would still only be the binary string conversion as apposed to
list-->numpy array-->binary string

 

Thanks to all for your input....

 

MJ

 

________________________________

From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Charles R
Harris
Sent: Tuesday, February 13, 2007 12:44 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] fromstring, tostring slow?

 

 


I am going to guess that a list would be faster for appending. Concat
and, I suspect, append make new arrays for each use, rather like string
concatenation in Python. A list, on the other hand, is no doubt
optimized for adding new values. Another option might be using PyTables
with extensible arrays. In any case, a bit of timing should show the way
if the performance is that crucial to your application. 

Chuck

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070213/32085549/attachment.html>

From stefan at sun.ac.za  Tue Feb 13 16:30:21 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Tue, 13 Feb 2007 23:30:21 +0200
Subject: [Numpy-discussion] Docstring formatting
In-Reply-To: <200702131610.09779.pgmdevlist@gmail.com>
References: <200702131610.09779.pgmdevlist@gmail.com>
Message-ID: <20070213213021.GQ6150@mentat.za.net>

Hi Pierre

On Tue, Feb 13, 2007 at 04:10:09PM -0500, Pierre GM wrote:
> Dear All,
> Did we come to some kind of agreement about the formatting of docstring ? In 
> any case, where could I find the 'official' format that Travis recommended ?

Travis checked numpy/doc/HOWTO_DOCUMENT.txt a couple of days ago.

Cheers
St?fan


From Chris.Barker at noaa.gov  Tue Feb 13 16:38:47 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 13 Feb 2007 13:38:47 -0800
Subject: [Numpy-discussion] fromstring, tostring slow?
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFB0@hardwire.esri.com>
References: <627102C921CD9745B070C3B10CB8199B010EBFB0@hardwire.esri.com>
Message-ID: <45D22FE7.3010304@noaa.gov>

Mark Janikas wrote:
> I am finding that directly packing numpy arrays into binary using the 
> tostring and fromstring methods

For starters, use fromfile and tofile, to save the overhead of creating 
an entire extra string.

fromfile is a function (as it is an alternate constructor for arrays):

numpy.fromfile()

ndarray.tofile() is an array method.

Enclosed is your test, including a test for tofile(), I needed to make 
the arrays much larger, and use time.time() rather than time.clock() to 
get enough time resolution to see anything, though if you really want to 
be accurate, you need to use the timeit module.

My results:
Using lists 0.457561016083
Using tostring 0.00922703742981
Using tofile 0.00431108474731

Another note: where is the data coming from -- there may be ways to 
optimize this whole process if we saw that.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: junk.py
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070213/0577b8d2/attachment.ksh>

From cjw at sympatico.ca  Tue Feb 13 18:26:22 2007
From: cjw at sympatico.ca (Colin J. Williams)
Date: Tue, 13 Feb 2007 18:26:22 -0500
Subject: [Numpy-discussion] Docstring formatting
In-Reply-To: <20070213213021.GQ6150@mentat.za.net>
References: <200702131610.09779.pgmdevlist@gmail.com>
	<20070213213021.GQ6150@mentat.za.net>
Message-ID: <eqthfj$ebu$1@sea.gmane.org>

Stefan van der Walt wrote:
> Hi Pierre
> 
> On Tue, Feb 13, 2007 at 04:10:09PM -0500, Pierre GM wrote:
>> Dear All,
>> Did we come to some kind of agreement about the formatting of docstring ? In 
>> any case, where could I find the 'official' format that Travis recommended ?
> 
> Travis checked numpy/doc/HOWTO_DOCUMENT.txt a couple of days ago.
> 
> Cheers
> St?fan
Could you give the URL please?

Colin W.



From robert.kern at gmail.com  Tue Feb 13 18:30:50 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 13 Feb 2007 17:30:50 -0600
Subject: [Numpy-discussion] Docstring formatting
In-Reply-To: <eqthfj$ebu$1@sea.gmane.org>
References: <200702131610.09779.pgmdevlist@gmail.com>	<20070213213021.GQ6150@mentat.za.net>
	<eqthfj$ebu$1@sea.gmane.org>
Message-ID: <45D24A2A.9050600@gmail.com>

Colin J. Williams wrote:

> Could you give the URL please?

http://svn.scipy.org/svn/numpy/trunk/numpy/doc/HOWTO_DOCUMENT.txt

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From mjanikas at esri.com  Tue Feb 13 18:44:37 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 13 Feb 2007 15:44:37 -0800
Subject: [Numpy-discussion] fromstring, tostring slow?
In-Reply-To: <45D22FE7.3010304@noaa.gov>
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFB4@hardwire.esri.com>

I don't think I can do that because I have heterogeneous rows of
data.... I.e. the columns in each row are different in length.
Furthermore, when reading it back in, I want to read only bytes of the
info at a time so I can save memory.  In this case, I only want to have
one record in mem at once.

Another issue has arisen from taking this routine cross-platform....
namely, if I write the file on Windows I cant read it on Solaris.  I
assume the big-little endian is at hand here.  I know using the struct
module that I can pack using either one.  Perhaps I will have to go back
to the drawing board.  I actually love these methods now because I get
back out directly what I put in.  Great kudos to the developers....

MJ


-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Christopher
Barker
Sent: Tuesday, February 13, 2007 1:39 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] fromstring, tostring slow?

Mark Janikas wrote:
> I am finding that directly packing numpy arrays into binary using the 
> tostring and fromstring methods

For starters, use fromfile and tofile, to save the overhead of creating 
an entire extra string.

fromfile is a function (as it is an alternate constructor for arrays):

numpy.fromfile()

ndarray.tofile() is an array method.

Enclosed is your test, including a test for tofile(), I needed to make 
the arrays much larger, and use time.time() rather than time.clock() to 
get enough time resolution to see anything, though if you really want to

be accurate, you need to use the timeit module.

My results:
Using lists 0.457561016083
Using tostring 0.00922703742981
Using tofile 0.00431108474731

Another note: where is the data coming from -- there may be ways to 
optimize this whole process if we saw that.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From stefan at sun.ac.za  Tue Feb 13 18:51:32 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Wed, 14 Feb 2007 01:51:32 +0200
Subject: [Numpy-discussion] fromstring, tostring slow?
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFB4@hardwire.esri.com>
References: <45D22FE7.3010304@noaa.gov>
	<627102C921CD9745B070C3B10CB8199B010EBFB4@hardwire.esri.com>
Message-ID: <20070213235132.GS6150@mentat.za.net>

On Tue, Feb 13, 2007 at 03:44:37PM -0800, Mark Janikas wrote:
> I don't think I can do that because I have heterogeneous rows of
> data.... I.e. the columns in each row are different in length.
> Furthermore, when reading it back in, I want to read only bytes of the
> info at a time so I can save memory.  In this case, I only want to have
> one record in mem at once.
> 
> Another issue has arisen from taking this routine cross-platform....
> namely, if I write the file on Windows I cant read it on Solaris.  I
> assume the big-little endian is at hand here.

Indeed.  You may want to take a look at npfile, the new IO module in
scipy written by Matthew Brett (you don't have to install the whole
scipy to use it, just grab the file).

Cheers
St?fan




From mjanikas at esri.com  Tue Feb 13 19:02:10 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 13 Feb 2007 16:02:10 -0800
Subject: [Numpy-discussion] fromstring, tostring slow?
In-Reply-To: <20070213235132.GS6150@mentat.za.net>
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFB5@hardwire.esri.com>

Yes, but does the code have the same license as NumPy?  As I work for a software company, where I help with the scripting interface, I must make sure everything I use is cited and has the appropriate license.  

MJ  

-----Original Message-----
From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Stefan van der Walt
Sent: Tuesday, February 13, 2007 3:52 PM
To: numpy-discussion at scipy.org
Subject: Re: [Numpy-discussion] fromstring, tostring slow?

On Tue, Feb 13, 2007 at 03:44:37PM -0800, Mark Janikas wrote:
> I don't think I can do that because I have heterogeneous rows of
> data.... I.e. the columns in each row are different in length.
> Furthermore, when reading it back in, I want to read only bytes of the
> info at a time so I can save memory.  In this case, I only want to have
> one record in mem at once.
> 
> Another issue has arisen from taking this routine cross-platform....
> namely, if I write the file on Windows I cant read it on Solaris.  I
> assume the big-little endian is at hand here.

Indeed.  You may want to take a look at npfile, the new IO module in
scipy written by Matthew Brett (you don't have to install the whole
scipy to use it, just grab the file).

Cheers
St?fan


_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From robert.kern at gmail.com  Tue Feb 13 19:04:49 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 13 Feb 2007 18:04:49 -0600
Subject: [Numpy-discussion] fromstring, tostring slow?
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFB5@hardwire.esri.com>
References: <627102C921CD9745B070C3B10CB8199B010EBFB5@hardwire.esri.com>
Message-ID: <45D25221.4050007@gmail.com>

Mark Janikas wrote:
> Yes, but does the code have the same license as NumPy?  As I work for a software company, where I help with the scripting interface, I must make sure everything I use is cited and has the appropriate license.  

The numpy and scipy licenses are the same except for the details like the
licensing party.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Tue Feb 13 19:07:03 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 13 Feb 2007 16:07:03 -0800
Subject: [Numpy-discussion] fromstring, tostring slow?
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFB4@hardwire.esri.com>
References: <627102C921CD9745B070C3B10CB8199B010EBFB4@hardwire.esri.com>
Message-ID: <45D252A7.3060405@noaa.gov>

Mark Janikas wrote:
> I don't think I can do that because I have heterogeneous rows of
> data.... I.e. the columns in each row are different in length.

like I said, show us your whole problem...

But you don't have to write.read all the data at once with from/tofile() 
anyway. Each of your "rows" has to be in a separate array anyway, as 
numpy arrays don't support "ragged" arrays, but each row can be written 
with tofile()

> Furthermore, when reading it back in, I want to read only bytes of the
> info at a time so I can save memory.  In this case, I only want to have
> one record in mem at once.

you can make multiple calls to fromfile(), thou you'll have to know how 
long each record is.

> Another issue has arisen from taking this routine cross-platform....
> namely, if I write the file on Windows I cant read it on Solaris.  I
> assume the big-little endian is at hand here.

yup.

> I know using the struct
> module that I can pack using either one.

so can numpy. see the "byteswap" method, and you can specify a 
particular endianess with a datatype when you read with fromfile():

a = N.fromfile(DataFile, dtype=N.dtype("<d"), count=20)

reads 20 little-endian doubles from DataFile, regardless of the native 
endianess of the machine you're on.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From stefan at sun.ac.za  Tue Feb 13 19:09:47 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Wed, 14 Feb 2007 02:09:47 +0200
Subject: [Numpy-discussion] fromstring, tostring slow?
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFB5@hardwire.esri.com>
References: <20070213235132.GS6150@mentat.za.net>
	<627102C921CD9745B070C3B10CB8199B010EBFB5@hardwire.esri.com>
Message-ID: <20070214000947.GT6150@mentat.za.net>

On Tue, Feb 13, 2007 at 04:02:10PM -0800, Mark Janikas wrote:
> Yes, but does the code have the same license as NumPy?  As I work
> for a software company, where I help with the scripting interface, I
> must make sure everything I use is cited and has the appropriate
> license.  

Yes, Scipy and Numpy are both released under BSD licenses.

Cheers
St?fan


From fred at ucar.edu  Tue Feb 13 19:15:35 2007
From: fred at ucar.edu (Fred Clare)
Date: Tue, 13 Feb 2007 17:15:35 -0700
Subject: [Numpy-discussion] Docstring formatting
In-Reply-To: <45D24A2A.9050600@gmail.com>
References: <200702131610.09779.pgmdevlist@gmail.com>	<20070213213021.GQ6150@mentat.za.net>
	<eqthfj$ebu$1@sea.gmane.org> <45D24A2A.9050600@gmail.com>
Message-ID: <C735E677-CD4A-4891-94C1-73F7DE079925@ucar.edu>

Mary,

Do you think we should adopt the "official" SciPy
docstring format?  It is pretty simple.

Did you see Konrad Hinson's comment on NetCDF
modules?

Fred

On Feb 13, 2007, at 4:30 PM, Robert Kern wrote:

> Colin J. Williams wrote:
>
>> Could you give the URL please?
>
> http://svn.scipy.org/svn/numpy/trunk/numpy/doc/HOWTO_DOCUMENT.txt
>
> --  
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a  
> harmless enigma
>  that is made terrible by our own mad attempt to interpret it as  
> though it had
>  an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From fred at ucar.edu  Tue Feb 13 19:17:43 2007
From: fred at ucar.edu (Fred Clare)
Date: Tue, 13 Feb 2007 17:17:43 -0700
Subject: [Numpy-discussion] Docstring formatting
In-Reply-To: <45D24A2A.9050600@gmail.com>
References: <200702131610.09779.pgmdevlist@gmail.com>	<20070213213021.GQ6150@mentat.za.net>
	<eqthfj$ebu$1@sea.gmane.org> <45D24A2A.9050600@gmail.com>
Message-ID: <C0BD1481-4C25-474C-968B-3CE412B90042@ucar.edu>

Sorry for a private e-mail escaping to this list.

Fred Clare

On Feb 13, 2007, at 4:30 PM, Robert Kern wrote:

> Colin J. Williams wrote:
>
>> Could you give the URL please?
>
> http://svn.scipy.org/svn/numpy/trunk/numpy/doc/HOWTO_DOCUMENT.txt
>
> --  
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a  
> harmless enigma
>  that is made terrible by our own mad attempt to interpret it as  
> though it had
>  an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From mjanikas at esri.com  Tue Feb 13 19:31:00 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 13 Feb 2007 16:31:00 -0800
Subject: [Numpy-discussion] fromstring, tostring slow?
In-Reply-To: <45D252A7.3060405@noaa.gov>
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFB6@hardwire.esri.com>

This is all very good info.  Especially, the byteswap.  Ill be testing
it momentarily.  As far as a detailed explanation of the problem....

In essence, I am applying sparse matrix multiplication.  The matrix of
which I am dealing with in the matter described is nxn.  Generally, this
matrix is 1-20% sparse.  I use it in spatial data analysis, where the
matrix W represents the spatial association between n observations.  The
operations I perform on it are generally related to the spatial lag of a
variable... or Wy, where y is a nxk matrix (usually k=1).  As k is
generally small, the y vector and the result vector are represented by
numpy arrays.  I can have nxkx2 pieces of info in mem (usually).  What I
cant have is n**2.  So, I store each row of W in a file as a record
consisting of 3 parts:

1) row, nn (# of neighbors)
2) nhs (nx1) vector of integers representing the columns in row[i] != 0
3) weights (nx1) vector of floats corresponding to the index in the
previous row

The first two parts of the record are known as a GAL or geographic
algorithm library.  Since a lot of my W matrices have distance metrics
associated with them I added the third.  I think this might be termed by
someone else as an enhanced GAL.  At any rate, this allows me to perform
this operation on large datasets w/o running out of mem.


-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Christopher
Barker
Sent: Tuesday, February 13, 2007 4:07 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] fromstring, tostring slow?

Mark Janikas wrote:
> I don't think I can do that because I have heterogeneous rows of
> data.... I.e. the columns in each row are different in length.

like I said, show us your whole problem...

But you don't have to write.read all the data at once with from/tofile()

anyway. Each of your "rows" has to be in a separate array anyway, as 
numpy arrays don't support "ragged" arrays, but each row can be written 
with tofile()

> Furthermore, when reading it back in, I want to read only bytes of the
> info at a time so I can save memory.  In this case, I only want to
have
> one record in mem at once.

you can make multiple calls to fromfile(), thou you'll have to know how 
long each record is.

> Another issue has arisen from taking this routine cross-platform....
> namely, if I write the file on Windows I cant read it on Solaris.  I
> assume the big-little endian is at hand here.

yup.

> I know using the struct
> module that I can pack using either one.

so can numpy. see the "byteswap" method, and you can specify a 
particular endianess with a datatype when you read with fromfile():

a = N.fromfile(DataFile, dtype=N.dtype("<d"), count=20)

reads 20 little-endian doubles from DataFile, regardless of the native 
endianess of the machine you're on.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From mjanikas at esri.com  Tue Feb 13 19:53:45 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 13 Feb 2007 16:53:45 -0800
Subject: [Numpy-discussion] fromstring, tostring slow?
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFB6@hardwire.esri.com>
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFB7@hardwire.esri.com>

Found a typo-or-two in my description.  #2 and #3 are nnx1 in shape

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Mark Janikas
Sent: Tuesday, February 13, 2007 4:31 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] fromstring, tostring slow?

This is all very good info.  Especially, the byteswap.  Ill be testing
it momentarily.  As far as a detailed explanation of the problem....

In essence, I am applying sparse matrix multiplication.  The matrix of
which I am dealing with in the matter described is nxn.  Generally, this
matrix is 1-20% sparse.  I use it in spatial data analysis, where the
matrix W represents the spatial association between n observations.  The
operations I perform on it are generally related to the spatial lag of a
variable... or Wy, where y is a nxk matrix (usually k=1).  As k is
generally small, the y vector and the result vector are represented by
numpy arrays.  I can have nxkx2 pieces of info in mem (usually).  What I
cant have is n**2.  So, I store each row of W in a file as a record
consisting of 3 parts:

1) row, nn (# of neighbors)
2) nhs (nx1) vector of integers representing the columns in row[i] != 0
3) weights (nx1) vector of floats corresponding to the index in the
previous row

The first two parts of the record are known as a GAL or geographic
algorithm library.  Since a lot of my W matrices have distance metrics
associated with them I added the third.  I think this might be termed by
someone else as an enhanced GAL.  At any rate, this allows me to perform
this operation on large datasets w/o running out of mem.


-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Christopher
Barker
Sent: Tuesday, February 13, 2007 4:07 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] fromstring, tostring slow?

Mark Janikas wrote:
> I don't think I can do that because I have heterogeneous rows of
> data.... I.e. the columns in each row are different in length.

like I said, show us your whole problem...

But you don't have to write.read all the data at once with from/tofile()

anyway. Each of your "rows" has to be in a separate array anyway, as 
numpy arrays don't support "ragged" arrays, but each row can be written 
with tofile()

> Furthermore, when reading it back in, I want to read only bytes of the
> info at a time so I can save memory.  In this case, I only want to
have
> one record in mem at once.

you can make multiple calls to fromfile(), thou you'll have to know how 
long each record is.

> Another issue has arisen from taking this routine cross-platform....
> namely, if I write the file on Windows I cant read it on Solaris.  I
> assume the big-little endian is at hand here.

yup.

> I know using the struct
> module that I can pack using either one.

so can numpy. see the "byteswap" method, and you can specify a 
particular endianess with a datatype when you read with fromfile():

a = N.fromfile(DataFile, dtype=N.dtype("<d"), count=20)

reads 20 little-endian doubles from DataFile, regardless of the native 
endianess of the machine you're on.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From puddleglum at o2.pl  Wed Feb 14 11:10:14 2007
From: puddleglum at o2.pl (=?UTF-8?B?V29qY2llY2ggxZptaWdhag==?=)
Date: Wed, 14 Feb 2007 17:10:14 +0100
Subject: [Numpy-discussion] Vectorizing a class method
Message-ID: <45D33466.7060207@o2.pl>

Hi,

I have a question about the vectorize function. I'd like to use it to 
create a vectorized version of a class method. I've tried the following 
code:

   from numpy import *

   class X:
       def func(self, n):
           return 2 * n  # example
       func = vectorize(func)

Now, when I declare an instance of the class X and invoke func() as an 
unbound method, it works:

   x = X()
   print X.func(x, [1, 2])  # output: [2 4]

But an attempt to invoke it "normally", i.e. like

   print x.func([1, 2])

fails with the message

Traceback (most recent call last):
   File "<stdin>", line 1, in ?
   File "/usr/lib/python2.4/site-packages/numpy/lib/function_base.py", 
line 823, in __call__
     raise ValueError, "mismatch between python function inputs"\
ValueError: mismatch between python function inputs and received arguments

It seems that in this case the class instance (x) isn't passed to the 
vectorize.__call__() method, and as a result the number of arguments 
does not agree with what this method expects.

Does anybody have an idea of how to do it correctly? As a workaround, I 
can write a wrapper function on the module level, which can be 
vectorized without problems, and call it from inside the class---but it 
looks ugly and is tedious given that I have multiple functions to be 
handled in this way.

Thanks in advance for any help,
Wojciech Smigaj


From tim.hochberg at ieee.org  Wed Feb 14 12:10:38 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 14 Feb 2007 10:10:38 -0700
Subject: [Numpy-discussion] Vectorizing a class method
In-Reply-To: <45D33466.7060207@o2.pl>
References: <45D33466.7060207@o2.pl>
Message-ID: <e4412d6b0702140910j1ceb7eddr962c7dbff0c97d18@mail.gmail.com>

On 2/14/07, Wojciech ?migaj <puddleglum at o2.pl> wrote:
>
> Hi,
>
> I have a question about the vectorize function. I'd like to use it to
> create a vectorized version of a class method. I've tried the following
> code:
>
>    from numpy import *
>
>    class X:
>        def func(self, n):
>            return 2 * n  # example
>        func = vectorize(func)
>
> Now, when I declare an instance of the class X and invoke func() as an
> unbound method, it works:
>
>    x = X()
>    print X.func(x, [1, 2])  # output: [2 4]
>
> But an attempt to invoke it "normally", i.e. like
>
>    print x.func([1, 2])
>
> fails with the message
>
> Traceback (most recent call last):
>    File "<stdin>", line 1, in ?
>    File "/usr/lib/python2.4/site-packages/numpy/lib/function_base.py",
> line 823, in __call__
>      raise ValueError, "mismatch between python function inputs"\
> ValueError: mismatch between python function inputs and received arguments
>
> It seems that in this case the class instance (x) isn't passed to the
> vectorize.__call__() method, and as a result the number of arguments
> does not agree with what this method expects.
>
> Does anybody have an idea of how to do it correctly? As a workaround, I
> can write a wrapper function on the module level, which can be
> vectorized without problems, and call it from inside the class---but it
> looks ugly and is tedious given that I have multiple functions to be
> handled in this way.



I think you want staticmethod. Something like:

class X:
   def f(x):
       return 2*x
   f = staticmethod(vectorize(x))

However, I don't have a Python distribution available here to check that. If
that doesn't work, as search on staticmethod should get you to the correct
syntax.

I'll just note in passing that if your function is composed of completely of
things that will operate on arrays as is (as your example is), then
vectorizing the function is counter productive. However, your real code may
well need vectorize to work.


Thanks in advance for any help,
> Wojciech Smigaj
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070214/1daeaf0d/attachment.html>

From puddleglum at o2.pl  Wed Feb 14 13:27:31 2007
From: puddleglum at o2.pl (=?ISO-8859-2?Q?Wojciech_=A6migaj?=)
Date: Wed, 14 Feb 2007 19:27:31 +0100
Subject: [Numpy-discussion] Vectorizing a class method
In-Reply-To: <e4412d6b0702140910j1ceb7eddr962c7dbff0c97d18@mail.gmail.com>
References: <45D33466.7060207@o2.pl>
	<e4412d6b0702140910j1ceb7eddr962c7dbff0c97d18@mail.gmail.com>
Message-ID: <45D35493.5090700@o2.pl>

Timothy Hochberg wrote:
> On 2/14/07, *Wojciech ?migaj* <puddleglum at o2.pl 
> <mailto:puddleglum at o2.pl>> wrote:
>>     I have a question about the vectorize function. I'd like to use it to
>>     create a vectorized version of a class method. I've tried the following
>>     code:
>> 
>>        from numpy import *
>> 
>>        class X:
>>            def func(self, n):
>>                return 2 * n  # example
>>            func = vectorize(func)
>> 
>> [...]

> I think you want staticmethod. Something like:
> 
> class X:
>    def f(x):
>        return 2*x
>    f = staticmethod(vectorize(x))
> 
> However, I don't have a Python distribution available here to check 
> that. If that doesn't work, as search on staticmethod should get you to 
> the correct syntax.
> 
> I'll just note in passing that if your function is composed of 
> completely of things that will operate on arrays as is (as your example 
> is), then vectorizing the function is counter productive. However, your 
> real code may well need vectorize to work.

Thank you for your answer. I see now that my example was oversimplified. 
In reality, the method func() accesses internal data of the object, so 
it cannot be made a staticmethod. In addition, it does not operate on 
the array as a whole: basically, it does calculations according to some 
formula if n != 0, and to another one otherwise. Perhaps both parts 
could be merged in some intelligent way, but right now I want to make 
the program work, and optimization will be done later. And vectorize is 
a very nice and quick way of making functions accept arrays, even if it 
is not super-efficient.

Best regards,
Wojciech Smigaj



From tom.denniston at alum.dartmouth.org  Wed Feb 14 14:10:36 2007
From: tom.denniston at alum.dartmouth.org (Tom Denniston)
Date: Wed, 14 Feb 2007 13:10:36 -0600
Subject: [Numpy-discussion] Vectorizing a class method
In-Reply-To: <45D35493.5090700@o2.pl>
References: <45D33466.7060207@o2.pl>
	<e4412d6b0702140910j1ceb7eddr962c7dbff0c97d18@mail.gmail.com>
	<45D35493.5090700@o2.pl>
Message-ID: <d6f2d3dd0702141110v5efb734di2159ab28358b91c6@mail.gmail.com>

I don't know if this helps but you could use where to do the dispatch
between the two different formulas.

I don't know the answer to your original question however.

On 2/14/07, Wojciech ?migaj <puddleglum at o2.pl> wrote:
> Timothy Hochberg wrote:
> > On 2/14/07, *Wojciech ?migaj* <puddleglum at o2.pl
> > <mailto:puddleglum at o2.pl>> wrote:
> >>     I have a question about the vectorize function. I'd like to use it to
> >>     create a vectorized version of a class method. I've tried the following
> >>     code:
> >>
> >>        from numpy import *
> >>
> >>        class X:
> >>            def func(self, n):
> >>                return 2 * n  # example
> >>            func = vectorize(func)
> >>
> >> [...]
>
> > I think you want staticmethod. Something like:
> >
> > class X:
> >    def f(x):
> >        return 2*x
> >    f = staticmethod(vectorize(x))
> >
> > However, I don't have a Python distribution available here to check
> > that. If that doesn't work, as search on staticmethod should get you to
> > the correct syntax.
> >
> > I'll just note in passing that if your function is composed of
> > completely of things that will operate on arrays as is (as your example
> > is), then vectorizing the function is counter productive. However, your
> > real code may well need vectorize to work.
>
> Thank you for your answer. I see now that my example was oversimplified.
> In reality, the method func() accesses internal data of the object, so
> it cannot be made a staticmethod. In addition, it does not operate on
> the array as a whole: basically, it does calculations according to some
> formula if n != 0, and to another one otherwise. Perhaps both parts
> could be merged in some intelligent way, but right now I want to make
> the program work, and optimization will be done later. And vectorize is
> a very nice and quick way of making functions accept arrays, even if it
> is not super-efficient.
>
> Best regards,
> Wojciech Smigaj
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

From peridot.faceted at gmail.com  Wed Feb 14 14:11:55 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 14 Feb 2007 14:11:55 -0500
Subject: [Numpy-discussion] Vectorizing a class method
In-Reply-To: <45D35493.5090700@o2.pl>
References: <45D33466.7060207@o2.pl>
	<e4412d6b0702140910j1ceb7eddr962c7dbff0c97d18@mail.gmail.com>
	<45D35493.5090700@o2.pl>
Message-ID: <ce557a360702141111u3b9f3940td74f44c8a89c23f6@mail.gmail.com>

On 14/02/07, Wojciech ?migaj <puddleglum at o2.pl> wrote:
> Timothy Hochberg wrote:
> > On 2/14/07, *Wojciech ?migaj* <puddleglum at o2.pl
> > <mailto:puddleglum at o2.pl>> wrote:
> >>     I have a question about the vectorize function. I'd like to use it to
> >>     create a vectorized version of a class method. I've tried the following
> >>     code:
> >>
> >>        from numpy import *
> >>
> >>        class X:
> >>            def func(self, n):
> >>                return 2 * n  # example
> >>            func = vectorize(func)

The appropriate spelling for this, in modern pythons, is

class X:
  @vectorize
  def func(self, n):
    return 2*n

Not that it makes it work any better.

There's no reason vectorize couldn't be made to do the Right Thing
when handed (bound and unbound) methods, for example by examining
X.func.im_self, X.func.im_class and X.func.im_func attributes. I can't
imagine a situation where a method would want to be vectorized over
its self parameter, or at least, not by default.

Anne

From puddleglum at o2.pl  Wed Feb 14 15:11:00 2007
From: puddleglum at o2.pl (=?UTF-8?B?V29qY2llY2ggxZptaWdhag==?=)
Date: Wed, 14 Feb 2007 21:11:00 +0100
Subject: [Numpy-discussion] Vectorizing a class method
In-Reply-To: <ce557a360702141111u3b9f3940td74f44c8a89c23f6@mail.gmail.com>
References: <45D33466.7060207@o2.pl>	<e4412d6b0702140910j1ceb7eddr962c7dbff0c97d18@mail.gmail.com>	<45D35493.5090700@o2.pl>
	<ce557a360702141111u3b9f3940td74f44c8a89c23f6@mail.gmail.com>
Message-ID: <45D36CD4.6040402@o2.pl>

Anne Archibald wrote:
> The appropriate spelling for this, in modern pythons, is
> 
> class X:
>   @vectorize
>   def func(self, n):
>     return 2*n
> 
> Not that it makes it work any better.
> 
> There's no reason vectorize couldn't be made to do the Right Thing
> when handed (bound and unbound) methods, for example by examining
> X.func.im_self, X.func.im_class and X.func.im_func attributes. 

This magic is well above my current level of expertise in Python, but 
perhaps one day I'll try to do it. Thanks for pointing me in this direction.

 > I can't imagine a situation where a method would want to be vectorized
 > over its self parameter, or at least, not by default.

Yes, I agree. I thought it would be harmless, though.

Anyway, since it seems it is not possible to vectorize() a method with 
today's NumPy, I think the class' docstring, which says "[...] 
somefunction -- a Python function or method" is rather misleading. 
Perhaps it ought to be changed?

Best regards,
Wojciech Smigaj


From puddleglum at o2.pl  Wed Feb 14 15:11:10 2007
From: puddleglum at o2.pl (=?ISO-8859-2?Q?Wojciech_=A6migaj?=)
Date: Wed, 14 Feb 2007 21:11:10 +0100
Subject: [Numpy-discussion] Vectorizing a class method
In-Reply-To: <d6f2d3dd0702141110v5efb734di2159ab28358b91c6@mail.gmail.com>
References: <45D33466.7060207@o2.pl>	<e4412d6b0702140910j1ceb7eddr962c7dbff0c97d18@mail.gmail.com>	<45D35493.5090700@o2.pl>
	<d6f2d3dd0702141110v5efb734di2159ab28358b91c6@mail.gmail.com>
Message-ID: <45D36CDE.10301@o2.pl>

Tom Denniston wrote:
> I don't know if this helps but you could use where to do the dispatch
> between the two different formulas.

Yes, that's a good idea. Thanks!

Wojciech Smigaj


From pgmdevlist at gmail.com  Tue Feb 13 18:52:51 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 13 Feb 2007 18:52:51 -0500
Subject: [Numpy-discussion] Docstring formatting
In-Reply-To: <20070213213021.GQ6150@mentat.za.net>
References: <200702131610.09779.pgmdevlist@gmail.com>
	<20070213213021.GQ6150@mentat.za.net>
Message-ID: <200702131852.51401.pgmdevlist@gmail.com>

Stefan,
> Travis checked numpy/doc/HOWTO_DOCUMENT.txt a couple of days ago.

Excellent, just what I needed. Thanks for the info


From tgrav at mac.com  Wed Feb 14 21:48:44 2007
From: tgrav at mac.com (Tommy Grav)
Date: Wed, 14 Feb 2007 21:48:44 -0500
Subject: [Numpy-discussion] lsq problem
References: <86E3078D-C9B7-4516-B573-9C731D6FC45E@mac.com>
Message-ID: <020E9168-95E9-4B5D-8E9D-3E0D925410FA@mac.com>

I need to fit a gaussian profile to a set of points and would like to  
use scipy (or numpy) to
do the least square fitting if possible. I am however unsure if the  
proper routines are
available, so I thought I would ask to get some hints to get going in  
the right direction.

The input are two 1-dimensional arrays x and flux, together with a  
function

def Gaussian(a,b,c,x1):
	return a*exp(-(pow(x1,2)/pow(c,2))) - c

I would like to find the values of (a,b,c), such that the difference  
between the gaussian
and fluxes are minimalized. Would scipy.linalg.lstsq be the right  
function to use, or is this
problem not linear? (I know I could find out this particular problem  
with a little research, but
I am under a little time pressure and I can not for the life of me  
remember my old math
classes). If the problem is not linear, is there another function  
that can be used or do I have
to code up my own lstsq function to solve the problem?

Thanks in advance for any hints to the answers.

Cheers
   Tommy



From kwgoodman at gmail.com  Wed Feb 14 22:11:59 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 14 Feb 2007 19:11:59 -0800
Subject: [Numpy-discussion] lsq problem
In-Reply-To: <020E9168-95E9-4B5D-8E9D-3E0D925410FA@mac.com>
References: <86E3078D-C9B7-4516-B573-9C731D6FC45E@mac.com>
	<020E9168-95E9-4B5D-8E9D-3E0D925410FA@mac.com>
Message-ID: <f4f93d420702141911u14615725hf596cd70a19e1bd3@mail.gmail.com>

On 2/14/07, Tommy Grav <tgrav at mac.com> wrote:
> I need to fit a gaussian profile to a set of points and would like to
> use scipy (or numpy) to
> do the least square fitting if possible. I am however unsure if the
> proper routines are
> available, so I thought I would ask to get some hints to get going in
> the right direction.
>
> The input are two 1-dimensional arrays x and flux, together with a
> function
>
> def Gaussian(a,b,c,x1):
>         return a*exp(-(pow(x1,2)/pow(c,2))) - c
>
> I would like to find the values of (a,b,c), such that the difference
> between the gaussian
> and fluxes are minimalized.

You left b out of your function:  a*exp(-power((x - b),2) / c).

If it is a one-off example you can always use brute force---just search a grid:

for a in as:
    for b in bs:
        for c in cs:

But if you have a lot of data or you need to do many fits I bet scipy
has what you need.


From tgrav at mac.com  Wed Feb 14 22:29:58 2007
From: tgrav at mac.com (Tommy Grav)
Date: Wed, 14 Feb 2007 22:29:58 -0500
Subject: [Numpy-discussion] lsq problem
In-Reply-To: <f4f93d420702141911u14615725hf596cd70a19e1bd3@mail.gmail.com>
References: <86E3078D-C9B7-4516-B573-9C731D6FC45E@mac.com>
	<020E9168-95E9-4B5D-8E9D-3E0D925410FA@mac.com>
	<f4f93d420702141911u14615725hf596cd70a19e1bd3@mail.gmail.com>
Message-ID: <562A0034-B385-463A-AC98-BC7D425D7014@mac.com>


On Feb 14, 2007, at 10:11 PM, Keith Goodman wrote:

> On 2/14/07, Tommy Grav <tgrav at mac.com> wrote:
>> I need to fit a gaussian profile to a set of points and would like to
>> use scipy (or numpy) to
>> do the least square fitting if possible. I am however unsure if the
>> proper routines are
>> available, so I thought I would ask to get some hints to get going in
>> the right direction.
>>
>> The input are two 1-dimensional arrays x and flux, together with a
>> function
>>
>> def Gaussian(a,b,c,x1):
>>         return a*exp(-(pow(x1,2)/pow(c,2))) - c
>>
>> I would like to find the values of (a,b,c), such that the difference
>> between the gaussian
>> and fluxes are minimalized.
>
> You left b out of your function:  a*exp(-power((x - b),2) / c).
>
> If it is a one-off example you can always use brute force---just  
> search a grid:
>
> for a in as:
>     for b in bs:
>         for c in cs:
>
> But if you have a lot of data or you need to do many fits I bet scipy
> has what you need.

I actually butchered the function, as it should read a*exp(-pow(x,2)/ 
b)) - c.
(I do not need to fit the center as that has already been when creating
the x array :).

Cheers
    Tommy


From kwgoodman at gmail.com  Wed Feb 14 22:35:08 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 14 Feb 2007 19:35:08 -0800
Subject: [Numpy-discussion] lsq problem
In-Reply-To: <562A0034-B385-463A-AC98-BC7D425D7014@mac.com>
References: <86E3078D-C9B7-4516-B573-9C731D6FC45E@mac.com>
	<020E9168-95E9-4B5D-8E9D-3E0D925410FA@mac.com>
	<f4f93d420702141911u14615725hf596cd70a19e1bd3@mail.gmail.com>
	<562A0034-B385-463A-AC98-BC7D425D7014@mac.com>
Message-ID: <f4f93d420702141935x6b68ff02ld692446abb5c3d46@mail.gmail.com>

On 2/14/07, Tommy Grav <tgrav at mac.com> wrote:
>
> On Feb 14, 2007, at 10:11 PM, Keith Goodman wrote:
>
> > On 2/14/07, Tommy Grav <tgrav at mac.com> wrote:
> >> I need to fit a gaussian profile to a set of points and would like to
> >> use scipy (or numpy) to
> >> do the least square fitting if possible. I am however unsure if the
> >> proper routines are
> >> available, so I thought I would ask to get some hints to get going in
> >> the right direction.
> >>
> >> The input are two 1-dimensional arrays x and flux, together with a
> >> function
> >>
> >> def Gaussian(a,b,c,x1):
> >>         return a*exp(-(pow(x1,2)/pow(c,2))) - c
> >>
> >> I would like to find the values of (a,b,c), such that the difference
> >> between the gaussian
> >> and fluxes are minimalized.
> >
> > You left b out of your function:  a*exp(-power((x - b),2) / c).
> >
> > If it is a one-off example you can always use brute force---just
> > search a grid:
> >
> > for a in as:
> >     for b in bs:
> >         for c in cs:
> >
> > But if you have a lot of data or you need to do many fits I bet scipy
> > has what you need.
>
> I actually butchered the function, as it should read a*exp(-pow(x,2)/
> b)) - c.
> (I do not need to fit the center as that has already been when creating
> the x array :).

Not having to fit the center is great because now you only have two
parameters to optimize. So

for a in as:
    for b in bs:


From charlesr.harris at gmail.com  Thu Feb 15 00:10:46 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 14 Feb 2007 22:10:46 -0700
Subject: [Numpy-discussion] lsq problem
In-Reply-To: <020E9168-95E9-4B5D-8E9D-3E0D925410FA@mac.com>
References: <86E3078D-C9B7-4516-B573-9C731D6FC45E@mac.com>
	<020E9168-95E9-4B5D-8E9D-3E0D925410FA@mac.com>
Message-ID: <e06186140702142110j169d995ewd3ba7b8f4d5b295d@mail.gmail.com>

On 2/14/07, Tommy Grav <tgrav at mac.com> wrote:
>
> I need to fit a gaussian profile to a set of points and would like to
> use scipy (or numpy) to
> do the least square fitting if possible. I am however unsure if the
> proper routines are
> available, so I thought I would ask to get some hints to get going in
> the right direction.
>
> The input are two 1-dimensional arrays x and flux, together with a
> function
>
> def Gaussian(a,b,c,x1):
>         return a*exp(-(pow(x1,2)/pow(c,2))) - c
>
> I would like to find the values of (a,b,c), such that the difference
> between the gaussian
> and fluxes are minimalized. Would scipy.linalg.lstsq be the right
> function to use, or is this
> problem not linear? (I know I could find out this particular problem
> with a little research, but
> I am under a little time pressure and I can not for the life of me
> remember my old math
> classes). If the problem is not linear, is there another function
> that can be used or do I have
> to code up my own lstsq function to solve the problem?


Ah, memories. This was the first problem I ever programmed. The professor
gave me a Fortran II manual, a paper on Gauss' method, and access to the IBM
360 with a world beating 2 megs of memory at the Institute for Space Studies
in New York City. That would have been about 1967.

Anyway, there are several approaches, the simplest being some quickly
available version of optimization, say Nelder-Mead, Fletcher-Powell, or
quasi Newtonian.  I believe at least the last two are in SciPy in the
optimization package and these would be your best bet for quick and dirty if
you can find the documentation. Lessee, the SciPy site says the following
are available:


   - fmin -- Nelder-Mead Simplex algorithm (uses only function calls)
   - fmin_powell -- Powell's (modified) level set method (uses only
   function calls)
   - fmin_cg -- Non-linear (Polak-Rubiere) conjugate gradient algorithm
   (can use function and gradient).
   - fmin_bfgs -- Quasi-Newton method (can use function and gradient)
   - fmin_ncg -- Line-search Newton Conjugate Gradient (can use function,
   gradient and hessian).
   - leastsq -- Minimize the sum of squares of M equations in N unknowns
   given a starting estimate.

I think either of the first two would be the easy if you are in a hurry and
of the two, fmin probably has fewer knobs. Then either of fmin_cg or
fmin_bfgs as the gradient in this case is easy to compute. Mind, it is
always helpful to get a reasonable starting point, and using the mean and
standard deviations is probably a good start.

As to Gauss' method, the trick is to linearize the problem about that last
best quess and iterate, essentially using repeated applications of leastsq.
The linearization is the usual Taylor's series thingee. Suppose you want to
minimize

sum( (d(x_i) - f(a; x_i))**2 )

where the x_i are the data points, f the fitting function, and a a (vector)
of parameters. Then expand f about a_0, you last best guess, and solve the
usual least squares problem

sum( (d(x_i) - f(a_0; x_i) - Df(a_0; x_i)*da)**2 )

where d(x_i) - f(a_0; x_i) is the new data to be fitted, Df is the
derivative (Jacobian) of the function f with respect to the parameters, and
the da (possibly a vector) are the corrections. The * multiplication in this
case is matrix multiplication. Anyway, the next estimate of the parameters
is then a_0 + da. Repeat until satisfied. You can also get the estimated
covariance of the fit in the usual way from the last linearization. The
method is quite good in many cases.

Good luck

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070214/699bf3de/attachment.html>

From mierle at gmail.com  Thu Feb 15 03:09:04 2007
From: mierle at gmail.com (Keir Mierle)
Date: Thu, 15 Feb 2007 03:09:04 -0500
Subject: [Numpy-discussion] Unifying numpy, scipy,
	and matplotlib docstring formats
Message-ID: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com>

I'd like to help the docstring formats of numpy, scipy. and matplotlib converge
on a high-quality standard (hence the cross-posting). However, before that can
happen all maintainers from all three packages need to agree on a format. In
the interest of speeding things along, I've taken the liberty of extending [1]
the docstring format proposed by Travis on January 10th on the scipy-dev
mailing list [2]. I also volunteer to convert a non-trivial portion of the
numpy, scipy, and matplotlib docstrings. If all goes well on the consensus
front, I'll try to coordinate an IRC docstring sprint or two.

The notable differences from the format Travis posted is that I have
capitalized headers for readability; the other changes are mostly modest, or
additions to handle the more demanding requirements of matplotlib docstrings.

On the DocstringStandard page I have also put a completely re-done docstring
for the 'contour' function from matplotlib. I think it is far more readable
than the original [3]. JDH and other matplotlibheads, what do you think?
Travis, do you find my additions reasonable? Scipy maintainers, would you
consider adopting this format (especially if someone helps with the gruntwork)?

Thanks in advance,
Keir

[1] http://scipy.org/DocstringStandard
[2] http://article.gmane.org/gmane.comp.python.scientific.devel/5572
[3] http://matplotlib.sourceforge.net/matplotlib.pylab.html#-contour

p.s. This is part of my plan to kick off http://scipy.org/PyLab


From tom.and at tiscalinet.it  Thu Feb 15 03:18:48 2007
From: tom.and at tiscalinet.it (Andrea Tomadin)
Date: Thu, 15 Feb 2007 09:18:48 +0100
Subject: [Numpy-discussion] Global Numpy vector with Swig
Message-ID: <0A4BC30F-9080-43F7-899B-6F246F345216@tiscalinet.it>

Hi,
I need to pass a Numpy array to a C code wrapped by Swig.
The array in the C code is a global variable declared as
double *vec
and I would like to set it in the calling Python module foo using e.g.
foo.cvar.vec = numpy.zeros(10)
so that the array is manipulated in place.

I found out the examples in the numpy/doc/swig dir and adapted them
to my code, so now I can pass a Numpy array to a function. But in the  
examples
I was not able to recognize any suggestion (any typedef?) to  
translate an array into a
variable. I'm just writing an intermediate function that does the job,
but I feel like writing a wrapping function that will be wrapped  
again by Swig...

Any practical example is highly appreciated!

Regards,
Andrea



From gzhu at peak6.com  Thu Feb 15 10:33:13 2007
From: gzhu at peak6.com (Geoffrey Zhu)
Date: Thu, 15 Feb 2007 09:33:13 -0600
Subject: [Numpy-discussion] Numpy and iterative procedures
Message-ID: <99F81FFD0EA54E4DA8D4F1BFE272F3410382797B@ppi-mail1.chicago.peak6.net>

Hi,

I am new to numpy. I'd like to know if it is possible to code efficient
iterative procedures with numpy. 

Specifically, I have the following problem.

M is an N*N matrix. Q is a N*1 vector. V is an N*1 vector I am trying to
find iteratively from the initial value V_0. The procedure is simply to
calculate

V_{n+1}[i]=3D1/M[I,i]*(q[i]-
(M[i,1]*v_{n+1}[1]+M[I,2]*v_{n+1}[2]+..+M[i,i-1]*v_{n+1}[i-1]) -
(M[I,i+1]*v_{n}[i+1]+M[I,i+2]*v_{n}[i+2]+..+M[I,N]*v_{n}[N]))

I do not see that this is something that can esaily be vectorized, is
it?

Thanks,
Geoffrey






_______________________________________________________=0A=
=0A=
The  information in this email or in any file attached=0A=
hereto is intended only for the personal and confiden-=0A=
tial  use  of  the individual or entity to which it is=0A=
addressed and may contain information that is  propri-=0A=
etary  and  confidential.  If you are not the intended=0A=
recipient of this message you are hereby notified that=0A=
any  review, dissemination, distribution or copying of=0A=
this message is strictly prohibited.  This  communica-=0A=
tion  is  for information purposes only and should not=0A=
be regarded as an offer to sell or as  a  solicitation=0A=
of an offer to buy any financial product. Email trans-=0A=
mission cannot be guaranteed to be  secure  or  error-=0A=
free. P6070214=0A=



From charlesr.harris at gmail.com  Thu Feb 15 11:10:59 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 15 Feb 2007 09:10:59 -0700
Subject: [Numpy-discussion] Numpy and iterative procedures
In-Reply-To: <99F81FFD0EA54E4DA8D4F1BFE272F3410382797B@ppi-mail1.chicago.peak6.net>
References: <99F81FFD0EA54E4DA8D4F1BFE272F3410382797B@ppi-mail1.chicago.peak6.net>
Message-ID: <e06186140702150810p4024f30jbbb3feed5111439d@mail.gmail.com>

On 2/15/07, Geoffrey Zhu <gzhu at peak6.com> wrote:
>
> Hi,
>
> I am new to numpy. I'd like to know if it is possible to code efficient
> iterative procedures with numpy.
>
> Specifically, I have the following problem.
>
> M is an N*N matrix. Q is a N*1 vector. V is an N*1 vector I am trying to
> find iteratively from the initial value V_0. The procedure is simply to
> calculate
>
> V_{n+1}[i]=3D1/M[I,i]*(q[i]-
> (M[i,1]*v_{n+1}[1]+M[I,2]*v_{n+1}[2]+..+M[i,i-1]*v_{n+1}[i-1]) -
> (M[I,i+1]*v_{n}[i+1]+M[I,i+2]*v_{n}[i+2]+..+M[I,N]*v_{n}[N]))
>
> I do not see that this is something that can esaily be vectorized, is
> it?


I think it would be better if you stated what the actual problem is. Is it a
differential equation, for instance. That way we can determine what the
problem class is and what algorithms are available to solve it.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070215/7b7db58c/attachment.html>

From gzhu at peak6.com  Thu Feb 15 11:32:32 2007
From: gzhu at peak6.com (Geoffrey Zhu)
Date: Thu, 15 Feb 2007 10:32:32 -0600
Subject: [Numpy-discussion] Numpy and iterative procedures
Message-ID: <99F81FFD0EA54E4DA8D4F1BFE272F341038279B5@ppi-mail1.chicago.peak6.net>

Thanks Chuck.
 
I am trying to use Successive Over-relaxation to solve linear equations
defined by M*v=3Dq. 
 
There are several goals:
 
1. Eventually (in production) I need it to be fast.
2. I am playing with the guts of the algorithm for now, to see how it
works. that means i need some control for now.
3. Even in production, there is a chance i'd like to have the ability to
tinker with the algorithm. 
 

  _____  

From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Charles R
Harris
Sent: Thursday, February 15, 2007 10:11 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Numpy and iterative procedures





On 2/15/07, Geoffrey Zhu <gzhu at peak6.com> wrote: 

	Hi,
	
	I am new to numpy. I'd like to know if it is possible to code
efficient
	iterative procedures with numpy.
	
	Specifically, I have the following problem.
	
	M is an N*N matrix. Q is a N*1 vector. V is an N*1 vector I am
trying to 
	find iteratively from the initial value V_0. The procedure is
simply to
	calculate
	
	V_{n+1}[i]=3D3D1/M[I,i]*(q[i]-
	(M[i,1]*v_{n+1}[1]+M[I,2]*v_{n+1}[2]+..+M[i,i-1]*v_{n+1}[i-1]) -
	(M[I,i+1]*v_{n}[i+1]+M[I,i+2]*v_{n}[i+2]+..+M[I,N]*v_{n}[N])) 
	
	I do not see that this is something that can esaily be
vectorized, is
	it?


I think it would be better if you stated what the actual problem is. Is
it a differential equation, for instance. That way we can determine what
the problem class is and what algorithms are available to solve it. 

Chuck




_______________________________________________________=0A=
=0A=
The  information in this email or in any file attached=0A=
hereto is intended only for the personal and confiden-=0A=
tial  use  of  the individual or entity to which it is=0A=
addressed and may contain information that is  propri-=0A=
etary  and  confidential.  If you are not the intended=0A=
recipient of this message you are hereby notified that=0A=
any  review, dissemination, distribution or copying of=0A=
this message is strictly prohibited.  This  communica-=0A=
tion  is  for information purposes only and should not=0A=
be regarded as an offer to sell or as  a  solicitation=0A=
of an offer to buy any financial product. Email trans-=0A=
mission cannot be guaranteed to be  secure  or  error-=0A=
free. P6070214=0A=

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070215/2f979e1e/attachment.html>

From nwagner at iam.uni-stuttgart.de  Thu Feb 15 11:46:05 2007
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Thu, 15 Feb 2007 17:46:05 +0100
Subject: [Numpy-discussion] Numpy and iterative procedures
In-Reply-To: <99F81FFD0EA54E4DA8D4F1BFE272F341038279B5@ppi-mail1.chicago.peak6.net>
References: <99F81FFD0EA54E4DA8D4F1BFE272F341038279B5@ppi-mail1.chicago.peak6.net>
Message-ID: <45D48E4D.5040907@iam.uni-stuttgart.de>

Geoffrey Zhu wrote:
> Thanks Chuck.
>  
> I am trying to use Successive Over-relaxation to solve linear
> equations defined by M*v=q.
>
AFAIK, splitting methods are not the best methods to solve a system of
linear equations, iteratively.
What can be said about your coefficient matrix M ? Is it spd (symmetric
positive definite) ?
In that case I suggest to use cg.

Anyway, you may try the following solvers which are available in scipy

     Iterative linear systems solutions

       cg         --- Conjugate gradient (symmetric systems only)
       cgs        --- Conjugate gradient squared
       qmr        --- Quasi-minimal residual
       gmres      --- Generalized minimal residual
       bicg       --- Bi-conjugate gradient
       bicgstab   --- Bi-conjugate gradient stabilized

Nils


> There are several goals:
>  
> 1. Eventually (in production) I need it to be fast.
> 2. I am playing with the guts of the algorithm for now, to see how it
> works. that means i need some control for now.
> 3. Even in production, there is a chance i'd like to have the ability
> to tinker with the algorithm.
>  
>
> ------------------------------------------------------------------------
> *From:* numpy-discussion-bounces at scipy.org
> [mailto:numpy-discussion-bounces at scipy.org] *On Behalf Of *Charles R
> Harris
> *Sent:* Thursday, February 15, 2007 10:11 AM
> *To:* Discussion of Numerical Python
> *Subject:* Re: [Numpy-discussion] Numpy and iterative procedures
>
>
>
> On 2/15/07, *Geoffrey Zhu* <gzhu at peak6.com <mailto:gzhu at peak6.com>>
> wrote:
>
>     Hi,
>
>     I am new to numpy. I'd like to know if it is possible to code
>     efficient
>     iterative procedures with numpy.
>
>     Specifically, I have the following problem.
>
>     M is an N*N matrix. Q is a N*1 vector. V is an N*1 vector I am
>     trying to
>     find iteratively from the initial value V_0. The procedure is
>     simply to
>     calculate
>
>     V_{n+1}[i]=3D1/M[I,i]*(q[i]-
>     (M[i,1]*v_{n+1}[1]+M[I,2]*v_{n+1}[2]+..+M[i,i-1]*v_{n+1}[i-1]) -
>     (M[I,i+1]*v_{n}[i+1]+M[I,i+2]*v_{n}[i+2]+..+M[I,N]*v_{n}[N]))
>
>     I do not see that this is something that can esaily be vectorized, is
>     it?
>
>
> I think it would be better if you stated what the actual problem is.
> Is it a differential equation, for instance. That way we can determine
> what the problem class is and what algorithms are available to solve it.
>
> Chuck
>
>
> ------------------------------------------------------------------------
>
> The information in this email or in any file attached hereto is
> intended only for the personal and confidential use of the individual
> or entity to which it is addressed and may contain information that is
> proprietary and confidential. If you are not the intended recipient of
> this message you are hereby notified that any review, dissemination,
> distribution or copying of this message is strictly prohibited. This
> communication is for information purposes only and should not be
> regarded as an offer to sell or as a solicitation of an offer to buy
> any financial product. Email transmission cannot be guaranteed to be
> secure or error-free. P6070214
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   
 


From robert.kern at gmail.com  Thu Feb 15 12:02:58 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 15 Feb 2007 12:02:58 -0500
Subject: [Numpy-discussion] Unifying numpy, scipy,
	and matplotlib docstring formats
In-Reply-To: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com>
References: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com>
Message-ID: <3d375d730702150902k31a1356ek4743a182902c90aa@mail.gmail.com>

On 2/15/07, Keir Mierle <mierle at gmail.com> wrote:
> On the DocstringStandard page I have also put a completely re-done docstring
> for the 'contour' function from matplotlib. I think it is far more readable
> than the original [3]. JDH and other matplotlibheads, what do you think?
> Travis, do you find my additions reasonable? Scipy maintainers, would you
> consider adopting this format (especially if someone helps with the gruntwork)?

It looks like you took the initial proposal rather than the result of
that discussion. Please see the document that we came up with:

http://svn.scipy.org/svn/numpy/trunk/numpy/doc/HOWTO_DOCUMENT.txt

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From gzhu at peak6.com  Thu Feb 15 12:08:21 2007
From: gzhu at peak6.com (Geoffrey Zhu)
Date: Thu, 15 Feb 2007 11:08:21 -0600
Subject: [Numpy-discussion] Confusing selector?
Message-ID: <99F81FFD0EA54E4DA8D4F1BFE272F341038279D6@ppi-mail1.chicago.peak6.net>

I am really new to numpy but I just found that v[2:4] means selecting
v[2] and v[3]. V[4] is not included! This is quite different from the
conventions of matlab and R. 




_______________________________________________________=0A=
=0A=
The  information in this email or in any file attached=0A=
hereto is intended only for the personal and confiden-=0A=
tial  use  of  the individual or entity to which it is=0A=
addressed and may contain information that is  propri-=0A=
etary  and  confidential.  If you are not the intended=0A=
recipient of this message you are hereby notified that=0A=
any  review, dissemination, distribution or copying of=0A=
this message is strictly prohibited.  This  communica-=0A=
tion  is  for information purposes only and should not=0A=
be regarded as an offer to sell or as  a  solicitation=0A=
of an offer to buy any financial product. Email trans-=0A=
mission cannot be guaranteed to be  secure  or  error-=0A=
free. P6070214=0A=



From lu.yangunb at gmail.com  Wed Feb 14 21:47:02 2007
From: lu.yangunb at gmail.com (lu yang)
Date: Wed, 14 Feb 2007 22:47:02 -0400
Subject: [Numpy-discussion] Numpy1.0.1 installation problem - urgent :(
Message-ID: <2768a9540702141847y568794f5vff110a4dbd4caf18@mail.gmail.com>

Hi,
This is the first time I install Numpy on a linux machine. I have been
working on it for several days without luck. I sincerely appreciate if
anybody can give any comments. My OS is Red Hat 8.
I have downloaded Python 2.5 and Numpy 1.0.1. Python 2.5 has been installed
on my machine. However, when I type:
"python2.5 setup.py install" to install Numpy in my home directory
/home/o6c11/numpy-1.0.1 , I got:

[o6c11 at chorus numpy-1.0.1]$ python2.5 setup.py install
Traceback (most recent call last):
  File "setup.py", line 89, in <module>
    setup_package()
  File "setup.py", line 59, in setup_package
    from numpy.distutils.core import setup
  File "numpy/__init__.py", line 36, in <module>
    import core
  File "/mnt/storage/home/o6c11/numpy-1.0.1/numpy/core/__init__.py", line 5,
in <module>
    import multiarray
ImportError: No module named multiarray

I changed the directory to /usr/local/lib/python2.5/site-packages/numpy, I
got:

[o6c11 at chorus numpy]$ python2.5 setup.py install
Traceback (most recent call last):
  File "setup.py", line 28, in <module>
    from numpy.distutils.core import setup
  File "/usr/local/lib/python2.5/site-packages/numpy/__init__.py", line 40,
in <module>
    import linalg
  File "/usr/local/lib/python2.5/site-packages/numpy/linalg/__init__.py",
line 4, in <module>
    from linalg import *
  File "/usr/local/lib/python2.5/site-packages/numpy/linalg/linalg.py", line
25, in <module>
    from numpy.linalg import lapack_lite
ImportError: /usr/lib/libblas.so.3: undefined symbol: e_wsfe

I also tried to import numpy in this way:

[o6c11 at chorus numpy]$ python2.5
Python 2.5 (r25:51908, Feb 12 2007, 22:36:33)
[GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.5/site-packages/numpy/__init__.py", line 40,
in <module>
    import linalg
  File "/usr/local/lib/python2.5/site-packages/numpy/linalg/__init__.py",
line 4, in <module>
    from linalg import *
  File "/usr/local/lib/python2.5/site-packages/numpy/linalg/linalg.py", line
25, in <module>
    from numpy.linalg import lapack_lite
ImportError: /usr/lib/libblas.so.3: undefined symbol: e_wsfe
>>>

I have no idea how to solve these problems. Please help a newbie. Thanks a
lot.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070214/66f4a9ef/attachment.html>

From gosselina at dfo-mpo.gc.ca  Sun Feb 11 21:24:02 2007
From: gosselina at dfo-mpo.gc.ca (Andre Gosselin)
Date: Sun, 11 Feb 2007 21:24:02 -0500
Subject: [Numpy-discussion]  Request for porting pycdf to NumPy
Message-ID: <1171247042.4560.6.camel@maboule>

Hi everybody.

I have just released version 0.6-3 of pycdf, which finally supports the
NumPy array package (along Numeric and numarray).

Here is the announcement.
==================================
Project "pysclint" ('pysclint') has released the new version of package
'pycdf'.
You can download it from SourceForge.net by following this link:
<https://sourceforge.net/project/showfiles.php?
group_id=143145&release_id=485773
>
or browse Release Notes and ChangeLog by visiting this link:
<https://sourceforge.net/project/shownotes.php?release_id=485773> 
==================================

I was pleased to follow this thread about pycdf, and see the number of
people who are pleased with it. 

In case anybody doubts it, I will still actively maintain pycdf.

Best regards,

Andr? Gosselin
gosselina at dfo-mpo.gc.ca





From satra at speech.mit.edu  Sun Feb 11 22:51:25 2007
From: satra at speech.mit.edu (Satrajit Ghosh)
Date: Sun, 11 Feb 2007 22:51:25 -0500
Subject: [Numpy-discussion] SVN Build, optimized libraries, site.cfg, windows
Message-ID: <9135A452397C4096AD412BD0A64C8553@rleendurance>

Hi,

I'm also not quite clear whether the optimized FFTW and UMFPACK libraries
are being used or required in numpy at all as show_config() doesn't report
it. 

I see that fftw and umfpack are being used for scipy.

I have attached my site.cfg. Any help would be much appreciated.

Cheers,

Satra


---- snip site.cfg ----
[atlas]
libraries = f77blas, cblas, atlas
library_dirs = C:\MinGW\lib;C:\DOWNLOADS\BUILDS\lib
include_dirs = C:\DOWNLOADS\BUILDS\include\ATLAS 

[lapack]
libraries = flapack, f77blas, cblas, atlas library_dirs =
C:\MinGW\lib;C:\DOWNLOADS\BUILDS\lib

# UMFPACK
# -------
# The UMFPACK library is used to factor large sparse matrices. It, in turn,
# depends on the AMD library for reordering the matrices for better
performance.
# Note that the AMD library has nothing to do with AMD (Advanced Micro
Devices), # the CPU company.
#
#   http://www.cise.ufl.edu/research/sparse/umfpack/
#   http://www.cise.ufl.edu/research/sparse/amd/
#
[amd]
library_dirs = C:\MinGW\lib;C:\DOWNLOADS\BUILDS\lib
include_dirs = C:\DOWNLOADS\BUILDS\include amd_libs = amd 

[umfpack]
library_dirs = C:\MinGW\lib;C:\DOWNLOADS\BUILDS\lib
include_dirs = C:\DOWNLOADS\BUILDS\include\UMFPACK
umfpack_libs = umfpack

# FFT libraries
# -------------
# There are two FFT libraries that we can configure here: FFTW (2 and 3) and
djbfft.
#
#   http://fftw.org/
#   http://cr.yp.to/djbfft.html
#
# Given only this section, numpy.distutils will try to figure out which
version # of FFTW you are using.
[fftw]
library_dirs = C:\MinGW\lib;C:\DOWNLOADS\BUILDS\lib
include_dirs = C:\DOWNLOADS\BUILDS\include libraries = fftw3
---- end snip



From shuntim.luk at polyu.edu.hk  Tue Feb 13 00:35:24 2007
From: shuntim.luk at polyu.edu.hk (LUK ShunTim)
Date: Tue, 13 Feb 2007 13:35:24 +0800
Subject: [Numpy-discussion] Possible solution to
 binary	distributionproblems for numpy on linux?
In-Reply-To: <45D118FC.7020403@ar.media.kyoto-u.ac.jp>
References: <45CFF4D0.5080401@ar.media.kyoto-u.ac.jp>	<f4f93d420702120945k59cd887h2f68d27c8c051a0b@mail.gmail.com>
	<45D118FC.7020403@ar.media.kyoto-u.ac.jp>
Message-ID: <45D14E1C.4010809@polyu.edu.hk>

David Cournapeau wrote:
> Keith Goodman wrote:
>> On 2/11/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
>>> My impression is that binary distribution of numpy is a big problem for many
>>> linux users, and that is entry barrier for many users (I may be wrong, that's just
>>> an impression from the ML).
>> Do all of the major GNU/Linux distributions have recent versions of NumPy?
>>
>> Debian Etch is at NumPy 1.0.1
> I think debian has numpy now (I am not using debian on workstation 
> anymore, so I am not really following), but what about new versions of 
> numpy/scipy ? If I want to give some of my code to people in my lab who 
> do not use the same distribution than me, can I give a 10 minutes 
> instructions set to have everything ?
> 

"Bleeding-edge apt-get repository" according to their web page.
"To use it add the following line to you /etc/apt/sources.list"

deb http://deb-scipy.alioth.debian.org/apt ./

Regards,
ST
--


From charlesr.harris at gmail.com  Thu Feb 15 12:28:59 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 15 Feb 2007 10:28:59 -0700
Subject: [Numpy-discussion] Confusing selector?
In-Reply-To: <99F81FFD0EA54E4DA8D4F1BFE272F341038279D6@ppi-mail1.chicago.peak6.net>
References: <99F81FFD0EA54E4DA8D4F1BFE272F341038279D6@ppi-mail1.chicago.peak6.net>
Message-ID: <e06186140702150928t4e80a13bx5b2d24160492340e@mail.gmail.com>

On 2/15/07, Geoffrey Zhu <gzhu at peak6.com> wrote:
>
> I am really new to numpy but I just found that v[2:4] means selecting
> v[2] and v[3]. V[4] is not included! This is quite different from the
> conventions of matlab and R.


Yes, it is the Python slicing convention and rather similar to for loops in
C where

for(i = 0; i < N; i++)

is a common construction. It is an consequence of zero based indexing.
Matlab started in Fortran and uses the conventions of the Fortran do loop,
which no doubt makes sense for arrays whose base index is 1. Slicing in
NumPy, and now Python, also includes a step, so you can write a[0:n:2] to
reference all the elements with even indices.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070215/2a1e872f/attachment.html>

From charlesr.harris at gmail.com  Thu Feb 15 12:36:20 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 15 Feb 2007 10:36:20 -0700
Subject: [Numpy-discussion] Possible solution to binary
	distributionproblems for numpy on linux?
In-Reply-To: <45D14E1C.4010809@polyu.edu.hk>
References: <45CFF4D0.5080401@ar.media.kyoto-u.ac.jp>
	<f4f93d420702120945k59cd887h2f68d27c8c051a0b@mail.gmail.com>
	<45D118FC.7020403@ar.media.kyoto-u.ac.jp>
	<45D14E1C.4010809@polyu.edu.hk>
Message-ID: <e06186140702150936sc27d1fx475412bfdc14e69c@mail.gmail.com>

On 2/12/07, LUK ShunTim <shuntim.luk at polyu.edu.hk> wrote:
>
> David Cournapeau wrote:
> > Keith Goodman wrote:
> >> On 2/11/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> >>> My impression is that binary distribution of numpy is a big problem
> for many
> >>> linux users, and that is entry barrier for many users (I may be wrong,
> that's just
> >>> an impression from the ML).
> >> Do all of the major GNU/Linux distributions have recent versions of
> NumPy?
> >>
> >> Debian Etch is at NumPy 1.0.1
> > I think debian has numpy now (I am not using debian on workstation
> > anymore, so I am not really following), but what about new versions of
> > numpy/scipy ? If I want to give some of my code to people in my lab who
> > do not use the same distribution than me, can I give a 10 minutes
> > instructions set to have everything ?
> >
>
> "Bleeding-edge apt-get repository" according to their web page.
> "To use it add the following line to you /etc/apt/sources.list"
>
> deb http://deb-scipy.alioth.debian.org/apt ./


Yep. In general, if you are using a Linux distro it is quite easy to use the
svn repository. It seem to be on the Mac and Windows that folks have
problems, particularly in filling the dependencies on Atlas if they want
efficiency.

That said, Numpy and MatplotLab have settled enough that the standard
packages are probably adequate.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070215/a4484c92/attachment.html>

From mierle at gmail.com  Thu Feb 15 13:05:20 2007
From: mierle at gmail.com (Keir Mierle)
Date: Thu, 15 Feb 2007 13:05:20 -0500
Subject: [Numpy-discussion] Possible solution to binary
	distributionproblems for numpy on linux?
In-Reply-To: <e06186140702150936sc27d1fx475412bfdc14e69c@mail.gmail.com>
References: <45CFF4D0.5080401@ar.media.kyoto-u.ac.jp>
	<f4f93d420702120945k59cd887h2f68d27c8c051a0b@mail.gmail.com>
	<45D118FC.7020403@ar.media.kyoto-u.ac.jp>
	<45D14E1C.4010809@polyu.edu.hk>
	<e06186140702150936sc27d1fx475412bfdc14e69c@mail.gmail.com>
Message-ID: <ef5675f30702151005q1d8046b5o1063f69ab8713aa0@mail.gmail.com>

On 2/15/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
> On 2/12/07, LUK ShunTim <shuntim.luk at polyu.edu.hk> wrote:
> > David Cournapeau wrote:
> > > Keith Goodman wrote:
> > >> On 2/11/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> > > > > My impression is that binary distribution of numpy is a big problem
> > > > > for many linux users, and that is entry barrier for many users (I may
> > > > > be wrong, that's just an impression from the ML).
> > > > > Do all of the major GNU/Linux distributions have recent versions of
> > > > > NumPy?
> > > > Debian Etch is at NumPy 1.0.1
> > > I think debian has numpy now (I am not using debian on workstation
> > > anymore, so I am not really following), but what about new versions of
> > > numpy/scipy ? If I want to give some of my code to people in my lab who
> > > do not use the same distribution than me, can I give a 10 minutes
> > > instructions set to have everything ?
> > >
> >
> > "Bleeding-edge apt-get repository" according to their web page.
> > "To use it add the following line to you /etc/apt/sources.list"
> >
> > deb http://deb-scipy.alioth.debian.org/apt ./
>
> Yep. In general, if you are using a Linux distro it is quite easy to use the
> svn repository. It seem to be on the Mac and Windows that folks have
> problems, particularly in filling the dependencies on Atlas if they want
> efficiency.
>
> That said, Numpy and MatplotLab have settled enough that the standard
> packages are probably adequate.

While this is true for people admining their own machines, this is not the case
for people without root access. For example, it is a major hassle to install
numpy on the computers at my university for exactly this reason.

It would be nice to have a self contained build of numpy/scipy/matplotlib for
these cases.

Keir


From mierle at gmail.com  Thu Feb 15 13:42:51 2007
From: mierle at gmail.com (Keir Mierle)
Date: Thu, 15 Feb 2007 13:42:51 -0500
Subject: [Numpy-discussion] Unifying numpy, scipy,
	and matplotlib docstring formats
In-Reply-To: <3d375d730702150902k31a1356ek4743a182902c90aa@mail.gmail.com>
References: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com>
	<3d375d730702150902k31a1356ek4743a182902c90aa@mail.gmail.com>
Message-ID: <ef5675f30702151042u543f508aucd1b0656895b273d@mail.gmail.com>

On 2/15/07, Robert Kern <robert.kern at gmail.com> wrote:
> On 2/15/07, Keir Mierle <mierle at gmail.com> wrote:
> > On the DocstringStandard page I have also put a completely re-done docstring
> > for the 'contour' function from matplotlib. I think it is far more readable
> > than the original [3]. JDH and other matplotlibheads, what do you think?
> > Travis, do you find my additions reasonable? Scipy maintainers, would you
> > consider adopting this format (especially if someone helps with the gruntwork)?
>
> It looks like you took the initial proposal rather than the result of
> that discussion. Please see the document that we came up with:
>
> http://svn.scipy.org/svn/numpy/trunk/numpy/doc/HOWTO_DOCUMENT.txt

Ah, I apologize for not checking the dates; I thought the HOWTO_DOCUMENT.txt
was the older proposal.

Nevertheless, I think the issues raised in my proposed version are significant
enough to warrent further discussion; especially for the more demanding needs
of matplotlib.

I would like to re-open this discussion to be sure there is consensus among the
numpy, scipy, and matplotlib folk before I invest signifcant time into
massaging the docstrings into the right form.

I am clearly biased as I invested time and thought into the proposed docstring
format I posted [1], but nevertheless I do not like the style listed in the
HOWTO_DOCUMENT.txt. The different sections have different styles of headings,
i.e. the difference style for :Pamaraters: and Examples, which is not good for
readability.  Furthermore, it does not specify enough formatting, for e.g.
keyword arguments with defaults.

For specifics, here are my issues with the current HOWTO:

 * Non-capitalized headers
   Capitalized headers are far more visually obvious when viewed on a text
   terminal (i.e. via function? in IPython)

 * Two different header styles
   The distinction between
   :Parameters:
   and
   Examples
   --------
   seems unnecessary; if this is necessary for reST, could a preprocessing step
   not fix this? The inconsistency appears unprofessional when viewed in a
   terminal.

 * No suggestions on how to handle functions which have multiple invocations,
   i.e. multiple function signatures. I have a proposal for this in [1].

 * Parameters / Returns instead of INPUTS / OUTPUTS. This is no doubt a
   preference, but nevertheless I vastly prefer having INPUTS / OUTPUTS instead
   of Parameters / Returns. I understand that the parameter/return form is more
   common for Python, so I realize this is contentious. Nevertheless, inputs /
   outputs has the clear advantage of being informative to someone who is just
   starting programming and may not realize the meanings of parameters /
   returns; but input/output is absolutely clear even to the non-programmer.

If it comes down to me writing a parser for my proposed format, I will do that.

Keir

[1] http://scipy.org/DocstringStandard


From mjanikas at esri.com  Thu Feb 15 13:48:39 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Thu, 15 Feb 2007 10:48:39 -0800
Subject: [Numpy-discussion] Update on sparse matrix in binary
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFB9@hardwire.esri.com>

I wanted to thank all of you who helped me with my making my sparse
matrix representation cross-platform in binary format!

 

I ended up writing and reading everything explicitly in little endian.
To recap, each row in the matrix is represented by three records:

 

1)       row#, nn (number of non-zero elements)

2)       col (indices of non-zero elements in row #)

3)       values (doubles corresponding to indices provided in the
previous array)

 

I created the arrays using: 

 

a = array([values], '<l or d').tostring(); f.write(a)

 

And read using: 

 

a = f.read(# of bytes, 8 for first record, nn*4 for longs, nn*8 for
doubles); 

a = fromstring(a, '<l or d')

 

The tofile() and fromfile() version work as well, but it wasn't any
faster.  So I went with what I thought was more intuitive in the context
of the algorithm.

 

So far, it has worked on the Windows, Linux, Solaris and Mac OSX boxes I
have tried.  I know it is hardware specific as opposed to platform, so I
will continue to seek out the rarer vintages of machines that can run
the software we provide.

 

Again, thanks to all; and if there are any more suggestions as to how
this process could be more optimized, please feel free to respond.  Best
wishes,

 

 

Mark Janikas

Product Engineer

ESRI, Geoprocessing

380 New York St.

Redlands, CA 92373

909-793-2853 (2563)

mjanikas at esri.com

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070215/1d00ee46/attachment.html>

From ellisonbg.net at gmail.com  Thu Feb 15 14:11:57 2007
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 15 Feb 2007 12:11:57 -0700
Subject: [Numpy-discussion] Numpy1.0.1 installation problem - urgent :(
In-Reply-To: <2768a9540702141847y568794f5vff110a4dbd4caf18@mail.gmail.com>
References: <2768a9540702141847y568794f5vff110a4dbd4caf18@mail.gmail.com>
Message-ID: <6ce0ac130702151111h55a60b6dl5228e11234b49179@mail.gmail.com>

I don't run numpy no linux often, but you shouldn't have any trouble.
I would do the following:

1.  Blast your current numpy install

rm -rf /usr/local/lib/python2.5/site-packages/numpy

2.  Get the lastest svn version

cd $HOME
svn co http://svn.scipy.org/svn/numpy/trunk numpy

3.  Try doing a fresh install (starting from $HOME):

cd numpy
python setup.py build
sudo python setup.py install

Let us know if that helps.

Brian

On 2/14/07, lu yang <lu.yangunb at gmail.com> wrote:
> Hi,
> This is the first time I install Numpy on a linux machine. I have been
> working on it for several days without luck. I sincerely appreciate if
> anybody can give any comments. My OS is Red Hat 8.
> I have downloaded Python 2.5 and Numpy 1.0.1. Python 2.5 has been installed
> on my machine. However, when I type:
> "python2.5 setup.py install" to install Numpy in my home directory
> /home/o6c11/numpy-1.0.1 , I got:
>
> [o6c11 at chorus numpy-1.0.1]$ python2.5 setup.py install
> Traceback (most recent call last):
>   File "setup.py", line 89, in <module>
>     setup_package()
>   File "setup.py", line 59, in setup_package
>     from numpy.distutils.core import setup
>   File "numpy/__init__.py", line 36, in <module>
>     import core
>   File "/mnt/storage/home/o6c11/numpy-1.0.1/numpy/core/__init__.py", line 5,
> in <module>
>     import multiarray
> ImportError: No module named multiarray
>
> I changed the directory to
> /usr/local/lib/python2.5/site-packages/numpy, I got:
>
> [o6c11 at chorus numpy]$ python2.5 setup.py install
> Traceback (most recent call last):
>   File "setup.py", line 28, in <module>
>     from numpy.distutils.core import setup
>   File
> "/usr/local/lib/python2.5/site-packages/numpy/__init__.py",
> line 40, in <module>
>     import linalg
>   File
> "/usr/local/lib/python2.5/site-packages/numpy/linalg/__init__.py",
> line 4, in <module>
>     from linalg import *
>   File
> "/usr/local/lib/python2.5/site-packages/numpy/linalg/linalg.py",
> line 25, in <module>
>     from numpy.linalg import lapack_lite
> ImportError: /usr/lib/libblas.so.3: undefined symbol: e_wsfe
>
> I also tried to import numpy in this way:
>
> [o6c11 at chorus numpy]$ python2.5
> Python 2.5 (r25:51908, Feb 12 2007, 22:36:33)
> [GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import numpy
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File
> "/usr/local/lib/python2.5/site-packages/numpy/__init__.py",
> line 40, in <module>
>     import linalg
>   File
> "/usr/local/lib/python2.5/site-packages/numpy/linalg/__init__.py",
> line 4, in <module>
>     from linalg import *
>   File
> "/usr/local/lib/python2.5/site-packages/numpy/linalg/linalg.py",
> line 25, in <module>
>     from numpy.linalg import lapack_lite
> ImportError: /usr/lib/libblas.so.3: undefined symbol: e_wsfe
> >>>
>
> I have no idea how to solve these problems. Please help a newbie. Thanks a
> lot.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From aisaac at american.edu  Thu Feb 15 14:40:18 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Thu, 15 Feb 2007 14:40:18 -0500
Subject: [Numpy-discussion] Unifying numpy, scipy,
	and matplotlib docstring formats
In-Reply-To: <ef5675f30702151042u543f508aucd1b0656895b273d@mail.gmail.com>
References: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com><3d375d730702150902k31a1356ek4743a182902c90aa@mail.gmail.com><ef5675f30702151042u543f508aucd1b0656895b273d@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-1448-20070215-144018.00@american.edu>

On Thu, 15 Feb 2007, Keir Mierle apparently wrote:
>  * Two different header styles
>    The distinction between
>    :Parameters:
>    and
>    Examples
>    --------

These are not two items of the same type.
``:Parameters:`` designates a consolidated field (for 
describing the documented object).  This is not a "section".

Cheers,
Alan Isaac






From wfspotz at sandia.gov  Thu Feb 15 16:26:50 2007
From: wfspotz at sandia.gov (Bill Spotz)
Date: Thu, 15 Feb 2007 14:26:50 -0700
Subject: [Numpy-discussion] Global Numpy vector with Swig
In-Reply-To: <0A4BC30F-9080-43F7-899B-6F246F345216@tiscalinet.it>
References: <0A4BC30F-9080-43F7-899B-6F246F345216@tiscalinet.it>
Message-ID: <A5AB0E0A-6FE0-4A17-8E98-B6DF512CDC03@sandia.gov>

It seems to me you would need to %ignore vec, so that it is not  
wrapped as a raw pointer to a double, and then in your interface file  
create a PyArrayObject whose data buffer points to vec (the most  
efficient way to do this is with %inline).  Then use %rename to  
rename whatever you called your PyArrayObject to "vec" (or not, if  
you do not care about the name).  Finally,

   foo.cvar.vec = numpy.zeros(10)

is going to decrement (and probably delete) your original vec and  
replace it with the new array of zeros (I think, unless there are  
some safeties put in place for cvars, in which case you would get an  
exception).  Use

   foo.cvar.vec[:] = ...

to manipulate the data in-place.

On Feb 15, 2007, at 1:18 AM, Andrea Tomadin wrote:

> Hi,
> I need to pass a Numpy array to a C code wrapped by Swig.
> The array in the C code is a global variable declared as
> double *vec
> and I would like to set it in the calling Python module foo using e.g.
> foo.cvar.vec = numpy.zeros(10)
> so that the array is manipulated in place.
>
> I found out the examples in the numpy/doc/swig dir and adapted them
> to my code, so now I can pass a Numpy array to a function. But in the
> examples
> I was not able to recognize any suggestion (any typedef?) to
> translate an array into a
> variable. I'm just writing an intermediate function that does the job,
> but I feel like writing a wrapping function that will be wrapped
> again by Swig...
>
> Any practical example is highly appreciated!
>
> Regards,
> Andrea
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

** Bill Spotz                                              **
** Sandia National Laboratories  Voice: (505)845-0170      **
** P.O. Box 5800                 Fax:   (505)284-5451      **
** Albuquerque, NM 87185-0370    Email: wfspotz at sandia.gov **





From tom.denniston at alum.dartmouth.org  Thu Feb 15 18:10:33 2007
From: tom.denniston at alum.dartmouth.org (Tom Denniston)
Date: Thu, 15 Feb 2007 17:10:33 -0600
Subject: [Numpy-discussion] problems with auto detection and broadcasting
	for userdefined numpy types
Message-ID: <d6f2d3dd0702151510p7001c13fo7ab87a84a8fe0f4@mail.gmail.com>

I have a userdefined numpy type for mx.DateTime and one for
mx.DateTimeDelta.  Info on these packages is available here
(http://www.egenix.com/files/python/eGenix-mx-Extensions.html).

In any case most things work.

What I am left with is three problems:


1. I cannot figure out how to cleanly or even at all, get type detection to work
So this works:
In [35]: numpy.array([D('2002-01-02 00:00:00.00'), D('2002-01-03
00:00:00.00')], dtype=numpy.mxDateTimeType)
Out[35]: array([2002-01-02 00:00:00.00, 2002-01-03 00:00:00.00],
dtype=mxNumpyDateTime)
But this doesn't:
In [36]: numpy.array([D('2002-01-02 00:00:00.00'), D('2002-01-03 00:00:00.00')])
Out[36]: array([2002-01-02 00:00:00.00, 2002-01-03 00:00:00.00], dtype=object)

I.e. if I don't specify the type it gives me an object dtype.  This is
unlike the behavior for strings, ints, etc.  I think I know where the
C code is that controls this but it is not obvious to me how to change
it without making it dependent on mx.DateTime at compile time or
having to put a bunch of #define #ifdef magic around it.  It would be
preferable to figure out a way to specify autodetection, from the
outside, so to speak.

2. I can get printing to work but only using a quasi hack in the numpy
arrayprint.py code.  It works with or without presense of the mx
libraries but doesn't seem to me entirely clean.

3. Broadcasting has bugs.  See example below:
The basic case (2,) - (2,) arrays works.
In [37]: numpy.array([D('2002-01-02 00:00:00.00'), D('2002-01-03
00:00:00.00')], dtype=numpy.mxDateTimeType) -
numpy.array([DateTime.DateTimeDeltaFromDays(1)],
dtype=numpy.mxDateTimeDeltaType)
Out[37]: array([2002-01-01 00:00:00.00, 2002-01-02 00:00:00.00],
dtype=mxNumpyDateTime)
The broadcasting case of (2,) - (1,) works:
In [39]: numpy.array([D('2002-01-02 00:00:00.00'), D('2002-01-03
00:00:00.00')], dtype=numpy.mxDateTimeType) -
numpy.array([DateTime.DateTimeDeltaFromDays(20)],
dtype=numpy.mxDateTimeDeltaType)
Out[39]: array([2001-12-13 00:00:00.00, 2001-12-14 00:00:00.00],
dtype=mxNumpyDateTime)
The extremely similar (2,) - () array:
In [40]: numpy.array([D('2002-01-02 00:00:00.00'), D('2002-01-03
00:00:00.00')], dtype=numpy.mxDateTimeType) -
numpy.array(DateTime.DateTimeDeltaFromDays(20),
dtype=numpy.mxDateTimeDeltaType)
---------------------------------------------------------------------------
<type 'exceptions.TypeError'>             Traceback (most recent call last)

/home/tom/src/<ipython console> in <module>()

<type 'exceptions.TypeError'>: function not supported for these types,
and can't coerce safely to supported types
Finally the (2,) - scalar doesn't work either:

In [41]: numpy.array([D('2002-01-02 00:00:00.00'), D('2002-01-03
00:00:00.00')], dtype=numpy.mxDateTimeType) -
DateTime.DateTimeDeltaFromDays(20)
---------------------------------------------------------------------------
<type 'exceptions.TypeError'>             Traceback (most recent call last)

/home/tom/src/<ipython console> in <module>()

<type 'exceptions.TypeError'>: function not supported for these types,
and can't coerce safely to supported types

The final one might not work even if broadcasting worked properly
simply because of the autodetection problem listed in 2.  The (2,) -
(), however, should work, I would think, because it has the type
explicitly set.



--Tom


From kwgoodman at gmail.com  Thu Feb 15 23:35:14 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 15 Feb 2007 20:35:14 -0800
Subject: [Numpy-discussion] Different results from repeated calculation
In-Reply-To: <f4f93d420701271225l18260e45x3ccc93f47ba982f0@mail.gmail.com>
References: <f4f93d420701271225l18260e45x3ccc93f47ba982f0@mail.gmail.com>
Message-ID: <f4f93d420702152035y746ec4cen2882d25e977cf545@mail.gmail.com>

On 1/27/07, Keith Goodman <kwgoodman at gmail.com> wrote:
> I get slightly different results when I repeat a calculation. In a
> long simulation the differences snowball and swamp the effects I am
> trying to measure.
>
> In the attach script there are three tests.
>
> In test1, I construct matrices x and y and then repeatedly calculate z
> = calc(x,y). The result z is the same every time. So this test passes.
>
> In test2, I construct matrices x and y each time before calculating z
> = calc(x,y). Sometimes z is slightly different---of the order of 1e-21
> to 1e-18. But the x's test to be equal and so do the y's. This test
> fails. (It doesn't fail on my friend's windows box; I'm running
> linux.)
>
> test3 is the same as test2 but I calculate z like this: z = calc(100*x
> ,y) / (100 * 100). This test passes.
>
> What is going on?
>
> Here is some sample output:
>
> >> import repeat
> >> repeat.all()
> 4 z different 8.47032947254e-22
> 5 z different 8.47032947254e-22
> 7 z different 8.47032947254e-22
> 9 z different 8.47032947254e-22
> 10 z different 8.47032947254e-22
> 16 z different 8.47032947254e-22
> 24 z different 8.47032947254e-22
> 25 z different 8.47032947254e-22
> 26 z different 8.47032947254e-22
> 27 z different 8.47032947254e-22
> 30 z different 8.47032947254e-22
> 32 z different 8.47032947254e-22
> 34 z different 8.47032947254e-22
> 35 z different 8.47032947254e-22
> 36 z different 8.47032947254e-22
> 39 z different 8.47032947254e-22
> 40 z different 8.47032947254e-22
> 41 z different 8.47032947254e-22
> 45 z different 8.47032947254e-22
> 46 z different 8.47032947254e-22
> 50 z different 8.47032947254e-22
> 52 z different 8.47032947254e-22
> 53 z different 8.47032947254e-22
> 55 z different 8.47032947254e-22
> 56 z different 8.47032947254e-22
> 58 z different 8.47032947254e-22
> 66 z different 8.47032947254e-22
> 67 z different 8.47032947254e-22
> 71 z different 8.47032947254e-22
> 73 z different 8.47032947254e-22
> 74 z different 8.47032947254e-22
> 83 z different 8.47032947254e-22
> 86 z different 8.47032947254e-22
> 87 z different 8.47032947254e-22
> 88 z different 8.47032947254e-22
> 89 z different 8.47032947254e-22
> 90 z different 8.47032947254e-22
> 92 z different 8.47032947254e-22
>
> test1:  0  differences
> test2:  38  differences
> test3:  0  differences
>
> Repeated runs tend to give me the same number of differences in test2
> for several runs. Then I get a new number of differences which last
> for severals runs...

I built a new computer: Core 2 Duo 32-bit Debian etch with numpy
1.0.2.dev3546. The repeatability test still fails. In order to make my
calculations repeatable I'll have to remove ATLAS. That really slows
things down.

Does anyone with Debian not have this problem?


From kwgoodman at gmail.com  Thu Feb 15 23:47:41 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 15 Feb 2007 20:47:41 -0800
Subject: [Numpy-discussion] Different results from repeated calculation
In-Reply-To: <f4f93d420702152035y746ec4cen2882d25e977cf545@mail.gmail.com>
References: <f4f93d420701271225l18260e45x3ccc93f47ba982f0@mail.gmail.com>
	<f4f93d420702152035y746ec4cen2882d25e977cf545@mail.gmail.com>
Message-ID: <f4f93d420702152047q269faf24l98df3d137e83587c@mail.gmail.com>

On 2/15/07, Keith Goodman <kwgoodman at gmail.com> wrote:
> I built a new computer: Core 2 Duo 32-bit Debian etch with numpy
> 1.0.2.dev3546. The repeatability test still fails. In order to make my
> calculations repeatable I'll have to remove ATLAS. That really slows
> things down.

Hey, I have no problem with atlas-base and atlas-sse! On my old debian
box all versions of atlas fail the repeatability test.


From haase at msg.ucsf.edu  Fri Feb 16 00:40:43 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Thu, 15 Feb 2007 21:40:43 -0800
Subject: [Numpy-discussion] Different results from repeated calculation
In-Reply-To: <f4f93d420702152047q269faf24l98df3d137e83587c@mail.gmail.com>
References: <f4f93d420701271225l18260e45x3ccc93f47ba982f0@mail.gmail.com>
	<f4f93d420702152035y746ec4cen2882d25e977cf545@mail.gmail.com>
	<f4f93d420702152047q269faf24l98df3d137e83587c@mail.gmail.com>
Message-ID: <bc657ead0702152140w45c451k587b62007c196db6@mail.gmail.com>

On 2/15/07, Keith Goodman <kwgoodman at gmail.com> wrote:
> On 2/15/07, Keith Goodman <kwgoodman at gmail.com> wrote:
> > I built a new computer: Core 2 Duo 32-bit Debian etch with numpy
> > 1.0.2.dev3546. The repeatability test still fails. In order to make my
> > calculations repeatable I'll have to remove ATLAS. That really slows
> > things down.
>
> Hey, I have no problem with atlas-base and atlas-sse! On my old debian
> box all versions of atlas fail the repeatability test.

You mean on the Core 2 Dua 32-bit only atlas-sse2 causes troubles ?
How does the speed compare atlas-sse2 vs. atlas-see (ignoring the
repeatablity problem)?
-Sebastian Haase


From cookedm at physics.mcmaster.ca  Fri Feb 16 03:03:56 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Fri, 16 Feb 2007 03:03:56 -0500
Subject: [Numpy-discussion] SVN Build, optimized libraries, site.cfg,
	windows
In-Reply-To: <9135A452397C4096AD412BD0A64C8553@rleendurance>
References: <9135A452397C4096AD412BD0A64C8553@rleendurance>
Message-ID: <19C77FCA-CFC0-4E54-8FDA-348918833655@physics.mcmaster.ca>

On Feb 11, 2007, at 22:51 , Satrajit Ghosh wrote:

> Hi,
>
> I'm also not quite clear whether the optimized FFTW and UMFPACK  
> libraries
> are being used or required in numpy at all as show_config() doesn't  
> report
> it.
>
> I see that fftw and umfpack are being used for scipy.
>
> I have attached my site.cfg. Any help would be much appreciated.

No, they're only in there for scipy (and for other packages that  
would like to use them). They're not required for Numpy.

-- 
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From joris at ster.kuleuven.ac.be  Fri Feb 16 04:43:20 2007
From: joris at ster.kuleuven.ac.be (joris at ster.kuleuven.ac.be)
Date: Fri, 16 Feb 2007 10:43:20 +0100
Subject: [Numpy-discussion] problem installing numpy on OS X
Message-ID: <1171619000.45d57cb8b8419@webmail.ster.kuleuven.be>

Hi,

I'm trying to install numpy 1.0.2 (trunk) on my intel mac with Tiger 10.4.8,
but I experience some problems, which I think may be compiler problems. From
the
beginning:

I installed MacPython 2.5:
Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) 
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

I installed the complete Apple developer tools from my retail CD which give me
gcc-3.3 and gcc-4.0. I used gcc_select to ensure that gcc-4.0 is the default
compiler, although this appeared unnecessary, as it already was. However, there
was no 'gcc' in /usr/bin/ so I made a link from gcc to gcc-4.0. 

In 
/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/
I downloaded the latest numpy from the trunk with svn. "python setup.py
install"
runs (with errors though, see later), and leads to the following error when
trying to import numpy:
~>python
Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) 
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy import *
Running from numpy source directory.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/__init__.py",
line 43, in <module>
    import ctypeslib
  File
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/ctypeslib.py",
line 5, in <module>
    from numpy import integer, ndarray, dtype as _dtype, deprecate
ImportError: cannot import name integer


The errors I was talking about seem to relate to the compiler, and are listed
below. Does anyone have a clue how I can resolve them?

Cheers,
Joris





compile options: '-Inumpy/core/src -Inumpy/core/include
-I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c'
gcc: _configtest.c
_configtest.c: In function 'main':
_configtest.c:4: error: 'isnan' undeclared (first use in this function)
_configtest.c:4: error: (Each undeclared identifier is reported only once
_configtest.c:4: error: for each function it appears in.)
_configtest.c: In function 'main':
_configtest.c:4: error: 'isnan' undeclared (first use in this function)
_configtest.c:4: error: (Each undeclared identifier is reported only once
_configtest.c:4: error: for each function it appears in.)
lipo: can't figure out the architecture type of: /var/tmp//ccIuW81W.out
_configtest.c: In function 'main':
_configtest.c:4: error: 'isnan' undeclared (first use in this function)
_configtest.c:4: error: (Each undeclared identifier is reported only once
_configtest.c:4: error: for each function it appears in.)
_configtest.c: In function 'main':
_configtest.c:4: error: 'isnan' undeclared (first use in this function)
_configtest.c:4: error: (Each undeclared identifier is reported only once
_configtest.c:4: error: for each function it appears in.)
lipo: can't figure out the architecture type of: /var/tmp//ccIuW81W.out
failure.
removing: _configtest.c _configtest.o
C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk
-fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd
-fno-common -dynamic -DNDEBUG -g -O3

compile options: '-Inumpy/core/src -Inumpy/core/include
-I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c'
gcc: _configtest.c
_configtest.c: In function 'main':
_configtest.c:4: error: 'isinf' undeclared (first use in this function)
_configtest.c:4: error: (Each undeclared identifier is reported only once
_configtest.c:4: error: for each function it appears in.)
_configtest.c: In function 'main':
_configtest.c:4: error: 'isinf' undeclared (first use in this function)
_configtest.c:4: error: (Each undeclared identifier is reported only once
_configtest.c:4: error: for each function it appears in.)
lipo: can't figure out the architecture type of: /var/tmp//ccbLrVkU.out
_configtest.c: In function 'main':
_configtest.c:4: error: 'isinf' undeclared (first use in this function)
_configtest.c:4: error: (Each undeclared identifier is reported only once
_configtest.c:4: error: for each function it appears in.)
_configtest.c: In function 'main':
_configtest.c:4: error: 'isinf' undeclared (first use in this function)
_configtest.c:4: error: (Each undeclared identifier is reported only once
_configtest.c:4: error: for each function it appears in.)
lipo: can't figure out the architecture type of: /var/tmp//ccbLrVkU.out
failure.
removing: _configtest.c _configtest.o
C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk
-fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd
-fno-common -dynamic -DNDEBUG -g -O3

compile options: '-Inumpy/core/src -Inumpy/core/include
-I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c'
gcc: _configtest.c
_configtest.c:7:2: error: #error No _WIN32
_configtest.c:7:2: error: #error No _WIN32
lipo: can't figure out the architecture type of: /var/tmp//cc10iUsH.out
_configtest.c:7:2: error: #error No _WIN32
_configtest.c:7:2: error: #error No _WIN32
lipo: can't figure out the architecture type of: /var/tmp//cc10iUsH.out
failure.


Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm



From kwgoodman at gmail.com  Fri Feb 16 09:32:01 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Fri, 16 Feb 2007 06:32:01 -0800
Subject: [Numpy-discussion] Different results from repeated calculation
In-Reply-To: <bc657ead0702152140w45c451k587b62007c196db6@mail.gmail.com>
References: <f4f93d420701271225l18260e45x3ccc93f47ba982f0@mail.gmail.com>
	<f4f93d420702152035y746ec4cen2882d25e977cf545@mail.gmail.com>
	<f4f93d420702152047q269faf24l98df3d137e83587c@mail.gmail.com>
	<bc657ead0702152140w45c451k587b62007c196db6@mail.gmail.com>
Message-ID: <f4f93d420702160632h2df65264x9be518eb35a4e0c4@mail.gmail.com>

On 2/15/07, Sebastian Haase <haase at msg.ucsf.edu> wrote:
> On 2/15/07, Keith Goodman <kwgoodman at gmail.com> wrote:
> > On 2/15/07, Keith Goodman <kwgoodman at gmail.com> wrote:
> > > I built a new computer: Core 2 Duo 32-bit Debian etch with numpy
> > > 1.0.2.dev3546. The repeatability test still fails. In order to make my
> > > calculations repeatable I'll have to remove ATLAS. That really slows
> > > things down.
> >
> > Hey, I have no problem with atlas-base and atlas-sse! On my old debian
> > box all versions of atlas fail the repeatability test.
>
> You mean on the Core 2 Dua 32-bit only atlas-sse2 causes troubles ?
> How does the speed compare atlas-sse2 vs. atlas-see (ignoring the
> repeatablity problem)?

Yes. On my old computer (P4) all three (atlas-base, -sse, -sse2) cause
problems. On my new computer only sse2 causes a problem.

I only want to know about the speed difference (sse, sse2) if the
difference is small.


From nadavh at visionsense.com  Fri Feb 16 09:52:28 2007
From: nadavh at visionsense.com (Nadav Horesh)
Date: Fri, 16 Feb 2007 16:52:28 +0200
Subject: [Numpy-discussion] Numpy and iterative procedures
Message-ID: <07C6A61102C94148B8104D42DE95F7E8C8F1C6@exchange2k.envision.co.il>

At first glance it doesn't look hard to, at least, avoid looping over i, by replacing [i] by [:-2], [i+1] by [1:-1] and [i+2] by [2:]. But I might be wrong. Can you submit the piece of code with at least the most internal loop?

   Nadav.

-----Original Message-----
From:	numpy-discussion-bounces at scipy.org on behalf of Geoffrey Zhu
Sent:	Thu 15-Feb-07 18:32
To:	Discussion of Numerical Python
Cc:	
Subject:	Re: [Numpy-discussion] Numpy and iterative procedures

Thanks Chuck.
 
I am trying to use Successive Over-relaxation to solve linear equations
defined by M*v=q. 
 
There are several goals:
 
1. Eventually (in production) I need it to be fast.
2. I am playing with the guts of the algorithm for now, to see how it
works. that means i need some control for now.
3. Even in production, there is a chance i'd like to have the ability to
tinker with the algorithm. 
 

  _____  

From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Charles R
Harris
Sent: Thursday, February 15, 2007 10:11 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Numpy and iterative procedures





On 2/15/07, Geoffrey Zhu <gzhu at peak6.com> wrote: 

	Hi,
	
	I am new to numpy. I'd like to know if it is possible to code
efficient
	iterative procedures with numpy.
	
	Specifically, I have the following problem.
	
	M is an N*N matrix. Q is a N*1 vector. V is an N*1 vector I am
trying to 
	find iteratively from the initial value V_0. The procedure is
simply to
	calculate
	
	V_{n+1}[i]=3D1/M[I,i]*(q[i]-
	(M[i,1]*v_{n+1}[1]+M[I,2]*v_{n+1}[2]+..+M[i,i-1]*v_{n+1}[i-1]) -
	(M[I,i+1]*v_{n}[i+1]+M[I,i+2]*v_{n}[i+2]+..+M[I,N]*v_{n}[N])) 
	
	I do not see that this is something that can esaily be
vectorized, is
	it?


I think it would be better if you stated what the actual problem is. Is
it a differential equation, for instance. That way we can determine what
the problem class is and what algorithms are available to solve it. 

Chuck




_______________________________________________________

The  information in this email or in any file attached
hereto is intended only for the personal and confiden-
tial  use  of  the individual or entity to which it is
addressed and may contain information that is  propri-
etary  and  confidential.  If you are not the intended
recipient of this message you are hereby notified that
any  review, dissemination, distribution or copying of
this message is strictly prohibited.  This  communica-
tion  is  for information purposes only and should not
be regarded as an offer to sell or as  a  solicitation
of an offer to buy any financial product. Email trans-
mission cannot be guaranteed to be  secure  or  error-
free. P6070214




-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 4150 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070216/8efeb511/attachment.bin>

From nwagner at iam.uni-stuttgart.de  Fri Feb 16 10:18:32 2007
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Fri, 16 Feb 2007 16:18:32 +0100
Subject: [Numpy-discussion] Numpy and iterative procedures
In-Reply-To: <07C6A61102C94148B8104D42DE95F7E8C8F1C6@exchange2k.envision.co.il>
References: <07C6A61102C94148B8104D42DE95F7E8C8F1C6@exchange2k.envision.co.il>
Message-ID: <45D5CB48.3080006@iam.uni-stuttgart.de>

Nadav Horesh wrote:
> At first glance it doesn't look hard to, at least, avoid looping over i, by replacing [i] by [:-2], [i+1] by [1:-1] and [i+2] by [2:]. But I might be wrong. Can you submit the piece of code with at least the most internal loop?
>
>    Nadav.
>   
I guess he is looking for an implementation of

http://en.wikipedia.org/wiki/Successive_over-relaxation

Nils


> -----Original Message-----
> From:	numpy-discussion-bounces at scipy.org on behalf of Geoffrey Zhu
> Sent:	Thu 15-Feb-07 18:32
> To:	Discussion of Numerical Python
> Cc:	
> Subject:	Re: [Numpy-discussion] Numpy and iterative procedures
>
> Thanks Chuck.
>  
> I am trying to use Successive Over-relaxation to solve linear equations
> defined by M*v=q. 
>  
> There are several goals:
>  
> 1. Eventually (in production) I need it to be fast.
> 2. I am playing with the guts of the algorithm for now, to see how it
> works. that means i need some control for now.
> 3. Even in production, there is a chance i'd like to have the ability to
> tinker with the algorithm. 
>  
>
>   _____  
>
> From: numpy-discussion-bounces at scipy.org
> [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Charles R
> Harris
> Sent: Thursday, February 15, 2007 10:11 AM
> To: Discussion of Numerical Python
> Subject: Re: [Numpy-discussion] Numpy and iterative procedures
>
>
>
>
>
> On 2/15/07, Geoffrey Zhu <gzhu at peak6.com> wrote: 
>
> 	Hi,
> 	
> 	I am new to numpy. I'd like to know if it is possible to code
> efficient
> 	iterative procedures with numpy.
> 	
> 	Specifically, I have the following problem.
> 	
> 	M is an N*N matrix. Q is a N*1 vector. V is an N*1 vector I am
> trying to 
> 	find iteratively from the initial value V_0. The procedure is
> simply to
> 	calculate
> 	
> 	V_{n+1}[i]=3D1/M[I,i]*(q[i]-
> 	(M[i,1]*v_{n+1}[1]+M[I,2]*v_{n+1}[2]+..+M[i,i-1]*v_{n+1}[i-1]) -
> 	(M[I,i+1]*v_{n}[i+1]+M[I,i+2]*v_{n}[i+2]+..+M[I,N]*v_{n}[N])) 
> 	
> 	I do not see that this is something that can esaily be
> vectorized, is
> 	it?
>
>
> I think it would be better if you stated what the actual problem is. Is
> it a differential equation, for instance. That way we can determine what
> the problem class is and what algorithms are available to solve it. 
>
> Chuck
>
>
>
>
> _______________________________________________________
>
> The  information in this email or in any file attached
> hereto is intended only for the personal and confiden-
> tial  use  of  the individual or entity to which it is
> addressed and may contain information that is  propri-
> etary  and  confidential.  If you are not the intended
> recipient of this message you are hereby notified that
> any  review, dissemination, distribution or copying of
> this message is strictly prohibited.  This  communica-
> tion  is  for information purposes only and should not
> be regarded as an offer to sell or as  a  solicitation
> of an offer to buy any financial product. Email trans-
> mission cannot be guaranteed to be  secure  or  error-
> free. P6070214
>
>
>
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   
 


From ellisonbg.net at gmail.com  Fri Feb 16 10:59:24 2007
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Fri, 16 Feb 2007 08:59:24 -0700
Subject: [Numpy-discussion] problem installing numpy on OS X
In-Reply-To: <1171619000.45d57cb8b8419@webmail.ster.kuleuven.be>
References: <1171619000.45d57cb8b8419@webmail.ster.kuleuven.be>
Message-ID: <6ce0ac130702160759v7c32b050h277404f74906dca1@mail.gmail.com>

Numpy should install fine on your system.  If there was no gcc in
/usr/bin, then something significant went wrong with your
DeveloperTools install.  I would do a full reinstall of that.  Also,
gcc 4.0.1 is the default so there is no reason to use gcc_select.

Where did you get your python.  I would use the 2.5 binary on the
python.org website.

Lastly, I would remove site-packages/numpy cd to your numpy source dir and do:

rm -rf build
python setup.py install

That should work.  We build numpy on this system regularly and have no
problems if gcc/python are setup properly.

Brian

On 2/16/07, joris at ster.kuleuven.ac.be <joris at ster.kuleuven.ac.be> wrote:
> Hi,
>
> I'm trying to install numpy 1.0.2 (trunk) on my intel mac with Tiger 10.4.8,
> but I experience some problems, which I think may be compiler problems. From
> the
> beginning:
>
> I installed MacPython 2.5:
> Python 2.5 (r25:51918, Sep 19 2006, 08:49:13)
> [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>
> I installed the complete Apple developer tools from my retail CD which give me
> gcc-3.3 and gcc-4.0. I used gcc_select to ensure that gcc-4.0 is the default
> compiler, although this appeared unnecessary, as it already was. However, there
> was no 'gcc' in /usr/bin/ so I made a link from gcc to gcc-4.0.
>
> In
> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/
> I downloaded the latest numpy from the trunk with svn. "python setup.py
> install"
> runs (with errors though, see later), and leads to the following error when
> trying to import numpy:
> ~>python
> Python 2.5 (r25:51918, Sep 19 2006, 08:49:13)
> [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from numpy import *
> Running from numpy source directory.
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File
> "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/__init__.py",
> line 43, in <module>
>     import ctypeslib
>   File
> "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/ctypeslib.py",
> line 5, in <module>
>     from numpy import integer, ndarray, dtype as _dtype, deprecate
> ImportError: cannot import name integer
>
>
> The errors I was talking about seem to relate to the compiler, and are listed
> below. Does anyone have a clue how I can resolve them?
>
> Cheers,
> Joris
>
>
>
>
>
> compile options: '-Inumpy/core/src -Inumpy/core/include
> -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c'
> gcc: _configtest.c
> _configtest.c: In function 'main':
> _configtest.c:4: error: 'isnan' undeclared (first use in this function)
> _configtest.c:4: error: (Each undeclared identifier is reported only once
> _configtest.c:4: error: for each function it appears in.)
> _configtest.c: In function 'main':
> _configtest.c:4: error: 'isnan' undeclared (first use in this function)
> _configtest.c:4: error: (Each undeclared identifier is reported only once
> _configtest.c:4: error: for each function it appears in.)
> lipo: can't figure out the architecture type of: /var/tmp//ccIuW81W.out
> _configtest.c: In function 'main':
> _configtest.c:4: error: 'isnan' undeclared (first use in this function)
> _configtest.c:4: error: (Each undeclared identifier is reported only once
> _configtest.c:4: error: for each function it appears in.)
> _configtest.c: In function 'main':
> _configtest.c:4: error: 'isnan' undeclared (first use in this function)
> _configtest.c:4: error: (Each undeclared identifier is reported only once
> _configtest.c:4: error: for each function it appears in.)
> lipo: can't figure out the architecture type of: /var/tmp//ccIuW81W.out
> failure.
> removing: _configtest.c _configtest.o
> C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk
> -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd
> -fno-common -dynamic -DNDEBUG -g -O3
>
> compile options: '-Inumpy/core/src -Inumpy/core/include
> -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c'
> gcc: _configtest.c
> _configtest.c: In function 'main':
> _configtest.c:4: error: 'isinf' undeclared (first use in this function)
> _configtest.c:4: error: (Each undeclared identifier is reported only once
> _configtest.c:4: error: for each function it appears in.)
> _configtest.c: In function 'main':
> _configtest.c:4: error: 'isinf' undeclared (first use in this function)
> _configtest.c:4: error: (Each undeclared identifier is reported only once
> _configtest.c:4: error: for each function it appears in.)
> lipo: can't figure out the architecture type of: /var/tmp//ccbLrVkU.out
> _configtest.c: In function 'main':
> _configtest.c:4: error: 'isinf' undeclared (first use in this function)
> _configtest.c:4: error: (Each undeclared identifier is reported only once
> _configtest.c:4: error: for each function it appears in.)
> _configtest.c: In function 'main':
> _configtest.c:4: error: 'isinf' undeclared (first use in this function)
> _configtest.c:4: error: (Each undeclared identifier is reported only once
> _configtest.c:4: error: for each function it appears in.)
> lipo: can't figure out the architecture type of: /var/tmp//ccbLrVkU.out
> failure.
> removing: _configtest.c _configtest.o
> C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk
> -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd
> -fno-common -dynamic -DNDEBUG -g -O3
>
> compile options: '-Inumpy/core/src -Inumpy/core/include
> -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c'
> gcc: _configtest.c
> _configtest.c:7:2: error: #error No _WIN32
> _configtest.c:7:2: error: #error No _WIN32
> lipo: can't figure out the architecture type of: /var/tmp//cc10iUsH.out
> _configtest.c:7:2: error: #error No _WIN32
> _configtest.c:7:2: error: #error No _WIN32
> lipo: can't figure out the architecture type of: /var/tmp//cc10iUsH.out
> failure.
>
>
> Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From gzhu at peak6.com  Fri Feb 16 11:34:50 2007
From: gzhu at peak6.com (gzhu at peak6.com)
Date: Fri, 16 Feb 2007 10:34:50 -0600
Subject: [Numpy-discussion] (no subject)
Message-ID: <1171643690.31219.1175011483@webmail.messagingengine.com>

Hi Nadav,

The code is attached at the end. There is probably still bugs in there
but it does not prevent me from showing the difficulity.

If you look at the inner loop below, you will see that vector v is
updated element by element. The new value of v[i] depends on the new
value of v[i-1] and the old value of v[i+1]. Updating an element
involves the new values of the already updated elements and the old
values of the rest of the elements that we have yet to update. This
makes vectorization difficult.


        for i in range(1,N-1):
            temp[i]=(1-w)*v[i]+w/D[i]*(q[i]-L[i-1]*v[i-1]-U[i]*v[i+1])
            err += (temp[i]-v[i])**2
            v[i]=temp[i]

Thanks,
Geoffrey

Complete code here;

def sor(v, L, D, U ,q, tol, w):
    '''solve M*v=q. return v.
    L, D, U are the sub-diagonal, diagonal, and super-diagonal of the
    matrix M.
    
    '''
    
    err=9999999
    N=D.shape[0] #number of elements
    temp=empty(N)
    while err> tol :
        err=0
        temp[0]=(1-w)*v[0]+w/D[0]*(q[0]-U[0]*v[1])
        err += (temp[0]-v[0])**2
        v[0]=temp[0]
        
        for i in range(1,N-1):
            temp[i]=(1-w)*v[i]+w/D[i]*(q[i]-L[i-1]*v[i-1]-U[i]*v[i+1])
            err += (temp[i]-v[i])**2
            v[i]=temp[i]
                
        temp[N-1]=(1-w)*v[N-1]+w/D[N-1]*(q[N-1]-L[N-2]*v[N-2])
        err += (temp[N-1]-v[N-1])**2
        v[N-1]=temp[N-1]
    return v
 

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Nadav Horesh
Sent: Friday, February 16, 2007 8:52 AM
To: Discussion of Numerical Python
Subject: RE: [Numpy-discussion] Numpy and iterative procedures

At first glance it doesn't look hard to, at least, avoid looping over i,
by replacing [i] by [:-2], [i+1] by [1:-1] and [i+2] by [2:]. But I
might be wrong. Can you submit the piece of code with at least the most
internal loop?

   Nadav.

-----Original Message-----
From:   numpy-discussion-bounces at scipy.org on behalf of Geoffrey Zhu
Sent:   Thu 15-Feb-07 18:32
To:     Discussion of Numerical Python
Cc:     
Subject:        Re: [Numpy-discussion] Numpy and iterative procedures

Thanks Chuck.
 
I am trying to use Successive Over-relaxation to solve linear equations
defined by M*v=q. 
 
There are several goals:
 
1. Eventually (in production) I need it to be fast.
2. I am playing with the guts of the algorithm for now, to see how it
works. that means i need some control for now.
3. Even in production, there is a chance i'd like to have the ability to
tinker with the algorithm. 
 

  _____  

From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Charles R
Harris
Sent: Thursday, February 15, 2007 10:11 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Numpy and iterative procedures





On 2/15/07, Geoffrey Zhu <gzhu at peak6.com> wrote: 

	Hi,
	
	I am new to numpy. I'd like to know if it is possible to code efficient
	iterative procedures with numpy.
	
	Specifically, I have the following problem.
	
	M is an N*N matrix. Q is a N*1 vector. V is an N*1 vector I am trying to 
	find iteratively from the initial value V_0. The procedure is simply to
	calculate
	
	V_{n+1}[i]=3D1/M[I,i]*(q[i]-
	(M[i,1]*v_{n+1}[1]+M[I,2]*v_{n+1}[2]+..+M[i,i-1]*v_{n+1}[i-1]) -
	(M[I,i+1]*v_{n}[i+1]+M[I,i+2]*v_{n}[i+2]+..+M[I,N]*v_{n}[N])) 
	
	I do not see that this is something that can esaily be vectorized, is
	it?


I think it would be better if you stated what the actual problem is. Is
it a differential equation, for instance. That way we can determine what
the problem class is and what algorithms are available to solve it. 

Chuck





From gzhu at peak6.com  Fri Feb 16 11:38:06 2007
From: gzhu at peak6.com (Geoffrey Zhu)
Date: Fri, 16 Feb 2007 10:38:06 -0600
Subject: [Numpy-discussion] Numpy and iterative procedures
Message-ID: <99F81FFD0EA54E4DA8D4F1BFE272F34103827C00@ppi-mail1.chicago.peak6.net>

In fact I am not looking for an implementation but a method to update a
vector iteratively when the value of an item of the vector depend partly
on the already updated items and partly depend on the old items. 

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Nils Wagner
Sent: Friday, February 16, 2007 9:19 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Numpy and iterative procedures

Nadav Horesh wrote:
> At first glance it doesn't look hard to, at least, avoid looping over
i, by replacing [i] by [:-2], [i+1] by [1:-1] and [i+2] by [2:]. But I
might be wrong. Can you submit the piece of code with at least the most
internal loop?
>
>    Nadav.
>   
I guess he is looking for an implementation of

http://en.wikipedia.org/wiki/Successive_over-relaxation

Nils


> -----Original Message-----
> From:	numpy-discussion-bounces at scipy.org on behalf of Geoffrey Zhu
> Sent:	Thu 15-Feb-07 18:32
> To:	Discussion of Numerical Python
> Cc:	
> Subject:	Re: [Numpy-discussion] Numpy and iterative procedures
>
> Thanks Chuck.
>  
> I am trying to use Successive Over-relaxation to solve linear 
> equations defined by M*v=q.
>  
> There are several goals:
>  
> 1. Eventually (in production) I need it to be fast.
> 2. I am playing with the guts of the algorithm for now, to see how it 
> works. that means i need some control for now.
> 3. Even in production, there is a chance i'd like to have the ability 
> to tinker with the algorithm.
>  
>
>   _____
>
> From: numpy-discussion-bounces at scipy.org
> [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Charles R 
> Harris
> Sent: Thursday, February 15, 2007 10:11 AM
> To: Discussion of Numerical Python
> Subject: Re: [Numpy-discussion] Numpy and iterative procedures
>
>
>
>
>
> On 2/15/07, Geoffrey Zhu <gzhu at peak6.com> wrote: 
>
> 	Hi,
> 	
> 	I am new to numpy. I'd like to know if it is possible to code 
> efficient
> 	iterative procedures with numpy.
> 	
> 	Specifically, I have the following problem.
> 	
> 	M is an N*N matrix. Q is a N*1 vector. V is an N*1 vector I am
trying 
> to
> 	find iteratively from the initial value V_0. The procedure is
simply 
> to
> 	calculate
> 	
> 	V_{n+1}[i]=3D1/M[I,i]*(q[i]-
> 	(M[i,1]*v_{n+1}[1]+M[I,2]*v_{n+1}[2]+..+M[i,i-1]*v_{n+1}[i-1]) -
> 	(M[I,i+1]*v_{n}[i+1]+M[I,i+2]*v_{n}[i+2]+..+M[I,N]*v_{n}[N]))
> 	
> 	I do not see that this is something that can esaily be
vectorized, is
> 	it?
>
>
> I think it would be better if you stated what the actual problem is.
Is
> it a differential equation, for instance. That way we can determine
what
> the problem class is and what algorithms are available to solve it. 
>
> Chuck
>
>
>
>
> _______________________________________________________
>
> The  information in this email or in any file attached
> hereto is intended only for the personal and confiden-
> tial  use  of  the individual or entity to which it is
> addressed and may contain information that is  propri-
> etary  and  confidential.  If you are not the intended
> recipient of this message you are hereby notified that
> any  review, dissemination, distribution or copying of
> this message is strictly prohibited.  This  communica-
> tion  is  for information purposes only and should not
> be regarded as an offer to sell or as  a  solicitation
> of an offer to buy any financial product. Email trans-
> mission cannot be guaranteed to be  secure  or  error-
> free. P6070214
>
>
>
>
>   
>
------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   
 
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion




From gzhu at peak6.com  Fri Feb 16 11:39:04 2007
From: gzhu at peak6.com (Geoffrey Zhu)
Date: Fri, 16 Feb 2007 10:39:04 -0600
Subject: [Numpy-discussion] Numpy and iterative procedures
Message-ID: <99F81FFD0EA54E4DA8D4F1BFE272F34103827C03@ppi-mail1.chicago.peak6.net>

Hi Nadav,

The code is attached at the end. There is probably still bugs in there
but it does not prevent me from showing the difficulity.

If you look at the inner loop below, you will see that vector v is
updated element by element. The new value of v[i] depends on the new
value of v[i-1] and the old value of v[i+1]. Updating an element
involves the new values of the already updated elements and the old
values of the rest of the elements that we have yet to update. This
makes vectorization difficult.


        for i in range(1,N-1):
            temp[i]=3D(1-w)*v[i]+w/D[i]*(q[i]-L[i-1]*v[i-1]-U[i]*v[i+1])
            err +=3D (temp[i]-v[i])**2
            v[i]=3Dtemp[i]

Thanks,
Geoffrey

Complete code here;

def sor(v, L, D, U ,q, tol, w):
    '''solve M*v=3Dq. return v.
    L, D, U are the sub-diagonal, diagonal, and super-diagonal of the
    matrix M.
    
    '''
    
    err=3D9999999
    N=3DD.shape[0] #number of elements
    temp=3Dempty(N)
    while err> tol :
        err=3D0
        temp[0]=3D(1-w)*v[0]+w/D[0]*(q[0]-U[0]*v[1])
        err +=3D (temp[0]-v[0])**2
        v[0]=3Dtemp[0]
        
        for i in range(1,N-1):
            temp[i]=3D(1-w)*v[i]+w/D[i]*(q[i]-L[i-1]*v[i-1]-U[i]*v[i+1])
            err +=3D (temp[i]-v[i])**2
            v[i]=3Dtemp[i]
                
        temp[N-1]=3D(1-w)*v[N-1]+w/D[N-1]*(q[N-1]-L[N-2]*v[N-2])
        err +=3D (temp[N-1]-v[N-1])**2
        v[N-1]=3Dtemp[N-1]
    return v
 

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Nadav Horesh
Sent: Friday, February 16, 2007 8:52 AM
To: Discussion of Numerical Python
Subject: RE: [Numpy-discussion] Numpy and iterative procedures

At first glance it doesn't look hard to, at least, avoid looping over i,
by replacing [i] by [:-2], [i+1] by [1:-1] and [i+2] by [2:]. But I
might be wrong. Can you submit the piece of code with at least the most
internal loop?

   Nadav.

-----Original Message-----
From:   numpy-discussion-bounces at scipy.org on behalf of Geoffrey Zhu
Sent:   Thu 15-Feb-07 18:32
To:     Discussion of Numerical Python
Cc:     
Subject:        Re: [Numpy-discussion] Numpy and iterative procedures

Thanks Chuck.
 
I am trying to use Successive Over-relaxation to solve linear equations
defined by M*v=3Dq. 
 
There are several goals:
 
1. Eventually (in production) I need it to be fast.
2. I am playing with the guts of the algorithm for now, to see how it
works. that means i need some control for now.
3. Even in production, there is a chance i'd like to have the ability to
tinker with the algorithm. 
 

  _____  

From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Charles R
Harris
Sent: Thursday, February 15, 2007 10:11 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Numpy and iterative procedures





On 2/15/07, Geoffrey Zhu <gzhu at peak6.com> wrote: 

	Hi,
	
	I am new to numpy. I'd like to know if it is possible to code efficient
	iterative procedures with numpy.
	
	Specifically, I have the following problem.
	
	M is an N*N matrix. Q is a N*1 vector. V is an N*1 vector I am trying to 
	find iteratively from the initial value V_0. The procedure is simply to
	calculate
	
	V_{n+1}[i]=3D3D1/M[I,i]*(q[i]-
	(M[i,1]*v_{n+1}[1]+M[I,2]*v_{n+1}[2]+..+M[i,i-1]*v_{n+1}[i-1]) -
	(M[I,i+1]*v_{n}[i+1]+M[I,i+2]*v_{n}[i+2]+..+M[I,N]*v_{n}[N])) 
	
	I do not see that this is something that can esaily be vectorized, is
	it?


I think it would be better if you stated what the actual problem is. Is
it a differential equation, for instance. That way we can determine what
the problem class is and what algorithms are available to solve it. 

Chuck





_______________________________________________________=0A=
=0A=
The  information in this email or in any file attached=0A=
hereto is intended only for the personal and confiden-=0A=
tial  use  of  the individual or entity to which it is=0A=
addressed and may contain information that is  propri-=0A=
etary  and  confidential.  If you are not the intended=0A=
recipient of this message you are hereby notified that=0A=
any  review, dissemination, distribution or copying of=0A=
this message is strictly prohibited.  This  communica-=0A=
tion  is  for information purposes only and should not=0A=
be regarded as an offer to sell or as  a  solicitation=0A=
of an offer to buy any financial product. Email trans-=0A=
mission cannot be guaranteed to be  secure  or  error-=0A=
free. P6070214=0A=



From shuntim.luk at polyu.edu.hk  Fri Feb 16 00:13:23 2007
From: shuntim.luk at polyu.edu.hk (LUK ShunTim)
Date: Fri, 16 Feb 2007 13:13:23 +0800
Subject: [Numpy-discussion] Possible solution to
 binary	distributionproblems for numpy on linux?
In-Reply-To: <ef5675f30702151005q1d8046b5o1063f69ab8713aa0@mail.gmail.com>
References: <45CFF4D0.5080401@ar.media.kyoto-u.ac.jp>	<f4f93d420702120945k59cd887h2f68d27c8c051a0b@mail.gmail.com>	<45D118FC.7020403@ar.media.kyoto-u.ac.jp>	<45D14E1C.4010809@polyu.edu.hk>	<e06186140702150936sc27d1fx475412bfdc14e69c@mail.gmail.com>
	<ef5675f30702151005q1d8046b5o1063f69ab8713aa0@mail.gmail.com>
Message-ID: <45D53D73.5080206@polyu.edu.hk>

Keir Mierle wrote:
> On 2/15/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
>> On 2/12/07, LUK ShunTim <shuntim.luk at polyu.edu.hk> wrote:
>>> David Cournapeau wrote:
>>>> Keith Goodman wrote:
>>>>> On 2/11/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
>>>>>> My impression is that binary distribution of numpy is a big problem
>>>>>> for many linux users, and that is entry barrier for many users (I may
>>>>>> be wrong, that's just an impression from the ML).
>>>>>> Do all of the major GNU/Linux distributions have recent versions of
>>>>>> NumPy?
>>>>> Debian Etch is at NumPy 1.0.1
>>>> I think debian has numpy now (I am not using debian on workstation
>>>> anymore, so I am not really following), but what about new versions of
>>>> numpy/scipy ? If I want to give some of my code to people in my lab who
>>>> do not use the same distribution than me, can I give a 10 minutes
>>>> instructions set to have everything ?
>>>>
>>> "Bleeding-edge apt-get repository" according to their web page.
>>> "To use it add the following line to you /etc/apt/sources.list"
>>>
>>> deb http://deb-scipy.alioth.debian.org/apt ./
>> Yep. In general, if you are using a Linux distro it is quite easy to use the
>> svn repository. It seem to be on the Mac and Windows that folks have
>> problems, particularly in filling the dependencies on Atlas if they want
>> efficiency.
>>
>> That said, Numpy and MatplotLab have settled enough that the standard
>> packages are probably adequate.
> 
> While this is true for people admining their own machines, this is not the case
> for people without root access. For example, it is a major hassle to install
> numpy on the computers at my university for exactly this reason.
> 

Probably OT now, ...

Ya, I can understand your feeling very well and therefore, I administer
my own box. :-)

A possible workaround for Debian is debootstrap, which enables one to
build a chroot debian system inside an existing one. Some infromation here
	http://wiki.debian.org/Debootstrap

Regards,
ST
--


From tom.and at tiscalinet.it  Fri Feb 16 12:59:35 2007
From: tom.and at tiscalinet.it (Andrea Tomadin)
Date: Fri, 16 Feb 2007 18:59:35 +0100
Subject: [Numpy-discussion] Global Numpy vector with Swig
In-Reply-To: <A5AB0E0A-6FE0-4A17-8E98-B6DF512CDC03@sandia.gov>
References: <0A4BC30F-9080-43F7-899B-6F246F345216@tiscalinet.it>
	<A5AB0E0A-6FE0-4A17-8E98-B6DF512CDC03@sandia.gov>
Message-ID: <CF554E5F-1F80-40D1-9850-D6ADE4BCB418@tiscalinet.it>

Il giorno 15/feb/07, alle ore 22:26, Bill Spotz ha scritto:

> It seems to me you would need to %ignore vec, so that it is not  
> wrapped as a raw pointer to a double, and then in your interface  
> file create a PyArrayObject whose data buffer points to vec (the  
> most efficient way to do this is with %inline).  Then use %rename  
> to rename whatever you called your PyArrayObject to "vec" (or not,  
> if you do not care about the name).  Finally,
>
>   foo.cvar.vec = numpy.zeros(10)
>
> is going to decrement (and probably delete) your original vec and  
> replace it with the new array of zeros (I think,

Thank you Bill for your help, even if writing this code is not  
straightforward for me. Anyway, while I was reading the docs, I  
became aware of the varin, varout typemap options. I wonder if I can  
define a
typemap(varin) double *
declaration, and insert in it the same code that in "numpy.i"  
translates the PyArrayObject into a C array and vice versa.
But I cannot find an example of typemp(varin) for Python, so I don't  
know what to use for the output, instead of $1. Where are the typemaps 
(varin) for Python?
It seems strange that nobody wrote a public code for such a simple  
task, as I read that Swig supports Python since the 90s! I suspect I  
have just not looked in the right places?!

Thank you again for help!
Andrea



From alec at mihailovs.com  Fri Feb 16 14:37:52 2007
From: alec at mihailovs.com (Alec Mihailovs)
Date: Fri, 16 Feb 2007 19:37:52 +0000 (UTC)
Subject: [Numpy-discussion] =?utf-8?q?magic=5Fsquare?=
Message-ID: <loom.20070216T202417-720@post.gmane.org>

I saw somewhere a comparison between numpy and Matlab/Octave. One of the 
Matlab/Octave commands that is missing in Numpy is magic(n) for producing n-by-
n magic squares.

Yesterday I posted a magic_square module in the CheeseShop, containing this 
missing magic(n) command together with 2 others, ismagic(a) and magic_constant
(a), 

http://www.python.org/pypi/magic_square/0.1
http://mihailovs.com/Alec/Python/magic_square.html

I wouldn't object adding them directly to the numpy package. It's, probably, 
RTFM question, but I couldn't find it on the web and I didn't BTFM yet - what 
is the standard way of submitting a module for including in numpy?

Another question that I have is that I have a link to NumPy on my magic_square 
page cited above. This is a kind of advertising and I've read that NumPy has a 
BSD type license including the advertising claim. Do I have to get a written 
permission from somebody to have a link to NumPy download page on my page?

Alec Mihailovs 
http://mihailovs.com/Alec/ 





From joris at ster.kuleuven.ac.be  Fri Feb 16 14:56:02 2007
From: joris at ster.kuleuven.ac.be (joris at ster.kuleuven.ac.be)
Date: Fri, 16 Feb 2007 20:56:02 +0100
Subject: [Numpy-discussion] problem installing numpy on OSX
Message-ID: <1171655762.45d60c52e876f@webmail.ster.kuleuven.be>


Brian, thanks for your answer!

I removed and completely reinstalled the Developer Tools. The issue with gcc
seems to be resolved now. My python binary is indeed the one of python.org, but
nevertheless I also reinstalled Python 2.5.

To install numpy I started all over again with svn. The results are, however,
exactly the same. 

Then I tried a version lower: numpy-1.0.1  (instead of 1.0.2 of the trunk). The
same error messages during the setup.py procedure, but they don't seem to matter
this time: afterwards I can import numpy without any obvious problems.

I know that numpy 1.0.2 installs well on my Linux box, but there I use gcc-3.4.
instead of gcc-4.0. Might this be a compiler issue?

What version of numpy did you install, did you notice any warning or error
messages at all?

Joris



Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm



From oliphant at ee.byu.edu  Fri Feb 16 15:09:43 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri, 16 Feb 2007 13:09:43 -0700
Subject: [Numpy-discussion] magic_square
In-Reply-To: <loom.20070216T202417-720@post.gmane.org>
References: <loom.20070216T202417-720@post.gmane.org>
Message-ID: <45D60F87.6040205@ee.byu.edu>

Alec Mihailovs wrote:

>I saw somewhere a comparison between numpy and Matlab/Octave. One of the 
>Matlab/Octave commands that is missing in Numpy is magic(n) for producing n-by-
>n magic squares.
>
>Yesterday I posted a magic_square module in the CheeseShop, containing this 
>missing magic(n) command together with 2 others, ismagic(a) and magic_constant
>(a), 
>
>http://www.python.org/pypi/magic_square/0.1
>http://mihailovs.com/Alec/Python/magic_square.html
>
>I wouldn't object adding them directly to the numpy package. It's, probably, 
>RTFM question, but I couldn't find it on the web and I didn't BTFM yet - what 
>is the standard way of submitting a module for including in numpy?
>  
>
You can submit a patch as a ticket on the Trac pages.   The trend is to 
put this kind of stuff into SciPy instead of NumPy.   I think we are 
trying to trim NumPy in the long term.

>Another question that I have is that I have a link to NumPy on my magic_square 
>page cited above. This is a kind of advertising and I've read that NumPy has a 
>BSD type license including the advertising claim. Do I have to get a written 
>permission from somebody to have a link to NumPy download page on my page?
>
>  
>
I'm not sure what the "advertising claim" is.  The only "advertising" 
claim is that you can't use the name of the NumPy developers or 
contributors in "support" of your project.   You can use the "NumPy" 
name because your package depends on it.

As far as I'm concerned just placing a link to NumPy does not require 
written permission.

-Travis





From wfspotz at sandia.gov  Fri Feb 16 15:10:12 2007
From: wfspotz at sandia.gov (Bill Spotz)
Date: Fri, 16 Feb 2007 13:10:12 -0700
Subject: [Numpy-discussion] Global Numpy vector with Swig
In-Reply-To: <CF554E5F-1F80-40D1-9850-D6ADE4BCB418@tiscalinet.it>
References: <0A4BC30F-9080-43F7-899B-6F246F345216@tiscalinet.it>
	<A5AB0E0A-6FE0-4A17-8E98-B6DF512CDC03@sandia.gov>
	<CF554E5F-1F80-40D1-9850-D6ADE4BCB418@tiscalinet.it>
Message-ID: <8D052BAE-4853-4F71-AD27-FDDB609C8392@sandia.gov>

Andrea,

It is my understanding that swig typemaps only apply to function  
arguments.  Since what you are talking about is a global variable, I  
don't believe typemaps will help you.  I would try

%{
#include "header-that-contains-vec.h"
npy_intp vec_dims[ ] = { (npy_intp) length_of_vec };
%}
%rename(vec) numpy_vec;
%ignore vec;
%include "header-that-contains-vec.h"
%inline %{
PyObject * numpy_vec = PyArray_SimpleNewFromData(1, vec_dims,  
NPY_DOUBLE, (void*)vec);
%}

I guess it would be nice if you could force a "Py_DECREF(numpy_vec);"  
before the module is destroyed, but I'll leave that as an exercise...

On Feb 16, 2007, at 10:59 AM, Andrea Tomadin wrote:

> Il giorno 15/feb/07, alle ore 22:26, Bill Spotz ha scritto:
>
>> It seems to me you would need to %ignore vec, so that it is not
>> wrapped as a raw pointer to a double, and then in your interface
>> file create a PyArrayObject whose data buffer points to vec (the
>> most efficient way to do this is with %inline).  Then use %rename
>> to rename whatever you called your PyArrayObject to "vec" (or not,
>> if you do not care about the name).  Finally,
>>
>>   foo.cvar.vec = numpy.zeros(10)
>>
>> is going to decrement (and probably delete) your original vec and
>> replace it with the new array of zeros (I think,
>
> Thank you Bill for your help, even if writing this code is not
> straightforward for me. Anyway, while I was reading the docs, I
> became aware of the varin, varout typemap options. I wonder if I can
> define a
> typemap(varin) double *
> declaration, and insert in it the same code that in "numpy.i"
> translates the PyArrayObject into a C array and vice versa.
> But I cannot find an example of typemp(varin) for Python, so I don't
> know what to use for the output, instead of $1. Where are the typemaps
> (varin) for Python?
> It seems strange that nobody wrote a public code for such a simple
> task, as I read that Swig supports Python since the 90s! I suspect I
> have just not looked in the right places?!
>
> Thank you again for help!
> Andrea

** Bill Spotz                                              **
** Sandia National Laboratories  Voice: (505)845-0170      **
** P.O. Box 5800                 Fax:   (505)284-5451      **
** Albuquerque, NM 87185-0370    Email: wfspotz at sandia.gov **





From wfspotz at sandia.gov  Fri Feb 16 15:18:07 2007
From: wfspotz at sandia.gov (Bill Spotz)
Date: Fri, 16 Feb 2007 13:18:07 -0700
Subject: [Numpy-discussion] Global Numpy vector with Swig
In-Reply-To: <8D052BAE-4853-4F71-AD27-FDDB609C8392@sandia.gov>
References: <0A4BC30F-9080-43F7-899B-6F246F345216@tiscalinet.it>
	<A5AB0E0A-6FE0-4A17-8E98-B6DF512CDC03@sandia.gov>
	<CF554E5F-1F80-40D1-9850-D6ADE4BCB418@tiscalinet.it>
	<8D052BAE-4853-4F71-AD27-FDDB609C8392@sandia.gov>
Message-ID: <EFAED3C3-7174-4393-95A1-C73F23F22F35@sandia.gov>

OK, I looked at the varin, varout descriptions in the online manual,  
and they specifically mention global variables, but WOW is that  
documentation minimal.  I would suggest asking Swig- 
user at lists.sourceforge.net for some assistance.

On Feb 16, 2007, at 1:10 PM, Bill Spotz wrote:

> Andrea,
>
> It is my understanding that swig typemaps only apply to function  
> arguments.  Since what you are talking about is a global variable,  
> I don't believe typemaps will help you.  I would try
>
> %{
> #include "header-that-contains-vec.h"
> npy_intp vec_dims[ ] = { (npy_intp) length_of_vec };
> %}
> %rename(vec) numpy_vec;
> %ignore vec;
> %include "header-that-contains-vec.h"
> %inline %{
> PyObject * numpy_vec = PyArray_SimpleNewFromData(1, vec_dims,  
> NPY_DOUBLE, (void*)vec);
> %}
>
> I guess it would be nice if you could force a "Py_DECREF 
> (numpy_vec);" before the module is destroyed, but I'll leave that  
> as an exercise...
>
> On Feb 16, 2007, at 10:59 AM, Andrea Tomadin wrote:
>
>> Il giorno 15/feb/07, alle ore 22:26, Bill Spotz ha scritto:
>>
>>> It seems to me you would need to %ignore vec, so that it is not
>>> wrapped as a raw pointer to a double, and then in your interface
>>> file create a PyArrayObject whose data buffer points to vec (the
>>> most efficient way to do this is with %inline).  Then use %rename
>>> to rename whatever you called your PyArrayObject to "vec" (or not,
>>> if you do not care about the name).  Finally,
>>>
>>>   foo.cvar.vec = numpy.zeros(10)
>>>
>>> is going to decrement (and probably delete) your original vec and
>>> replace it with the new array of zeros (I think,
>>
>> Thank you Bill for your help, even if writing this code is not
>> straightforward for me. Anyway, while I was reading the docs, I
>> became aware of the varin, varout typemap options. I wonder if I can
>> define a
>> typemap(varin) double *
>> declaration, and insert in it the same code that in "numpy.i"
>> translates the PyArrayObject into a C array and vice versa.
>> But I cannot find an example of typemp(varin) for Python, so I don't
>> know what to use for the output, instead of $1. Where are the  
>> typemaps
>> (varin) for Python?
>> It seems strange that nobody wrote a public code for such a simple
>> task, as I read that Swig supports Python since the 90s! I suspect I
>> have just not looked in the right places?!
>>
>> Thank you again for help!
>> Andrea
>
> ** Bill Spotz                                              **
> ** Sandia National Laboratories  Voice: (505)845-0170      **
> ** P.O. Box 5800                 Fax:   (505)284-5451      **
> ** Albuquerque, NM 87185-0370    Email: wfspotz at sandia.gov **
>
>

** Bill Spotz                                              **
** Sandia National Laboratories  Voice: (505)845-0170      **
** P.O. Box 5800                 Fax:   (505)284-5451      **
** Albuquerque, NM 87185-0370    Email: wfspotz at sandia.gov **





From seb.haase at gmx.net  Fri Feb 16 16:51:17 2007
From: seb.haase at gmx.net (Sebastian Haase)
Date: Fri, 16 Feb 2007 13:51:17 -0800
Subject: [Numpy-discussion] Different results from repeated calculation
In-Reply-To: <f4f93d420702160632h2df65264x9be518eb35a4e0c4@mail.gmail.com>
References: <f4f93d420701271225l18260e45x3ccc93f47ba982f0@mail.gmail.com>
	<f4f93d420702152035y746ec4cen2882d25e977cf545@mail.gmail.com>
	<f4f93d420702152047q269faf24l98df3d137e83587c@mail.gmail.com>
	<bc657ead0702152140w45c451k587b62007c196db6@mail.gmail.com>
	<f4f93d420702160632h2df65264x9be518eb35a4e0c4@mail.gmail.com>
Message-ID: <bc657ead0702161351i343b803dnaf6904f67168d9b7@mail.gmail.com>

On 2/16/07, Keith Goodman <kwgoodman at gmail.com> wrote:
> On 2/15/07, Sebastian Haase <haase at msg.ucsf.edu> wrote:
> > On 2/15/07, Keith Goodman <kwgoodman at gmail.com> wrote:
> > > On 2/15/07, Keith Goodman <kwgoodman at gmail.com> wrote:
> > > > I built a new computer: Core 2 Duo 32-bit Debian etch with numpy
> > > > 1.0.2.dev3546. The repeatability test still fails. In order to make my
> > > > calculations repeatable I'll have to remove ATLAS. That really slows
> > > > things down.
> > >
> > > Hey, I have no problem with atlas-base and atlas-sse! On my old debian
> > > box all versions of atlas fail the repeatability test.
> >
> > You mean on the Core 2 Dua 32-bit only atlas-sse2 causes troubles ?
> > How does the speed compare atlas-sse2 vs. atlas-see (ignoring the
> > repeatablity problem)?
>
> Yes. On my old computer (P4) all three (atlas-base, -sse, -sse2) cause
> problems. On my new computer only sse2 causes a problem.
>
> I only want to know about the speed difference (sse, sse2) if the
> difference is small.

I was just wondering what generally the speed improvement from sse to sse2 is ?
Any tentative number would be fine...
-S.


From alec at mihailovs.com  Fri Feb 16 20:23:35 2007
From: alec at mihailovs.com (Alec Mihailovs)
Date: Sat, 17 Feb 2007 01:23:35 +0000 (UTC)
Subject: [Numpy-discussion] =?utf-8?q?magic=5Fsquare?=
References: <loom.20070216T202417-720@post.gmane.org>
	<45D60F87.6040205@ee.byu.edu>
Message-ID: <loom.20070217T021856-420@post.gmane.org>

Travis Oliphant <oliphant <at> ee.byu.edu> writes:  

> You can submit a patch as a ticket on the Trac pages.   The trend is to 
> put this kind of stuff into SciPy instead of NumPy.   I think we are 
> trying to trim NumPy in the long term.

I'll look into this. 

> As far as I'm concerned just placing a link to NumPy does not require 
> written permission.

Glad to hear that.

Thank you, 
Alec Mihailovs 
http://mihailovs.com/Alec/ 





From nadavh at visionsense.com  Sat Feb 17 02:14:58 2007
From: nadavh at visionsense.com (Nadav Horesh)
Date: Sat, 17 Feb 2007 09:14:58 +0200
Subject: [Numpy-discussion] (no subject)
Message-ID: <07C6A61102C94148B8104D42DE95F7E8C8F1C9@exchange2k.envision.co.il>


It looks like you encountered a fundamental short come of numpy (or in fact any similar system like octave, matlab etc...): The dependence on values calculated in previous iteration can not vectorize easily. If you have an access to C compiler, I urge you to write (at least) the inner loop with pyrex  or a similar package that can easily link C code with numpy. It would help you a lot to realize these kind of algorithms with a reasonable execution time without loosing much of the python's benefits.

  Nadav


-----Original Message-----
From:	numpy-discussion-bounces at scipy.org on behalf of gzhu at peak6.com
Sent:	Fri 16-Feb-07 18:34
To:	numpy-discussion at scipy.org
Cc:	
Subject:	[Numpy-discussion] (no subject)

Hi Nadav,

The code is attached at the end. There is probably still bugs in there
but it does not prevent me from showing the difficulty.

If you look at the inner loop below, you will see that vector v is
updated element by element. The new value of v[i] depends on the new
value of v[i-1] and the old value of v[i+1]. Updating an element
involves the new values of the already updated elements and the old
values of the rest of the elements that we have yet to update. This
makes vectorization difficult.


        for i in range(1,N-1):
            temp[i]=(1-w)*v[i]+w/D[i]*(q[i]-L[i-1]*v[i-1]-U[i]*v[i+1])
            err += (temp[i]-v[i])**2
            v[i]=temp[i]

Thanks,
Geoffrey

Complete code here;

def sor(v, L, D, U ,q, tol, w):
    '''solve M*v=q. return v.
    L, D, U are the sub-diagonal, diagonal, and super-diagonal of the
    matrix M.
    
    '''
    
    err=9999999
    N=D.shape[0] #number of elements
    temp=empty(N)
    while err> tol :
        err=0
        temp[0]=(1-w)*v[0]+w/D[0]*(q[0]-U[0]*v[1])
        err += (temp[0]-v[0])**2
        v[0]=temp[0]
        
        for i in range(1,N-1):
            temp[i]=(1-w)*v[i]+w/D[i]*(q[i]-L[i-1]*v[i-1]-U[i]*v[i+1])
            err += (temp[i]-v[i])**2
            v[i]=temp[i]
                
        temp[N-1]=(1-w)*v[N-1]+w/D[N-1]*(q[N-1]-L[N-2]*v[N-2])
        err += (temp[N-1]-v[N-1])**2
        v[N-1]=temp[N-1]
    return v
 

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Nadav Horesh
Sent: Friday, February 16, 2007 8:52 AM
To: Discussion of Numerical Python
Subject: RE: [Numpy-discussion] Numpy and iterative procedures

At first glance it doesn't look hard to, at least, avoid looping over i,
by replacing [i] by [:-2], [i+1] by [1:-1] and [i+2] by [2:]. But I
might be wrong. Can you submit the piece of code with at least the most
internal loop?

   Nadav.

-----Original Message-----
From:   numpy-discussion-bounces at scipy.org on behalf of Geoffrey Zhu
Sent:   Thu 15-Feb-07 18:32
To:     Discussion of Numerical Python
Cc:     
Subject:        Re: [Numpy-discussion] Numpy and iterative procedures

Thanks Chuck.
 
I am trying to use Successive Over-relaxation to solve linear equations
defined by M*v=q. 
 
There are several goals:
 
1. Eventually (in production) I need it to be fast.
2. I am playing with the guts of the algorithm for now, to see how it
works. that means i need some control for now.
3. Even in production, there is a chance i'd like to have the ability to
tinker with the algorithm. 
 

  _____  

From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Charles R
Harris
Sent: Thursday, February 15, 2007 10:11 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Numpy and iterative procedures





On 2/15/07, Geoffrey Zhu <gzhu at peak6.com> wrote: 

	Hi,
	
	I am new to numpy. I'd like to know if it is possible to code efficient
	iterative procedures with numpy.
	
	Specifically, I have the following problem.
	
	M is an N*N matrix. Q is a N*1 vector. V is an N*1 vector I am trying to 
	find iteratively from the initial value V_0. The procedure is simply to
	calculate
	
	V_{n+1}[i]=3D1/M[I,i]*(q[i]-
	(M[i,1]*v_{n+1}[1]+M[I,2]*v_{n+1}[2]+..+M[i,i-1]*v_{n+1}[i-1]) -
	(M[I,i+1]*v_{n}[i+1]+M[I,i+2]*v_{n}[i+2]+..+M[I,N]*v_{n}[N])) 
	
	I do not see that this is something that can esaily be vectorized, is
	it?


I think it would be better if you stated what the actual problem is. Is
it a differential equation, for instance. That way we can determine what
the problem class is and what algorithms are available to solve it. 

Chuck



_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion





From barrywark at gmail.com  Sun Feb 18 02:41:14 2007
From: barrywark at gmail.com (Barry Wark)
Date: Sat, 17 Feb 2007 23:41:14 -0800
Subject: [Numpy-discussion] [Matplotlib-users] [matplotlib-devel]
	Unifying numpy, scipy, and matplotlib docstring formats
In-Reply-To: <1e2af89e0702160148u6f3b4ce4re11423f4ca9a35ec@mail.gmail.com>
References: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com>
	<45D4ED02.1050908@noaa.gov>
	<ef5675f30702151607h187c69d4t6360d490a88292ce@mail.gmail.com>
	<45D50245.1080102@noaa.gov>
	<1e2af89e0702160148u6f3b4ce4re11423f4ca9a35ec@mail.gmail.com>
Message-ID: <cd7634ce0702172341jfb16e96md36453e174de13a3@mail.gmail.com>

Perhaps we should consider two use cases: interactive use ala Matlab
and larger code bases. In the first case, being able to import * saves
a lot of typing and the namespace polution problem isn't a big deal.
The second use, obviously, benefits from avoiding import *.

Returning to the OP's questions, why couldn't both cases be helped by
creating a "meta-package" for numpy, scipy, and matplotlib? For the
sake of argument, lets call the package "plab". Existing code could be
affected by changing the individual packages, but a package that
essentially does

from pylab import *
from numpy import *
from scipy import *

would give a standard API that future code and interactive use could
use. Code could do
import plab

plab.plot() #etc.

and interactive use could do from plab import *.

Just a thought...

Barry

On 2/16/07, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> > I think a consensus is building in the python community that you should
> > NEVER use import *!
>
> Well, I have only been coding python for a few years, but I would say,
> along with writing unit tests, the great importance of not using
> import * is one of the secrets that you learn slowly and painfully
> with experience.  Chris' point about the movement of big projects away
> from that idiom is a very good one. It is convenient, but over time
> you realize that the value of convenience is far outweighed by the
> namespace mess and loss of clarity that results.
>
> Best,
>
> Matthew
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Matplotlib-users mailing list
> Matplotlib-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>


From david at ar.media.kyoto-u.ac.jp  Sun Feb 18 05:14:30 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 18 Feb 2007 19:14:30 +0900
Subject: [Numpy-discussion] Forcing the use of unoptimized blas/lapack even
	when atlas is present
Message-ID: <45D82706.3000604@ar.media.kyoto-u.ac.jp>

Hi there,
   
    I am developing a building tool to automatically build the whole 
numpy/scipy/matplotlib set from sources including dependencies, and one 
of the problem I got is to force which blas/lapack version to use when 
building numpy and scipy.
    I thought that doing a BLAS=blaslib LAPACK=lapacklib python 
setup.config was enough when build numpy, but numpy still wants to use 
atlas. I would like to avoid using site.cfg if possible, as I want to 
build everything automatically,

    cheers,

    David


From matthew.brett at gmail.com  Sun Feb 18 06:46:55 2007
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sun, 18 Feb 2007 11:46:55 +0000
Subject: [Numpy-discussion] [Matplotlib-users] [matplotlib-devel]
	Unifying numpy, scipy, and matplotlib docstring formats
In-Reply-To: <cd7634ce0702172341jfb16e96md36453e174de13a3@mail.gmail.com>
References: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com>
	<45D4ED02.1050908@noaa.gov>
	<ef5675f30702151607h187c69d4t6360d490a88292ce@mail.gmail.com>
	<45D50245.1080102@noaa.gov>
	<1e2af89e0702160148u6f3b4ce4re11423f4ca9a35ec@mail.gmail.com>
	<cd7634ce0702172341jfb16e96md36453e174de13a3@mail.gmail.com>
Message-ID: <1e2af89e0702180346g24f324a7je1f535f5fd6e0b96@mail.gmail.com>

Hi,

> import plab
>
> plab.plot() #etc.
>
> and interactive use could do from plab import *.

Yes...  It's a hard call of course.  I am a long term matlab user, and
switched to python relatively recently.  I do see the attraction of
persuading people that you can get something very similar to matlab
easily.  The downside about making numpy / python like matlab is that
you soon realize that you really have to think about your problems
differently, and write code in a different way.  I know that's
obvious, but the variables as pointers, mutable / immutable types,
zero based indexing, arrays vs matrices are all (fruitful) stumbling
blocks.  Then there is the very large change of thinking in an OO way,
pulling in other large packages for doing other tasks, writing
well-structured code with tests - all the features that python gives
you for an industrial strength code base.  And, the more pylab looks
like matlab, the more surprised and confused people will be when they
switch.  So,  I would argue that getting as close to matlab as
possible should not be the unqualified goal here - it is a real
change, with real pain, but great benefits.

Best,

Matthew


From ndbecker2 at gmail.com  Sun Feb 18 13:10:53 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sun, 18 Feb 2007 13:10:53 -0500
Subject: [Numpy-discussion] [matplotlib-devel] Unifying numpy, scipy,
	and matplotlib docstring formats
References: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com>
	<45D4ED02.1050908@noaa.gov>
	<ef5675f30702151607h187c69d4t6360d490a88292ce@mail.gmail.com>
	<45D50245.1080102@noaa.gov>
	<1e2af89e0702160148u6f3b4ce4re11423f4ca9a35ec@mail.gmail.com>
	<cd7634ce0702172341jfb16e96md36453e174de13a3@mail.gmail.com>
	<1e2af89e0702180346g24f324a7je1f535f5fd6e0b96@mail.gmail.com>
Message-ID: <era4re$o90$1@sea.gmane.org>

I have never used matlab, but a lot of my colleagues do.  Can anyone give me
some good references that I could show them to explain the advantages of
python over matlab?



From williams at astro.ox.ac.uk  Sun Feb 18 18:46:21 2007
From: williams at astro.ox.ac.uk (Michael Williams)
Date: Sun, 18 Feb 2007 23:46:21 +0000
Subject: [Numpy-discussion] [matplotlib-devel] Unifying numpy, scipy,
	and matplotlib docstring formats
In-Reply-To: <era4re$o90$1@sea.gmane.org>
References: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com>
	<45D4ED02.1050908@noaa.gov>
	<ef5675f30702151607h187c69d4t6360d490a88292ce@mail.gmail.com>
	<45D50245.1080102@noaa.gov>
	<1e2af89e0702160148u6f3b4ce4re11423f4ca9a35ec@mail.gmail.com>
	<cd7634ce0702172341jfb16e96md36453e174de13a3@mail.gmail.com>
	<1e2af89e0702180346g24f324a7je1f535f5fd6e0b96@mail.gmail.com>
	<era4re$o90$1@sea.gmane.org>
Message-ID: <20070218234621.GA398@astro.ox.ac.uk>

On Sun, Feb 18, 2007 at 01:10:53PM -0500, Neal Becker wrote:
> I have never used matlab, but a lot of my colleagues do.  Can anyone give me
> some good references that I could show them to explain the advantages of
> python over matlab?

http://www.scipy.org/NumPyProConPage
http://thread.gmane.org/gmane.comp.python.scientific.user/8950
http://groups.google.com/group/comp.lang.python/browse_frm/thread/a71af37fd9372868/

-- Mike


From barrywark at gmail.com  Sun Feb 18 19:03:38 2007
From: barrywark at gmail.com (Barry Wark)
Date: Sun, 18 Feb 2007 16:03:38 -0800
Subject: [Numpy-discussion] [Matplotlib-users] [matplotlib-devel]
	Unifying numpy, scipy, and matplotlib docstring formats
In-Reply-To: <1e2af89e0702180346g24f324a7je1f535f5fd6e0b96@mail.gmail.com>
References: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com>
	<45D4ED02.1050908@noaa.gov>
	<ef5675f30702151607h187c69d4t6360d490a88292ce@mail.gmail.com>
	<45D50245.1080102@noaa.gov>
	<1e2af89e0702160148u6f3b4ce4re11423f4ca9a35ec@mail.gmail.com>
	<cd7634ce0702172341jfb16e96md36453e174de13a3@mail.gmail.com>
	<1e2af89e0702180346g24f324a7je1f535f5fd6e0b96@mail.gmail.com>
Message-ID: <cd7634ce0702181603t56baedf7hf8deeb8d468ca8b3@mail.gmail.com>

Matt,

Yes, I agree. I wasn't coming at so much from the goal of making Pylab
a Matlab clone (as you point out, that's silly, and misses much of the
advantage of Python), but rather from the goal of making interactive
use as efficient as possible. When I fire up ipython -pylab to do some
quick exploration, it's nice not to have to type N.blah or pylab.plot
etc. If I just import pylab *, however, then the commands I use may
not be what I expect from more formal coding where I use N.blah numpy,
S.foo for scipy, and pylab.bar for matplotlib. Making it easy for
users to have either namespace strategy, with consistent bindings, ala
the start of this thread is a good idea, IMO.

Well, I've said my piece. I'll get out of the way and let others have a crack...

Barry

On 2/18/07, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> > import plab
> >
> > plab.plot() #etc.
> >
> > and interactive use could do from plab import *.
>
> Yes...  It's a hard call of course.  I am a long term matlab user, and
> switched to python relatively recently.  I do see the attraction of
> persuading people that you can get something very similar to matlab
> easily.  The downside about making numpy / python like matlab is that
> you soon realize that you really have to think about your problems
> differently, and write code in a different way.  I know that's
> obvious, but the variables as pointers, mutable / immutable types,
> zero based indexing, arrays vs matrices are all (fruitful) stumbling
> blocks.  Then there is the very large change of thinking in an OO way,
> pulling in other large packages for doing other tasks, writing
> well-structured code with tests - all the features that python gives
> you for an industrial strength code base.  And, the more pylab looks
> like matlab, the more surprised and confused people will be when they
> switch.  So,  I would argue that getting as close to matlab as
> possible should not be the unqualified goal here - it is a real
> change, with real pain, but great benefits.
>
> Best,
>
> Matthew
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Matplotlib-users mailing list
> Matplotlib-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>


From zpincus at stanford.edu  Mon Feb 19 05:56:34 2007
From: zpincus at stanford.edu (Zachary Pincus)
Date: Mon, 19 Feb 2007 02:56:34 -0800
Subject: [Numpy-discussion] linalg.eigh orders eigenvalues/eigenvectors
	differently than linalg.eig
Message-ID: <C86CE02D-C4EA-4BAC-920A-854E9449369C@stanford.edu>

Hello all,

It seems that the 'eigh' routine from numpy.linalg does not follow  
the same convention as numpy.linalg.eig in terms of the order of the  
returned eigenvalues. (And thus eigenvectors as well...)

Specifically, eig returns eigenvalues in order from largest to  
smallest, while eigh returns them from smallest to largest.

Example:
 >>> a = numpy.array([[21, 28, 35],[28, 38, 48],[35, 48, 61]])
 >>> numpy.linalg.eigh(a)
(array([ -1.02825542e-14,   7.04131679e-01,   1.19295868e+02]),
array([[ 0.40824829, -0.81314396, -0.41488581],
        [-0.81649658, -0.12200588, -0.56431188],
        [ 0.40824829,  0.56913221, -0.71373795]]))

 >>> numpy.linalg.eig(a)
(array([  1.19295868e+02,   7.04131679e-01,   4.62814557e-15]),
array([[-0.41488581, -0.81314396,  0.40824829],
        [-0.56431188, -0.12200588, -0.81649658],
        [-0.71373795,  0.56913221,  0.40824829]]))

Is this a bug? If it is, though, fixing it now might break code that  
depends on this 'wrong' order. (This is also present in  
scipy.linalg.) If not a bug, or not-fixable-now, then at least some  
documentation as to the convention regarding ordering of eigenvalues  
is probably worthwhile...

Any thoughts?

Zach


From svetosch at gmx.net  Mon Feb 19 06:06:31 2007
From: svetosch at gmx.net (Sven Schreiber)
Date: Mon, 19 Feb 2007 12:06:31 +0100
Subject: [Numpy-discussion] linalg.eigh orders eigenvalues/eigenvectors
 differently than linalg.eig
In-Reply-To: <C86CE02D-C4EA-4BAC-920A-854E9449369C@stanford.edu>
References: <C86CE02D-C4EA-4BAC-920A-854E9449369C@stanford.edu>
Message-ID: <45D984B7.9060200@gmx.net>

Zachary Pincus schrieb:
> Hello all,
> 
> It seems that the 'eigh' routine from numpy.linalg does not follow  
> the same convention as numpy.linalg.eig in terms of the order of the  
> returned eigenvalues. (And thus eigenvectors as well...)

I was told on this list that the ordering should not be relied upon, and
that it might change in the future. So it seems that user code should
explicitly re-order the eigenvalues (and corresponding eigenvectors,
probably using argsort and fancy indexing -- if those are the right terms).

...

> scipy.linalg.) If not a bug, or not-fixable-now, then at least some  
> documentation as to the convention regarding ordering of eigenvalues  
> is probably worthwhile...
> 

true

cheers,
sven



From a.u.r.e.l.i.a.n at gmx.net  Mon Feb 19 07:14:50 2007
From: a.u.r.e.l.i.a.n at gmx.net (Johannes Loehnert)
Date: Mon, 19 Feb 2007 13:14:50 +0100
Subject: [Numpy-discussion] linalg.eigh orders eigenvalues/eigenvectors
	differently than linalg.eig
In-Reply-To: <45D984B7.9060200@gmx.net>
References: <C86CE02D-C4EA-4BAC-920A-854E9449369C@stanford.edu>
	<45D984B7.9060200@gmx.net>
Message-ID: <200702191314.50846.a.u.r.e.l.i.a.n@gmx.net>

On Monday 19 February 2007 12:06, Sven Schreiber wrote:
> Zachary Pincus schrieb:
> > Hello all,
> >
> > It seems that the 'eigh' routine from numpy.linalg does not follow
> > the same convention as numpy.linalg.eig in terms of the order of the
> > returned eigenvalues. (And thus eigenvectors as well...)
>
> I was told on this list that the ordering should not be relied upon, and
> that it might change in the future. So it seems that user code should
> explicitly re-order the eigenvalues (and corresponding eigenvectors,
> probably using argsort and fancy indexing -- if those are the right terms).

Indeed. eig and eigh are wrappers for lapack functions, so the result is 
whatever those give back. Do not rely on a particular order of eigenvalues, 
sort yourself.

Short example for convenience:
#---------
eigvals, eigvecs = eig(some_matrix)
ind = argsort(eigvals)
eigvals = eigvals[ind]
eigvecs = eigvecs[:, ind]   # second axis !!
# etc.
#------------

Johannes


From lyang at unb.ca  Mon Feb 19 07:51:07 2007
From: lyang at unb.ca (Yang, Lu)
Date: Mon, 19 Feb 2007 08:51:07 -0400
Subject: [Numpy-discussion] Numpy1.0.1 installation problem - urgent :(
In-Reply-To: <6ce0ac130702151111h55a60b6dl5228e11234b49179@mail.gmail.com>
References: <2768a9540702141847y568794f5vff110a4dbd4caf18@mail.gmail.com>
	<6ce0ac130702151111h55a60b6dl5228e11234b49179@mail.gmail.com>
Message-ID: <1171889467.45d99d3b00dd7@webmail.unb.ca>

Thanks so much. It works.

Lu


Quoting Brian Granger <ellisonbg.net at gmail.com>:

> I don't run numpy no linux often, but you shouldn't have any trouble.
> I would do the following:
> 
> 1.  Blast your current numpy install
> 
> rm -rf /usr/local/lib/python2.5/site-packages/numpy
> 
> 2.  Get the lastest svn version
> 
> cd $HOME
> svn co http://svn.scipy.org/svn/numpy/trunk numpy
> 
> 3.  Try doing a fresh install (starting from $HOME):
> 
> cd numpy
> python setup.py build
> sudo python setup.py install
> 
> Let us know if that helps.
> 
> Brian
> 
> On 2/14/07, lu yang <lu.yangunb at gmail.com> wrote:
> > Hi,
> > This is the first time I install Numpy on a linux machine. I have been
> > working on it for several days without luck. I sincerely appreciate if
> > anybody can give any comments. My OS is Red Hat 8.
> > I have downloaded Python 2.5 and Numpy 1.0.1. Python 2.5 has been installed
> > on my machine. However, when I type:
> > "python2.5 setup.py install" to install Numpy in my home directory
> > /home/o6c11/numpy-1.0.1 , I got:
> >
> > [o6c11 at chorus numpy-1.0.1]$ python2.5 setup.py install
> > Traceback (most recent call last):
> >   File "setup.py", line 89, in <module>
> >     setup_package()
> >   File "setup.py", line 59, in setup_package
> >     from numpy.distutils.core import setup
> >   File "numpy/__init__.py", line 36, in <module>
> >     import core
> >   File "/mnt/storage/home/o6c11/numpy-1.0.1/numpy/core/__init__.py", line 5,
> > in <module>
> >     import multiarray
> > ImportError: No module named multiarray
> >
> > I changed the directory to
> > /usr/local/lib/python2.5/site-packages/numpy, I got:
> >
> > [o6c11 at chorus numpy]$ python2.5 setup.py install
> > Traceback (most recent call last):
> >   File "setup.py", line 28, in <module>
> >     from numpy.distutils.core import setup
> >   File
> > "/usr/local/lib/python2.5/site-packages/numpy/__init__.py",
> > line 40, in <module>
> >     import linalg
> >   File
> > "/usr/local/lib/python2.5/site-packages/numpy/linalg/__init__.py",
> > line 4, in <module>
> >     from linalg import *
> >   File
> > "/usr/local/lib/python2.5/site-packages/numpy/linalg/linalg.py",
> > line 25, in <module>
> >     from numpy.linalg import lapack_lite
> > ImportError: /usr/lib/libblas.so.3: undefined symbol: e_wsfe
> >
> > I also tried to import numpy in this way:
> >
> > [o6c11 at chorus numpy]$ python2.5
> > Python 2.5 (r25:51908, Feb 12 2007, 22:36:33)
> > [GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> import numpy
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> >   File
> > "/usr/local/lib/python2.5/site-packages/numpy/__init__.py",
> > line 40, in <module>
> >     import linalg
> >   File
> > "/usr/local/lib/python2.5/site-packages/numpy/linalg/__init__.py",
> > line 4, in <module>
> >     from linalg import *
> >   File
> > "/usr/local/lib/python2.5/site-packages/numpy/linalg/linalg.py",
> > line 25, in <module>
> >     from numpy.linalg import lapack_lite
> > ImportError: /usr/lib/libblas.so.3: undefined symbol: e_wsfe
> > >>>
> >
> > I have no idea how to solve these problems. Please help a newbie. Thanks a
> > lot.
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 





From nwagner at iam.uni-stuttgart.de  Mon Feb 19 08:19:20 2007
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 19 Feb 2007 14:19:20 +0100
Subject: [Numpy-discussion] linalg.eigh orders eigenvalues/eigenvectors
 differently than linalg.eig
In-Reply-To: <200702191314.50846.a.u.r.e.l.i.a.n@gmx.net>
References: <C86CE02D-C4EA-4BAC-920A-854E9449369C@stanford.edu>	<45D984B7.9060200@gmx.net>
	<200702191314.50846.a.u.r.e.l.i.a.n@gmx.net>
Message-ID: <45D9A3D8.4060506@iam.uni-stuttgart.de>

Johannes Loehnert wrote:
> On Monday 19 February 2007 12:06, Sven Schreiber wrote:
>   
>> Zachary Pincus schrieb:
>>     
>>> Hello all,
>>>
>>> It seems that the 'eigh' routine from numpy.linalg does not follow
>>> the same convention as numpy.linalg.eig in terms of the order of the
>>> returned eigenvalues. (And thus eigenvectors as well...)
>>>       
>> I was told on this list that the ordering should not be relied upon, and
>> that it might change in the future. So it seems that user code should
>> explicitly re-order the eigenvalues (and corresponding eigenvectors,
>> probably using argsort and fancy indexing -- if those are the right terms).
>>     
>
> Indeed. eig and eigh are wrappers for lapack functions, so the result is 
> whatever those give back. Do not rely on a particular order of eigenvalues, 
> sort yourself.
>
> Short example for convenience:
> #---------
> eigvals, eigvecs = eig(some_matrix)
> ind = argsort(eigvals)
> eigvals = eigvals[ind]
> eigvecs = eigvecs[:, ind]   # second axis !!
> # etc.
> #------------
>
> Johannes
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   
 
eigh is based on *heev  (* placeholder for s,d,c,z)
http://www.netlib.org/lapack/complex16/zheev.f

*  W       (output) DOUBLE PRECISION array, dimension (N)
*          If INFO = 0, the eigenvalues in ascending order.


Apparently the eigenvalues are in ascending order for eigh.


Hence, It would be nice if scipy could offer more wrappers for matrices
with certain properties, e.g.

http://www.netlib.org/lapack/double/dsygv.f
http://www.netlib.org/lapack/double/dsyevr.f
http://www.netlib.org/lapack/double/dsygvx.f
http://www.netlib.org/lapack/double/dstevr.f

Any comments ?

Nils

Also LAPACK3.1 has many improved eigensolvers. Please try suzuki.py with

LAPACK3.0

>>> w_e
array([-200.        +0.j        , -100.        +0.j        ,
        700.00307816+0.00103379j,  700.00307816-0.00103379j,
        699.9993562 +0.00318262j,  699.9993562 -0.00318262j,
        699.99756564+0.00214883j,  699.99756564-0.00214883j])


versus

LAPACK3.1.
>>> w_e
array([-200.        +0.j        , -100.        +0.j        ,
        699.99891191+0.00188463j,  699.99891191-0.00188463j,
        700.00217619+0.j        ,  700.00294523+0.j        ,
        699.99852738+0.0025507j ,  699.99852738-0.0025507j ])


-------------- next part --------------
A non-text attachment was scrubbed...
Name: suzuki.py
Type: text/x-python
Size: 890 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070219/eca2c2d7/attachment.py>

From robert.kern at gmail.com  Mon Feb 19 11:04:58 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 19 Feb 2007 10:04:58 -0600
Subject: [Numpy-discussion] Forcing the use of unoptimized blas/lapack
 even	when atlas is present
In-Reply-To: <45D82706.3000604@ar.media.kyoto-u.ac.jp>
References: <45D82706.3000604@ar.media.kyoto-u.ac.jp>
Message-ID: <45D9CAAA.3030409@gmail.com>

David Cournapeau wrote:
> Hi there,
>    
>     I am developing a building tool to automatically build the whole 
> numpy/scipy/matplotlib set from sources including dependencies, and one 
> of the problem I got is to force which blas/lapack version to use when 
> building numpy and scipy.
>     I thought that doing a BLAS=blaslib LAPACK=lapacklib python 
> setup.config was enough when build numpy, but numpy still wants to use 
> atlas. I would like to avoid using site.cfg if possible, as I want to 
> build everything automatically,

Set ATLAS=0, I believe.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From nwagner at iam.uni-stuttgart.de  Mon Feb 19 11:09:51 2007
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 19 Feb 2007 17:09:51 +0100
Subject: [Numpy-discussion] Forcing the use of unoptimized blas/lapack
 even	when atlas is present
In-Reply-To: <45D9CAAA.3030409@gmail.com>
References: <45D82706.3000604@ar.media.kyoto-u.ac.jp>
	<45D9CAAA.3030409@gmail.com>
Message-ID: <45D9CBCF.6060909@iam.uni-stuttgart.de>

Robert Kern wrote:
> David Cournapeau wrote:
>   
>> Hi there,
>>    
>>     I am developing a building tool to automatically build the whole 
>> numpy/scipy/matplotlib set from sources including dependencies, and one 
>> of the problem I got is to force which blas/lapack version to use when 
>> building numpy and scipy.
>>     I thought that doing a BLAS=blaslib LAPACK=lapacklib python 
>> setup.config was enough when build numpy, but numpy still wants to use 
>> atlas. I would like to avoid using site.cfg if possible, as I want to 
>> build everything automatically,
>>     
>
> Set ATLAS=0, I believe.
>
>   

 IIRC, export ATLAS=None should also work.

Nils



From cookedm at physics.mcmaster.ca  Mon Feb 19 11:15:04 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Mon, 19 Feb 2007 11:15:04 -0500
Subject: [Numpy-discussion] Forcing the use of unoptimized blas/lapack
	even	when atlas is present
In-Reply-To: <45D9CAAA.3030409@gmail.com>
References: <45D82706.3000604@ar.media.kyoto-u.ac.jp>
	<45D9CAAA.3030409@gmail.com>
Message-ID: <40111626-9A15-43F6-85E6-6419F07A7CAC@physics.mcmaster.ca>

On Feb 19, 2007, at 11:04 , Robert Kern wrote:

> David Cournapeau wrote:
>> Hi there,
>>
>>     I am developing a building tool to automatically build the whole
>> numpy/scipy/matplotlib set from sources including dependencies,  
>> and one
>> of the problem I got is to force which blas/lapack version to use  
>> when
>> building numpy and scipy.
>>     I thought that doing a BLAS=blaslib LAPACK=lapacklib python
>> setup.config was enough when build numpy, but numpy still wants to  
>> use
>> atlas. I would like to avoid using site.cfg if possible, as I want to
>> build everything automatically,
>
> Set ATLAS=0, I believe.

Not quite, you need
LAPACK=None BLAS=None

(ATLAS=None is only needed if ATLAS is being looked for specifically,  
i.e., system_info.atlas_info is used instead of  
system_info.lapack_opt_info in the setup.py. AFAIK, that's only used  
when debugging ATLAS installs in scipy).

-- 
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From yves.frederix at gmail.com  Mon Feb 19 12:11:20 2007
From: yves.frederix at gmail.com (Yves Frederix)
Date: Mon, 19 Feb 2007 18:11:20 +0100
Subject: [Numpy-discussion] numpy.dot and ACML
Message-ID: <20070219171120.GA5609@kotnet.org>

Hi all,

I have managed to compile numpy using pathscale and ACML on a 64 bit AMD
system. Now I wanted to verify that numpy.dot indeed uses the ACML
libs. The example for dot()
(http://www.scipy.org/Numpy_Example_List?highlight=%28example%29#head-c7a573f030ff7cbaea62baf219599b3976136bac) suggest a way of doing this:

	1 u0050015 at lo-03-02 .../core $ python -c "import numpy; print id(numpy.dot)==id(numpy.core.multiarray.dot);"
	True

This indicates that I am not using the acml libraries. 

When running a benchmark (see attach) and comparing to a non-ACML
installation though, the strange thing is that there is a clear
speed difference, suggesting again that the acml libraries are indeed
used.

Because this is not all that clear to me, I was wondering whether there
exists an alternative way of verifying what libraries are used.

Many thanks,
YVES
-------------- next part --------------
ACML:

dim        x.T*y        x*y.T      A*x        A*B        A.T*x       
-----------------------------------------------------------------
5000       0.002492   0.002417   0.002412   0.002399   0.002416  
50000      0.020074   0.020024   0.020004   0.020003   0.020024  
100000     0.092777   0.093690   0.100220   0.093787   0.094250  
200000     0.184933   0.198623   0.196120   0.197089   0.197273  
300000     0.276583   0.279177   0.280898   0.284016   0.276204  
500000     0.476340   0.481987   0.471875   0.480868   0.481501  
1000000.0  0.892623   0.895500   0.915173   0.894815   0.922501  
5000000.0  4.450555   4.465748   4.467870   4.468188   4.469083 

No ACML:

dim        x.T*y        x*y.T      A*x        A*B        A.T*x       
-----------------------------------------------------------------
5000       0.002523   0.002428   0.002410   0.002430   0.002419  
50000      0.024756   0.061520   0.036575   0.036399   0.036450  
100000     0.338576   0.353074   0.169472   0.302087   0.334633  
200000     0.670803   0.735732   0.538166   0.649335   0.744496  
300000     1.004381   1.269259   0.482542   2.194308   0.611997  
500000     1.110656   1.504701   1.571736   1.656021   1.491146  
1000000.0  2.182746   2.234478   2.254645   2.439508   2.537558  
5000000.0  10.878910  16.578266  8.265109   8.905976   17.124400 


From zpincus at stanford.edu  Tue Feb 20 14:33:16 2007
From: zpincus at stanford.edu (Zachary Pincus)
Date: Tue, 20 Feb 2007 11:33:16 -0800
Subject: [Numpy-discussion] Distributing prebuilt numpy and other extensions
Message-ID: <D3A11D21-A268-41B3-9819-3C6F13D0A8B5@stanford.edu>

Hello folks,

I've developed some command-line tools for biologists using python/ 
numpy and some custom C and Fortran extensions, and I'm trying to  
figure out how to easily distribute them...

For people using linux, I figure a source distribution is no problem  
at all. (Right?)
On the other hand, for Mac users (whose computers by default don't  
have the dev tools, and even then would need to get a fortran  
compiler elsewhere) I'd like to figure out something a bit easier.

I'd like to somehow provide an installer (double-clickable or python  
script) that does a version check and then installs an appropriate  
version of prebuilt binaries for numpy and my C and Fortran  
extensions. Is this possible within the bounds of the python or numpy  
distutils? Would setuptools be a better way to go? Preferably it  
would be a dead easy, one-step thing...

Or is this whole idea problematic, and better to stick with source  
distribution in all cases?

Thanks for any advice,

Zach Pincus


Program in Biomedical Informatics and Department of Biochemistry
Stanford University School of Medicine



From mjanikas at esri.com  Tue Feb 20 18:56:20 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 20 Feb 2007 15:56:20 -0800
Subject: [Numpy-discussion] Greek Letters
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFBB@hardwire.esri.com>

Hello all,

 

I was wondering how I could print the chi-squared symbol in python.  I
have been looking at the Unicode docs, but I figured I would ask for
assistance here while I delve into it.  Thanks for any help in advance.

 

Mark Janikas

Product Engineer

ESRI, Geoprocessing

380 New York St.

Redlands, CA 92373

909-793-2853 (2563)

mjanikas at esri.com

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070220/8770f778/attachment.html>

From zpincus at stanford.edu  Tue Feb 20 19:18:05 2007
From: zpincus at stanford.edu (Zachary Pincus)
Date: Tue, 20 Feb 2007 16:18:05 -0800
Subject: [Numpy-discussion] Greek Letters
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFBB@hardwire.esri.com>
References: <627102C921CD9745B070C3B10CB8199B010EBFBB@hardwire.esri.com>
Message-ID: <7E429C6F-DF05-42C1-BDE6-53CFF1090DDA@stanford.edu>

I have found that the python 'unicode name' escape sequence, combined  
with the canonical list of unicode names ( http://unicode.org/Public/ 
UNIDATA/NamesList.txt ), is a good way of getting the symbols you  
want and still keeping the python code legible.

 From the above list, we see that the symbol name we want is GREEK  
SMALL LETTER CHI, so:
chi = u'\N{GREEK SMALL LETTER CHI}'
will do the trick. For chi^2, use:
chi2 = u'\N{GREEK SMALL LETTER CHI}\N{SUPERSCRIPT TWO}'

Note that to print these characters, we usually need to encode them  
somehow. My terminal supports UTF-8, so the following works for me:
import codecs
print codecs.encode(chi2, 'utf8')

giving (if your mail reader supports utf8 and mine encodes it  
properly...):
??

Zach Pincus

Program in Biomedical Informatics and Department of Biochemistry
Stanford University School of Medicine


On Feb 20, 2007, at 3:56 PM, Mark Janikas wrote:

> Hello all,
>
>
>
> I was wondering how I could print the chi-squared symbol in  
> python.  I have been looking at the Unicode docs, but I figured I  
> would ask for assistance here while I delve into it.  Thanks for  
> any help in advance.
>
>
>
> Mark Janikas
>
> Product Engineer
>
> ESRI, Geoprocessing
>
> 380 New York St.
>
> Redlands, CA 92373
>
> 909-793-2853 (2563)
>
> mjanikas at esri.com
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From robert.kern at gmail.com  Tue Feb 20 19:19:55 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 20 Feb 2007 18:19:55 -0600
Subject: [Numpy-discussion] Greek Letters
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFBB@hardwire.esri.com>
References: <627102C921CD9745B070C3B10CB8199B010EBFBB@hardwire.esri.com>
Message-ID: <45DB902B.70009@gmail.com>

Mark Janikas wrote:
> Hello all,
> 
> I was wondering how I could print the chi-squared symbol in python.  I
> have been looking at the Unicode docs, but I figured I would ask for
> assistance here while I delve into it.  Thanks for any help in advance.

Print it where? To the terminal (which one?)? In HTML? With some GUI?

Assuming that you have a Unicode-capable terminal, you can find out the encoding
it uses by looking at sys.stdout.encoding. Encode your Unicode string with that
encoding, and print it. E.g., I use iTerm on OS X and set it to use UTF-8 as the
encoding:

In [5]: import sys

In [6]: sys.stdout.encoding
Out[6]: 'UTF-8'

In [7]: print u'\u03a7\u00b2'.encode(sys.stdout.encoding)
??

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From mjanikas at esri.com  Tue Feb 20 20:11:42 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 20 Feb 2007 17:11:42 -0800
Subject: [Numpy-discussion] Greek Letters
In-Reply-To: <7E429C6F-DF05-42C1-BDE6-53CFF1090DDA@stanford.edu>
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFBC@hardwire.esri.com>

Thanks for all the info.  That website with all the codes is great.  

MJ

-----Original Message-----
From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Zachary Pincus
Sent: Tuesday, February 20, 2007 4:18 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Greek Letters

I have found that the python 'unicode name' escape sequence, combined  
with the canonical list of unicode names ( http://unicode.org/Public/ 
UNIDATA/NamesList.txt ), is a good way of getting the symbols you  
want and still keeping the python code legible.

 From the above list, we see that the symbol name we want is GREEK  
SMALL LETTER CHI, so:
chi = u'\N{GREEK SMALL LETTER CHI}'
will do the trick. For chi^2, use:
chi2 = u'\N{GREEK SMALL LETTER CHI}\N{SUPERSCRIPT TWO}'

Note that to print these characters, we usually need to encode them  
somehow. My terminal supports UTF-8, so the following works for me:
import codecs
print codecs.encode(chi2, 'utf8')

giving (if your mail reader supports utf8 and mine encodes it  
properly...):
??

Zach Pincus

Program in Biomedical Informatics and Department of Biochemistry
Stanford University School of Medicine


On Feb 20, 2007, at 3:56 PM, Mark Janikas wrote:

> Hello all,
>
>
>
> I was wondering how I could print the chi-squared symbol in  
> python.  I have been looking at the Unicode docs, but I figured I  
> would ask for assistance here while I delve into it.  Thanks for  
> any help in advance.
>
>
>
> Mark Janikas
>
> Product Engineer
>
> ESRI, Geoprocessing
>
> 380 New York St.
>
> Redlands, CA 92373
>
> 909-793-2853 (2563)
>
> mjanikas at esri.com
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion


From mjanikas at esri.com  Tue Feb 20 20:16:23 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 20 Feb 2007 17:16:23 -0800
Subject: [Numpy-discussion] Greek Letters
In-Reply-To: <45DB902B.70009@gmail.com>
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFBD@hardwire.esri.com>

Thanks Robert.... but alas, I get.....

>>> import sys
>>> sys.stdout.encoding
'cp437'
>>> print u'\u03a7\u00b2'.encode(sys.stdout.encoding)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "C:\Python24\lib\encodings\cp437.py", line 18, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u03a7' in position
 0: character maps to <undefined>
>>>


Ill keep at it.... please let me know if you have any solutions....

Thanks again,

MJ

-----Original Message-----
From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Robert Kern
Sent: Tuesday, February 20, 2007 4:20 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Greek Letters

Mark Janikas wrote:
> Hello all,
> 
> I was wondering how I could print the chi-squared symbol in python.  I
> have been looking at the Unicode docs, but I figured I would ask for
> assistance here while I delve into it.  Thanks for any help in advance.

Print it where? To the terminal (which one?)? In HTML? With some GUI?

Assuming that you have a Unicode-capable terminal, you can find out the encoding
it uses by looking at sys.stdout.encoding. Encode your Unicode string with that
encoding, and print it. E.g., I use iTerm on OS X and set it to use UTF-8 as the
encoding:

In [5]: import sys

In [6]: sys.stdout.encoding
Out[6]: 'UTF-8'

In [7]: print u'\u03a7\u00b2'.encode(sys.stdout.encoding)
??

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion


From mjanikas at esri.com  Tue Feb 20 20:29:25 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 20 Feb 2007 17:29:25 -0800
Subject: [Numpy-discussion] Greek Letters
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFBD@hardwire.esri.com>
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFBE@hardwire.esri.com>

Oh.  I am using CygWin, and the website I just went to:

http://www.cygwin.com/faq/faq_3.html


stated that: " The short answer is that Cygwin is not Unicode-aware"

Not sure if this is going to apply to python in general, but I suspect it will.  Ugh, I dislike Windows a lot, but it pays the bills.  The interesting thing to note is that the print out to gui interface is 'UTF-8' so it works.  It just wont work on my terminal where I do all of my testing.  I might just have to put a try statement in and put a "chi-square" in the except.

MJ

-----Original Message-----
From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Mark Janikas
Sent: Tuesday, February 20, 2007 5:16 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Greek Letters

Thanks Robert.... but alas, I get.....

>>> import sys
>>> sys.stdout.encoding
'cp437'
>>> print u'\u03a7\u00b2'.encode(sys.stdout.encoding)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "C:\Python24\lib\encodings\cp437.py", line 18, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u03a7' in position
 0: character maps to <undefined>
>>>


Ill keep at it.... please let me know if you have any solutions....

Thanks again,

MJ

-----Original Message-----
From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Robert Kern
Sent: Tuesday, February 20, 2007 4:20 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Greek Letters

Mark Janikas wrote:
> Hello all,
> 
> I was wondering how I could print the chi-squared symbol in python.  I
> have been looking at the Unicode docs, but I figured I would ask for
> assistance here while I delve into it.  Thanks for any help in advance.

Print it where? To the terminal (which one?)? In HTML? With some GUI?

Assuming that you have a Unicode-capable terminal, you can find out the encoding
it uses by looking at sys.stdout.encoding. Encode your Unicode string with that
encoding, and print it. E.g., I use iTerm on OS X and set it to use UTF-8 as the
encoding:

In [5]: import sys

In [6]: sys.stdout.encoding
Out[6]: 'UTF-8'

In [7]: print u'\u03a7\u00b2'.encode(sys.stdout.encoding)
??

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion


From robert.kern at gmail.com  Tue Feb 20 20:36:19 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 20 Feb 2007 19:36:19 -0600
Subject: [Numpy-discussion] Greek Letters
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFBD@hardwire.esri.com>
References: <627102C921CD9745B070C3B10CB8199B010EBFBD@hardwire.esri.com>
Message-ID: <45DBA213.9050805@gmail.com>

Mark Janikas wrote:
> Thanks Robert.... but alas, I get.....
> 
>>>> import sys
>>>> sys.stdout.encoding
> 'cp437'
>>>> print u'\u03a7\u00b2'.encode(sys.stdout.encoding)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "C:\Python24\lib\encodings\cp437.py", line 18, in encode
>     return codecs.charmap_encode(input,errors,encoding_map)
> UnicodeEncodeError: 'charmap' codec can't encode character u'\u03a7' in position
>  0: character maps to <undefined>
> 
> 
> Ill keep at it.... please let me know if you have any solutions....

Yup, CP437 doesn't support the ? character. You will have to use a terminal that
can accept a full-Unicode encoding like UTF-8 and a font that has the relevant
characters. Most of the modern terminal emulators for Linux et al. are capable
of UTF-8. On Windows, you may be out of luck. I don't know of any
fully-Unicode-capable terminal. Google tells me that cmd.exe takes a /U argument
that should put it in Unicode mode, but that doesn't change anything for me.

I also ran into a case where sys.stdout.encoding is None, so my solution is not
as robust as I thought.

If you use something else for your interpreter shell like IDLE, it is possible
that it supports Unicode. sys.stdout.encoding may not be set appropriately, though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Tue Feb 20 20:39:31 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Wed, 21 Feb 2007 03:39:31 +0200
Subject: [Numpy-discussion] Greek Letters
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFBE@hardwire.esri.com>
References: <627102C921CD9745B070C3B10CB8199B010EBFBD@hardwire.esri.com>
	<627102C921CD9745B070C3B10CB8199B010EBFBE@hardwire.esri.com>
Message-ID: <20070221013931.GO5912@mentat.za.net>

On Tue, Feb 20, 2007 at 05:29:25PM -0800, Mark Janikas wrote:
> Oh.  I am using CygWin, and the website I just went to:
> 
> http://www.cygwin.com/faq/faq_3.html
> 
> 
> stated that: " The short answer is that Cygwin is not Unicode-aware"
> 
> Not sure if this is going to apply to python in general, but I
> suspect it will.  Ugh, I dislike Windows a lot, but it pays the bills.

Actually you pay Bill :)

Maybe try the free vmplayer with a linux session?

Cheers
St?fan


From mjanikas at esri.com  Tue Feb 20 20:47:19 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Tue, 20 Feb 2007 17:47:19 -0800
Subject: [Numpy-discussion] Greek Letters
In-Reply-To: <45DBA213.9050805@gmail.com>
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFBF@hardwire.esri.com>

Great info.  The last part about the sys.stdout not working everywhere is a bit troubling.  But For now, I am just gonna add a try statement as stated in the previous message.  Thanks again,

MJ

PS... The software company I work for has a version that runs on Unix/Linux so I am awaiting a new laptop with which to employ Linux!  I cant wait to get back to my roots.  Debian and Fluxbox.... could there be a better working environment?  

-----Original Message-----
From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Robert Kern
Sent: Tuesday, February 20, 2007 5:36 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Greek Letters

Mark Janikas wrote:
> Thanks Robert.... but alas, I get.....
> 
>>>> import sys
>>>> sys.stdout.encoding
> 'cp437'
>>>> print u'\u03a7\u00b2'.encode(sys.stdout.encoding)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "C:\Python24\lib\encodings\cp437.py", line 18, in encode
>     return codecs.charmap_encode(input,errors,encoding_map)
> UnicodeEncodeError: 'charmap' codec can't encode character u'\u03a7' in position
>  0: character maps to <undefined>
> 
> 
> Ill keep at it.... please let me know if you have any solutions....

Yup, CP437 doesn't support the ? character. You will have to use a terminal that
can accept a full-Unicode encoding like UTF-8 and a font that has the relevant
characters. Most of the modern terminal emulators for Linux et al. are capable
of UTF-8. On Windows, you may be out of luck. I don't know of any
fully-Unicode-capable terminal. Google tells me that cmd.exe takes a /U argument
that should put it in Unicode mode, but that doesn't change anything for me.

I also ran into a case where sys.stdout.encoding is None, so my solution is not
as robust as I thought.

If you use something else for your interpreter shell like IDLE, it is possible
that it supports Unicode. sys.stdout.encoding may not be set appropriately, though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion


From strawman at astraw.com  Tue Feb 20 20:51:54 2007
From: strawman at astraw.com (Andrew Straw)
Date: Tue, 20 Feb 2007 17:51:54 -0800
Subject: [Numpy-discussion] Greek Letters
In-Reply-To: <45DBA213.9050805@gmail.com>
References: <627102C921CD9745B070C3B10CB8199B010EBFBD@hardwire.esri.com>
	<45DBA213.9050805@gmail.com>
Message-ID: <45DBA5BA.3080004@astraw.com>

Robert Kern wrote:
> On Windows, you may be out of luck. I don't know of any
> fully-Unicode-capable terminal.
The lack of a decent console application is one of the most problematic 
issues I face whenever attempting to do serious programming in Windows. 
I wish I knew of a better terminal program. Here's one that seems like 
it might work, but I haven't tried it yet: 
http://software.jessies.org/terminator




From david at ar.media.kyoto-u.ac.jp  Tue Feb 20 21:50:11 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 21 Feb 2007 11:50:11 +0900
Subject: [Numpy-discussion] Distributing prebuilt numpy and other
	extensions
In-Reply-To: <D3A11D21-A268-41B3-9819-3C6F13D0A8B5@stanford.edu>
References: <D3A11D21-A268-41B3-9819-3C6F13D0A8B5@stanford.edu>
Message-ID: <45DBB363.2060706@ar.media.kyoto-u.ac.jp>

Zachary Pincus wrote:
> Hello folks,
>
> I've developed some command-line tools for biologists using python/ 
> numpy and some custom C and Fortran extensions, and I'm trying to  
> figure out how to easily distribute them...
>
> For people using linux, I figure a source distribution is no problem  
> at all. (Right?)
If they already have dependencies, and can easily install blas/lapack, 
then it is indeed relatively easy. Incidentally, I am working on a 
system to build a complete self-contained numpy/scipy from sources, 
including LAPACK/BLAS/ATLAS if necessary, for reasons similar to you. On 
linux at least, I still think source distribution is the most reliable 
(compiler version mismatch, etc... makes it really difficult to 
distribute binaries across several distributions).
> On the other hand, for Mac users (whose computers by default don't  
> have the dev tools, and even then would need to get a fortran  
> compiler elsewhere) I'd like to figure out something a bit easier.
>
I don't know much about Mac OS X, but PyMC distributes binaries for 
intel and ppc Mac in a way similar to what you are looking for:

http://trichech.us/?page_id=3

Unfortunately, the build scripts are not included in the sources, but 
maybe the author of PyMC can give you some hints ?

cheers,

David



From robert.kern at gmail.com  Tue Feb 20 22:25:23 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 20 Feb 2007 21:25:23 -0600
Subject: [Numpy-discussion] Distributing prebuilt numpy and
	other	extensions
In-Reply-To: <45DBB363.2060706@ar.media.kyoto-u.ac.jp>
References: <D3A11D21-A268-41B3-9819-3C6F13D0A8B5@stanford.edu>
	<45DBB363.2060706@ar.media.kyoto-u.ac.jp>
Message-ID: <45DBBBA3.3040008@gmail.com>

David Cournapeau wrote:
> I don't know much about Mac OS X, but PyMC distributes binaries for 
> intel and ppc Mac in a way similar to what you are looking for:
> 
> http://trichech.us/?page_id=3

Or rather

  http://trichech.us/?page_id=4

Unfortunately, those builds are currently broken.

> Unfortunately, the build scripts are not included in the sources, but 
> maybe the author of PyMC can give you some hints ?

Primarily, you just use bdist_mpkg to generate the Installer.app packages.

  http://cheeseshop.python.org/pypi/bdist_mpkg/

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From gnurser at googlemail.com  Wed Feb 21 08:26:35 2007
From: gnurser at googlemail.com (George Nurser)
Date: Wed, 21 Feb 2007 13:26:35 +0000
Subject: [Numpy-discussion] New release of pycdf package ported to NumPy
In-Reply-To: <1171314114.6259.4.camel@maboule>
References: <1171314114.6259.4.camel@maboule>
Message-ID: <1d1e6ea70702210526k2130039wfb55fcc93ff6d33c@mail.gmail.com>

Hi Andre,
I've downloaded bpycdf and it works very nicely with  numpy; thanks
very much for all your effort.

One small problem; I'm probably being stupid, but I cannot see how to
set a _Fillvalue as Float32.

regards, George Nurser.

On 12/02/07, Andre Gosselin <gosselina at dfo-mpo.gc.ca> wrote:
> A small error slipped into the pycdf-0.6-3 release. Attribute deletion
> through del(), and dimension length inquiry through len() were missing
> in that release.
>
> A new pycdf-0.6.3b fixes those problems. I have withdrawn pycdf-0.6.3
> from Sourceforge.net . Those people who have already downloaded this
> release can safely continue to use it,if they do not mind missing the
> del() and len() features.
>
> Reagrds,
> Andre Gosselin
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From toon.knapen at fft.be  Wed Feb 21 09:56:03 2007
From: toon.knapen at fft.be (Toon Knapen)
Date: Wed, 21 Feb 2007 15:56:03 +0100
Subject: [Numpy-discussion] installation documentation
Message-ID: <45DC5D83.70603@fft.be>

Hi all,

Is there detailed info on the installation process available.

I'm asking because in addition to installing numpy on linux-x86, I'm 
also trying to install numpy on aix-power and windows-x86. So before 
bombarding the ml with questions, I would like to get my hands on all 
doc available (I already have the 'Guide to Numpy' of sept. 15 2006 but 
can not find all the info I'm looking for in there neither)

Thanks in advance,

Toon Knapen


From lxander.m at gmail.com  Wed Feb 21 12:39:59 2007
From: lxander.m at gmail.com (Alexander Michael)
Date: Wed, 21 Feb 2007 12:39:59 -0500
Subject: [Numpy-discussion] Managing Rolling Data
Message-ID: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>

I'm new to numpy and looking for advice on setting up and managing
array data for my particular problem. I'm collecting observations of P
properties for N objects over a rolling horizon of H sample times. I
could conceptually store the data in three-dimensional array with
shape (N,P,H) that would allow me to easily (and efficiently with
strided slices) compute the statistics over both N and H that I am
interested in. This is great, but the rub is that H, an interval of T,
 is a rolling horizon. T is to large to fit in memory, so I need to
load up H, perform my calculations, pop the oldest N x P slice and
push the newest N x P slice into the data cube. What's the best way to
do this that will maintain fast computations along the one-dimensional
slices over N and H? Is there a commonly accepted idiom?

Fundamentally, I see two solutions. The first would be to essentially
perform a memcpy to propagate the data. The second would be to manage
the N x P slices as H discontiguous memory blocks and merely reorder
the pointers with each new sample. Can I do either of these with
numpy?

Thanks,
Alex


From Chris.Barker at noaa.gov  Wed Feb 21 12:52:35 2007
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Wed, 21 Feb 2007 09:52:35 -0800
Subject: [Numpy-discussion] Greek Letters
In-Reply-To: <45DBA5BA.3080004@astraw.com>
References: <627102C921CD9745B070C3B10CB8199B010EBFBD@hardwire.esri.com>
	<45DBA213.9050805@gmail.com> <45DBA5BA.3080004@astraw.com>
Message-ID: <45DC86E3.4080501@noaa.gov>

Andrew Straw wrote:
>  Here's one that seems like
> it might work, but I haven't tried it yet: 
> http://software.jessies.org/terminator

Now if only there was a decent terminal emulator for Windows that didn't 
use cygwin...

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From Chris.Barker at noaa.gov  Wed Feb 21 13:14:15 2007
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Wed, 21 Feb 2007 10:14:15 -0800
Subject: [Numpy-discussion] [Matplotlib-users] [matplotlib-devel]
 Unifying numpy, scipy, and matplotlib docstring formats
In-Reply-To: <cd7634ce0702172341jfb16e96md36453e174de13a3@mail.gmail.com>
References: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com>
	<45D4ED02.1050908@noaa.gov>
	<ef5675f30702151607h187c69d4t6360d490a88292ce@mail.gmail.com>
	<45D50245.1080102@noaa.gov>
	<1e2af89e0702160148u6f3b4ce4re11423f4ca9a35ec@mail.gmail.com>
	<cd7634ce0702172341jfb16e96md36453e174de13a3@mail.gmail.com>
Message-ID: <45DC8BF7.6040609@noaa.gov>

There's probably a better forum for this conversation, but...

Barry Wark wrote:
> Perhaps we should consider two use cases: interactive use ala Matlab
> and larger code bases.

A couple key points -- yes, interactive use is different than larger 
code bases, but I think it's a "Bad Idea" to promite totally different 
coding styles for these cases for a couple reasons:

-- One usually is doing both at once. I was a long-time, every day 
Matlab user, and hardly did anything of consequence interactively. I 
learned very quickly that it made a whole lot more sense to write a five 
line script that I could save, edit, etc. than do stuff interactively. 
Once I got something working, parts of that five line script might get 
cut&pasted into "real" code.

I do still test one or two lines interactively, but even then, I want 
the style to be something I can put in my code.

2) consistency in docs and examples is important, recommending different 
styles for interactive and programming use is just going to confuse 
people more.

3) even for folks that do a lot of interactive use, they are likely to 
write larger scale code at some point, and then they would need to learn 
something new.

> In the first case, being able to import * saves
> a lot of typing

No, it saves a little typing, if you're using an OOO style anyway.

 > and the namespace polution problem isn't a big deal.

Yes, it can be. A good interactive environment will be able to do things 
like method and command completion -- namespace pollution keeps that 
from working well.

> Returning to the OP's questions, why couldn't both cases be helped by
> creating a "meta-package" for numpy, scipy, and matplotlib? For the
> sake of argument, lets call the package "plab". Existing code could be
> affected by changing the individual packages, but a package that
> essentially does
> 
> from pylab import *
> from numpy import *
> from scipy import *


The issue with this is that you've now hidden where things are coming 
from. People seeing examples using that package will have no idea where 
things come from.

and by the way, the current "pylab", as delivered with MPL, pretty much 
does this already. I think we need to move away from that, rather than 
putting even more into pylab.

Matthew Brett wrote:
>  The downside about making numpy / python like matlab is that
> you soon realize that you really have to think about your problems
> differently, and write code in a different way.

Good point. A part of good Pythonic code is namespaces and OOO style. 
New users might as well learn the whole pile at once.

That all being said, it would be nice to establish a standard convention 
for how to import the key packages. I use:

import numpy as N
import matplotlib as MPL

But I don't really care that much, if we can come to any kind of 
community consensus, I'll follow it. The goal would be for all docs, 
Wiki entries, examples on the mailing lists, etc. to use the same style.

-Chris






-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From wollez at gmx.net  Wed Feb 21 13:11:31 2007
From: wollez at gmx.net (WolfgangZillig)
Date: Wed, 21 Feb 2007 19:11:31 +0100
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
Message-ID: <eri20i$91i$1@sea.gmane.org>

Hi,

I'm quite new to numpy/scipy so please excuse if my problem is too obvious.

example code:

import numpy as n
print n.sin(n.pi)
print n.cos(n.pi/2.0)

results in:
1.22460635382e-016
6.12303176911e-017

I've expected something around 0. Can anybody explain what I am doing 
wrong here?



From peridot.faceted at gmail.com  Wed Feb 21 13:16:32 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 21 Feb 2007 13:16:32 -0500
Subject: [Numpy-discussion] Managing Rolling Data
In-Reply-To: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
References: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
Message-ID: <ce557a360702211016p1f73704ejecca37674fe8e988@mail.gmail.com>

On 21/02/07, Alexander Michael <lxander.m at gmail.com> wrote:
> I'm new to numpy and looking for advice on setting up and managing
> array data for my particular problem. I'm collecting observations of P
> properties for N objects over a rolling horizon of H sample times. I
> could conceptually store the data in three-dimensional array with
> shape (N,P,H) that would allow me to easily (and efficiently with
> strided slices) compute the statistics over both N and H that I am
> interested in. This is great, but the rub is that H, an interval of T,
>  is a rolling horizon. T is to large to fit in memory, so I need to
> load up H, perform my calculations, pop the oldest N x P slice and
> push the newest N x P slice into the data cube. What's the best way to
> do this that will maintain fast computations along the one-dimensional
> slices over N and H? Is there a commonly accepted idiom?
>
> Fundamentally, I see two solutions. The first would be to essentially
> perform a memcpy to propagate the data. The second would be to manage
> the N x P slices as H discontiguous memory blocks and merely reorder
> the pointers with each new sample. Can I do either of these with
> numpy?

Yes, and several other possibilities as well.

To do a memcpy, all you need is
buffer[...,:-1] = buffer[...,1:]
buffer[...,-1] = new_data()
somecomputation(buffer[13,5,:])

Discontiguous blocks are somewhat inconvenient; one of the key
assumptions of numpy is that memory is stored in contiguous,
homogeneous blocks. You can use python lists (which are lists of
pointers), though:
listofbuffers = listofbuffers[:-1]+(new_data(),)
Extracting slices is now more awkward, either
somecomputation([buffer[13,5] for buffer in listofbuffers])
or convert the list to an array (which involves copying all the elements):
buffer = array(listofbuffers)
somecomputation(buffer[13,5,:])

Alternatively, if most of your statistics don't care about the order
of the data, you could maintain a rolling buffer:
buffer[...,oldest] = new_data()
oldest += 1
oldest %= H
somecomputationwhereorderdoesntmatter(buffer[13,6,:])
somecomputation(concatenate((buffer[13,5,oldest:],buffer[13,5,:oldest])))
(this last copies the data for that one test.) Fancy indexing can also
be used here to pull out the elements in the right order, and if your
statistics can be easily rewritten to work on a wrapped array, this is
probably the most efficient.

Incidentally, if the array wants to be inhomogeneous along one
dimension, you can use recarrays (apparently; I've never investigated
them).

Good luck,
Anne


From charlesr.harris at gmail.com  Wed Feb 21 13:25:50 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 21 Feb 2007 11:25:50 -0700
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <eri20i$91i$1@sea.gmane.org>
References: <eri20i$91i$1@sea.gmane.org>
Message-ID: <e06186140702211025u57414ff0pc04fbf6ecbb613f3@mail.gmail.com>

On 2/21/07, WolfgangZillig <wollez at gmx.net> wrote:
>
> Hi,
>
> I'm quite new to numpy/scipy so please excuse if my problem is too
> obvious.
>
> example code:
>
> import numpy as n
> print n.sin(n.pi)
> print n.cos(n.pi/2.0)
>
> results in:
> 1.22460635382e-016
> 6.12303176911e-017
>
> I've expected something around 0. Can anybody explain what I am doing
> wrong here?


They *are* around zero. You are seeing rounding error, probably both in the
value of pi and the approximations used to evaluate the sin and cos, most
likely some rational min/max fit. I recall teaching PDEs to students who
used Mathematica and they had the same sort of problem because terms
involving sin(pi) didn't go exactly to zero. Made their fourier series not
quite match the text answers. If you want these sort of terms to be exact
you will have to use some sort of symbolic program.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070221/06e9173f/attachment.html>

From robert.kern at gmail.com  Wed Feb 21 13:26:39 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 21 Feb 2007 12:26:39 -0600
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <eri20i$91i$1@sea.gmane.org>
References: <eri20i$91i$1@sea.gmane.org>
Message-ID: <45DC8EDF.7000209@gmail.com>

WolfgangZillig wrote:
> Hi,
> 
> I'm quite new to numpy/scipy so please excuse if my problem is too obvious.
> 
> example code:
> 
> import numpy as n
> print n.sin(n.pi)
> print n.cos(n.pi/2.0)
> 
> results in:
> 1.22460635382e-016
> 6.12303176911e-017
> 
> I've expected something around 0. Can anybody explain what I am doing 
> wrong here?

Nothing. You are getting something around 0 to the limit of double-precision
floating point. numpy.pi cannot be exactly ?, but within about 1e-16 of it. That
imprecision carries through the computations.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From zpincus at stanford.edu  Wed Feb 21 13:26:58 2007
From: zpincus at stanford.edu (Zachary Pincus)
Date: Wed, 21 Feb 2007 10:26:58 -0800
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <eri20i$91i$1@sea.gmane.org>
References: <eri20i$91i$1@sea.gmane.org>
Message-ID: <42A22E13-899B-45B8-A447-AC09587BF866@stanford.edu>

Your results are indeed around zero.

 >>> numpy.allclose(0, 1.22460635382e-016)
True

It's not exactly zero because floating point math is in general not  
exact. You'll need to check out a reference about doing floating  
point operations numerically for more details, but in general you  
should not expect exact results due to the limited precision of any  
fixed-width digital representation of floats.

A corrolary: in general do not two floating-point values for equality  
-- use something like numpy.allclose. (Exception -- equality is  
expected if the exact sequence of operations to generate two numbers  
were identical.)

Zach Pincus

Program in Biomedical Informatics and Department of Biochemistry
Stanford University School of Medicine

On Feb 21, 2007, at 10:11 AM, WolfgangZillig wrote:

> Hi,
>
> I'm quite new to numpy/scipy so please excuse if my problem is too  
> obvious.
>
> example code:
>
> import numpy as n
> print n.sin(n.pi)
> print n.cos(n.pi/2.0)
>
> results in:
> 1.22460635382e-016
> 6.12303176911e-017
>
> I've expected something around 0. Can anybody explain what I am doing
> wrong here?
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From peridot.faceted at gmail.com  Wed Feb 21 13:27:16 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 21 Feb 2007 13:27:16 -0500
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <eri20i$91i$1@sea.gmane.org>
References: <eri20i$91i$1@sea.gmane.org>
Message-ID: <ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>

On 21/02/07, WolfgangZillig <wollez at gmx.net> wrote:
> Hi,
>
> I'm quite new to numpy/scipy so please excuse if my problem is too obvious.
>
> example code:
>
> import numpy as n
> print n.sin(n.pi)
> print n.cos(n.pi/2.0)
>
> results in:
> 1.22460635382e-016
> 6.12303176911e-017
>
> I've expected something around 0. Can anybody explain what I am doing
> wrong here?

Well, nothing. It is around zero. Try (without numpy)
>>> (1.+1e-16)-1.
0.0
That is, for floating-point numbers, 1e-16 is about the fractional
error you can expect from any calculation, and since pi is of order
unity, you should expect its representation to have about that big an
error on it; feed that through sin and you get an error of about the
same size.

Or, to see more clearly, try taking (on a pocket calculator, say)
sin(3.14) (or even sin(pi)). Roundoff error is a basic fact of life in
numerical computations.

Anne


From peridot.faceted at gmail.com  Wed Feb 21 13:29:24 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 21 Feb 2007 13:29:24 -0500
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <42A22E13-899B-45B8-A447-AC09587BF866@stanford.edu>
References: <eri20i$91i$1@sea.gmane.org>
	<42A22E13-899B-45B8-A447-AC09587BF866@stanford.edu>
Message-ID: <ce557a360702211029x54d0c168hc08dcf00685a5959@mail.gmail.com>

On 21/02/07, Zachary Pincus <zpincus at stanford.edu> wrote:

> A corrolary: in general do not two floating-point values for equality
> -- use something like numpy.allclose. (Exception -- equality is
> expected if the exact sequence of operations to generate two numbers
> were identical.)

I really can't agree that blindly using allclose() is a good idea. For
example, allclose(plancks_constant,0) and the difference leads to
quantum mechanics... you really have to know how much difference you
expect and how big your numbers are going to be.

Anne


From wollez at gmx.net  Wed Feb 21 13:36:16 2007
From: wollez at gmx.net (WolfgangZillig)
Date: Wed, 21 Feb 2007 19:36:16 +0100
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
Message-ID: <eri3f0$dam$1@sea.gmane.org>

Anne Archibald schrieb:
> On 21/02/07, WolfgangZillig <wollez at gmx.net> wrote:
>> Hi,
>>
>> I'm quite new to numpy/scipy so please excuse if my problem is too obvious.
>>
>> example code:
>>
>> import numpy as n
>> print n.sin(n.pi)
>> print n.cos(n.pi/2.0)
>>
>> results in:
>> 1.22460635382e-016
>> 6.12303176911e-017
>>
>> I've expected something around 0. Can anybody explain what I am doing
>> wrong here?
> 
> Well, nothing. It is around zero. Try (without numpy)
>>>> (1.+1e-16)-1.
> 0.0
> That is, for floating-point numbers, 1e-16 is about the fractional
> error you can expect from any calculation, and since pi is of order
> unity, you should expect its representation to have about that big an
> error on it; feed that through sin and you get an error of about the
> same size.
> 
> Or, to see more clearly, try taking (on a pocket calculator, say)
> sin(3.14) (or even sin(pi)). Roundoff error is a basic fact of life in
> numerical computations.
> 
> Anne


Thanks for your answers. I was already aware that it won't be exactly 0 
but it wasn't clear to me that the rounding precision is around 1e-16.

Thanks for your help!
Wolfgang



From zpincus at stanford.edu  Wed Feb 21 13:43:57 2007
From: zpincus at stanford.edu (Zachary Pincus)
Date: Wed, 21 Feb 2007 10:43:57 -0800
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <ce557a360702211029x54d0c168hc08dcf00685a5959@mail.gmail.com>
References: <eri20i$91i$1@sea.gmane.org>
	<42A22E13-899B-45B8-A447-AC09587BF866@stanford.edu>
	<ce557a360702211029x54d0c168hc08dcf00685a5959@mail.gmail.com>
Message-ID: <5D401631-E2FE-41B6-A65A-2CEFA2DE1A31@stanford.edu>

It's true -- blindly using allclose isn't a lot better than blindly  
using equality testing. (Though given the choice between blindly  
using one and blindly using the other, I'd still probably vote for  
allclose... it won't get you quantum mechanics, but it'll do fine for  
a lot of other things.)

On the other hand, *properly* using allclose (e.g. setting the  
absolute and expected relative error tolerances) is better than  
properly using equality testing because in many cases there is no  
proper use for equality testing.

Zach

On Feb 21, 2007, at 10:29 AM, Anne Archibald wrote:

> On 21/02/07, Zachary Pincus <zpincus at stanford.edu> wrote:
>
>> A corrolary: in general do not two floating-point values for equality
>> -- use something like numpy.allclose. (Exception -- equality is
>> expected if the exact sequence of operations to generate two numbers
>> were identical.)
>
> I really can't agree that blindly using allclose() is a good idea. For
> example, allclose(plancks_constant,0) and the difference leads to
> quantum mechanics... you really have to know how much difference you
> expect and how big your numbers are going to be.
>
> Anne
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From mike.ressler at alum.mit.edu  Wed Feb 21 14:13:26 2007
From: mike.ressler at alum.mit.edu (Mike Ressler)
Date: Wed, 21 Feb 2007 11:13:26 -0800
Subject: [Numpy-discussion] Managing Rolling Data
In-Reply-To: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
References: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
Message-ID: <268febdf0702211113h6ba7a1a3j76d434c7429db44b@mail.gmail.com>

On 2/21/07, Alexander Michael <lxander.m at gmail.com> wrote:
> ... T is to large to fit in memory, so I need to
> load up H, perform my calculations, pop the oldest N x P slice and
> push the newest N x P slice into the data cube. What's the best way to
> do this that will maintain fast computations along the one-dimensional
> slices over N and H? Is there a commonly accepted idiom?

Would loading your data via memmap, then slicing it, do your job
(using numpy.memmap)? I work on 12 GB files with 4 GB of memory, but
it is transparent to me since the OS takes care of moving data in and
out of memory. May not be the fastest solution possible, but for me it
is a case where dev time is more significant than run time.

Mike


-- 
mike.ressler at alum.mit.edu


From David.L.Goldsmith at noaa.gov  Wed Feb 21 14:24:19 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Wed, 21 Feb 2007 11:24:19 -0800
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <eri20i$91i$1@sea.gmane.org>
References: <eri20i$91i$1@sea.gmane.org>
Message-ID: <45DC9C63.2080408@noaa.gov>

As far as a computer is concerned, those numbers are "around" zero - 
"growing-up" w/ Matlab, e.g., one quickly learns to recognize these 
numbers for what they are.  One way to return zero for numbers like 
these is

if numpy.allclose(x, 0): return 0 (or 0*x to assure that 0 is the same 
type as x),

but caveat emptor: sometimes, of course, 1e-16 really is supposed to be 
1e-16, not just the best the algorithm can do to get to zero.  Also, 
(help me out here guys) I thought there was something like 
zeroifclose(x) which does the above, or does numpy only have realifclose 
to return a real when an imaginary part is close to zero?

DG

WolfgangZillig wrote:
> Hi,
>
> I'm quite new to numpy/scipy so please excuse if my problem is too obvious.
>
> example code:
>
> import numpy as n
> print n.sin(n.pi)
> print n.cos(n.pi/2.0)
>
> results in:
> 1.22460635382e-016
> 6.12303176911e-017
>
> I've expected something around 0. Can anybody explain what I am doing 
> wrong here?
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From lxander.m at gmail.com  Wed Feb 21 14:46:51 2007
From: lxander.m at gmail.com (Alexander Michael)
Date: Wed, 21 Feb 2007 14:46:51 -0500
Subject: [Numpy-discussion] Managing Rolling Data
In-Reply-To: <268febdf0702211113h6ba7a1a3j76d434c7429db44b@mail.gmail.com>
References: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
	<268febdf0702211113h6ba7a1a3j76d434c7429db44b@mail.gmail.com>
Message-ID: <525f23e80702211146s441cdcedn4eca9c573646eba4@mail.gmail.com>

On 2/21/07, Mike Ressler <mike.ressler at alum.mit.edu> wrote:
> Would loading your data via memmap, then slicing it, do your job
> (using numpy.memmap)? ...

Interesting idea. I think Anne's suggestion that sliced assignment
will reduce to an efficient memcpy fits my needs a bit better than
memmap because I'll be pushing new N x P samples into the array that
will be arriving while the monitor is running.

Actually, I'm hoping sliced self-assignment is as efficient as memmove
(i.e. without creating temporaries), since the dst and src are
overlapping, but I haven't tried it yet to confirm if this is
relatively efficient.

Thank you both for your ideas,
Alex


From David.L.Goldsmith at noaa.gov  Wed Feb 21 14:52:39 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Wed, 21 Feb 2007 11:52:39 -0800
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <ce557a360702211029x54d0c168hc08dcf00685a5959@mail.gmail.com>
References: <eri20i$91i$1@sea.gmane.org>
	<42A22E13-899B-45B8-A447-AC09587BF866@stanford.edu>
	<ce557a360702211029x54d0c168hc08dcf00685a5959@mail.gmail.com>
Message-ID: <45DCA307.9050301@noaa.gov>

Anne Archibald wrote:
> On 21/02/07, Zachary Pincus <zpincus at stanford.edu> wrote:
>
>   
>> A corrolary: in general do not two floating-point values for equality
>> -- use something like numpy.allclose. (Exception -- equality is
>> expected if the exact sequence of operations to generate two numbers
>> were identical.)
>>     
>
> I really can't agree that blindly using allclose() is a good idea. For
> example, allclose(plancks_constant,0) and the difference leads to
> quantum mechanics... you really have to know how much difference you
> expect and how big your numbers are going to be.
>
> Anne
>   
"Precisely!" ;-)

Last time we had a posting like this, didn't one of the respondents 
include a link to something within the numpy Web docs that talks about 
floating point precision?

DG
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From Chris.Barker at noaa.gov  Wed Feb 21 14:54:14 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 21 Feb 2007 11:54:14 -0800
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
Message-ID: <45DCA366.1030104@noaa.gov>

Anne Archibald wrote:
> Or, to see more clearly, try taking (on a pocket calculator, say)
> sin(3.14) (or even sin(pi)).

This is an interesting point. I took a class from William Kahan once 
(pass/fail, thank god!), and one question he posed to us was:

How many digits of pi is used in an HP calculator?

I never figured out the answer myself, and the question involved the 
info that certain calculations involving pi being accurate to a certain 
degree. I don't have my HP calculator with me right now but I suspect 
that sin(pi) might well evaluate to exactly zero, or closer than 1e-16 
anyway.

I wonder if there are any C math libs that do a better job than you'd 
expect from standard FP? (short of unlimited precision ones)

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From oliphant at ee.byu.edu  Wed Feb 21 14:55:20 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed, 21 Feb 2007 12:55:20 -0700
Subject: [Numpy-discussion] installation documentation
In-Reply-To: <45DC5D83.70603@fft.be>
References: <45DC5D83.70603@fft.be>
Message-ID: <45DCA3A8.6050103@ee.byu.edu>

Toon Knapen wrote:

>Hi all,
>
>Is there detailed info on the installation process available.
>
>I'm asking because in addition to installing numpy on linux-x86, I'm 
>also trying to install numpy on aix-power and windows-x86. So before 
>bombarding the ml with questions, I would like to get my hands on all 
>doc available (I already have the 'Guide to Numpy' of sept. 15 2006 but 
>can not find all the info I'm looking for in there neither)
>
>  
>
 
If you don't care about using "vendor-specific math libraries" then it's 
as easy as

python setup.py install

on linux and aix (it's also that easy on Windows if you have the right 
compiler and/or configuration file set up).

If you care about vendor specific libraries, then you have to either put 
them in a "known" place or write a site.cfg file.   (We should document 
where the "known" places are...)

The installation process uses distutils and so information about 
distutils that you can glean from other places is also relevant.

Ask on the mailing list for more specifics.

-Travis





From oliphant at ee.byu.edu  Wed Feb 21 14:59:27 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed, 21 Feb 2007 12:59:27 -0700
Subject: [Numpy-discussion] Managing Rolling Data
In-Reply-To: <ce557a360702211016p1f73704ejecca37674fe8e988@mail.gmail.com>
References: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
	<ce557a360702211016p1f73704ejecca37674fe8e988@mail.gmail.com>
Message-ID: <45DCA49F.7050005@ee.byu.edu>

Anne Archibald wrote:

>Discontiguous blocks are somewhat inconvenient; one of the key
>assumptions of numpy is that memory is stored in contiguous,
>homogeneous blocks. 
>

Not to add anything really useful to this discussion, but I should 
correct this wording before it gives incorrect conceptions.  Actually 
the key assumption NumPy makes is that memory is accessible through 
uniform "striding" and not that it is necessarily contiguous.   You are 
correct, however that each element of the array must be of the same 
"data-type."

>Incidentally, if the array wants to be inhomogeneous along one
>dimension, you can use recarrays (apparently; I've never investigated
>them).
>  
>

Your arrays are still "homogeneous" when you use record arrays (i.e. 
each element of the array is still the same "data-type")  It's just that 
the data-type can be complicated.  You can have arrays whose elements 
consist of a 4-byte float and a 3-byte string for example.

-Travis



From robert.kern at gmail.com  Wed Feb 21 15:08:44 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 21 Feb 2007 14:08:44 -0600
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <45DCA366.1030104@noaa.gov>
References: <eri20i$91i$1@sea.gmane.org>	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
	<45DCA366.1030104@noaa.gov>
Message-ID: <45DCA6CC.8060906@gmail.com>

Christopher Barker wrote:
> I wonder if there are any C math libs that do a better job than you'd 
> expect from standard FP? (short of unlimited precision ones)

With respect to ? and the zeros of sin() and cos()? Not really. If
numpy.sin(numpy.pi) were to give you 0.0, it would be *wrong*. numpy.sin() is
supposed to give you the most accurate result representable in double-precision
for the input you gave it. numpy.pi is not ?.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From David.L.Goldsmith at noaa.gov  Wed Feb 21 15:27:02 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Wed, 21 Feb 2007 12:27:02 -0800
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <45DCA6CC.8060906@gmail.com>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
	<45DCA366.1030104@noaa.gov> <45DCA6CC.8060906@gmail.com>
Message-ID: <45DCAB16.6090304@noaa.gov>

Robert Kern wrote:
> Christopher Barker wrote:
>   
>> I wonder if there are any C math libs that do a better job than you'd 
>> expect from standard FP? (short of unlimited precision ones)
>>     
>
> With respect to ? and the zeros of sin() and cos()? Not really. If
> numpy.sin(numpy.pi) were to give you 0.0, it would be *wrong*. numpy.sin() is
> supposed to give you the most accurate result representable in double-precision
> for the input you gave it. numpy.pi is not ?.
>
>   
True, but since the string literal "numpy.pi" is not the same as the 
string literal "3.1415926535897931" there's no reason in principle why 
"numpy.py" couldn't symbolically represent the exact value; of course, 
since numpy does not otherwise support symbolic computation, such an 
adoption would be frivolous and rather silly (but then the Pythons were 
never afraid of being silly). :-)


DG


From Chris.Barker at noaa.gov  Wed Feb 21 15:27:56 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 21 Feb 2007 12:27:56 -0800
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <45DCA6CC.8060906@gmail.com>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
	<45DCA366.1030104@noaa.gov> <45DCA6CC.8060906@gmail.com>
Message-ID: <45DCAB4C.2060805@noaa.gov>

Robert Kern wrote:
> Christopher Barker wrote:
>> I wonder if there are any C math libs that do a better job than you'd 
>> expect from standard FP? (short of unlimited precision ones)
> 
> With respect to ? and the zeros of sin() and cos()? Not really. If
> numpy.sin(numpy.pi) were to give you 0.0, it would be *wrong*. numpy.sin() is
> supposed to give you the most accurate result representable in double-precision
> for the input you gave it.

But does it?

> numpy.pi is not ?.

More precisely, it's the best IEEE754 64 bit FP approximation of pi.

Right. I think that was the trick that HP used -- they somehow stored 
and worked with pi with more digits. The things you can do if you're 
making dedicated hardware.

I do wonder if there would be some way to use the extended precision 
built in to Intel FP hardware -- i.e. have a pi that you can pass in 
that has the full 80 bits that can be used internally. I don't know if 
the trig functions can be done with extended precision though.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From David.L.Goldsmith at noaa.gov  Wed Feb 21 15:40:58 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Wed, 21 Feb 2007 12:40:58 -0800
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <45DCAB4C.2060805@noaa.gov>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
	<45DCA366.1030104@noaa.gov> <45DCA6CC.8060906@gmail.com>
	<45DCAB4C.2060805@noaa.gov>
Message-ID: <45DCAE5A.6010309@noaa.gov>

I grew up a TI guy - my recollection is that they stated in the user 
manual that though the display could show "only" 10 decimal digits, 
memory saved and computations used 16; perhaps nowadays it is even more, 
but unless you're doing millions of sequential calculations (how often 
do you do that on a handheld scientific calculator?) you shouldn't be 
seeing cumulative error problems, right?

DG

Christopher Barker wrote:
> Robert Kern wrote:
>   
>> Christopher Barker wrote:
>>     
>>> I wonder if there are any C math libs that do a better job than you'd 
>>> expect from standard FP? (short of unlimited precision ones)
>>>       
>> With respect to ? and the zeros of sin() and cos()? Not really. If
>> numpy.sin(numpy.pi) were to give you 0.0, it would be *wrong*. numpy.sin() is
>> supposed to give you the most accurate result representable in double-precision
>> for the input you gave it.
>>     
>
> But does it?
>
>   
>> numpy.pi is not ?.
>>     
>
> More precisely, it's the best IEEE754 64 bit FP approximation of pi.
>
> Right. I think that was the trick that HP used -- they somehow stored 
> and worked with pi with more digits. The things you can do if you're 
> making dedicated hardware.
>
> I do wonder if there would be some way to use the extended precision 
> built in to Intel FP hardware -- i.e. have a pi that you can pass in 
> that has the full 80 bits that can be used internally. I don't know if 
> the trig functions can be done with extended precision though.
>
> -Chris
>
>
>   



From lxander.m at gmail.com  Wed Feb 21 15:48:35 2007
From: lxander.m at gmail.com (Alexander Michael)
Date: Wed, 21 Feb 2007 15:48:35 -0500
Subject: [Numpy-discussion] Managing Rolling Data
In-Reply-To: <268febdf0702211113h6ba7a1a3j76d434c7429db44b@mail.gmail.com>
References: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
	<268febdf0702211113h6ba7a1a3j76d434c7429db44b@mail.gmail.com>
Message-ID: <525f23e80702211248v6e6e7039xd2a70b5fb900dabd@mail.gmail.com>

On 2/21/07, Mike Ressler <mike.ressler at alum.mit.edu> wrote:
> Would loading your data via memmap, then slicing it, do your job
> (using numpy.memmap)? ...

Interesting idea. I think Anne's suggestion that sliced assignment
will reduce to an efficient memcpy fits my needs a bit better than
memmap because I'll be pushing new N x P samples into the array that
will be arriving while the monitor is running.

Actually, I'm hoping sliced self-assignment is as efficient as memmove
(i.e. without creating temporaries), since the dst and src are
overlapping, but I haven't tried it yet to confirm if this is
relatively efficient.

Thank you both for your ideas,
Alex


From tim.hochberg at ieee.org  Wed Feb 21 16:35:36 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 21 Feb 2007 14:35:36 -0700
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <45DCA366.1030104@noaa.gov>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
	<45DCA366.1030104@noaa.gov>
Message-ID: <e4412d6b0702211335n3c22c2fdv29791fa68b7a3a8e@mail.gmail.com>

On 2/21/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
>
> Anne Archibald wrote:
> > Or, to see more clearly, try taking (on a pocket calculator, say)
> > sin(3.14) (or even sin(pi)).
>
> This is an interesting point. I took a class from William Kahan once
> (pass/fail, thank god!), and one question he posed to us was:
>
> How many digits of pi is used in an HP calculator?
>
> I never figured out the answer myself, and the question involved the
> info that certain calculations involving pi being accurate to a certain
> degree. I don't have my HP calculator with me right now but I suspect
> that sin(pi) might well evaluate to exactly zero, or closer than 1e-16
> anyway.


Not on my HP calcularor. However, mine is 20+ years old and the label has
worn off so I'm not even sure what model it is any more (15c?). However,
sin(pi) is -4.1e-10 on this calculator, FWIW.


I wonder if there are any C math libs that do a better job than you'd
> expect from standard FP? (short of unlimited precision ones)
>
> -Chris
>
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070221/a5a46693/attachment.html>

From robert.kern at gmail.com  Wed Feb 21 16:47:51 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 21 Feb 2007 15:47:51 -0600
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <45DCAB4C.2060805@noaa.gov>
References: <eri20i$91i$1@sea.gmane.org>	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>	<45DCA366.1030104@noaa.gov>
	<45DCA6CC.8060906@gmail.com> <45DCAB4C.2060805@noaa.gov>
Message-ID: <45DCBE07.7070809@gmail.com>

Christopher Barker wrote:
> Robert Kern wrote:
>> Christopher Barker wrote:
>>> I wonder if there are any C math libs that do a better job than you'd 
>>> expect from standard FP? (short of unlimited precision ones)
>> With respect to ? and the zeros of sin() and cos()? Not really.

I'll back off on this a little bit. There are some approaches that will work;
they're not floating point, but they're not really "symbolic" computation either.

  http://keithbriggs.info/xrc.html

>> If
>> numpy.sin(numpy.pi) were to give you 0.0, it would be *wrong*. numpy.sin() is
>> supposed to give you the most accurate result representable in double-precision
>> for the input you gave it.
> 
> But does it?

Not quite, it seems, but 0 is even farther from the correct answer, apparently:

[sage-2.0-intelmac-i386-Darwin]$ ./sage
----------------------------------------------------------------------
| SAGE Version 2.0, Release Date: 2007-01-28                         |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------

sage: npi = RealNumber(3.1415926535897931, min_prec=53)
sage: npi.sin()
0.00000000000000013634246772254977
sage: import numpy
sage: numpy.sin(numpy.pi)
1.22460635382e-16

>> numpy.pi is not ?.
> 
> More precisely, it's the best IEEE754 64 bit FP approximation of pi.
> 
> Right. I think that was the trick that HP used -- they somehow stored 
> and worked with pi with more digits. The things you can do if you're 
> making dedicated hardware.
> 
> I do wonder if there would be some way to use the extended precision 
> built in to Intel FP hardware -- i.e. have a pi that you can pass in 
> that has the full 80 bits that can be used internally. I don't know if 
> the trig functions can be done with extended precision though.

Well, you can always use long double if it is implemented on your platform. You
will have to construct a value for ? yourself, though. I'm afraid that we don't
really make that easy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From peridot.faceted at gmail.com  Wed Feb 21 17:03:11 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 21 Feb 2007 17:03:11 -0500
Subject: [Numpy-discussion] Managing Rolling Data
In-Reply-To: <525f23e80702211146s441cdcedn4eca9c573646eba4@mail.gmail.com>
References: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
	<268febdf0702211113h6ba7a1a3j76d434c7429db44b@mail.gmail.com>
	<525f23e80702211146s441cdcedn4eca9c573646eba4@mail.gmail.com>
Message-ID: <ce557a360702211403p44688661uf1ae4c3f51609a08@mail.gmail.com>

On 21/02/07, Alexander Michael <lxander.m at gmail.com> wrote:
> On 2/21/07, Mike Ressler <mike.ressler at alum.mit.edu> wrote:
> > Would loading your data via memmap, then slicing it, do your job
> > (using numpy.memmap)? ...
>
> Interesting idea. I think Anne's suggestion that sliced assignment
> will reduce to an efficient memcpy fits my needs a bit better than
> memmap because I'll be pushing new N x P samples into the array that
> will be arriving while the monitor is running.

If you want a record of all of them on disk anyway, with careful
management you can read from a file as it's being written, though
you'll need some rather exotic numpy hackery to have an ever-growing
array. It might actually be nice to have a little wrapper object for
this sort of thing, as you can't be the only one who needs it.

> Actually, I'm hoping sliced self-assignment is as efficient as memmove
> (i.e. without creating temporaries), since the dst and src are
> overlapping, but I haven't tried it yet to confirm if this is
> relatively efficient.

I think it is almost as efficient as memmove; in particular, it
doesn't create any temporaries (be careful which way you do the
assignment, in fact, or you'll have an array of all the same thing)
but it does use a for loop in C (maybe several) instead of a
highly-optimized C library bulk copy. (I'm not sure about this - it
could be special-cased in the assignment code, but I don't think it
is.)

Anne


From peridot.faceted at gmail.com  Wed Feb 21 17:09:39 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 21 Feb 2007 17:09:39 -0500
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <45DCBE07.7070809@gmail.com>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
	<45DCA366.1030104@noaa.gov> <45DCA6CC.8060906@gmail.com>
	<45DCAB4C.2060805@noaa.gov> <45DCBE07.7070809@gmail.com>
Message-ID: <ce557a360702211409s5ed88636x9dfe48bf8881ce1d@mail.gmail.com>

On 21/02/07, Robert Kern <robert.kern at gmail.com> wrote:

> Well, you can always use long double if it is implemented on your platform. You
> will have to construct a value for ? yourself, though. I'm afraid that we don't
> really make that easy.

If the trig functions are implemented at all, you can probably use
atan2(-1,0) and get a decent approximation; alternatively, if you want
to use sin(pi)=0 as a definition you could use scipy's bisection
routine (which is datatype-independent, IIRC) to find pi. Or you could
probably use a rational approximation to pi, then convert numerator
and denominator to long doubles, or better yet compare the rational
number with your long double approximation of pi.

But no, not particularly easy.

Anne

From tim.hochberg at ieee.org  Wed Feb 21 17:20:25 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 21 Feb 2007 15:20:25 -0700
Subject: [Numpy-discussion] Managing Rolling Data
In-Reply-To: <ce557a360702211403p44688661uf1ae4c3f51609a08@mail.gmail.com>
References: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
	<268febdf0702211113h6ba7a1a3j76d434c7429db44b@mail.gmail.com>
	<525f23e80702211146s441cdcedn4eca9c573646eba4@mail.gmail.com>
	<ce557a360702211403p44688661uf1ae4c3f51609a08@mail.gmail.com>
Message-ID: <e4412d6b0702211420n74434813vd8cfb02e7dbd67f6@mail.gmail.com>

If none of the suggested methods turn out to be efficient enough due to
copying overhead, here's a way to reduce the copying overhead by trading
memory (and a bit of complexity) for copying overhead. The general thrust is
to allocate M extra slices of memory and then shift the data every M time
slices instead of every time slice.

First you would allocate a block of memory N*P*(H+M) in size.
>>> buffer = zeros([H+M,N,P], float)
Then you'd look at the first H time slices.
>>> data = buffer[:H]
Top pop one piece of data off the stack you'd simply shift data to look at a
different place in the buffer. The first time, you'd have something like
this:
>>> data = buffer[1:1+H]
Every M time steps you need to recopy the data. I expect that this should
reduce your copying overhead a bunch since your not copying as frequently.
It's pretty tunable too. You'd want to wrap some convenience functions
around stuff to automate the copying and popping, but that should be easy
enough.

I haven't tried this though, so caveat emptor.

-tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070221/34f8e67b/attachment.html>

From charlesr.harris at gmail.com  Wed Feb 21 18:24:47 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 21 Feb 2007 16:24:47 -0700
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <45DCBE07.7070809@gmail.com>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
	<45DCA366.1030104@noaa.gov> <45DCA6CC.8060906@gmail.com>
	<45DCAB4C.2060805@noaa.gov> <45DCBE07.7070809@gmail.com>
Message-ID: <e06186140702211524w7c0cd8a9h3b6ccec1a0fdc7d7@mail.gmail.com>

On 2/21/07, Robert Kern <robert.kern at gmail.com> wrote:
>
> Christopher Barker wrote:
> > Robert Kern wrote:
> >> Christopher Barker wrote:
> >>> I wonder if there are any C math libs that do a better job than you'd
> >>> expect from standard FP? (short of unlimited precision ones)
> >> With respect to ? and the zeros of sin() and cos()? Not really.


<snip>

Well, you can always use long double if it is implemented on your platform.
> You
> will have to construct a value for ? yourself, though. I'm afraid that we
> don't
> really make that easy.
>
> --


pi = 3. 1415926535 8979323846 2643383279 5028841971 6939937510 5820974944
5923078164 0628620899 8628034825 3421170679 8214808651 *...

*
I dont know what that looks like when converted to long double. Lessee,

In [1]: import numpy

In [2]: pi = numpy.float128(3.1415926535897932384626433832795028841971)

In [3]: numpy.pi - pi
Out[3]: 0.0

In [7]: '%25.20f'%numpy.pi
Out[7]: '   3.14159265358979311600'

In [8]: '%25.20f'%pi
Out[8]: '   3.14159265358979311600'

I think we have a bug. Or else extended arithmetic isn't supported on this
machine.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070221/e0f7a654/attachment.html>

From tim.hochberg at ieee.org  Wed Feb 21 18:33:32 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 21 Feb 2007 16:33:32 -0700
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <e06186140702211524w7c0cd8a9h3b6ccec1a0fdc7d7@mail.gmail.com>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
	<45DCA366.1030104@noaa.gov> <45DCA6CC.8060906@gmail.com>
	<45DCAB4C.2060805@noaa.gov> <45DCBE07.7070809@gmail.com>
	<e06186140702211524w7c0cd8a9h3b6ccec1a0fdc7d7@mail.gmail.com>
Message-ID: <e4412d6b0702211533y68c01b0dwd3c6e778173e5d39@mail.gmail.com>

On 2/21/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
>
>
>
> On 2/21/07, Robert Kern <robert.kern at gmail.com> wrote:
> >
> > Christopher Barker wrote:
> > > Robert Kern wrote:
> > >> Christopher Barker wrote:
> > >>> I wonder if there are any C math libs that do a better job than
> > you'd
> > >>> expect from standard FP? (short of unlimited precision ones)
> > >> With respect to ? and the zeros of sin() and cos()? Not really.
>
>
> <snip>
>
> Well, you can always use long double if it is implemented on your
> > platform. You
> > will have to construct a value for ? yourself, though. I'm afraid that
> > we don't
> > really make that easy.
> >
> > --
>
>
> pi = 3. 1415926535 8979323846 2643383279 5028841971 6939937510 5820974944
> 5923078164 0628620899 8628034825 3421170679 8214808651 *...
>
> *
> I dont know what that looks like when converted to long double. Lessee,
>
> In [1]: import numpy
>
> In [2]: pi = numpy.float128(3.1415926535897932384626433832795028841971)
>

I think this is where you go wrong. Your string of digits is first a python
float and *then* is converted to a long double. In the intermediate stage it
gets truncated and you don't get the precision back.


In [3]: numpy.pi - pi
> Out[3]: 0.0
>
> In [7]: '%25.20f'%numpy.pi
> Out[7]: '   3.14159265358979311600 '
>
> In [8]: '%25.20f'%pi
> Out[8]: '   3.14159265358979311600'
>
> I think we have a bug. Or else extended arithmetic isn't supported on this
> machine.
>
> Chuck
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070221/03b855f1/attachment.html>

From rowen at cesmail.net  Wed Feb 21 18:39:14 2007
From: rowen at cesmail.net (Russell E. Owen)
Date: Wed, 21 Feb 2007 15:39:14 -0800
Subject: [Numpy-discussion] Distributing prebuilt numpy and other
	extensions
References: <D3A11D21-A268-41B3-9819-3C6F13D0A8B5@stanford.edu>
Message-ID: <rowen-D38DF1.15391421022007@sea.gmane.org>

In article <D3A11D21-A268-41B3-9819-3C6F13D0A8B5 at stanford.edu>,
 Zachary Pincus <zpincus at stanford.edu> wrote:

> Hello folks,
> 
> I've developed some command-line tools for biologists using python/ 
> numpy and some custom C and Fortran extensions, and I'm trying to  
> figure out how to easily distribute them...
> 
> For people using linux, I figure a source distribution is no problem  
> at all. (Right?)
> On the other hand, for Mac users (whose computers by default don't  
> have the dev tools, and even then would need to get a fortran  
> compiler elsewhere) I'd like to figure out something a bit easier.
> 
> I'd like to somehow provide an installer (double-clickable or python  
> script) that does a version check and then installs an appropriate  
> version of prebuilt binaries for numpy and my C and Fortran  
> extensions. Is this possible within the bounds of the python or numpy  
> distutils? Would setuptools be a better way to go? Preferably it  
> would be a dead easy, one-step thing...
> 
> Or is this whole idea problematic, and better to stick with source  
> distribution in all cases?

As Robert Kern said, using bdist_mpkg is a nice easy way to create a 
double-clickable Mac installer for python code. It builds an installer 
package using the normal setup.py file for your stuff.

Lots of packages built this way are available at:
<http://pythonmac.org/packages>

But if you want one installer that installs everything then, you have to 
figure what to do if the user already has some of the your python 
packages installed (e.g. numpy). Overwrite the existing package? Somehow 
install it in parallel and have the user pick which version to use?

None of this is automated in bdist_mpkg. It is set up to install one 
python package at a time. So...

For your project I suspect you would be better off using easy_install
<http://peak.telecommunity.com/DevCenter/EasyInstall>
and packaging your project as a python "egg". easy_install is 
cross-platform, handles dependencies automatically and can install from 
source or precompiled binaries. That said, I've not actually used it 
except to install existing eggs, though I'd like to find some time to 
learn it.

-- Russell



From charlesr.harris at gmail.com  Wed Feb 21 18:43:21 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 21 Feb 2007 16:43:21 -0700
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <e4412d6b0702211533y68c01b0dwd3c6e778173e5d39@mail.gmail.com>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
	<45DCA366.1030104@noaa.gov> <45DCA6CC.8060906@gmail.com>
	<45DCAB4C.2060805@noaa.gov> <45DCBE07.7070809@gmail.com>
	<e06186140702211524w7c0cd8a9h3b6ccec1a0fdc7d7@mail.gmail.com>
	<e4412d6b0702211533y68c01b0dwd3c6e778173e5d39@mail.gmail.com>
Message-ID: <e06186140702211543k33586c00kfb29c484ee20b05@mail.gmail.com>

On 2/21/07, Timothy Hochberg <tim.hochberg at ieee.org> wrote:
>
>
>
> On 2/21/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
> >
> >
> >
> > On 2/21/07, Robert Kern < robert.kern at gmail.com> wrote:
> > >
> > > Christopher Barker wrote:
> > > > Robert Kern wrote:
> > > >> Christopher Barker wrote:
> > > >>> I wonder if there are any C math libs that do a better job than
> > > you'd
> > > >>> expect from standard FP? (short of unlimited precision ones)
> > > >> With respect to ? and the zeros of sin() and cos()? Not really.
> >
> >
> > <snip>
> >
> > Well, you can always use long double if it is implemented on your
> > > platform. You
> > > will have to construct a value for ? yourself, though. I'm afraid that
> > > we don't
> > > really make that easy.
> > >
> > > --
> >
> >
> > pi = 3. 1415926535 8979323846 2643383279 5028841971 6939937510
> > 5820974944 5923078164 0628620899 8628034825 3421170679 8214808651 *...
> >
> > *
> > I dont know what that looks like when converted to long double. Lessee,
> >
> > In [1]: import numpy
> >
> > In [2]: pi = numpy.float128(3.1415926535897932384626433832795028841971)
> >
>
> I think this is where you go wrong. Your string of digits is first a
> python float and *then* is converted to a long double. In the intermediate
> stage it gets truncated and you don't get the precision back.
>

True. But there is missing functionality here.

In [4]: pi = numpy.float128('3.1415926535897932384626433832795028841971')
---------------------------------------------------------------------------
exceptions.TypeError                                 Traceback (most recent
call last)

/home/charris/workspace/microsat/daemon/<ipython console>

TypeError: a float is required

It's somewhat pointless to have a data type that you can't properly
initialize. I think the string value should work, it works for python types.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070221/64b40658/attachment.html>

From charlesr.harris at gmail.com  Wed Feb 21 18:54:52 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 21 Feb 2007 16:54:52 -0700
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <e06186140702211543k33586c00kfb29c484ee20b05@mail.gmail.com>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
	<45DCA366.1030104@noaa.gov> <45DCA6CC.8060906@gmail.com>
	<45DCAB4C.2060805@noaa.gov> <45DCBE07.7070809@gmail.com>
	<e06186140702211524w7c0cd8a9h3b6ccec1a0fdc7d7@mail.gmail.com>
	<e4412d6b0702211533y68c01b0dwd3c6e778173e5d39@mail.gmail.com>
	<e06186140702211543k33586c00kfb29c484ee20b05@mail.gmail.com>
Message-ID: <e06186140702211554w5099e704oc8a008282d09041b@mail.gmail.com>

On 2/21/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
>
>
>
> On 2/21/07, Timothy Hochberg <tim.hochberg at ieee.org> wrote:
> >
> >
> >
> > On 2/21/07, Charles R Harris < charlesr.harris at gmail.com> wrote:
> > >
> > >
> > >
> > > On 2/21/07, Robert Kern < robert.kern at gmail.com> wrote:
> > > >
> > > > Christopher Barker wrote:
> > > > > Robert Kern wrote:
> > > > >> Christopher Barker wrote:
> > > > >>> I wonder if there are any C math libs that do a better job than
> > > > you'd
> > > > >>> expect from standard FP? (short of unlimited precision ones)
> > > > >> With respect to ? and the zeros of sin() and cos()? Not really.
> > >
> > >
> > > <snip>
> > >
> > > Well, you can always use long double if it is implemented on your
> > > > platform. You
> > > > will have to construct a value for ? yourself, though. I'm afraid
> > > > that we don't
> > > > really make that easy.
> > > >
> > > > --
> > >
> > >
> > > pi = 3. 1415926535 8979323846 2643383279 5028841971 6939937510
> > > 5820974944 5923078164 0628620899 8628034825 3421170679 8214808651 *...
> > >
> > > *
> > > I dont know what that looks like when converted to long double.
> > > Lessee,
> > >
> > > In [1]: import numpy
> > >
> > > In [2]: pi = numpy.float128(3.1415926535897932384626433832795028841971
> > > )
> > >
> >
> > I think this is where you go wrong. Your string of digits is first a
> > python float and *then* is converted to a long double. In the intermediate
> > stage it gets truncated and you don't get the precision back.
> >
>
> True. But there is missing functionality here.
>
> In [4]: pi = numpy.float128('3.1415926535897932384626433832795028841971')
> ---------------------------------------------------------------------------
>
> exceptions.TypeError                                 Traceback (most
> recent call last)
>
> /home/charris/workspace/microsat/daemon/<ipython console>
>
> TypeError: a float is required
>
> It's somewhat pointless to have a data type that you can't properly
> initialize. I think the string value should work, it works for python types.
>
>

OTOH, the old way does give extended precision for pi

In [8]: numpy.arctan2(numpy.float128(1), numpy.float128(1))*4
Out[8]: 3.14159265358979323851

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070221/3094fdef/attachment.html>

From cimrman3 at ntc.zcu.cz  Thu Feb 22 04:02:46 2007
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Thu, 22 Feb 2007 10:02:46 +0100
Subject: [Numpy-discussion] Managing Rolling Data
In-Reply-To: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
References: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
Message-ID: <45DD5C36.2030609@ntc.zcu.cz>

Alexander Michael wrote:
> I'm new to numpy and looking for advice on setting up and managing
> array data for my particular problem. I'm collecting observations of P
> properties for N objects over a rolling horizon of H sample times. I
> could conceptually store the data in three-dimensional array with
> shape (N,P,H) that would allow me to easily (and efficiently with
> strided slices) compute the statistics over both N and H that I am
> interested in. This is great, but the rub is that H, an interval of T,
>  is a rolling horizon. T is to large to fit in memory, so I need to
> load up H, perform my calculations, pop the oldest N x P slice and
> push the newest N x P slice into the data cube. What's the best way to
> do this that will maintain fast computations along the one-dimensional
> slices over N and H? Is there a commonly accepted idiom?
> 
> Fundamentally, I see two solutions. The first would be to essentially
> perform a memcpy to propagate the data. The second would be to manage
> the N x P slices as H discontiguous memory blocks and merely reorder
> the pointers with each new sample. Can I do either of these with
> numpy?

The 'pointers reordering' can be very nicely done via
deque from collections module - it behaves like a list, but has some
additional methods like rotate(). I have used it successfully as a
circular buffer for numpy arrays, but I can see that you need an
efficient slicing over H, so a contiguous memory would be better for you
IMHO.

r.


From sturla at molden.no  Thu Feb 22 07:29:46 2007
From: sturla at molden.no (Sturla Molden)
Date: Thu, 22 Feb 2007 13:29:46 +0100
Subject: [Numpy-discussion] Managing Rolling Data
In-Reply-To: <ce557a360702211403p44688661uf1ae4c3f51609a08@mail.gmail.com>
References: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>	<268febdf0702211113h6ba7a1a3j76d434c7429db44b@mail.gmail.com>	<525f23e80702211146s441cdcedn4eca9c573646eba4@mail.gmail.com>
	<ce557a360702211403p44688661uf1ae4c3f51609a08@mail.gmail.com>
Message-ID: <45DD8CBA.5020401@molden.no>

On 2/21/2007 11:03 PM, Anne Archibald wrote:

> I think it is almost as efficient as memmove; in particular, it
> doesn't create any temporaries 

A ring buffer is O(1) whereas a memmove is O(N). Unless the amount of 
data to be moved are very small, this makes the ringbuffer the more 
attractive solution.

Slicing becomes a little bit more complicated with a ring, but not very 
much. The data are stored in two ordered and contiguous segments: after 
and before the stack pointer (in that order).


Sturla Molden


From toon.knapen at fft.be  Thu Feb 22 09:59:53 2007
From: toon.knapen at fft.be (Toon Knapen)
Date: Thu, 22 Feb 2007 15:59:53 +0100
Subject: [Numpy-discussion] installation documentation
In-Reply-To: <45DCA3A8.6050103@ee.byu.edu>
References: <45DC5D83.70603@fft.be> <45DCA3A8.6050103@ee.byu.edu>
Message-ID: <45DDAFE9.2000503@fft.be>

Travis Oliphant wrote:
> If you don't care about using "vendor-specific math libraries" then it's 
> as easy as
> 
> python setup.py install
> 
> on linux and aix (it's also that easy on Windows if you have the right 
> compiler and/or configuration file set up).
> 
> If you care about vendor specific libraries, then you have to either put 
> them in a "known" place or write a site.cfg file.   (We should document 
> where the "known" places are...)
> 
> The installation process uses distutils and so information about 
> distutils that you can glean from other places is also relevant.
> 
> Ask on the mailing list for more specifics.
> 

Ok, here goes ;-)

First of all, when installing on windows-x86 using msvc7.1 and 
python-2.5, the build of numpy crashes because it was looking for 
python25.lib and was looking in the lib directory of the python 
installation. However, I compiled python-2.5 using the solution-file 
that comes with it and this one generated the python25.lib file in the 
pcbuild subdirectory (and does not copy it to the lib directory). After 
copying it manually, the build worked.

However when trying to use vendor specific libraries, I run into 
trouble. First of all, on linux-x86 I think I have set it up right but 
how do I verify this? I have the same problem as mentioned in:
http://projects.scipy.org/pipermail/numpy-discussion/2007-February/026130.html
I also found numpy.__config__.show() but this does not show any tuned 
lapack library although I had one in my site.cfg. Any ideas ?

Thanks in advance,

Toon
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: setup.log
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070222/1b5e9748/attachment.ksh>

From lxander.m at gmail.com  Thu Feb 22 11:23:58 2007
From: lxander.m at gmail.com (Alexander Michael)
Date: Thu, 22 Feb 2007 11:23:58 -0500
Subject: [Numpy-discussion] Managing Rolling Data
In-Reply-To: <45DD8CBA.5020401@molden.no>
References: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
	<268febdf0702211113h6ba7a1a3j76d434c7429db44b@mail.gmail.com>
	<525f23e80702211146s441cdcedn4eca9c573646eba4@mail.gmail.com>
	<ce557a360702211403p44688661uf1ae4c3f51609a08@mail.gmail.com>
	<45DD8CBA.5020401@molden.no>
Message-ID: <525f23e80702220823k199edf8awa77b762c875fc515@mail.gmail.com>

On 2/22/07, Sturla Molden <sturla at molden.no> wrote:
> A ring buffer is O(1) whereas a memmove is O(N). Unless the amount of
> data to be moved are very small, this makes the ringbuffer the more
> attractive solution.
>
> Slicing becomes a little bit more complicated with a ring, but not very
> much. The data are stored in two ordered and contiguous segments: after
> and before the stack pointer (in that order).

I like the ring buffer idea as well. As order matters for some of my
operations, I will need to put the two sides together somehow, but
want to avoid blindly copying into a new array every time I slice on
H. I would also like to avoid writing wrappers for every possible
array operation. Does numpy provide a facility for creating a
concatenated "view" that can be handled like a "normal" array and only
gets copied into contiguous memory when push comes to shove?

This discussion has been really helpful for me as a numpy neophyte,
thanks everyone!

Alex


From David.L.Goldsmith at noaa.gov  Thu Feb 22 13:29:47 2007
From: David.L.Goldsmith at noaa.gov (David L Goldsmith)
Date: Thu, 22 Feb 2007 10:29:47 -0800
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <e06186140702211543k33586c00kfb29c484ee20b05@mail.gmail.com>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
	<45DCA366.1030104@noaa.gov> <45DCA6CC.8060906@gmail.com>
	<45DCAB4C.2060805@noaa.gov> <45DCBE07.7070809@gmail.com>
	<e06186140702211524w7c0cd8a9h3b6ccec1a0fdc7d7@mail.gmail.com>
	<e4412d6b0702211533y68c01b0dwd3c6e778173e5d39@mail.gmail.com>
	<e06186140702211543k33586c00kfb29c484ee20b05@mail.gmail.com>
Message-ID: <45DDE11B.8010308@noaa.gov>

I agree w/ Chuck - I'd consider what Tim describes is happening a "bug".

DG

Charles R Harris wrote:
>
>
> On 2/21/07, *Timothy Hochberg* <tim.hochberg at ieee.org 
> <mailto:tim.hochberg at ieee.org>> wrote:
>
>
>
>     On 2/21/07, *Charles R Harris* < charlesr.harris at gmail.com
>     <mailto:charlesr.harris at gmail.com>> wrote:
>
>
>
>         On 2/21/07, *Robert Kern* < robert.kern at gmail.com
>         <mailto:robert.kern at gmail.com>> wrote:
>
>             Christopher Barker wrote:
>             > Robert Kern wrote:
>             >> Christopher Barker wrote:
>             >>> I wonder if there are any C math libs that do a better
>             job than you'd
>             >>> expect from standard FP? (short of unlimited precision
>             ones)
>             >> With respect to ? and the zeros of sin() and cos()? Not
>             really.
>
>
>         <snip>
>
>             Well, you can always use long double if it is implemented
>             on your platform. You
>             will have to construct a value for ? yourself, though. I'm
>             afraid that we don't
>             really make that easy.
>
>             --
>
>
>         pi = 3. 1415926535 8979323846 2643383279 5028841971 6939937510
>         5820974944 5923078164 0628620899 8628034825 3421170679
>         8214808651 *...
>
>         *
>         I dont know what that looks like when converted to long
>         double. Lessee,
>
>         In [1]: import numpy
>
>         In [2]: pi =
>         numpy.float128(3.1415926535897932384626433832795028841971)
>
>
>     I think this is where you go wrong. Your string of digits is first
>     a python float and *then* is converted to a long double. In the
>     intermediate stage it gets truncated and you don't get the
>     precision back.
>
>
> True. But there is missing functionality here.
>
> In [4]: pi = numpy.float128('3.1415926535897932384626433832795028841971')
> --------------------------------------------------------------------------- 
>
> exceptions.TypeError                                 Traceback (most 
> recent call last)
>
> /home/charris/workspace/microsat/daemon/<ipython console>
>
> TypeError: a float is required
>  
> It's somewhat pointless to have a data type that you can't properly 
> initialize. I think the string value should work, it works for python 
> types.
>
> Chuck
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   


-- 
ERD/ORR/NOS/NOAA <http://response.restoration.noaa.gov/emergencyresponse/>


From Chris.Barker at noaa.gov  Thu Feb 22 13:46:01 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 22 Feb 2007 10:46:01 -0800
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <45DDE11B.8010308@noaa.gov>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
	<45DCA366.1030104@noaa.gov> <45DCA6CC.8060906@gmail.com>
	<45DCAB4C.2060805@noaa.gov> <45DCBE07.7070809@gmail.com>
	<e06186140702211524w7c0cd8a9h3b6ccec1a0fdc7d7@mail.gmail.com>
	<e4412d6b0702211533y68c01b0dwd3c6e778173e5d39@mail.gmail.com>
	<e06186140702211543k33586c00kfb29c484ee20b05@mail.gmail.com>
	<45DDE11B.8010308@noaa.gov>
Message-ID: <45DDE4E9.7080900@noaa.gov>

David L Goldsmith wrote:
> I agree w/ Chuck - I'd consider what Tim describes is happening a "bug".

It's not a bug, but it is a missing feature. numpy doesn't appear to 
convert strings to numbers for any of its own types:

 >>> N.float128("4.3")
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: a float is required
 >>> N.uint8("4")
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ValueError: setting an array element with a sequence.
 >>> N.uint16("4")
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ValueError: setting an array element with a sequence.
 >>> N.int16("4")
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ValueError: setting an array element with a sequence.

It works for "float" and "int":
 >>> N.float("4")
4.0
 >>> N.int("4")
4

Because these are built-in python types, and those constructors support 
initialization with strings.

Given that there are some numpy types that hold values that can not be 
initialized by standard python literals, it would be nice to be able to 
do this.

Oh, and here's another inconsistency:

 >>> N.int32("676")
676
 >>> N.uint32("676")
array([6, 7, 6], dtype=uint32)

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From cookedm at physics.mcmaster.ca  Thu Feb 22 15:23:32 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu, 22 Feb 2007 15:23:32 -0500
Subject: [Numpy-discussion] what goes wrong with cos(), sin()
In-Reply-To: <45DCA366.1030104@noaa.gov>
References: <eri20i$91i$1@sea.gmane.org>
	<ce557a360702211027tc727e32xdd70a670254592cf@mail.gmail.com>
	<45DCA366.1030104@noaa.gov>
Message-ID: <BC248C37-0E25-4799-AFE1-99D34A097C20@physics.mcmaster.ca>

On Feb 21, 2007, at 14:54 , Christopher Barker wrote:

> Anne Archibald wrote:
>> Or, to see more clearly, try taking (on a pocket calculator, say)
>> sin(3.14) (or even sin(pi)).
>
> This is an interesting point. I took a class from William Kahan once
> (pass/fail, thank god!), and one question he posed to us was:
>
> How many digits of pi is used in an HP calculator?

FWIW,

There are two data types for reals (at least on the HP 28 and 48  
series, and others in that line): a 12 decimal digit real used for  
communicating with the user, and an extended 15 decimal digit real  
used internally. All calculations are done in base 10.

The exponent e for the 12-digit real is in the range -500 < e < 500,  
and for the 15-digit, -50000 < e < 50000.

AFAIK, most of HP's calculators are like this.

-- 
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From myeates at jpl.nasa.gov  Thu Feb 22 17:50:16 2007
From: myeates at jpl.nasa.gov (Mathew Yeates)
Date: Thu, 22 Feb 2007 14:50:16 -0800
Subject: [Numpy-discussion] dumb question about creating a complex array
Message-ID: <45DE1E28.3070907@jpl.nasa.gov>

given an array of floats, 2N columns and M rows, where the elements 
A[r,2*j] and A[r,2*j+1] form the real and imaginary parts of a complex 
number ....... What is the simplest way to create a complex array? It's 
a fairly large array so I want to keep copying to a minimum.

(Actually, it's not a float array, its elements are byte sized, in case 
that matters)

Mathew

-- 






From stefan at sun.ac.za  Thu Feb 22 18:21:01 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Fri, 23 Feb 2007 01:21:01 +0200
Subject: [Numpy-discussion] dumb question about creating a complex array
In-Reply-To: <45DE1E28.3070907@jpl.nasa.gov>
References: <45DE1E28.3070907@jpl.nasa.gov>
Message-ID: <20070222232101.GE5912@mentat.za.net>

Hi Matthew

On Thu, Feb 22, 2007 at 02:50:16PM -0800, Mathew Yeates wrote:
> given an array of floats, 2N columns and M rows, where the elements 
> A[r,2*j] and A[r,2*j+1] form the real and imaginary parts of a complex 
> number ....... What is the simplest way to create a complex array? It's 
> a fairly large array so I want to keep copying to a minimum.
> 
> (Actually, it's not a float array, its elements are byte sized, in case 
> that matters)

That's a bit problematic, since numpy doesn't have Complex16.  If you
had two 32-bit floats, you could do:

In [14]: x = N.array([10,10,20,20],dtype=N.float32)

In [15]: x.view(N.complex64)
Out[15]: array([ 10.+10.j,  20.+20.j], dtype=complex64)

As things stand, you may have to copy:

In [16]: x = N.array([10,10,20,20],dtype=N.uint8)

In [19]: x.astype(N.float32).view(N.complex64)
Out[19]: array([ 10.+10.j,  20.+20.j], dtype=complex64)

Maybe someone else comes up with a better answer.

Cheers
St?fan


From stefan at sun.ac.za  Thu Feb 22 18:26:25 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Fri, 23 Feb 2007 01:26:25 +0200
Subject: [Numpy-discussion] dumb question about creating a complex array
In-Reply-To: <45DE1E28.3070907@jpl.nasa.gov>
References: <45DE1E28.3070907@jpl.nasa.gov>
Message-ID: <20070222232625.GF5912@mentat.za.net>

On Thu, Feb 22, 2007 at 02:50:16PM -0800, Mathew Yeates wrote:
> given an array of floats, 2N columns and M rows, where the elements 
> A[r,2*j] and A[r,2*j+1] form the real and imaginary parts of a complex 
> number ....... What is the simplest way to create a complex array? It's 
> a fairly large array so I want to keep copying to a minimum.
> 
> (Actually, it's not a float array, its elements are byte sized, in case 
> that matters)

One other solution may be to use record arrays:

In [40]: x = N.array([[10,10,20,20],[20,20,30,30]],N.uint8)

In [41]: y = x.view([('real',N.uint8),('imag',N.uint8)]).reshape(x.shape[0],-1)

In [42]: y['real']
Out[42]: 
array([[10, 20],
       [20, 30]], dtype=uint8)

In [43]: y['imag']
Out[43]: 
array([[10, 20],
       [20, 30]], dtype=uint8)

Then you can at least do some arithmetic manually without any copying
of data taking place.

Regards
St?fan


From Chris.Barker at noaa.gov  Thu Feb 22 18:28:22 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 22 Feb 2007 15:28:22 -0800
Subject: [Numpy-discussion] dumb question about creating a complex array
In-Reply-To: <45DE1E28.3070907@jpl.nasa.gov>
References: <45DE1E28.3070907@jpl.nasa.gov>
Message-ID: <45DE2716.4020105@noaa.gov>

Mathew Yeates wrote:
> given an array of floats, 2N columns and M rows, where the elements 
> A[r,2*j] and A[r,2*j+1] form the real and imaginary parts of a complex 
> number ....... What is the simplest way to create a complex array? It's 
> a fairly large array so I want to keep copying to a minimum.
> 
> (Actually, it's not a float array, its elements are byte sized, in case 
> that matters)

It does. If it was a float array, you may even be able to do it without 
any copying at all. Anyway, this should work:

 >>> a = N.array([[1,2,3,4],[2,3,4,5],[4,5,6,7]], dtype=N.byte)
 >>> a
array([[1, 2, 3, 4],
        [2, 3, 4, 5],
        [4, 5, 6, 7]], dtype=int8)
# have I got that right?
 >>> b = N.empty((a.shape[0],a.shape[1]/2), dtype=N.complex)
 >>> b.real = a[:,range(0,a.shape[1],2)]
 >>> b.imag = a[:,range(1,a.shape[1],2)]
 >>> b
array([[ 1.+2.j,  3.+4.j],
        [ 2.+3.j,  4.+5.j],
        [ 4.+5.j,  6.+7.j]])


Is that what you wanted?

By the way, I think there is a trick for doing the every other column 
trick without using range(), but I can't find it at the moment.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From myeates at jpl.nasa.gov  Thu Feb 22 18:52:06 2007
From: myeates at jpl.nasa.gov (Mathew Yeates)
Date: Thu, 22 Feb 2007 15:52:06 -0800
Subject: [Numpy-discussion] dumb question about creating a complex array
In-Reply-To: <45DE2716.4020105@noaa.gov>
References: <45DE1E28.3070907@jpl.nasa.gov> <45DE2716.4020105@noaa.gov>
Message-ID: <45DE2CA6.60109@jpl.nasa.gov>

Thanks for the replies. I think I have enough to work with.

Mathew


Christopher Barker wrote:
> Mathew Yeates wrote:
>   
>> given an array of floats, 2N columns and M rows, where the elements 
>> A[r,2*j] and A[r,2*j+1] form the real and imaginary parts of a complex 
>> number ....... What is the simplest way to create a complex array? It's 
>> a fairly large array so I want to keep copying to a minimum.
>>
>> (Actually, it's not a float array, its elements are byte sized, in case 
>> that matters)
>>     
>
> It does. If it was a float array, you may even be able to do it without 
> any copying at all. Anyway, this should work:
>
>  >>> a = N.array([[1,2,3,4],[2,3,4,5],[4,5,6,7]], dtype=N.byte)
>  >>> a
> array([[1, 2, 3, 4],
>         [2, 3, 4, 5],
>         [4, 5, 6, 7]], dtype=int8)
> # have I got that right?
>  >>> b = N.empty((a.shape[0],a.shape[1]/2), dtype=N.complex)
>  >>> b.real = a[:,range(0,a.shape[1],2)]
>  >>> b.imag = a[:,range(1,a.shape[1],2)]
>  >>> b
> array([[ 1.+2.j,  3.+4.j],
>         [ 2.+3.j,  4.+5.j],
>         [ 4.+5.j,  6.+7.j]])
>
>
> Is that what you wanted?
>
> By the way, I think there is a trick for doing the every other column 
> trick without using range(), but I can't find it at the moment.
>
> -Chris
>
>
>
>   


-- 






From HansJoachim.Ehlers at eumetsat.int  Fri Feb 23 07:06:28 2007
From: HansJoachim.Ehlers at eumetsat.int (Hans-Joachim Ehlers)
Date: Fri, 23 Feb 2007 13:06:28 +0100
Subject: [Numpy-discussion] installating numpy 1.0.1 on AIX 5.3 using python
	2.5 and xlf	v10.1
Message-ID: <45DEE6D4020000AE000045A8@gwn0201.eumetsat.int>

Given :
AIX 5.3
xlf v10.1
python 2.5 
numpy 1.0.1

Task:
Building/installaing numpy on AIX 5.3 

Problem:
The install process can not determine the IBM Fortran compiler
Running   python   ./numpy/distutils/fcompiler/ibm.py gives pretty much
garbage on the screen and the resolut is 'None' .
So i changed the line in ./numpy/distutils/fcompiler/ibm.py
from
        'version_cmd'  : ["xlf"],
to
        'version_cmd'  : ["xlf -qversion"],

since xlf v10 supports the option where xlf v8 does not.

Now running  python   ./numpy/distutils/fcompiler/ibm.py gives me:

customize IbmFCompiler
Couldn't match compiler version for 'IBM XL Fortran Enterprise Edition
V10.1 for AIX \nVersion: 10.01.0000.0004'
None

Could anybody give me a solution/hint since i do not know what else has
to be changed to make a match. BTW: I am not a python programmer. I just
try to build numpy.

Tia
Hajo



From lxander.m at gmail.com  Fri Feb 23 11:09:29 2007
From: lxander.m at gmail.com (Alexander Michael)
Date: Fri, 23 Feb 2007 11:09:29 -0500
Subject: [Numpy-discussion] Managing Rolling Data
In-Reply-To: <525f23e80702220823k199edf8awa77b762c875fc515@mail.gmail.com>
References: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
	<268febdf0702211113h6ba7a1a3j76d434c7429db44b@mail.gmail.com>
	<525f23e80702211146s441cdcedn4eca9c573646eba4@mail.gmail.com>
	<ce557a360702211403p44688661uf1ae4c3f51609a08@mail.gmail.com>
	<45DD8CBA.5020401@molden.no>
	<525f23e80702220823k199edf8awa77b762c875fc515@mail.gmail.com>
Message-ID: <525f23e80702230809t4a4620a7mf0752c25e0b6b3cb@mail.gmail.com>

Timothy's refinement of Anne's idea will work for me:

>>> import timeit
>>> print '%.2fms/push' % (1000 * timeit.Timer(
...    "a[...,:-1] = a[...,1:]",
...    "from numpy import empty; a = empty((5000,20,1000))"
...    ).timeit(number=10)/10)

537.86ms/push

I still find the ring buffer solution appealing, but I did not see a
way to stack two arrays together without creating copies. Am I missing
a bit of numpy cleverness?

Thank you everyone for your help,
Alex


From joris at ster.kuleuven.ac.be  Fri Feb 23 11:38:19 2007
From: joris at ster.kuleuven.ac.be (joris at ster.kuleuven.ac.be)
Date: Fri, 23 Feb 2007 17:38:19 +0100
Subject: [Numpy-discussion] the neighbourhood of each element of an array
Message-ID: <1172248699.45df187b7f9f6@webmail.ster.kuleuven.be>

Hi,

Given a (possibly masked) 2d array x, is there a fast(er) way in Numpy to obtain
the same result as the following few lines?

d = 1                                  # neighbourhood 'radius'
Nrow = x.shape[0]
Ncol = x.shape[1]
y = array([[x[i-d:i+d+1,j-d:j+d+1].ravel() for j in range(d,Ncol-d)]      \
                                           for i in range(d,Nrow-d)])
                 
What you get is an array containing all the elements in a neighbourhood for each
element, disregarding the edges to avoid out-of-range problems. The code above
becomes quite slow for e.g. a 2000x2000 array. Does anyone know a better
approach?

Ciao,
Joris



Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm



From peridot.faceted at gmail.com  Fri Feb 23 13:26:45 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 23 Feb 2007 13:26:45 -0500
Subject: [Numpy-discussion] Managing Rolling Data
In-Reply-To: <525f23e80702230809t4a4620a7mf0752c25e0b6b3cb@mail.gmail.com>
References: <525f23e80702210939r1ed80363q789feb8626de0461@mail.gmail.com>
	<268febdf0702211113h6ba7a1a3j76d434c7429db44b@mail.gmail.com>
	<525f23e80702211146s441cdcedn4eca9c573646eba4@mail.gmail.com>
	<ce557a360702211403p44688661uf1ae4c3f51609a08@mail.gmail.com>
	<45DD8CBA.5020401@molden.no>
	<525f23e80702220823k199edf8awa77b762c875fc515@mail.gmail.com>
	<525f23e80702230809t4a4620a7mf0752c25e0b6b3cb@mail.gmail.com>
Message-ID: <ce557a360702231026m6825aab3rc2f0346668eee24@mail.gmail.com>

On 23/02/07, Alexander Michael <lxander.m at gmail.com> wrote:

> I still find the ring buffer solution appealing, but I did not see a
> way to stack two arrays together without creating copies. Am I missing
> a bit of numpy cleverness?

The short answer is no; the stride in memory from one element to the
next must always be the same in an array, and so it's impossible to
join two arrays like that without copying.

There is a ring-buffer-like idea that lets you do less copying; if you
need slices of length N, you can choose M>=2N and maintain a
ringbuffer of length M instead; when you push something into the
M-N+jth position, you also push (i.e., copy) it into the jth position.
Then you can always take a contiguous slice starting anywhere from 0
to M-N-1, and you need only copy a small fraction (N/M or so) of the
values pushed on. You never move values in the ringbuffer, and there
are no big spikes in CPU time.

Anne


From faltet at carabos.com  Fri Feb 23 14:18:48 2007
From: faltet at carabos.com (Francesc Altet)
Date: Fri, 23 Feb 2007 20:18:48 +0100
Subject: [Numpy-discussion] the neighbourhood of each element of an array
In-Reply-To: <1172248699.45df187b7f9f6@webmail.ster.kuleuven.be>
References: <1172248699.45df187b7f9f6@webmail.ster.kuleuven.be>
Message-ID: <200702232018.50199.faltet@carabos.com>

A Divendres 23 Febrer 2007 17:38, joris at ster.kuleuven.ac.be escrigu?:
> Hi,
>
> Given a (possibly masked) 2d array x, is there a fast(er) way in Numpy to
> obtain the same result as the following few lines?
>
> d = 1                                  # neighbourhood 'radius'
> Nrow = x.shape[0]
> Ncol = x.shape[1]
> y = array([[x[i-d:i+d+1,j-d:j+d+1].ravel() for j in range(d,Ncol-d)]      \
>                                            for i in range(d,Nrow-d)])
>
> What you get is an array containing all the elements in a neighbourhood for
> each element, disregarding the edges to avoid out-of-range problems. The
> code above becomes quite slow for e.g. a 2000x2000 array. Does anyone know
> a better approach?

Well, it seems that copying data here is taking most of the CPU. Perhaps you 
may want to try getting *references* to the original slices better. For 
example, if rd = 2+d, you can write:

def get_neighbors_views_ravel(x):
    # The next is for an array of references to *views* of neighborgs
    y = numpy.empty((Nrow-2*d, Ncol-2*d), dtype='object')

    for i in xrange(0, Nrow-2*d):
        x2 = x[i:i+rd]   # Get a view of the first dimension slice
        for j in xrange(0, Ncol-2*d):
            y[i, j] = x2[:,j:j+rd].ravel()
    return y

which is a 1.34x (on my machine) faster than your current approach.

If you want more speed, you may want to not .ravel() in the new array creation 
time (you can always use .ravel() when you are going to use the data). The 
removal of the .ravel() call makes the above function 2.56x faster.

Finally, if your machine has an x86 architecture, you can also take advantge 
of Psyco so as to accelerate a bit more. With Psyco and not raveling, you can 
get up to 3x better times than your original approach (without using Psyco). 
Of course, more speed-ups could be possible by using Pyrex or any other easy 
method for doing C-extensions.

I'm attaching a small benchmark, and here are the results for my machine:

ref time--> 3.021
views (ravel) time--> 2.258   speed-up--> 1.34
views (no ravel) time--> 1.179   speed-up--> 2.56

and if we use psyco:

ref time--> 2.368
views (ravel) time--> 1.636   speed-up--> 1.45
views (no ravel) time--> 0.935   speed-up--> 2.53

Cheers,

-- 
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: prova13.py
Type: application/x-python
Size: 1675 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070223/762867a5/attachment.bin>

From bryan at cole.uklinux.net  Fri Feb 23 14:39:00 2007
From: bryan at cole.uklinux.net (Bryan Cole)
Date: Fri, 23 Feb 2007 19:39:00 +0000
Subject: [Numpy-discussion] the neighbourhood of each element of an array
In-Reply-To: <1172248699.45df187b7f9f6@webmail.ster.kuleuven.be>
References: <1172248699.45df187b7f9f6@webmail.ster.kuleuven.be>
Message-ID: <1172259540.21554.14.camel@pc1.cole.uklinux.net>

On Fri, 2007-02-23 at 17:38 +0100, joris at ster.kuleuven.ac.be wrote:
> Hi,
> 
> Given a (possibly masked) 2d array x, is there a fast(er) way in Numpy to obtain
> the same result as the following few lines?
> 
> d = 1                                  # neighbourhood 'radius'
> Nrow = x.shape[0]
> Ncol = x.shape[1]
> y = array([[x[i-d:i+d+1,j-d:j+d+1].ravel() for j in range(d,Ncol-d)]      \
>                                            for i in range(d,Nrow-d)])
>                  

how about something like

er = Nrow - d
ec = Ncol - d
y = array([x[i:er+i, j:ec+j] for j in arange(-d,d)
				for i in arange(-d,d)])

now you're looping over a small array and combining slices of the big
array (as opposed to looping over the big array and combining slices
from a small one). This should be faster for large Nrow, Ncol.

BC


> What you get is an array containing all the elements in a neighbourhood for each
> element, disregarding the edges to avoid out-of-range problems. The code above
> becomes quite slow for e.g. a 2000x2000 array. Does anyone know a better
> approach?
> 
> Ciao,
> Joris
> 
> 
> 
> Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm




From zpincus at stanford.edu  Fri Feb 23 14:53:05 2007
From: zpincus at stanford.edu (Zachary Pincus)
Date: Fri, 23 Feb 2007 11:53:05 -0800
Subject: [Numpy-discussion] the neighbourhood of each element of an array
In-Reply-To: <1172259540.21554.14.camel@pc1.cole.uklinux.net>
References: <1172248699.45df187b7f9f6@webmail.ster.kuleuven.be>
	<1172259540.21554.14.camel@pc1.cole.uklinux.net>
Message-ID: <A521F336-4C9C-4394-AA67-00ADBE24A49B@stanford.edu>

Scipy's ndimage module has a function that takes a generic callback  
and calls it with the values of each neighborhood (of a given size,  
and optionally with a particular "mask" footprint) centered on each  
array element. That function handles boundary conditions, etc nicely.

Unfortunately, I'm not sure if it works with masked arrays, and I  
think it hands a ravel'd set of pixels back to the callback function.  
You could probably hack masking in there by passing it the mask  
concatenated to the array, and then deal with the mask explicitly.

Zach


On Feb 23, 2007, at 11:39 AM, Bryan Cole wrote:

> On Fri, 2007-02-23 at 17:38 +0100, joris at ster.kuleuven.ac.be wrote:
>> Hi,
>>
>> Given a (possibly masked) 2d array x, is there a fast(er) way in  
>> Numpy to obtain
>> the same result as the following few lines?
>>
>> d = 1                                  # neighbourhood 'radius'
>> Nrow = x.shape[0]
>> Ncol = x.shape[1]
>> y = array([[x[i-d:i+d+1,j-d:j+d+1].ravel() for j in range(d,Ncol- 
>> d)]      \
>>                                            for i in range(d,Nrow-d)])
>>
>
> how about something like
>
> er = Nrow - d
> ec = Ncol - d
> y = array([x[i:er+i, j:ec+j] for j in arange(-d,d)
> 				for i in arange(-d,d)])
>
> now you're looping over a small array and combining slices of the big
> array (as opposed to looping over the big array and combining slices
> from a small one). This should be faster for large Nrow, Ncol.
>
> BC
>
>
>> What you get is an array containing all the elements in a  
>> neighbourhood for each
>> element, disregarding the edges to avoid out-of-range problems.  
>> The code above
>> becomes quite slow for e.g. a 2000x2000 array. Does anyone know a  
>> better
>> approach?
>>
>> Ciao,
>> Joris
>>
>>
>>
>> Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From pgmdevlist at gmail.com  Fri Feb 23 15:19:17 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Fri, 23 Feb 2007 15:19:17 -0500
Subject: [Numpy-discussion] the neighbourhood of each element of an array
In-Reply-To: <A521F336-4C9C-4394-AA67-00ADBE24A49B@stanford.edu>
References: <1172248699.45df187b7f9f6@webmail.ster.kuleuven.be>
	<1172259540.21554.14.camel@pc1.cole.uklinux.net>
	<A521F336-4C9C-4394-AA67-00ADBE24A49B@stanford.edu>
Message-ID: <200702231519.18510.pgmdevlist@gmail.com>

On Friday 23 February 2007 14:53:05 Zachary Pincus wrote:
> Scipy's ndimage module has a function that takes a generic callback
> and calls it with the values of each neighborhood (of a given size,
> and optionally with a particular "mask" footprint) centered on each
> array element. That function handles boundary conditions, etc nicely.
>
> Unfortunately, I'm not sure if it works with masked arrays, and I
> think it hands a ravel'd set of pixels back to the callback function.
> You could probably hack masking in there by passing it the mask
> concatenated to the array, and then deal with the mask explicitly.

Without really thinking about it:
The easiest would be to process the masked array in steps:
* process the _data part of the maskedarray (or its filled version) with the 
function: that will be your new _data. 
* if the mask is not nomask, process the _mask part of the maskedarray to get 
a new _mask
* Set to True any element of the new mask that contains a True value: in other 
terms, mask the values that have a masked neighbor. 


From a.schmolck at gmx.net  Fri Feb 23 16:19:05 2007
From: a.schmolck at gmx.net (Alexander Schmolck)
Date: 23 Feb 2007 21:19:05 +0000
Subject: [Numpy-discussion] Fortran order arrays to and from numpy arrays
Message-ID: <yfsodnklghi.fsf@oc.ex.ac.uk>

Hi,

I'm currently puzzling over how to best convert (column major order) matlab
arrays to numpy arrays and vice versa -- I'm looking for a solution that's
simple, general and reasonably fast -- being also applicable to Numeric arrays
would be a plus (I'd like to retain Numeric compatibility for the current
version of the code).

The solution that mlabwrap (http://mlabwrap.sf.net) currently employs to
transfer arrays between matlab and python is to have a low level C++ extension
(mlabraw) copying the data into a new numpy array created with
Pyarray_SimpleNew and using a simple or nested for-loop to copy the data
around (only rank up to 2 is currently handled), e.g.
    
    //matlab matrix -> numpy
    for (mwIndex lCol = 0; lCol < lCols; lCol++) {
      double *lDst = (double *)(lRetval->data) + lCol;
      for (mwIndex lRow = 0; lRow < lRows; lRow++, lDst += lCols) {
        *lDst = *lPR++;}}

    //numpy 2d array -> matlab
    npy_intp lRowDelta = pStrides[0]/sizeof(T);
    npy_intp lColDelta = pStrides[1]/sizeof(T);
    for (npy_intp lCol=0; lCol < pCols; lCol++) {
      T *lSrc = pSrc + lCol*lColDelta;
      for (npy_intp lRow=0; lRow < pRows; lRow++, lSrc += lRowDelta) {
        *pDst++ = *lSrc;}}

Some simple timing results suggest 2 things:

1. There is an enormous overhead associated with calling into matlab (e.g.
   just using mlabraw to evaluate ``sin(1);`` in matlab, without accessing the
   result the result is about 1000 times slower than ``math.sin(1)``. I have
   no idea where this rather huge overhead comes from, but I suspect there
   isn't much one can do about it.

2. Despite this overhead, copying around large arrays (e.g. >=1e5 elements) in
   above way causes notable additional overhead. Whilst I don't think there's
   a sane way to avoid copying by sharing data between numpy and matlab the
   copying could likely be done better.

I guess the moral from 1 might be that ctypes will be the way to go for
future version, I would assume that the overhead associated with ctypes
doesn't matter that much and that it will be much less painful to maintain
than having a C or C++ extension module, especially given that the Mathworks
seem fond of changing the C API every minor release. I haven't used ctypes yet
so I'd be interested to know what people think.

The moral from 2. might be just to memcpy the data wholesale where possible
and transpose and reshape afterwards or, in numpy, to set the fortran-order
flag (and query fortran-contiguousness on copying to matlab) -- I suppose
that specifying fortran order and tranposing/reshaping are pretty much
equivalent? So matlab to numpy would look roughly so:

  a_fortran_order = memcpy_raw_matlab_data_into_numpy_array(ptr)
  a = a_fortran_order.T.reshape(a_fortran_order.shape)

I would assume that this ought be on average more efficient than always
transforming to C-contiguous (as some of the time the returned array will
anyway be transposed afterwards or accessed in a fashion were column major is
faster; the case that several operations that profit from row-major storage
are carried out on the same array seems unlikely).

Unfortunately I don't see an easy way to use the same approach the other way
(matlab doesn't seem to offer much on the C level to manipulate arrays), so
I'd presumably need something like:

  stuff_into_matlab_array(a.T.reshape(a.shape).copy())

the question is how to avoid doing two copies.

Any comments appreciated,

'as






From charlesr.harris at gmail.com  Fri Feb 23 16:39:23 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 23 Feb 2007 14:39:23 -0700
Subject: [Numpy-discussion] Fortran order arrays to and from numpy arrays
In-Reply-To: <yfsodnklghi.fsf@oc.ex.ac.uk>
References: <yfsodnklghi.fsf@oc.ex.ac.uk>
Message-ID: <e06186140702231339l57e3caf4l39514724c4e782f5@mail.gmail.com>

On 23 Feb 2007 21:19:05 +0000, Alexander Schmolck <a.schmolck at gmx.net>
wrote:
>
> Hi,
>
> I'm currently puzzling over how to best convert (column major order)
> matlab
> arrays to numpy arrays and vice versa -- I'm looking for a solution that's
> simple, general and reasonably fast -- being also applicable to Numeric
> arrays
> would be a plus (I'd like to retain Numeric compatibility for the current
> version of the code).
>
> The solution that mlabwrap (http://mlabwrap.sf.net) currently employs to
> transfer arrays between matlab and python is to have a low level C++
> extension
> (mlabraw) copying the data into a new numpy array created with
> Pyarray_SimpleNew and using a simple or nested for-loop to copy the data
> around (only rank up to 2 is currently handled), e.g.
>
>     //matlab matrix -> numpy
>     for (mwIndex lCol = 0; lCol < lCols; lCol++) {
>       double *lDst = (double *)(lRetval->data) + lCol;
>       for (mwIndex lRow = 0; lRow < lRows; lRow++, lDst += lCols) {
>         *lDst = *lPR++;}}
>
>     //numpy 2d array -> matlab
>     npy_intp lRowDelta = pStrides[0]/sizeof(T);
>     npy_intp lColDelta = pStrides[1]/sizeof(T);
>     for (npy_intp lCol=0; lCol < pCols; lCol++) {
>       T *lSrc = pSrc + lCol*lColDelta;
>       for (npy_intp lRow=0; lRow < pRows; lRow++, lSrc += lRowDelta) {
>         *pDst++ = *lSrc;}}
>
> Some simple timing results suggest 2 things:
>
> 1. There is an enormous overhead associated with calling into matlab (e.g.
>    just using mlabraw to evaluate ``sin(1);`` in matlab, without accessing
> the
>    result the result is about 1000 times slower than ``math.sin(1)``. I
> have
>    no idea where this rather huge overhead comes from, but I suspect there
>    isn't much one can do about it.
>
> 2. Despite this overhead, copying around large arrays (e.g. >=1e5
> elements) in
>    above way causes notable additional overhead. Whilst I don't think
> there's
>    a sane way to avoid copying by sharing data between numpy and matlab
> the
>    copying could likely be done better.
>
> I guess the moral from 1 might be that ctypes will be the way to go for
> future version, I would assume that the overhead associated with ctypes
> doesn't matter that much and that it will be much less painful to maintain
> than having a C or C++ extension module, especially given that the
> Mathworks
> seem fond of changing the C API every minor release. I haven't used ctypes
> yet
> so I'd be interested to know what people think.
>
> The moral from 2. might be just to memcpy the data wholesale where
> possible
> and transpose and reshape afterwards or, in numpy, to set the
> fortran-order
> flag (and query fortran-contiguousness on copying to matlab) -- I suppose
> that specifying fortran order and tranposing/reshaping are pretty much
> equivalent? So matlab to numpy would look roughly so:
>
>   a_fortran_order = memcpy_raw_matlab_data_into_numpy_array(ptr)
>   a = a_fortran_order.T.reshape(a_fortran_order.shape)
>
> I would assume that this ought be on average more efficient than always
> transforming to C-contiguous (as some of the time the returned array will
> anyway be transposed afterwards or accessed in a fashion were column major
> is
> faster; the case that several operations that profit from row-major
> storage
> are carried out on the same array seems unlikely).
>
> Unfortunately I don't see an easy way to use the same approach the other
> way
> (matlab doesn't seem to offer much on the C level to manipulate arrays),
> so
> I'd presumably need something like:
>
>   stuff_into_matlab_array(a.T.reshape(a.shape).copy())
>
> the question is how to avoid doing two copies.
>
> Any comments appreciated,


The easiest way to deal with the ordering is to use the order keyword in
numpy:

In [4]: a = array([0,1,2,3]).reshape((2,2), order='F')

In [5]: a
Out[5]:
array([[0, 2],
       [1, 3]])

You would still need to get access to something to reshape, shared memory or
something, but the key is that you don't have to reorder the elements, you
just need the correct strides and offsets to address the elements in Fortran
order. I have no idea if this works in numeric.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070223/8c828dc3/attachment.html>

From strawman at astraw.com  Sat Feb 24 03:46:32 2007
From: strawman at astraw.com (Andrew Straw)
Date: Sat, 24 Feb 2007 00:46:32 -0800
Subject: [Numpy-discussion] Fortran order arrays to and from numpy arrays
In-Reply-To: <yfsodnklghi.fsf@oc.ex.ac.uk>
References: <yfsodnklghi.fsf@oc.ex.ac.uk>
Message-ID: <45DFFB68.5010303@astraw.com>

Alexander Schmolck wrote:
> 2. Despite this overhead, copying around large arrays (e.g. >=1e5 elements) in
>    above way causes notable additional overhead. Whilst I don't think there's
>    a sane way to avoid copying by sharing data between numpy and matlab the
>    copying could likely be done better.
>   
Alex, what do you think about "hybrid arrays"?

http://www.mail-archive.com/numpy-discussion at lists.sourceforge.net/msg03748.html


From david at ar.media.kyoto-u.ac.jp  Sat Feb 24 09:25:38 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 24 Feb 2007 23:25:38 +0900
Subject: [Numpy-discussion] How to tell numpy to use gfortran as a compiler ?
Message-ID: <45E04AE2.6020009@ar.media.kyoto-u.ac.jp>

Hi,

    I try to compile numpy using gfortran, using:

    python setup.py config --fcompiler=gnu

    But this does not work. Whatever option I try, numpy build system 
uses g77, and as a result, I have problems with my ATLAS library 
compiled with gfortran. What should I do to compiler numpy with 
blas/lapack compiler with gfortran ?

    cheers,

    David


From robert.kern at gmail.com  Sat Feb 24 10:45:15 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 24 Feb 2007 09:45:15 -0600
Subject: [Numpy-discussion] How to tell numpy to use gfortran as a
 compiler ?
In-Reply-To: <45E04AE2.6020009@ar.media.kyoto-u.ac.jp>
References: <45E04AE2.6020009@ar.media.kyoto-u.ac.jp>
Message-ID: <45E05D8B.4070500@gmail.com>

David Cournapeau wrote:
> Hi,
> 
>     I try to compile numpy using gfortran, using:
> 
>     python setup.py config --fcompiler=gnu
> 
>     But this does not work. Whatever option I try, numpy build system 
> uses g77, and as a result, I have problems with my ATLAS library 
> compiled with gfortran. What should I do to compiler numpy with 
> blas/lapack compiler with gfortran ?

--fcompiler=gnu95

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From zpincus at stanford.edu  Sat Feb 24 15:24:25 2007
From: zpincus at stanford.edu (Zachary Pincus)
Date: Sat, 24 Feb 2007 12:24:25 -0800
Subject: [Numpy-discussion] Questions about cross-compiling extensions for
	mac-ppc and mac-intel
Message-ID: <A69A9808-77C3-4E0A-8EA5-B34566B391F9@stanford.edu>

Hi folks,

I've been doing a lot of web-reading on the subject, but have not  
been completely able to synthesize all of the disparate bits of  
advice about building python extensions as Mac-PPC and Mac-Intel fat  
binaries, so I'm turning to the wisdom of this list for a few questions.

My general goal is to make a double-clickable Mac installer of a set  
of tools built around numpy, numpy's distutils, a very hacked-up  
version of PIL, and some fortran code too. To this end, I need to  
figure out how to get the numpy distutils to cross-compile,  
generating PPC code and Intel code in separate builds -- and/or  
generating a universal binary all in one go. (I'd like to distribute  
a universal version of numpy, but I think that my own code needs to  
be built/distributed separately for each architecture due to endian- 
ness issues.)

Is there explicit support in distutils for this, or is it a matter of  
setting the proper environment variables to entice gcc and gfortran  
to generate code for a specific architecture?

One problem is that PIL is a tricky beast, even in the neutered form  
that I'm using it. It does a compile-time check for the endian-ness  
of the system, and a compile-time search for the zlib to use, both of  
which are problematic.

To address the former, I'd like to be able to (say) include something  
like 'config_endian --big' on the 'python setup.py' command-line, and  
have that information trickle down to the PIL config script (a few  
subpackages deep). Is this easy or possible?

To address the latter, I think I need to have the PIL extensions  
dynamically link against '/Developer/SDKs/MacOSX10.4u.sdk/usr/lib/ 
libz.dylib' which is the fat-binary version of the library, using the  
headers from '/Developer/SDKs/MacOSX10.4u.sdk/usr/include/zlib.h
'. Right now, PIL is using system_info from numpy.distutils to find  
the valid library paths on which libz and its headers might live.  
This is nice and more or less platform-neutral, which I like. How  
best should I convince/configure numpy.distutils.system_info to put '/ 
Developer/SDKs/MacOSX10.4u.sdk/usr/{lib,include}' on the output to  
get_include_dirs() and get_lib_dirs()?

Thanks for any advice or counsel,

Zach Pincus

Program in Biomedical Informatics and Department of Biochemistry
Stanford University School of Medicine



From a.schmolck at gmx.net  Sat Feb 24 20:40:33 2007
From: a.schmolck at gmx.net (Alexander Schmolck)
Date: 25 Feb 2007 01:40:33 +0000
Subject: [Numpy-discussion] Fortran order arrays to and from numpy arrays
In-Reply-To: <45DFFB68.5010303@astraw.com>
References: <yfsodnklghi.fsf@oc.ex.ac.uk> <45DFFB68.5010303@astraw.com>
Message-ID: <yfsps7zj9pq.fsf@oc.ex.ac.uk>

Andrew Straw <strawman at astraw.com> writes:

> Alexander Schmolck wrote:
> > 2. Despite this overhead, copying around large arrays (e.g. >=1e5 elements) in
> >    above way causes notable additional overhead. Whilst I don't think there's
> >    a sane way to avoid copying by sharing data between numpy and matlab the
> >    copying could likely be done better.
> >
> 
> Alex, what do you think about "hybrid arrays"?
> 
> http://www.mail-archive.com/numpy-discussion at lists.sourceforge.net/msg03748.html

Oh, that's why I said no *sane* way :) 

I read about hybrid arrays, but as far as I can tell the only way to use them
to avoid copying stuff around is to create your own hybrid array memory pool
(as you suggested downthread), which seems a highly unattractive pain-for-gain
trade-off, especially given that you *do* have to reorder the data as you go
from numpy to matlab -- unless of course I'm missing something. I have the
impression that just memcpy'ing a single chunk of data isn't that much of a
performance sink, but the reordering copying that mlabwrap currently does
seems rather expensive.

In other words, I think going from matlab to numpy is fine (just memcpy into a
newly created fortran-order numpy array, or more or less equivalently, memcpy
into a C-order array, transpose and reshape), the question appears to be how
to best go from numpy to matlab when the numpy array isn't fortran-contiguous.
I assume that it makes more sense to rely on some numpy functionality than to
use a custom reordering-copy routine, especially if I want to move to ctypes
later. Is there anything better than

1. allocating a matlab array
2. transposing and reshaping the numpy array
3. allocating (or keeping around) a temporary numpy array with data
   pointing to the matlab array data
4. using some function (PyArray_CopyInto?) to copy from the transposed,
   reshaped numpy array into the temporary numpy array thereby filling the
   matlab array with an appropriately reordered copy of the original array

?

cheers,

alex


From a.schmolck at gmx.net  Sat Feb 24 20:44:01 2007
From: a.schmolck at gmx.net (Alexander Schmolck)
Date: 25 Feb 2007 01:44:01 +0000
Subject: [Numpy-discussion] Fortran order arrays to and from numpy arrays
In-Reply-To: <e06186140702231339l57e3caf4l39514724c4e782f5@mail.gmail.com>
References: <yfsodnklghi.fsf@oc.ex.ac.uk>
	<e06186140702231339l57e3caf4l39514724c4e782f5@mail.gmail.com>
Message-ID: <yfsmz33j9jy.fsf@oc.ex.ac.uk>

"Charles R Harris" <charlesr.harris at gmail.com> writes:

> > Unfortunately I don't see an easy way to use the same approach the other
> > way
> > (matlab doesn't seem to offer much on the C level to manipulate arrays),
> > so
> > I'd presumably need something like:
> >
> >   stuff_into_matlab_array(a.T.reshape(a.shape).copy())
> >
> > the question is how to avoid doing two copies.
> >
> > Any comments appreciated,
> 
> 
> The easiest way to deal with the ordering is to use the order keyword in
> numpy:
> 
> In [4]: a = array([0,1,2,3]).reshape((2,2), order='F')
> 
> In [5]: a
> Out[5]:
> array([[0, 2],
>        [1, 3]])
> 
> You would still need to get access to something to reshape, shared memory or
> something, but the key is that you don't have to reorder the elements, you
> just need the correct strides and offsets to address the elements in Fortran
> order. I have no idea if this works in numeric.

It doesn't work in Numeric, but that isn't much of any issue because I think
it ought to be pretty much equivalent by transposing and reshaping. However
the problem is that I *do* need to reorder the elements for numpy->matlab and
I'm not sure how to best do this (without unnecessary copying and temporary
numpy array creation but using numpy functionality if possible).

'as


From charlesr.harris at gmail.com  Sun Feb 25 00:30:49 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 24 Feb 2007 22:30:49 -0700
Subject: [Numpy-discussion] Fortran order arrays to and from numpy arrays
In-Reply-To: <yfsmz33j9jy.fsf@oc.ex.ac.uk>
References: <yfsodnklghi.fsf@oc.ex.ac.uk>
	<e06186140702231339l57e3caf4l39514724c4e782f5@mail.gmail.com>
	<yfsmz33j9jy.fsf@oc.ex.ac.uk>
Message-ID: <e06186140702242130j1c32d8fcr75c3425c465e13a5@mail.gmail.com>

On 25 Feb 2007 01:44:01 +0000, Alexander Schmolck <a.schmolck at gmx.net>
wrote:
>
> "Charles R Harris" <charlesr.harris at gmail.com> writes:
>
> > > Unfortunately I don't see an easy way to use the same approach the
> other
> > > way
> > > (matlab doesn't seem to offer much on the C level to manipulate
> arrays),
> > > so
> > > I'd presumably need something like:
> > >
> > >   stuff_into_matlab_array(a.T.reshape(a.shape).copy())
> > >
> > > the question is how to avoid doing two copies.
> > >
> > > Any comments appreciated,
> >
> >
> > The easiest way to deal with the ordering is to use the order keyword in
> > numpy:
> >
> > In [4]: a = array([0,1,2,3]).reshape((2,2), order='F')
> >
> > In [5]: a
> > Out[5]:
> > array([[0, 2],
> >        [1, 3]])
> >
> > You would still need to get access to something to reshape, shared
> memory or
> > something, but the key is that you don't have to reorder the elements,
> you
> > just need the correct strides and offsets to address the elements in
> Fortran
> > order. I have no idea if this works in numeric.
>
> It doesn't work in Numeric, but that isn't much of any issue because I
> think
> it ought to be pretty much equivalent by transposing and reshaping.
> However
> the problem is that I *do* need to reorder the elements for numpy->matlab
> and
> I'm not sure how to best do this (without unnecessary copying and
> temporary
> numpy array creation but using numpy functionality if possible).


I don't see any way to get around a copy, but you can make numpy do the
work. For example:

In [12]: a = array([[0,1],[2,3]])

In [13]: b = array(a, order='f')

In [14]: a.flags
Out[14]:
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

In [15]: b.flags
Out[15]:
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

F_CONTIGUOUS is what you want. The trick is to somehow use memory in the
construction of the reordered array that is already designated for matlab. I
don't know how to do this, but I think it might be doable. Travis is your
best bet to answer that question.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070224/a79b75cc/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Sun Feb 25 01:41:01 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 25 Feb 2007 15:41:01 +0900
Subject: [Numpy-discussion] How to tell numpy to use gfortran as a
 compiler ?
In-Reply-To: <45E05D8B.4070500@gmail.com>
References: <45E04AE2.6020009@ar.media.kyoto-u.ac.jp>
	<45E05D8B.4070500@gmail.com>
Message-ID: <45E12F7D.30702@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
> David Cournapeau wrote:
>> Hi,
>>
>>     I try to compile numpy using gfortran, using:
>>
>>     python setup.py config --fcompiler=gnu
>>
>>     But this does not work. Whatever option I try, numpy build system 
>> uses g77, and as a result, I have problems with my ATLAS library 
>> compiled with gfortran. What should I do to compiler numpy with 
>> blas/lapack compiler with gfortran ?
>
> --fcompiler=gnu95
I also tried this option, without any luck. numpy still uses g77 (which 
is gcc 3.4 for fortran on my ubuntu system) instead of gfortran.

But now, I think I misunderstand some things : I thought that g77 was 
the 3.* version of the fortran compiler, and gfortran the 4.* one. But 
it looks like they are also different in the fortran dialect they are 
supporting (I know nothing about fortran). So should I use gfortran at 
all to compile the fortran wrapper for BLAS/LAPACK by ATLAS ? Or should 
I use g77 ? How can I be sure then that I won't have problems using gcc 
3* serie for Fortran and gcc 4* serie for everything else (C, C++ and 
most if not all libraries compiled on my system) ?

cheers,

David


From sturla at molden.no  Sun Feb 25 09:43:03 2007
From: sturla at molden.no (Sturla Molden)
Date: Sun, 25 Feb 2007 15:43:03 +0100 (CET)
Subject: [Numpy-discussion] How to tell numpy to use gfortran as a
 compiler ?
In-Reply-To: <45E12F7D.30702@ar.media.kyoto-u.ac.jp>
References: <45E04AE2.6020009@ar.media.kyoto-u.ac.jp>
	<45E05D8B.4070500@gmail.com> <45E12F7D.30702@ar.media.kyoto-u.ac.jp>
Message-ID: <4688.89.8.47.113.1172414583.squirrel@webmail.uio.no>

> Robert Kern wrote:

> But now, I think I misunderstand some things : I thought that g77 was
> the 3.* version of the fortran compiler, and gfortran the 4.* one. But
> it looks like they are also different in the fortran dialect they are
> supporting (I know nothing about fortran). So should I use gfortran at
> all to compile the fortran wrapper for BLAS/LAPACK by ATLAS ? Or should
> I use g77 ? How can I be sure then that I won't have problems using gcc
> 3* serie for Fortran and gcc 4* serie for everything else (C, C++ and
> most if not all libraries compiled on my system) ?


g77 is a Fortran 77 compiler. The development of g77 is halted.

gfortran is a Fortran 77, 90, and 95 compiler. It is the current Fortran
compiler in the GNU Compiler Collection (GCC).


You can compile the reference implementation of BLAS and LAPACK with both
g77 and gfortran, as these libraries are written in Fortran 77. ATLAS is
written in C and some Fortran 77.

gfortran are able to do some optimizations that g77 cannot, e.g.
autovectorization using SSE and MMX extensions and profile-guided
optimizations. Also be aware that if you use gfortran and GCC 4, the C
compiler is better as well.

There is a third Fortran compiler based on the GCC backend called "g95".
It is not a part of GCC and uses copyrighted source code illegally
(numerous GPL violations). The head developer is rumored to have serious
bad karma. Apart from that, g95 is an ok Fortran 95 compiler. Always
ensure that ambiguous switches like "gnu95" means gfortran and not g95. I
don't know what NumPy does.














From jks at iki.fi  Sun Feb 25 11:44:37 2007
From: jks at iki.fi (=?iso-8859-1?Q?Jouni_K=2E_Sepp=E4nen?=)
Date: Sun, 25 Feb 2007 18:44:37 +0200
Subject: [Numpy-discussion] Unifying numpy, scipy,
	and matplotlib docstring formats
References: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com>
	<45D4ED02.1050908@noaa.gov>
	<ef5675f30702151607h187c69d4t6360d490a88292ce@mail.gmail.com>
	<45D50245.1080102@noaa.gov>
	<1e2af89e0702160148u6f3b4ce4re11423f4ca9a35ec@mail.gmail.com>
	<cd7634ce0702172341jfb16e96md36453e174de13a3@mail.gmail.com>
	<1e2af89e0702180346g24f324a7je1f535f5fd6e0b96@mail.gmail.com>
	<cd7634ce0702181603t56baedf7hf8deeb8d468ca8b3@mail.gmail.com>
Message-ID: <m2bqjinq4q.fsf_-_@iki.fi>

"Barry Wark" <barrywark at gmail.com> writes:

> Yes, I agree. I wasn't coming at so much from the goal of making Pylab
> a Matlab clone (as you point out, that's silly, and misses much of the
> advantage of Python), but rather from the goal of making interactive
> use as efficient as possible. When I fire up ipython -pylab to do some
> quick exploration, it's nice not to have to type N.blah or pylab.plot

IMHO the greatest strength of Matlab in interactive use is the matrix
input format. For one thing, it is easier to type something like

  [0 1 0; 1 0 0; 0 0 1]

than

  array([[0,1,0],[1,0,0],[0,0,1]])

Granted, you can often leave out the array() wrapper, but typing all
the commas and brackets and getting the nesting right slows me down
enough that using Python feels like tedious work where Matlab is more
like an Emacs-like extension of the mind. Another neat feature is
auto-flattening: to e.g. add row- and column-wise sums and a grand
total to a matrix M, you can type

  [M sum(M,2); sum(M,1) sum(M(:))]

compared to which the r_[] and c_[] syntax feels like an ugly hack.

(Of course, the auto-flattening feature is a disaster for serious
programming (as opposed to quick interactive work), so Matlab has
added cell arrays which don't auto-flatten, leading to no end of
confusion between [] and {} indexing and the need to add {:} in
seemingly random spots in Matlab code.)

I suppose these things could be addressed quite neatly by IPython. 
It could even modify your history similarly to what it currently 
does with the %magic commands, so that when you type

  a = [0 1 0; 1 0 0; 0 0 1]

and then examine your history, you see

  a = array([[0,1,0],[1,0,0],[0,0,1]])

which you could copy-paste into the program you are developing.

Perhaps the namespace issue could also be addressed at the IPython
level. The pylab profile could import the various packages, perhaps
with some kind of abbreviated names, and rewrite commands like

  a = array(...)
  plot(sin(a))

to

  a = numpy.array(...)
  pylab.plot(numpy.sin(a))

so again you could copy/paste from the history to an editor and get
correctly Pythonic code without any "from ... import *". Probably a
100% solution is quite difficult because of the dynamic features in
Python, but it seems to me that a 80% solution should be feasible.
(Parse the input to an AST using the parser facility in the Python
library, use a tree walker to find all references to functions or
variables, and if they don't exist in locals() or globals() and are
not the target of an assignment anywhere in the AST, replace them by
references to the appropriate package.)

-- 
Jouni K. Sepp?nen
http://www.iki.fi/jks



From joris at ster.kuleuven.ac.be  Sun Feb 25 17:15:33 2007
From: joris at ster.kuleuven.ac.be (joris at ster.kuleuven.ac.be)
Date: Sun, 25 Feb 2007 23:15:33 +0100
Subject: [Numpy-discussion] the neighbourhood of each element of an array
In-Reply-To: <200702232018.50199.faltet@carabos.com>
References: <1172248699.45df187b7f9f6@webmail.ster.kuleuven.be>
	<200702232018.50199.faltet@carabos.com>
Message-ID: <1172441733.45e20a857395c@webmail.ster.kuleuven.be>


Thanks for all the useful comments. Some feedback about the improved version of
my code snippet. For a 40x40 matrix and d=1 the new version is 44 times faster,
and for d=2 it's 27 times faster. For my astronomical images (typical
2000x2000)
the new version saves my day.

# improved version
d = 1
Nrow = x.shape[0]
Ncol = x.shape[1]
s = 2*d+1
y = empty((s*s,Nrow-2*d,Ncol-2*d), dtype=x.dtype)    
for i in xrange(-d,d+1):
    for j in xrange(-d,d+1):
        y[(j+d)+s*(i+d)]= x[d+i:Nrow-d+i,d+j:Ncol-d+j]  
y = y.swapaxes(0,2).swapaxes(0,1)


The suggestion of Fransesc is a really cool one. But combining it with the
suggestion of Bryan does not seem possible in this particular case as the
swapaxis operations would no longer be possible as the resulting array would be
one with shape (s*s,) containing objects with shape (Nrow-2*d,Ncol-2*d).

Cheers,
Joris





Quoting Francesc Altet <faltet at carabos.com>:

> A Divendres 23 Febrer 2007 17:38, joris at ster.kuleuven.ac.be escrigu?:
> > Hi,
> >
> > Given a (possibly masked) 2d array x, is there a fast(er) way in Numpy to
> > obtain the same result as the following few lines?
> >
> > d = 1                                  # neighbourhood 'radius'
> > Nrow = x.shape[0]
> > Ncol = x.shape[1]
> > y = array([[x[i-d:i+d+1,j-d:j+d+1].ravel() for j in range(d,Ncol-d)]   \
> >                                            for i in range(d,Nrow-d)])
> >
> > What you get is an array containing all the elements in a neighbourhood
> for
> > each element, disregarding the edges to avoid out-of-range problems. The
> > code above becomes quite slow for e.g. a 2000x2000 array. Does anyone know
> > a better approach?
> 
> Well, it seems that copying data here is taking most of the CPU. Perhaps you
> 
> may want to try getting *references* to the original slices better. For 
> example, if rd = 2+d, you can write:
> 
> def get_neighbors_views_ravel(x):
>     # The next is for an array of references to *views* of neighborgs
>     y = numpy.empty((Nrow-2*d, Ncol-2*d), dtype='object')
> 
>     for i in xrange(0, Nrow-2*d):
>         x2 = x[i:i+rd]   # Get a view of the first dimension slice
>         for j in xrange(0, Ncol-2*d):
>             y[i, j] = x2[:,j:j+rd].ravel()
>     return y
> 
> which is a 1.34x (on my machine) faster than your current approach.
> 
> If you want more speed, you may want to not .ravel() in the new array
> creation 
> time (you can always use .ravel() when you are going to use the data). The 
> removal of the .ravel() call makes the above function 2.56x faster.
> 
> Finally, if your machine has an x86 architecture, you can also take advantge
> 
> of Psyco so as to accelerate a bit more. With Psyco and not raveling, you can
> 
> get up to 3x better times than your original approach (without using Psyco).
> 
> Of course, more speed-ups could be possible by using Pyrex or any other easy
> 
> method for doing C-extensions.
> 
> I'm attaching a small benchmark, and here are the results for my machine:
> 
> ref time--> 3.021
> views (ravel) time--> 2.258   speed-up--> 1.34
> views (no ravel) time--> 1.179   speed-up--> 2.56
> 
> and if we use psyco:
> 
> ref time--> 2.368
> views (ravel) time--> 1.636   speed-up--> 1.45
> views (no ravel) time--> 0.935   speed-up--> 2.53
> 
> Cheers,
> 
> -- 
> >0,0<   Francesc Altet ? ? http://www.carabos.com/
> V   V   C?rabos Coop. V. ??Enjoy Data
>  "-"
> 




Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm



From fperez.net at gmail.com  Sun Feb 25 17:16:17 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 25 Feb 2007 15:16:17 -0700
Subject: [Numpy-discussion] Unifying numpy, scipy,
	and matplotlib docstring formats
In-Reply-To: <m2bqjinq4q.fsf_-_@iki.fi>
References: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com>
	<45D4ED02.1050908@noaa.gov>
	<ef5675f30702151607h187c69d4t6360d490a88292ce@mail.gmail.com>
	<45D50245.1080102@noaa.gov>
	<1e2af89e0702160148u6f3b4ce4re11423f4ca9a35ec@mail.gmail.com>
	<cd7634ce0702172341jfb16e96md36453e174de13a3@mail.gmail.com>
	<1e2af89e0702180346g24f324a7je1f535f5fd6e0b96@mail.gmail.com>
	<cd7634ce0702181603t56baedf7hf8deeb8d468ca8b3@mail.gmail.com>
	<m2bqjinq4q.fsf_-_@iki.fi>
Message-ID: <db6b5ecc0702251416y19a9bc91y5ab0bb85af901a68@mail.gmail.com>

Hi,

On 2/25/07, Jouni K. Sepp?nen <jks at iki.fi> wrote:

> I suppose these things could be addressed quite neatly by IPython.
> It could even modify your history similarly to what it currently
> does with the %magic commands, so that when you type

Feel free to play with implementing this, it's easy to do so on your
personal setup, since input prefilter can be trivially added by any
user.  Once you find a set of tools that you're happy with, just send
them my way and we'll include them officially.

Here's some links you may find useful:

http://ipython.scipy.org/doc/manual/node7.html#SECTION00073000000000000000
http://ipython.scipy.org/doc/manual/node11.html

the code for these extensions ships already with ipython, under
IPython/Extensions.

Look at the one for quantities with units, it's a good starting point
for what you want to do.

> Perhaps the namespace issue could also be addressed at the IPython
> level. The pylab profile could import the various packages, perhaps
> with some kind of abbreviated names, and rewrite commands like

Ditto.

Regards,

f


From david at ar.media.kyoto-u.ac.jp  Sun Feb 25 21:47:58 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 26 Feb 2007 11:47:58 +0900
Subject: [Numpy-discussion] How to tell numpy to use gfortran as a
 compiler ?
In-Reply-To: <4688.89.8.47.113.1172414583.squirrel@webmail.uio.no>
References: <45E04AE2.6020009@ar.media.kyoto-u.ac.jp>	<45E05D8B.4070500@gmail.com>
	<45E12F7D.30702@ar.media.kyoto-u.ac.jp>
	<4688.89.8.47.113.1172414583.squirrel@webmail.uio.no>
Message-ID: <45E24A5E.9070707@ar.media.kyoto-u.ac.jp>

Sturla Molden wrote:
>
>
> g77 is a Fortran 77 compiler. The development of g77 is halted.
>
> gfortran is a Fortran 77, 90, and 95 compiler. It is the current Fortran
> compiler in the GNU Compiler Collection (GCC).
>
>
> You can compile the reference implementation of BLAS and LAPACK with both
> g77 and gfortran, as these libraries are written in Fortran 77. ATLAS is
> written in C and some Fortran 77.
>
> gfortran are able to do some optimizations that g77 cannot, e.g.
> autovectorization using SSE and MMX extensions and profile-guided
> optimizations. Also be aware that if you use gfortran and GCC 4, the C
> compiler is better as well.
>
Ok, that clears things you, thank you. Now, I have to understand why 
--fcompiler=gnu95 still calls gfortran....

cheers,

David


From robert.kern at gmail.com  Sun Feb 25 22:00:42 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 25 Feb 2007 21:00:42 -0600
Subject: [Numpy-discussion] How to tell numpy to use gfortran as a
 compiler ?
In-Reply-To: <45E24A5E.9070707@ar.media.kyoto-u.ac.jp>
References: <45E04AE2.6020009@ar.media.kyoto-u.ac.jp>	<45E05D8B.4070500@gmail.com>	<45E12F7D.30702@ar.media.kyoto-u.ac.jp>	<4688.89.8.47.113.1172414583.squirrel@webmail.uio.no>
	<45E24A5E.9070707@ar.media.kyoto-u.ac.jp>
Message-ID: <45E24D5A.5080803@gmail.com>

David Cournapeau wrote:
> Sturla Molden wrote:
>>
>> g77 is a Fortran 77 compiler. The development of g77 is halted.
>>
>> gfortran is a Fortran 77, 90, and 95 compiler. It is the current Fortran
>> compiler in the GNU Compiler Collection (GCC).
>>
>>
>> You can compile the reference implementation of BLAS and LAPACK with both
>> g77 and gfortran, as these libraries are written in Fortran 77. ATLAS is
>> written in C and some Fortran 77.
>>
>> gfortran are able to do some optimizations that g77 cannot, e.g.
>> autovectorization using SSE and MMX extensions and profile-guided
>> optimizations. Also be aware that if you use gfortran and GCC 4, the C
>> compiler is better as well.
>>
> Ok, that clears things you, thank you. Now, I have to understand why 
> --fcompiler=gnu95 still calls gfortran....

You mean g77? Anyways, I think I know why you are having problems. Passing
--fcompiler to the config command only affects the Fortran compiler that is used
during configuration phase (where we compile small C programs to determine what
your platform supports, like isnan() and the like). It does not propagate to the
rest of the build_ext phase where you want it. Use config_fc to set up your
Fortran compiler for all of the phases:

  $ python setup.py config_fc --fcompiler=gnu95 build

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From andrew.mcmurry at astro.uio.no  Mon Feb 26 06:13:14 2007
From: andrew.mcmurry at astro.uio.no (Andrew McMurry)
Date: Mon, 26 Feb 2007 12:13:14 +0100
Subject: [Numpy-discussion] NumPy array from ctypes data
Message-ID: <997DB91B-FA63-4F65-84C7-FA57F151B7DE@astro.uio.no>

How can I create a NumPy array which is backed by ctypes data?

ie create a NumPy array from a ctypes one without any data copying?

So that changing an element of the NumPy array will alter the  
underlying ctypes array.  I have C structures that contain numerical  
data and would like to access that data through the NumPy array  
interface.

I have tried the following:

PyObject *make_dbl_array(double *data, int n)
{
   npy_intp dim = n;
   return PyArray_SimpleNewFromData(1, &dim, NPY_DOUBLE, (void *)data);
}

But when called via ctypes (with restype set to ctypes.py_object) it  
gives a bus error.

	Andrew



From aisaac at american.edu  Mon Feb 26 12:39:12 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Mon, 26 Feb 2007 12:39:12 -0500
Subject: [Numpy-discussion] [Matplotlib-users] Unifying numpy, scipy,
	and matplotlib docstring formats
In-Reply-To: <m2bqjinq4q.fsf_-_@iki.fi>
References: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com><45D4ED02.1050908@noaa.gov><ef5675f30702151607h187c69d4t6360d490a88292ce@mail.gmail.com><45D50245.1080102@noaa.gov><1e2af89e0702160148u6f3b4ce4re11423f4ca9a35ec@mail.gmail.com><cd7634ce0702172341jfb16e96md36453e174de13a3@mail.gmail.com><1e2af89e0702180346g24f324a7je1f535f5fd6e0b96@mail.gmail.com><cd7634ce0702181603t56baedf7hf8deeb8d468ca8b3@mail.gmail.com><m2bqjinq4q.fsf_-_@iki.fi>
Message-ID: <Mahogany-0.67.0-1728-20070226-123912.00@american.edu>

On Sun, 25 Feb 2007, Jouni K. Sepp?nen apparently wrote: 
> it is easier to type something like 
>   [0 1 0; 1 0 0; 0 0 1] 
> than 
>   array([[0,1,0],[1,0,0],[0,0,1]]) 


x = numpy.mat('0 1 0; 1 0 0; 0 0 1').A

hth,
Alan Isaac






From Chris.Barker at noaa.gov  Mon Feb 26 14:05:11 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 26 Feb 2007 11:05:11 -0800
Subject: [Numpy-discussion] Questions about cross-compiling extensions
	for	mac-ppc and mac-intel
In-Reply-To: <A69A9808-77C3-4E0A-8EA5-B34566B391F9@stanford.edu>
References: <A69A9808-77C3-4E0A-8EA5-B34566B391F9@stanford.edu>
Message-ID: <45E32F67.70809@noaa.gov>

Zachary Pincus wrote:
> building python extensions as Mac-PPC and Mac-Intel fat  
> binaries, so I'm turning to the wisdom of this list for a few questions.

I'd try the pythonmac list too -- there are folks there that actually 
understand all this!

> My general goal is to make a double-clickable Mac installer of a set  
> of tools built around numpy, numpy's distutils, a very hacked-up  
> version of PIL, and some fortran code too. To this end, I need to  
> figure out how to get the numpy distutils to cross-compile,  
> generating PPC code and Intel code in separate builds -- and/or  
> generating a universal binary all in one go. (I'd like to distribute  
> a universal version of numpy, but I think that my own code needs to  
> be built/distributed separately for each architecture due to endian- 
> ness issues.)

hmm -- maybe you'd be better off dealing with the endian issues in your 
code -- i.e. dealing with it at runtime, rather than compile time.

> Is there explicit support in distutils for this, or is it a matter of  
> setting the proper environment variables to entice gcc and gfortran  
> to generate code for a specific architecture?

I'm no expert, but the glory of distutils is that it will, by default 
build extensions the same way as python itself was built. So if you use 
a PPC python, you'll get PPC extensions, same with Intel, and if you use 
  a Universal Python, you'll get a Universal extension.

The trick is that while you can build Universal on either platform, you 
can't use this trick to build Intel extensions on a PPC mac, as the 
Python would have to be intel, and a PPC mac won't run an Intel Python. 
It may be possible to run a PPC Python on an Intel Mac with Rosettta, 
though.

In any case, Universal is probably the best bet except for your Fortran 
code - non one has made a Fortan compiler that can do Universal. You may 
be able to build the two part independently an use Lipo to put them 
together, however.

Googling this list and the pythonmac one should get you some discussion 
of this, but AFAIK, no one has done it yet.

If you do need to have your Fortran stuff separate, you can still make 
the rest of it Universal

> One problem is that PIL is a tricky beast, even in the neutered form  
> that I'm using it. It does a compile-time check for the endian-ness  
> of the system, and a compile-time search for the zlib to use, both of  
> which are problematic.

Well, I know it's possible to build Universal. There are binaries on 
pythonmac.org/packages. The folks on the pythonmac list should be able 
to tell you how. ( zlib is included with OS-X, so that shouldn't be an 
issue)

> To address the former, I'd like to be able to (say) include something  
> like 'config_endian --big' on the 'python setup.py' command-line, and  
> have that information trickle down to the PIL config script (a few  
> subpackages deep). Is this easy or possible?

I doubt it, but there has got to be a way to tie endianess to platform. 
You'd want the Intel code built one way, and the PPC code another. I 
think distutils may take care of this for you.

Good luck! And if you find a way to build a universal Fortran extension 
-- be sure to let us know!

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From robert.kern at gmail.com  Mon Feb 26 14:38:49 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 26 Feb 2007 13:38:49 -0600
Subject: [Numpy-discussion] Questions about cross-compiling extensions
 for	mac-ppc and mac-intel
In-Reply-To: <45E32F67.70809@noaa.gov>
References: <A69A9808-77C3-4E0A-8EA5-B34566B391F9@stanford.edu>
	<45E32F67.70809@noaa.gov>
Message-ID: <45E33749.1080604@gmail.com>

Christopher Barker wrote:

> I'm no expert, but the glory of distutils is that it will, by default 
> build extensions the same way as python itself was built. So if you use 
> a PPC python, you'll get PPC extensions, same with Intel, and if you use 
>   a Universal Python, you'll get a Universal extension.

There is a little wrinkle in that numpy configures itself by compiling and
running small C programs to determine what is supported on its platform. When
building on an Intel machine even with a Universal Python, the results of that
configuration will only be for the system it was compiled on. Thus, even though
Universal binaries built on 10.4 systems would usually work on 10.3.9, numpy
doesn't.

> The trick is that while you can build Universal on either platform, you 
> can't use this trick to build Intel extensions on a PPC mac, as the 
> Python would have to be intel, and a PPC mac won't run an Intel Python. 
> It may be possible to run a PPC Python on an Intel Mac with Rosettta, 
> though.
> 
> In any case, Universal is probably the best bet except for your Fortran 
> code - non one has made a Fortan compiler that can do Universal.

The R folks have a package containing gcc 4.0.3 with gfortran that looks like it
might be Universal. I haven't tried to build scipy with it, yet.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Mon Feb 26 14:43:09 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 26 Feb 2007 13:43:09 -0600
Subject: [Numpy-discussion] Questions about cross-compiling extensions
 for	mac-ppc and mac-intel
In-Reply-To: <A69A9808-77C3-4E0A-8EA5-B34566B391F9@stanford.edu>
References: <A69A9808-77C3-4E0A-8EA5-B34566B391F9@stanford.edu>
Message-ID: <45E3384D.5080005@gmail.com>

Zachary Pincus wrote:

> To address the former, I'd like to be able to (say) include something  
> like 'config_endian --big' on the 'python setup.py' command-line, and  
> have that information trickle down to the PIL config script (a few  
> subpackages deep). Is this easy or possible?

I'd just do separate builds on PPC and Intel machines.

> To address the latter, I think I need to have the PIL extensions  
> dynamically link against '/Developer/SDKs/MacOSX10.4u.sdk/usr/lib/ 
> libz.dylib' which is the fat-binary version of the library, using the  
> headers from '/Developer/SDKs/MacOSX10.4u.sdk/usr/include/zlib.h
> '. Right now, PIL is using system_info from numpy.distutils to find  
> the valid library paths on which libz and its headers might live.  
> This is nice and more or less platform-neutral, which I like. How  
> best should I convince/configure numpy.distutils.system_info to put '/ 
> Developer/SDKs/MacOSX10.4u.sdk/usr/{lib,include}' on the output to  
> get_include_dirs() and get_lib_dirs()?

distutils ought to be including an argument like this:

  -isysroot /Developer/SDKs/MacOSX10.4u.sdk

That ought to be sufficient for building PIL. Don't hack up PIL's build process
if at all possible.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Mon Feb 26 14:49:53 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 26 Feb 2007 11:49:53 -0800
Subject: [Numpy-discussion] Questions about cross-compiling extensions
	for	mac-ppc and mac-intel
In-Reply-To: <45E33749.1080604@gmail.com>
References: <A69A9808-77C3-4E0A-8EA5-B34566B391F9@stanford.edu>
	<45E32F67.70809@noaa.gov> <45E33749.1080604@gmail.com>
Message-ID: <45E339E1.2040803@noaa.gov>

Robert Kern wrote:
> even though
> Universal binaries built on 10.4 systems would usually work on 10.3.9, numpy
> doesn't.

Darn, but I for one, can live without 10.3.9 support -- it does build 
Universal properly for 10.4 doesn't it?

> The R folks have a package containing gcc 4.0.3 with gfortran that looks like it
> might be Universal. I haven't tried to build scipy with it, yet.

cool!

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From fred at ucar.edu  Mon Feb 26 14:51:42 2007
From: fred at ucar.edu (Fred Clare)
Date: Mon, 26 Feb 2007 12:51:42 -0700
Subject: [Numpy-discussion] Advice on masked array implementation
Message-ID: <3805FA8C-E28F-4C2B-BCBB-627396418F77@ucar.edu>

We would like some advice on how to proceed with implementing
masked array capabilities in a large package of climate-related
analysis functions.  We are in the initial stages of trying to
duplicate functionality from an existing package written in a
locally-developed scripting language.  The existing functionality
depends heavily on masked arrays (actually on variables with
attributes, with "fill_value" being one such).

We have polled our user base and, while all responders plan to
convert to NumPy, many have not started the conversion or are in
transition.  It is our experience that converting users to
new ways is usually a multi-year undertaking, so it seems
that we may need to support both Numeric and NumPy installations
for some time to come.  Even if people have converted to the
NumPy version of our package, they may still be importing
packages that have not been converted.

Our initial design attempts to deal with the possibility
of users potentially mixing (or using exclusively) Numeric
arrays, Numeric masked arrays, NumPy arrays, and NumPy masked
arrays.  For example, suppose you have a function:

    result = func(arg0, arg1)

where the two arguments and return variable can be any one of
the four types of arrays mentioned.  Currently we are testing
to see if either argument is a NumPy or Numeric masked array.
If just Numeric masked arrays, then we return a Numeric masked
array.  If just NumPy masked arrays, then we return a NumPy
masked array.  If one is a Numeric masked array and the other
is a NumPy masked array, then we return a NumPy masked array.
Similar checking is done for using just Numeric and/or NumPy
non-masked arrays.  Does this seem like a reasonable approach?

It is tempting just to go with NumPy, but then we will have a
large class of users who cannot access the new functionality.

We have followed the discussion on the development of the new
maskedarray module, but have not used it.  I went to

http://projects.scipy.org/scipy/numpy/attachment/wiki/MaskedArray/ 
maskedarray.py

as referenced in a posting from Pierre GM, but I got "Internal Error."

Has there been any decision as to whether maskedarray will be
in NumPy version 1.1?  Any estimate as to when 1.1 would be out?
If we commit to numpy.core.ma now, how much trouble will it be
to convert to the new maskedarray?  Is there any user documentation
on maskedarray and details on the differences between it and
numpy.core.ma?

Thanks,

Fred Clare



From robert.kern at gmail.com  Mon Feb 26 14:52:59 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 26 Feb 2007 13:52:59 -0600
Subject: [Numpy-discussion] Questions about cross-compiling extensions
 for	mac-ppc and mac-intel
In-Reply-To: <45E339E1.2040803@noaa.gov>
References: <A69A9808-77C3-4E0A-8EA5-B34566B391F9@stanford.edu>	<45E32F67.70809@noaa.gov>
	<45E33749.1080604@gmail.com> <45E339E1.2040803@noaa.gov>
Message-ID: <45E33A9B.8070202@gmail.com>

Christopher Barker wrote:
> Robert Kern wrote:
>> even though
>> Universal binaries built on 10.4 systems would usually work on 10.3.9, numpy
>> doesn't.
> 
> Darn, but I for one, can live without 10.3.9 support -- it does build 
> Universal properly for 10.4 doesn't it?

I've never tested it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From pgmdevlist at gmail.com  Mon Feb 26 15:42:58 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 26 Feb 2007 15:42:58 -0500
Subject: [Numpy-discussion] Advice on masked array implementation
In-Reply-To: <3805FA8C-E28F-4C2B-BCBB-627396418F77@ucar.edu>
References: <3805FA8C-E28F-4C2B-BCBB-627396418F77@ucar.edu>
Message-ID: <200702261542.58719.pgmdevlist@gmail.com>

On Monday 26 February 2007 14:51:42 Fred Clare wrote:
> We would like some advice on how to proceed with implementing
> masked array capabilities in a large package of climate-related
> analysis functions. 

Sounds great ! I'm working on the same field basically, and I needed 
MaskedArrays to deal with missing values in environmental series. But we can 
chat about that off-list.

> If one is a Numeric masked array and the other
> is a NumPy masked array, then we return a NumPy masked array.
> Similar checking is done for using just Numeric and/or NumPy
> non-masked arrays.  Does this seem like a reasonable approach?

> We have followed the discussion on the development of the new
> maskedarray module, but have not used it.  I went to
>
> http://projects.scipy.org/scipy/numpy/attachment/wiki/MaskedArray/
> maskedarray.py
>
> as referenced in a posting from Pierre GM, but I got "Internal Error."

Yes, I had to take the package off the projects.scipy.org site when I got 
write access to the svn server, as that particular version was really 
outdated. You can find the latest version on the scipy svn server, in the 
sandbox:
http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/maskedarray/

Note that I made some major updates a couple of weeks ago, without advertising 
them. A description is available at:
http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/maskedarray/CHANGELOG

> Has there been any decision as to whether maskedarray will be
> in NumPy version 1.1?  

The decision is out of my hands. My understanding is that before the new 
implementation can be taken seriously, more feedback is needed from actual 
users (and I fully agree with that). Moreover, there are some vague plans 
about porting it to C. My naive initial attempts with Pyrex having failed 
dramatically, I will have to learn C, so it probably won't happen in the very 
next weeks... But porting to C should solve some minor issues I'm unhappy 
with for now, and can't implement in python without significantly degrading 
the performances.

> Any estimate as to when 1.1 would be out? 
> If we commit to numpy.core.ma now, how much trouble will it be
> to convert to the new maskedarray?  

It shouldn't be that a problem. Normally, the following should work (even if 
some warnings are raised)
>>> import numpy.core.ma as ma
>>> import maskedarray as MA
>>> x = ma.array([1,2,3,4,5], mask=[1,0,1,0,0])
>>> x
array(data =
 [999999      2      3 999999      5],
      mask =
 [ True False False  True False],
      fill_value=999999)
>>> X = MA.array(x)
>>> X
masked_array(data = [-- 2 3 -- 5],
      mask = [ True False False  True False],
      fill_value=999999)

That is, maskedarray.MaskedArrays recognize numpy.core.ma.maskedarray

I tried to keep as much backward compatibility as I could, but without really 
testing it, so no guarantee.

> Is there any user documentation 
> on maskedarray and details on the differences between it and
> numpy.core.ma?

Not at this point, unfortunately. Note that the "new" implementation follows 
very closely Paul Dubois' initial code. (In fact, a bit too closely for its 
own good. Reggie Dugard suggested some modifications I tried to take into 
account in the latest version that seem to solve that). Therefore, switching 
from numpy.core.ma to maskedarray should be relatively painless. I'd be more 
than happy to help you on that.

Basically the main differences between the two implementations are:
- MaskedArray are regular subclasses of ndarray, so you can use asanyarray 
without losing your mask.
- Subclassing MaskedArray is far easier with the new implementation than it 
was with numpy.core.ma
- the fill_value attribute is now a property
- the _data attribute is now a view of the MaskedArray, instead of an 
independent object.
- the underlying _data can be any subclass of ndarray (such as matrix)
- some of the MaskedArray methods (ravel, transpose...) are implemented 
through wrappers that must have a __get__ method. That works well w/ 
Python2.4, I'm not sure it would work w/ 2.3.
- some methods that were not available in numpy.core.ma are now in 
maskedarray, either .core or .extras
- there's a prototype of MaskedRecords objects, that gives the possibility to 
mask specific fields in a recarray.

All in all, I think that the new implementation gets rid of some of the 
limitations of numpy.core.ma, without affecting too badly performances. The 
latest test showed that yes, maskedarray is slightly slower than 
numpy.core.ma (10%), but it provides more functionality: for example, you can 
prevent a mask to be overwritten, it is very easy to subclass, it interacts 
nicely with ndarray... I'm using the new implementation systematically for my 
own projects (which explains why there are regularly some tweakings to the 
implementation), and Matt Knox and I have been using it for our common 
TimeSeries project without any difficulty so far. 

Once again, please do not hesitate to contact me on or off-list if you have 
any questions/comments/requests.




From david at ar.media.kyoto-u.ac.jp  Tue Feb 27 02:00:52 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 27 Feb 2007 16:00:52 +0900
Subject: [Numpy-discussion] How to tell numpy to use gfortran as a
 compiler ?
In-Reply-To: <45E24D5A.5080803@gmail.com>
References: <45E04AE2.6020009@ar.media.kyoto-u.ac.jp>	<45E05D8B.4070500@gmail.com>	<45E12F7D.30702@ar.media.kyoto-u.ac.jp>	<4688.89.8.47.113.1172414583.squirrel@webmail.uio.no>	<45E24A5E.9070707@ar.media.kyoto-u.ac.jp>
	<45E24D5A.5080803@gmail.com>
Message-ID: <45E3D724.4060301@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
> David Cournapeau wrote:
>   
>> Sturla Molden wrote:
>>     
>>> g77 is a Fortran 77 compiler. The development of g77 is halted.
>>>
>>> gfortran is a Fortran 77, 90, and 95 compiler. It is the current Fortran
>>> compiler in the GNU Compiler Collection (GCC).
>>>
>>>
>>> You can compile the reference implementation of BLAS and LAPACK with both
>>> g77 and gfortran, as these libraries are written in Fortran 77. ATLAS is
>>> written in C and some Fortran 77.
>>>
>>> gfortran are able to do some optimizations that g77 cannot, e.g.
>>> autovectorization using SSE and MMX extensions and profile-guided
>>> optimizations. Also be aware that if you use gfortran and GCC 4, the C
>>> compiler is better as well.
>>>
>>>       
>> Ok, that clears things you, thank you. Now, I have to understand why 
>> --fcompiler=gnu95 still calls gfortran....
>>     
>
> You mean g77? Anyways, I think I know why you are having problems. Passing
> --fcompiler to the config command only affects the Fortran compiler that is used
> during configuration phase (where we compile small C programs to determine what
> your platform supports, like isnan() and the like). It does not propagate to the
> rest of the build_ext phase where you want it. Use config_fc to set up your
> Fortran compiler for all of the phases:
>
>   $ python setup.py config_fc --fcompiler=gnu95 build
>   
Thanks, that indeed was the cause of my problem.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Tue Feb 27 02:04:23 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 27 Feb 2007 16:04:23 +0900
Subject: [Numpy-discussion] nan, warning and error modes
Message-ID: <45E3D7F7.6040500@ar.media.kyoto-u.ac.jp>

Hi,

    I am developing some numpy code, which sometimes fail because of 
nan. This is likely to be due to some bad coding on my side, and as such 
any NaN is a bug for this particular piece of code.
    Is there a way to get a warning when the first Nan is detected in 
the code (or even a faulty assertion) ? It looks like there are some 
variables/functions related to that in numpy, but I didn't find any 
useful document on the matter, either in the numpy book nor on the scipy 
website.

    cheers,

    David


From robert.kern at gmail.com  Tue Feb 27 02:16:46 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 27 Feb 2007 01:16:46 -0600
Subject: [Numpy-discussion] nan, warning and error modes
In-Reply-To: <45E3D7F7.6040500@ar.media.kyoto-u.ac.jp>
References: <45E3D7F7.6040500@ar.media.kyoto-u.ac.jp>
Message-ID: <45E3DADE.8000504@gmail.com>

David Cournapeau wrote:
> Hi,
> 
>     I am developing some numpy code, which sometimes fail because of 
> nan. This is likely to be due to some bad coding on my side, and as such 
> any NaN is a bug for this particular piece of code.
>     Is there a way to get a warning when the first Nan is detected in 
> the code (or even a faulty assertion) ? It looks like there are some 
> variables/functions related to that in numpy, but I didn't find any 
> useful document on the matter, either in the numpy book nor on the scipy 
> website.

seterr(invalid='warning')

Described in section "9.1.4 Error Handling" of the _Guide to NumPy_ (dated
2006-12-07).

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Tue Feb 27 02:18:12 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 27 Feb 2007 01:18:12 -0600
Subject: [Numpy-discussion] nan, warning and error modes
In-Reply-To: <45E3DADE.8000504@gmail.com>
References: <45E3D7F7.6040500@ar.media.kyoto-u.ac.jp>
	<45E3DADE.8000504@gmail.com>
Message-ID: <3d375d730702262318l23810b83k6e32d795dac6f4f4@mail.gmail.com>

On 2/27/07, Robert Kern <robert.kern at gmail.com> wrote:
> David Cournapeau wrote:
> > Hi,
> >
> >     I am developing some numpy code, which sometimes fail because of
> > nan. This is likely to be due to some bad coding on my side, and as such
> > any NaN is a bug for this particular piece of code.
> >     Is there a way to get a warning when the first Nan is detected in
> > the code (or even a faulty assertion) ? It looks like there are some
> > variables/functions related to that in numpy, but I didn't find any
> > useful document on the matter, either in the numpy book nor on the scipy
> > website.
>
> seterr(invalid='warning')

Ahem.

  seterr(invalid='warn')

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Tue Feb 27 02:32:36 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 27 Feb 2007 16:32:36 +0900
Subject: [Numpy-discussion] nan, warning and error modes
In-Reply-To: <45E3DADE.8000504@gmail.com>
References: <45E3D7F7.6040500@ar.media.kyoto-u.ac.jp>
	<45E3DADE.8000504@gmail.com>
Message-ID: <45E3DE94.3010700@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
> David Cournapeau wrote:
>   
>> Hi,
>>
>>     I am developing some numpy code, which sometimes fail because of 
>> nan. This is likely to be due to some bad coding on my side, and as such 
>> any NaN is a bug for this particular piece of code.
>>     Is there a way to get a warning when the first Nan is detected in 
>> the code (or even a faulty assertion) ? It looks like there are some 
>> variables/functions related to that in numpy, but I didn't find any 
>> useful document on the matter, either in the numpy book nor on the scipy 
>> website.
>>     
>
> seterr(invalid='warning')
>
> Described in section "9.1.4 Error Handling" of the _Guide to NumPy_ (dated
> 2006-12-07).
>
>   
Well, my bad, even if I don't have this version, this section is present 
in my own version.... I looked at the wrong keywords,

thank you,

David


From alexandre.fayolle at logilab.fr  Tue Feb 27 04:50:43 2007
From: alexandre.fayolle at logilab.fr (Alexandre Fayolle)
Date: Tue, 27 Feb 2007 10:50:43 +0100
Subject: [Numpy-discussion] Unifying numpy, scipy,
	and matplotlib docstring formats
In-Reply-To: <m2bqjinq4q.fsf_-_@iki.fi>
References: <ef5675f30702150009i1ac9c809yd0fbbb9655b4b8b3@mail.gmail.com>
	<45D4ED02.1050908@noaa.gov>
	<ef5675f30702151607h187c69d4t6360d490a88292ce@mail.gmail.com>
	<45D50245.1080102@noaa.gov>
	<1e2af89e0702160148u6f3b4ce4re11423f4ca9a35ec@mail.gmail.com>
	<cd7634ce0702172341jfb16e96md36453e174de13a3@mail.gmail.com>
	<1e2af89e0702180346g24f324a7je1f535f5fd6e0b96@mail.gmail.com>
	<cd7634ce0702181603t56baedf7hf8deeb8d468ca8b3@mail.gmail.com>
	<m2bqjinq4q.fsf_-_@iki.fi>
Message-ID: <20070227095043.GB26292@crater.logilab.fr>

On Sun, Feb 25, 2007 at 06:44:37PM +0200, Jouni K. Sepp??nen wrote:
> "Barry Wark" <barrywark at gmail.com> writes:
> 
> > Yes, I agree. I wasn't coming at so much from the goal of making Pylab
> > a Matlab clone (as you point out, that's silly, and misses much of the
> > advantage of Python), but rather from the goal of making interactive
> > use as efficient as possible. When I fire up ipython -pylab to do some
> > quick exploration, it's nice not to have to type N.blah or pylab.plot
> 
> IMHO the greatest strength of Matlab in interactive use is the matrix
> input format. For one thing, it is easier to type something like
> 
>   [0 1 0; 1 0 0; 0 0 1]
> 
> than
> 
>   array([[0,1,0],[1,0,0],[0,0,1]])


A very nice shortcut in my opinion is the one supported by the HP
scientific calculators (HP28, HP48 in my days), which would look like:

array([[0,1,0], 1, 0, 0, 0, 0, 1])

I'm not quite sure how this could be generalized nicely for arbitrary
shapes, but for arrays of rank 2 (which are a very common case where
people are actually typing the values) it feels nice (especially on a
french keyboard where the square brackets are awkward to type in) 

-- 
Alexandre Fayolle                              LOGILAB, Paris (France)
Formations Python, Zope, Plone, Debian:  http://www.logilab.fr/formations
D?veloppement logiciel sur mesure:       http://www.logilab.fr/services
Informatique scientifique:               http://www.logilab.fr/science
Reprise et maintenance de sites CPS:     http://www.migration-cms.com/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 481 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070227/7b55bd68/attachment.sig>

From acorriga at gmu.edu  Tue Feb 27 13:26:30 2007
From: acorriga at gmu.edu (Andrew Corrigan)
Date: Tue, 27 Feb 2007 13:26:30 -0500
Subject: [Numpy-discussion] inconsistent mgrid results
Message-ID: <45E477D6.6000206@gmu.edu>

I'm confused about the following:

 >>> print mgrid[2.45:2.6:0.05, 0:5:1]
[[[ 2.45  2.45  2.45  2.45  2.45]
 [ 2.5   2.5   2.5   2.5   2.5 ]]

[[ 0.    1.    2.    3.    4.  ]
 [ 0.    1.    2.    3.    4.  ]]]
 >>> print mgrid[2.45:2.6:0.05]
[ 2.45  2.5   2.55]

In the first case in the first dimension I get 2.45, 2.5.   In the 
second case in the first dimension I get 2.45, 2.5, 2.55   In both cases 
I'm using 2.45:2.6:0.05  to specify the grid in the first dimension.  
Why is there a difference?  Am I using mgrid incorrectly?   I'm using 
the latest numpy release from sourceforge.
Thanks, Andrew




From rudolf.sykora at gmail.com  Tue Feb 27 14:53:57 2007
From: rudolf.sykora at gmail.com (Rudolf Sykora)
Date: Tue, 27 Feb 2007 20:53:57 +0100
Subject: [Numpy-discussion] building an array using smaller ones
Message-ID: <a560a5d00702271153g1df222bbp500f0c0f1ae1b774@mail.gmail.com>

Hello everybody,
I wonder how I could most easily accomplish the following:

Say I have sth like:
a = array( [1, 2] )
and I want to use this array to build another array in the following sence:
b = array( [[1, 2, 3, a], [5, a, 6, 7], [0, 2-a, 3, 4]])  # this doesn't
work

I would like to obtain
b = array( [[1, 2, 3, 1, 2],  [5 ,1 ,2 ,6 ,7], [0, 1, 0, 3, 4]] )

I know a rather complicated way but believe there must be an easy one.
Thank you very much.

Ruda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070227/4a8400ef/attachment.html>

From a.schmolck at gmx.net  Tue Feb 27 16:35:53 2007
From: a.schmolck at gmx.net (Alexander Schmolck)
Date: 27 Feb 2007 21:35:53 +0000
Subject: [Numpy-discussion] [ANN] mlabrap-1.0b: a high level python to matlab
Message-ID: <yfsr6sbffly.fsf@oc.ex.ac.uk>

URL
---
<http://mlabwrap.sourceforge.net>

Description
-----------

Mlabwrap-1.0 is a high-level python to matlab(tm) bridge that makes calling
matlab functions from python almost as convenient as using a normal python
library. It is available under a very liberal license (BSD/MIT) and should
work on all major platforms and (non-ancient) python and matlab versions and
either numpy or Numeric (Numeric support will be dropped in the future).

News
----

version 1.0b brings python 2.5 compatibility and various small fixes (improved
error handling for 7.3, improvements to setup.py etc.). Provided I don't get
any bug reports within the next two weeks or so the only difference between
this version and 1.0 final will be cheeseshop support. Since mlabwrap 1.0 will
be the last version that offers Numeric support anyone who wants to put off
the switch to numpy a bit longer and is interested in using mlabwrap is
strongly encouraged to download, test and possibly submit a bug report now.


Examples
--------

Creating a simple line plot:

>>> from mlabwrap import mlab; mlab.plot([1,2,3],'-o')

Creating a surface plot:

>>> from mlabwrap import mlab; from numpy import *
>>> xx = arange(-2*pi, 2*pi, 0.2)
>>> mlab.surf(subtract.outer(sin(xx),cos(xx)))

Creating a neural network and training it on the xor problem (requires netlab)

>>> net = mlab.mlp(2,3,1,'logistic')
>>> net = mlab.mlptrain(net, [[1,1], [0,0], [1,0], [0,1]], [0,0,1,1], 1000)

Thanks go to Taylor Berg and many others who tested and provided feedback for
the upcoming 1.0 release; more acknowledgements can be found on the website.

cheers,

'as


From oliphant at ee.byu.edu  Tue Feb 27 18:29:51 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Tue, 27 Feb 2007 16:29:51 -0700
Subject: [Numpy-discussion] Buffer Interface for Python 3.0
Message-ID: <45E4BEEF.4030609@ee.byu.edu>


An update for those of you who did not get the chance to come to PyCon.  
PyCon was very well attended this year and there were some excellent 
discussions and presentations.

 From PyCon I learned that Python 3000 is closer than I had previously 
thought.  What this means for me, is that I am now focussing toward 
getting a re-vamped buffer interface into Python 3.0

This will help us interact with the Python developers more effectively.  
Once we have the buffer interface hammered out for Python 3.0 we can 
back-port the result to Python 2.6

Thus my buffer PEP is being re-vamped.  Anybody who would like to 
comment or contribute to the design of the new buffer interface is 
welcome to voice their opinion. 

Basically, what we are going to do now is

1) Return the data-format specification in an extended struct-style string
2) Return the shape information in a tuple of lists: (shape, strides)

There are two questions I'm grappling with right now:

1) Do we propose the inclusion of offsets in the shape information?  
NumPy does not use offsets internally but simply has a pointer to the 
start of the array.

2) The buffer interface needs to understand the idea of discontiguous 
arrays.  If the shape/stride information is separate from the 
pointer-to-data call, then the user needs to know if that 
pointer-to-data is a "contiguous chunk" or just the beginning of a 
strided memory area (and so should not be treated as a single-segment).

3) If we support strided memory areas, then we should probably also 
allow some way for PIL-like objects to report their buffer sequence (I'm 
sure this was the origin of the multi-segment buffer protocol to begin 
with).  Or we could just ignore that possibility.  The PIL would have to 
copy memory in order to share it's images.

Anybody with ideas is welcome to participate.  What I have so far is at

http://wiki.python.org/moin/ArrayInterface


Thanks,

-Travis



From oliphant at ee.byu.edu  Tue Feb 27 20:06:17 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Tue, 27 Feb 2007 18:06:17 -0700
Subject: [Numpy-discussion] Draft PEP for the new buffer interface to be in
	Python 3000
Message-ID: <45E4D589.8080804@ee.byu.edu>

PEP: <unassigned>
Title: Revising the buffer protocol
Version: $Revision: $
Last-Modified: $Date:  $
Author: Travis Oliphant <oliphant at ee.byu.edu>
Status: Draft
Type: Standards Track
Created: 28-Aug-2006
Python-Version: 3000

Abstract

   This PEP proposes re-designing the buffer API (PyBufferProcs
   function pointers) to improve the way Python allows memory sharing
   in Python 3.0

   In particular, it is proposed that the multiple-segment and
   character buffer portions of the buffer API are eliminated and
   additional function pointers are provided to allow sharing any
   multi-dimensional nature of the memory and what data-format the
   memory contains.

Rationale

   The buffer protocol allows different Python types to exchange a
   pointer to a sequence of internal buffers.  This functionality is
   '''extremely''' useful for sharing large segments of memory between
   different high-level objects, but it's too limited and has issues.

    1. There is the little (never?) used "sequence-of-segments" option
       (bf_getsegcount)

    2. There is the apparently redundant character-buffer option
       (bf_getcharbuffer)

    3. There is no way for a consumer to tell the buffer-API-exporting
       object it is "finished" with its view of the memory and
       therefore no way for the exporting object to be sure that it is
       safe to reallocate the pointer to the memory that it owns (the
       array object reallocating its memory after sharing it with the
       buffer object which held the original pointer led to the
       infamous buffer-object problem).

    4. Memory is just a pointer with a length. There is no way to
       describe what's "in" the memory (float, int, C-structure, etc.)

    5. There is no shape information provided for the memory.  But,
       several array-like Python types could make use of a standard
       way to describe the shape-interpretation of the memory
       (!wxPython, GTK, pyQT, CVXOPT, !PyVox, Audio and Video
       Libraries, ctypes, !NumPy, data-base interfaces, etc.)

    There are two widely used libraries that use the concept of
    discontiguous memory: PIL and NumPy.  Their view of discontiguous
    arrays is a bit different, though.  NumPy uses the notion of
    constant striding in each dimension as it's basic concept of an
    array. In this way a simple sub-region of a larger array can be
    described without copying the data.  Strided memory is a common
    way to describe data to many computing libraries (such as the BLAS
    and LAPACK).

    The PIL uses a more opaque memory representation. Sometimes an
    image is contained in a contiguous segment of memory, but
    sometimes it is contained in an array of pointers to the
    contiguous segments (usually lines) of the image.  This allows the
    image to not be loaded entirely into memory.  The PIL is where the
    idea of multiple buffer segments in the original buffer interface
    came from, I believe.

    The buffer interface should allow discontiguous memory areas to
    share standard striding information.  However, consumers that do
    not want to deal with strided memory should also be able to
    request a contiguous segment easily.    


Proposal Overview

   * Eliminate the char-buffer and multiple-segment sections of the
     buffer-protocol.

   * Unify the read/write versions of getting the buffer.

   * Add a new function to the protocol that should be called when
     the consumer object is "done" with the view.

   * Add a new function to allow the protocol to describe what is in
     memory (unifying what is currently done now in struct and
     array)

   * Add a new function to allow the protocol to share shape
     information

   * Fix all objects in core and standard library to conform to the
     new interface

   * Extend the struct module to handle more format specifiers

Specification

    Change the PyBufferProcs structure to

    typedef struct {
         getbufferproc bf_getbuffer
         releasebufferproc bf_releasebuffer
         formatbufferproc bf_getbufferformat
         shapebufferproc bf_getbuffershape
    }

    typedef PyObject *(*getbufferproc)(PyObject *obj, void **buf,
                                       Py_ssize_t *len, int requires)
 
      Return a pointer to memory in buf and the length of that memory
      buffer in buf.  Requirements for the memory are provided in
      requires (PYBUFFER_WRITE, PYBUFFER_ONESEGMENT).  NULL is
      returned and an error raised if the object cannot return a view
      with those requirements.  Otherwise, an object-specific "view"
      object is returned (which can just be a borrowed reference to
      obj).

      This view object should be used in the other API calls and
      does not need to be decref'd.  It should be "released" if the
      interface exporter provides the bf_releasebuffer function.

    typedef int (*releasebufferproc)(PyObject *view)

      This function is called when a view of memory previously
      acquired from the object is no longer needed.  It is up to the
      exporter of the API to make sure all views have been released
      before eliminating a reference to a previously returned pointer.
      It is up to consumers of the API to call this function on the
      object whose view is obtained when it is no longer needed.  A -1
      is returned on error and 0 on success.

    typedef char *(*formatbufferproc)(PyObject *view, int *itemsize)

      Get the format-string of the memory using the struct-module
      string syntax (see below for proposed additions to that syntax).
      Also, there is never an alignment assumption in this
      string---the full byte-layout is always required.  If the
      implied size of this string is smaller than the length of the
      buffer then it is assumed that the string is repeated.

      If itemsize is not NULL, then return the size implied by the
      format string.  This could be the entire length of the buffer or
      just the length of each element.  It is equivalent to *itemsize
      = PyObject_SizeFromFormat(ret) if ret is the returned string.
      However, very often objects already know the itemsize without
      having to compute it separately.

    typedef PyObject *(*shapebufferproc)(PyObject *view)

      Return a 2-tuple of lists containing shape information: (shape,
      strides).  The strides object can be None if the memory is
      C-style contiguous) otherwise it provides the striding in each
      dimension.

    All of these routines are optional for a type object (but the last
    three make no sense unless the first one is implemented).


New C-API calls are proposed

   int
   PyObject_CheckBuffer(PyObject *obj)

      return 1 if the getbuffer function is available otherwise 0

   PyObject *
   PyObject_GetBuffer(PyObject *obj, void **buf, Py_ssize_t *len,
                      int requires)

      return a borrowed reference to a "view" object of memory for the
      object.  Requirements for the memory should be given in requires
      (PYBUFFER_WRITE, PYBUFFER_ONESEGMENT).  The memory pointer is in
      *buf and its length in *len.

      Note, the memory is not considered a single segment of memory
      unless PYBUFFER_ONESEGMENT is used in requires. Get possible
      striding using PyObject_GetBufferShape on the view object.
      
   int
   PyObject_ReleaseBuffer(PyObject *view)
      
      call this function to tell obj that you are done with your "view"
      This is a no-op if the object doesn't implement a release function.
      Only call this after a previous PyObject_GetBuffer has succeeded.
      Return -1 on error.
      
   char *
   PyObject_GetBufferFormat(PyObject *view, int *itemsize)

      Return a NULL-terminated string indicating the data-format of
      the memory buffer.  The string is in struct-module syntax with
      the exception that there is never an alignment assumption (all
      bytes must be accounted for). If the length of the buffer
      indicated by this string is smaller than the total length of the
      buffer, then a repeat of the string is implied to fill the
      length of the buffer.

      If itemsize is not NULL, then return the implied size
      of each item (this could be calculated from the format string
      but it is often known by the view object anyway).

   PyObject *
   PyObject_GetBufferShape(PyObject *view)

      Return a 2-tuple of lists (shape, stride) providing the
      multi-dimensional shape of the memory area.  The stride
      shows how many bytes to skip in each dimension to move
      in that dimension from the start of the array.

      Memory that is not a single contiguous-buffer can be represented
      with the pointer returned from GetBuffer and the shape and
      strides returned from GetBufferShape.

   int PyObject_SizeFromFormat(char *)
      Return the implied size of the data-format area from a struct-style
      description.

Additions to the struct string-syntax

   The struct string-syntax is missing some characters to fully
   implement data-format descriptions already available elsewhere (in
   ctypes and NumPy for example).  Here are the proposed additions:

   Character         Description
   ==================================
   '1'               bit (number before states how many bits)
   '?'               platform _Bool type
   'g'               long double  
   'F'               complex float  
   'D'               complex double
   'G'               complex long double
   'c'               ucs-1 (latin-1) encoding
   'u'               ucs-2
   'w'               ucs-4
   'O'               pointer to Python Object
   'T{}'             structure (detailed layout inside {})
   '(k1,k2,...,kn)'  multi-dimensional array of whatever follows
   ':name:'          optional name of the preceeding element
   '&'               specific pointer (prefix before another charater)
   'X{}'             pointer to a function (optional function
                                             signature inside {})

   The struct module will be changed to understand these as well and
   return appropriate Python objects on unpacking.  Un-packing a
   long-double will return a c-types long_double.  Unpacking 'u' or
   'w' will return Python unicode.  Unpacking a multi-dimensional
   array will return a list of lists.  Un-packing a pointer will
   return a ctypes pointer object.  Un-packing a bit will return a
   Python Bool.

   Endian-specification ('=','>','<') is also allowed inside the
   string so that it can change if needed.  The previously-specified
   endian string is enforce at all times.  The default endian is '='.

   According to the struct-module, a number can preceed a character
   code to specify how many of that type there are.  The
   (k1,k2,...,kn) extension also allows specifying if the data is
   supposed to be viewed as a (C-style contiguous, last-dimension
   varies the fastest) multi-dimensional array of a particular format.

   Functions should be added to ctypes to create a ctypes object from
   a struct description, and add long-double, and ucs-2 to ctypes.

Code to be affected

   All objects and modules in Python that export or consume the old
   buffer interface will be modified.  Here is a partial list.
     
   * buffer object
   * bytes object
   * string object
   * array module
   * struct module
   * mmap module
   * ctypes module

   anything else using the buffer API

Issues and Details

   The proposed locking mechanism relies entirely on the objects
   implementing the buffer interface to do their own thing.  Ideally
   an object that implements the buffer interface should keep at least
   a number indicating how many releases are extant.

   The handling of discontiguous memory is new and can be seen as a
   modification of the multiple-segment interface.  It is motivated by
   NumPy (used to be Numeric).  NumPy objects should be able to share
   their strided memory with code that understands how to manage
   strided memory.

   Code should also be able to request contiguous memory if needed and
   objects exporting the buffer interface should be able to handle
   that either by raising an error (or constructing a read-only
   contiguous object and returning that as the view).

   Currently the struct module does not allow specification of nested
   structures.  It seems like specifying a nested structure should be
   specified as several ways of viewing memory areas (ctypes and
   NumPy) already allow this.

Copyright

   This PEP is placed in the public domain



From peridot.faceted at gmail.com  Tue Feb 27 20:42:53 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 27 Feb 2007 20:42:53 -0500
Subject: [Numpy-discussion] Buffer Interface for Python 3.0
In-Reply-To: <45E4BEEF.4030609@ee.byu.edu>
References: <45E4BEEF.4030609@ee.byu.edu>
Message-ID: <ce557a360702271742w21f5fd34jafeed5bf4a7121cf@mail.gmail.com>

On 27/02/07, Travis Oliphant <oliphant at ee.byu.edu> wrote:

> Basically, what we are going to do now is
>
> 1) Return the data-format specification in an extended struct-style string
> 2) Return the shape information in a tuple of lists: (shape, strides)
>
> There are two questions I'm grappling with right now:
>
> 1) Do we propose the inclusion of offsets in the shape information?
> NumPy does not use offsets internally but simply has a pointer to the
> start of the array.

I'm not quite sure I understand what this means. Correct me if I'm
wrong, but within numpy, an array typically lives inside a hunk of
memory allocated with malloc(); the first data element is somewhere
inside that, and any data elements are distributed according to
strides. Is that about right? The array object needs to know the
location of the first element, the strides and sizes, the data type of
each element, and it seems to me it also needs the address of the data
area, so that that can be free()d when the last array using that hunk
of memory is deallocated. In fact it would need a refcounted link to
the array...

Or, if this isn't how it works, how does numpy arrange for the array's
memory to be deleted at the right time? Do numpy arrays keep a
refcounted link to the array that "owns" the memory?

How is memory deallocation managed for the buffer protocol? It seems
like what one needs to access the memory is a buffer object plus an
offset (plus the usual strides and whatnot).

> 2) The buffer interface needs to understand the idea of discontiguous
> arrays.  If the shape/stride information is separate from the
> pointer-to-data call, then the user needs to know if that
> pointer-to-data is a "contiguous chunk" or just the beginning of a
> strided memory area (and so should not be treated as a single-segment).
>
> 3) If we support strided memory areas, then we should probably also
> allow some way for PIL-like objects to report their buffer sequence (I'm
> sure this was the origin of the multi-segment buffer protocol to begin
> with).  Or we could just ignore that possibility.  The PIL would have to
> copy memory in order to share it's images.

I'm not quite sure I understand what you mean by "contiguous" here.
One interpretation would be that any array that uses every byte
between the first and last is contiguous, and any other is
discontiguous. Another would be that any array that can be described
by strides and an offest is contiguous, as it must live in a
contiguous block of malloc()ed (or mmap()ed or whatever) memory;
discontiguous arrays would then be things like C's
array-of-pointers-to-arrays arrangement, for which each row would be a
single malloc()ed chunk but the chunks might be arranged arbitrarily
in memory.

If the former, I can't see why we would not support them, since they
naturally occur in numpy and are tidily handled by the
(shape,strides,offset) information. If the latter, supporting them is
going to be a real challenge, involving a great deal of indirection...
would the goal be to make them accessible through an interface
resembling numpy's indexing?

Anne


From charlesr.harris at gmail.com  Tue Feb 27 21:55:16 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 27 Feb 2007 19:55:16 -0700
Subject: [Numpy-discussion] Draft PEP for the new buffer interface to be
	in Python 3000
In-Reply-To: <45E4D589.8080804@ee.byu.edu>
References: <45E4D589.8080804@ee.byu.edu>
Message-ID: <e06186140702271855r3787b232s22aa60aea4a65ab@mail.gmail.com>

On 2/27/07, Travis Oliphant <oliphant at ee.byu.edu> wrote:
>
> PEP: <unassigned>
> Title: Revising the buffer protocol
> Version: $Revision: $
> Last-Modified: $Date:  $
> Author: Travis Oliphant <oliphant at ee.byu.edu>
> Status: Draft
> Type: Standards Track
> Created: 28-Aug-2006
> Python-Version: 3000


<snip>

>
> Additions to the struct string-syntax
>
>    The struct string-syntax is missing some characters to fully
>    implement data-format descriptions already available elsewhere (in
>    ctypes and NumPy for example).  Here are the proposed additions:
>
>    Character         Description
>    ==================================
>    '1'               bit (number before states how many bits)
>    '?'               platform _Bool type
>    'g'               long double
>    'F'               complex float
>    'D'               complex double
>    'G'               complex long double
>    'c'               ucs-1 (latin-1) encoding
>    'u'               ucs-2
>    'w'               ucs-4
>    'O'               pointer to Python Object
>    'T{}'             structure (detailed layout inside {})
>    '(k1,k2,...,kn)'  multi-dimensional array of whatever follows
>    ':name:'          optional name of the preceeding element
>    '&'               specific pointer (prefix before another charater)
>    'X{}'             pointer to a function (optional function
>                                              signature inside {})


I think it might be good to have something for the quad and half precision
floats that will be coming along in the next IEEE754  specification. Quad
precision isn't used that much, but when you need it, it is useful. Half
precision (16 bits) is used in some GPU's and I have seen it used for such
things as recording side looking radar returns.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070227/4ec9a929/attachment.html>

From oliphant at ee.byu.edu  Tue Feb 27 22:13:31 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Tue, 27 Feb 2007 20:13:31 -0700
Subject: [Numpy-discussion] Draft PEP for the new buffer interface to
 be	in Python 3000
In-Reply-To: <e06186140702271855r3787b232s22aa60aea4a65ab@mail.gmail.com>
References: <45E4D589.8080804@ee.byu.edu>
	<e06186140702271855r3787b232s22aa60aea4a65ab@mail.gmail.com>
Message-ID: <45E4F35B.9010301@ee.byu.edu>

Charles R Harris wrote:
>
>
> On 2/27/07, *Travis Oliphant* <oliphant at ee.byu.edu 
> <mailto:oliphant at ee.byu.edu>> wrote:
>
>     PEP: <unassigned>
>     Title: Revising the buffer protocol
>     Version: $Revision: $
>     Last-Modified: $Date:  $
>     Author: Travis Oliphant <oliphant at ee.byu.edu
>     <mailto:oliphant at ee.byu.edu>>
>     Status: Draft
>     Type: Standards Track
>     Created: 28-Aug-2006
>     Python-Version: 3000
>
>
> <snip>
>
>
>     Additions to the struct string-syntax
>
>        The struct string-syntax is missing some characters to fully
>        implement data-format descriptions already available elsewhere (in
>        ctypes and NumPy for example).  Here are the proposed additions:
>
>        Character         Description
>        ==================================
>        '1'               bit (number before states how many bits)
>        '?'               platform _Bool type
>        'g'               long double
>        'F'               complex float
>        'D'               complex double
>        'G'               complex long double
>        'c'               ucs-1 (latin-1) encoding
>        'u'               ucs-2
>        'w'               ucs-4
>        'O'               pointer to Python Object
>        'T{}'             structure (detailed layout inside {})
>        '(k1,k2,...,kn)'  multi-dimensional array of whatever follows
>        ':name:'          optional name of the preceeding element
>        '&'               specific pointer (prefix before another charater)
>        'X{}'             pointer to a function (optional function
>                                                  signature inside {})
>
>
> I think it might be good to have something for the quad and half 
> precision floats that will be coming along in the next IEEE754  
> specification. Quad precision isn't used that much, but when you need 
> it, it is useful. Half precision (16 bits) is used in some GPU's and I 
> have seen it used for such things as recording side looking radar 
> returns.

The problem is that we aren't really specifying floating-point 
standards, we are specifying float, double and long double as whatever 
the compiler understands.  

There are some platforms which don't follow the IEEE 754 standard.   
This format specification will not be able to describe 
platform-independent floating-point descriptions.  

It would be nice to have such a description, but that is not what 
struct-style syntax does.  Perhaps we could add it in the specification, 
but I'm not sure if the added complexity is worth holding it up over.

-Travis



From steve at shrogers.com  Tue Feb 27 23:05:58 2007
From: steve at shrogers.com (Steven H. Rogers)
Date: Tue, 27 Feb 2007 21:05:58 -0700
Subject: [Numpy-discussion] NumPy in Teaching
Message-ID: <45E4FFA6.9010408@shrogers.com>

I'm doing an informal survey on the use of Array Programming Languages 
for teaching.  If you're using NumPy in this manner I'd like to hear 
from you.  What subject was/is taught, academic level, results, lessons 
learned, etc.

Regards,
Steve



From peridot.faceted at gmail.com  Tue Feb 27 23:08:41 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 27 Feb 2007 23:08:41 -0500
Subject: [Numpy-discussion] Draft PEP for the new buffer interface to be
	in Python 3000
In-Reply-To: <45E4F35B.9010301@ee.byu.edu>
References: <45E4D589.8080804@ee.byu.edu>
	<e06186140702271855r3787b232s22aa60aea4a65ab@mail.gmail.com>
	<45E4F35B.9010301@ee.byu.edu>
Message-ID: <ce557a360702272008j7ca51e35s3753d67066c85c46@mail.gmail.com>

On 27/02/07, Travis Oliphant <oliphant at ee.byu.edu> wrote:

> The problem is that we aren't really specifying floating-point
> standards, we are specifying float, double and long double as whatever
> the compiler understands.
>
> There are some platforms which don't follow the IEEE 754 standard.
> This format specification will not be able to describe
> platform-independent floating-point descriptions.
>
> It would be nice to have such a description, but that is not what
> struct-style syntax does.  Perhaps we could add it in the specification,
> but I'm not sure if the added complexity is worth holding it up over.

Hmm. If this is to be used to describe, say, binary data in files on
disk, having machine-independent formats would be very handy. The
endianness specifiers are there to provide this for integers, because
it's so useful. I realize that if a machine doesn't implement IEEE
floats it will be pretty much impossible to implement python functions
to work with them, or even just decode them, but it would be nice to
be able to at least *specify* them.

How much more complicated would it be to allow their specification?
One letter for each IEEE type, in addition to the existing letters for
platform-specific floats?

Anne M. Archibald


From charlesr.harris at gmail.com  Tue Feb 27 23:10:22 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 27 Feb 2007 21:10:22 -0700
Subject: [Numpy-discussion] Draft PEP for the new buffer interface to be
	in Python 3000
In-Reply-To: <45E4F35B.9010301@ee.byu.edu>
References: <45E4D589.8080804@ee.byu.edu>
	<e06186140702271855r3787b232s22aa60aea4a65ab@mail.gmail.com>
	<45E4F35B.9010301@ee.byu.edu>
Message-ID: <e06186140702272010n52439c3ck2106775529246714@mail.gmail.com>

On 2/27/07, Travis Oliphant <oliphant at ee.byu.edu> wrote:
>
> Charles R Harris wrote:
> >
> >
> > On 2/27/07, *Travis Oliphant* <oliphant at ee.byu.edu
> > <mailto:oliphant at ee.byu.edu>> wrote:
> >
> >     PEP: <unassigned>
> >     Title: Revising the buffer protocol
> >     Version: $Revision: $
> >     Last-Modified: $Date:  $
> >     Author: Travis Oliphant <oliphant at ee.byu.edu
> >     <mailto:oliphant at ee.byu.edu>>
> >     Status: Draft
> >     Type: Standards Track
> >     Created: 28-Aug-2006
> >     Python-Version: 3000
> >
> >
> > <snip>
> >
> >
> >     Additions to the struct string-syntax
> >
> >        The struct string-syntax is missing some characters to fully
> >        implement data-format descriptions already available elsewhere
> (in
> >        ctypes and NumPy for example).  Here are the proposed additions:
> >
> >        Character         Description
> >        ==================================
> >        '1'               bit (number before states how many bits)
> >        '?'               platform _Bool type
> >        'g'               long double
> >        'F'               complex float
> >        'D'               complex double
> >        'G'               complex long double
> >        'c'               ucs-1 (latin-1) encoding
> >        'u'               ucs-2
> >        'w'               ucs-4
> >        'O'               pointer to Python Object
> >        'T{}'             structure (detailed layout inside {})
> >        '(k1,k2,...,kn)'  multi-dimensional array of whatever follows
> >        ':name:'          optional name of the preceeding element
> >        '&'               specific pointer (prefix before another
> charater)
> >        'X{}'             pointer to a function (optional function
> >                                                  signature inside {})
> >
> >
> > I think it might be good to have something for the quad and half
> > precision floats that will be coming along in the next IEEE754
> > specification. Quad precision isn't used that much, but when you need
> > it, it is useful. Half precision (16 bits) is used in some GPU's and I
> > have seen it used for such things as recording side looking radar
> > returns.
>
> The problem is that we aren't really specifying floating-point
> standards, we are specifying float, double and long double as whatever
> the compiler understands.
>
> There are some platforms which don't follow the IEEE 754 standard.
> This format specification will not be able to describe
> platform-independent floating-point descriptions.
>
> It would be nice to have such a description, but that is not what
> struct-style syntax does.  Perhaps we could add it in the specification,
> but I'm not sure if the added complexity is worth holding it up over.


True enough, and it may not make that much sense until it is in the c
standard. But it might be nice to reserve something for the future and maybe
give some thought of how to deal with new data types as they come along. I
can't think of any really flexible methods that don't require some sort of
verbose table that goes along with the data, and the single letter codes are
starting to get out of hand. Hmmm. It would actually be nice to redo things
so that there was a prefix, say z for complex, f for float, then something
for precision. The designation wouldn't be much use without some arithmetic
to go with it and it doesn't make sense to write code for things that don't
exist. I wonder how much of the arithmetic can be abstracted from the data
type?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070227/48da295a/attachment.html>

From oliphant at ee.byu.edu  Tue Feb 27 23:38:55 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Tue, 27 Feb 2007 21:38:55 -0700
Subject: [Numpy-discussion] Draft PEP for the new buffer interface to
 be	in Python 3000
In-Reply-To: <e06186140702272010n52439c3ck2106775529246714@mail.gmail.com>
References: <45E4D589.8080804@ee.byu.edu>	<e06186140702271855r3787b232s22aa60aea4a65ab@mail.gmail.com>	<45E4F35B.9010301@ee.byu.edu>
	<e06186140702272010n52439c3ck2106775529246714@mail.gmail.com>
Message-ID: <45E5075F.6090608@ee.byu.edu>

Charles R Harris wrote:
>
>
>     The problem is that we aren't really specifying floating-point
>     standards, we are specifying float, double and long double as whatever
>     the compiler understands.
>
>     There are some platforms which don't follow the IEEE 754 standard.
>     This format specification will not be able to describe
>     platform-independent floating-point descriptions.
>
>     It would be nice to have such a description, but that is not what
>     struct-style syntax does.  Perhaps we could add it in the
>     specification,
>     but I'm not sure if the added complexity is worth holding it up over.
>
>
> True enough, and it may not make that much sense until it is in the c 
> standard. But it might be nice to reserve something for the future and 
> maybe give some thought of how to deal with new data types as they 
> come along. I can't think of any really flexible methods that don't 
> require some sort of verbose table that goes along with the data, and 
> the single letter codes are starting to get out of hand. Hmmm. It 
> would actually be nice to redo things so that there was a prefix, say 
> z for complex, f for float, then something for precision. The 
> designation wouldn't be much use without some arithmetic to go with it 
> and it doesn't make sense to write code for things that don't exist. I 
> wonder how much of the arithmetic can be abstracted from the data type?

I suspect we may have to do this separately in the NumPy world.   
Perhaps we could get such a specification into Python itself, but I'm 
not hopeful.  Notice, though that we could use the struct syntax to 
specify a floating-point structure using  the bit-field and naming.

In other words an IEEE 754 32-bit float would be represented in 
struct-style syntax as

'>1t:sign: 8t:exp: 23t:mantissa:'



-Travis





From charlesr.harris at gmail.com  Tue Feb 27 23:54:42 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 27 Feb 2007 21:54:42 -0700
Subject: [Numpy-discussion] Draft PEP for the new buffer interface to be
	in Python 3000
In-Reply-To: <45E5075F.6090608@ee.byu.edu>
References: <45E4D589.8080804@ee.byu.edu>
	<e06186140702271855r3787b232s22aa60aea4a65ab@mail.gmail.com>
	<45E4F35B.9010301@ee.byu.edu>
	<e06186140702272010n52439c3ck2106775529246714@mail.gmail.com>
	<45E5075F.6090608@ee.byu.edu>
Message-ID: <e06186140702272054w7f30ec47l7dce6b5ad3416c52@mail.gmail.com>

On 2/27/07, Travis Oliphant <oliphant at ee.byu.edu> wrote:
>
> Charles R Harris wrote:
> >
> >
> >     The problem is that we aren't really specifying floating-point
> >     standards, we are specifying float, double and long double as
> whatever
> >     the compiler understands.
> >
> >     There are some platforms which don't follow the IEEE 754 standard.
> >     This format specification will not be able to describe
> >     platform-independent floating-point descriptions.
> >
> >     It would be nice to have such a description, but that is not what
> >     struct-style syntax does.  Perhaps we could add it in the
> >     specification,
> >     but I'm not sure if the added complexity is worth holding it up
> over.
> >
> >
> > True enough, and it may not make that much sense until it is in the c
> > standard. But it might be nice to reserve something for the future and
> > maybe give some thought of how to deal with new data types as they
> > come along. I can't think of any really flexible methods that don't
> > require some sort of verbose table that goes along with the data, and
> > the single letter codes are starting to get out of hand. Hmmm. It
> > would actually be nice to redo things so that there was a prefix, say
> > z for complex, f for float, then something for precision. The
> > designation wouldn't be much use without some arithmetic to go with it
> > and it doesn't make sense to write code for things that don't exist. I
> > wonder how much of the arithmetic can be abstracted from the data type?
>
> I suspect we may have to do this separately in the NumPy world.
> Perhaps we could get such a specification into Python itself, but I'm
> not hopeful.  Notice, though that we could use the struct syntax to
> specify a floating-point structure using  the bit-field and naming.
>
> In other words an IEEE 754 32-bit float would be represented in
> struct-style syntax as
>
> '>1t:sign: 8t:exp: 23t:mantissa:'


That would probably do nicely. There are potential ambiguities but nothing
worth worrying about. Is there a way to assign names to such a type? I
suppose that it is just another string constant so one could write something
like

float32 = '>1t:sign: 8t:exp: 23t:mantissa:'

and use that. Can those bit fields be of arbitrary length?

Now for something completely different ;) In some things, like the socket
module, it is possible to ask for a filelike interface which buffers the
input and has the usual read, readline, etc function interface, but fromfile
doesn't work with it. This isn't a biggie and I suppose fromfile is looking
for a 'real' file, but I wonder if this would be a difficult thing to
implement? I could look at the code but I thought I would ask you first.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070227/9b17cef7/attachment.html>

From oliphant at ee.byu.edu  Wed Feb 28 00:11:26 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Tue, 27 Feb 2007 22:11:26 -0700
Subject: [Numpy-discussion] Draft PEP for the new buffer interface to
 be	in Python 3000
In-Reply-To: <e06186140702272054w7f30ec47l7dce6b5ad3416c52@mail.gmail.com>
References: <45E4D589.8080804@ee.byu.edu>	<e06186140702271855r3787b232s22aa60aea4a65ab@mail.gmail.com>	<45E4F35B.9010301@ee.byu.edu>	<e06186140702272010n52439c3ck2106775529246714@mail.gmail.com>	<45E5075F.6090608@ee.byu.edu>
	<e06186140702272054w7f30ec47l7dce6b5ad3416c52@mail.gmail.com>
Message-ID: <45E50EFE.3080802@ee.byu.edu>

Charles R Harris wrote:
>
>
> On 2/27/07, *Travis Oliphant* <oliphant at ee.byu.edu 
> <mailto:oliphant at ee.byu.edu>> wrote:
>
>     Charles R Harris wrote:
>     >
>     >
>     >     The problem is that we aren't really specifying floating-point
>     >     standards, we are specifying float, double and long double
>     as whatever
>     >     the compiler understands.
>     >
>     >     There are some platforms which don't follow the IEEE 754
>     standard.
>     >     This format specification will not be able to describe
>     >     platform-independent floating-point descriptions.
>     >
>     >     It would be nice to have such a description, but that is not
>     what
>     >     struct-style syntax does.  Perhaps we could add it in the
>     >     specification,
>     >     but I'm not sure if the added complexity is worth holding it
>     up over.
>     >
>     >
>     > True enough, and it may not make that much sense until it is in
>     the c
>     > standard. But it might be nice to reserve something for the
>     future and
>     > maybe give some thought of how to deal with new data types as they
>     > come along. I can't think of any really flexible methods that don't
>     > require some sort of verbose table that goes along with the
>     data, and
>     > the single letter codes are starting to get out of hand. Hmmm. It
>     > would actually be nice to redo things so that there was a
>     prefix, say
>     > z for complex, f for float, then something for precision. The
>     > designation wouldn't be much use without some arithmetic to go
>     with it
>     > and it doesn't make sense to write code for things that don't
>     exist. I
>     > wonder how much of the arithmetic can be abstracted from the
>     data type?
>
>     I suspect we may have to do this separately in the NumPy world.
>     Perhaps we could get such a specification into Python itself, but I'm
>     not hopeful.  Notice, though that we could use the struct syntax to
>     specify a floating-point structure using  the bit-field and naming.
>
>     In other words an IEEE 754 32-bit float would be represented in
>     struct-style syntax as
>
>     '>1t:sign: 8t:exp: 23t:mantissa:'
>
>
> That would probably do nicely. There are potential ambiguities but 
> nothing worth worrying about. Is there a way to assign names to such a 
> type? I suppose that it is just another string constant so one could 
> write something like
>
> float32 = '>1t:sign: 8t:exp: 23t:mantissa:'
>
> and use that. Can those bit fields be of arbitrary length?
>
> Now for something completely different ;) In some things, like the 
> socket module, it is possible to ask for a filelike interface which 
> buffers the input and has the usual read, readline, etc function 
> interface, but fromfile doesn't work with it. This isn't a biggie and 
> I suppose fromfile is looking for a 'real' file, but I wonder if this 
> would be a difficult thing to implement? I could look at the code but 
> I thought I would ask you first.

The problem here is that fromfile is using the raw stdio fscanf commands 
which require an actual file id.    It is not using the Python-level 
fread.   It's pretty low-level.   On the other-hand there is the 
fromstring approach which works with any stream.   I suspect a function 
that uses one or the other could be implemented.

The relevant functions are  XXXX_scan and XXX_fromstr in arraytypes.c.src

These are used for each data-type.  Notice that PyArray_FromFile 
actually requires a FILE *fp pointer.   You might be able to use 
PyArray_FromString which allows a char * to read data from.

-Travis





>
> Chuck
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From wright at esrf.fr  Wed Feb 28 04:58:52 2007
From: wright at esrf.fr (Jon Wright)
Date: Wed, 28 Feb 2007 10:58:52 +0100
Subject: [Numpy-discussion] How to tell numpy to use gfortran as a,
	compiler
In-Reply-To: <mailman.21164.1172421917.8339.numpy-discussion@scipy.org>
References: <mailman.21164.1172421917.8339.numpy-discussion@scipy.org>
Message-ID: <45E5525C.10005@esrf.fr>


>Date: Sun, 25 Feb 2007 15:43:03 +0100 (CET)
>From: "Sturla Molden" <sturla at molden.no>
>Subject: Re: [Numpy-discussion] How to tell numpy to use gfortran as a
>	compiler ?
>Content-Type: text/plain;charset=iso-8859-1
>  
>
[...]

>There is a third Fortran compiler based on the GCC backend called "g95".
>It is not a part of GCC and uses copyrighted source code illegally
>(numerous GPL violations). The head developer is rumored to have serious
>bad karma. Apart from that, g95 is an ok Fortran 95 compiler. Always
>ensure that ambiguous switches like "gnu95" means gfortran and not g95. I
>don't know what NumPy does.
>  
>

These comments are out of order - even for the internet. I remember 
gfortran was a fork of g95 which occured when the developers wanted to 
go in different directions. It does not violate the gpl, source is 
available for download.

Jon





From hayes.tyler at gmail.com  Wed Feb 28 09:16:27 2007
From: hayes.tyler at gmail.com (Tyler Hayes)
Date: Wed, 28 Feb 2007 09:16:27 -0500
Subject: [Numpy-discussion] f2py and Fortran90 gfortran_filename error
Message-ID: <92bcafc50702280616n4fc74500ic8750b96ca9a2ad3@mail.gmail.com>

Hello All:

I was suggested to post this question here with the f2py experts from
comp.lang.python.

I have been able to use the example F77 files suggested in the f2py
User Manual, but the problems happened when I tried my own F90 file
(attached below.

I am able to create a <module_name>.so file using the following method:

(1) Created a Fortran90 program matsolve.f90

Note: The program compiles fine and prints the proper output for the
simple matrix specified.

(2) f2py matsolve.f90 -m matsolve2 -h matsolve2.pyf

This created the matsolve2.pyf fine

(3) f2py -c matsolve2.pyf --f90exec=/usr/bin/gfortran matsolve.f90

Note: I had to specify the f90exec path as f2py did not automatically find
it. This was the only way I could generate a *.so file.

The rub:

When I import it into Python, I receive the following
message:

>>> import matsolve2

Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ImportError: ./matsolve2.so: undefined symbol: _gfortran_filename

Any suggestions are greatly appreciated. (I almost started smoking
again last night! ARGH!)

Cheers,

t.

PS - Below I have attached the F90 program for LU decomposition with a
simple test case.


! MATSOLVE.f90
!
! Start main program
PROGRAM MATSOLVE
  IMPLICIT NONE
  INTEGER,PARAMETER :: n=3
  INTEGER :: i,j
  REAL,DIMENSION(n) :: x,b
  REAL,DIMENSION(n,n) :: A,L,U
! Initialize the vectors and matrices with a test case from text
! Using the one given in Appendix A from Thompson.
! Known vector "b"
  b(1) = 12.
  b(2) = 11.
  b(3) =  2.

! Known coefficient matrix "A", and initialize L and U
  DO i=1,n
     DO j=1,n
        L(i,j) = 0.
        U(i,j) = 0.
     END DO
  END DO

! Create matrix A
  A(1,1) =  3.
  A(1,2) = -1.
  A(1,3) =  2.

  A(2,1) = 1.
  A(2,2) = 2.
  A(2,3) = 3.

  A(3,1) =  2.
  A(3,2) = -2.
  A(3,3) = -1.

! Call subroutine to create L and U matrices from A
  CALL lumake(L,U,A,n)
! Print results
  PRINT *, '-----------------------'
  DO i=1,n
     DO j=1,n
        PRINT *, i, j, A(i,j), L(i,j), U(i,j)
     END DO
  END DO
  PRINT *, '-----------------------'

! Call subroutine to solve for "x" using L and U
  CALL lusolve(x,L,U,b,n)
! Print results
  PRINT *, '-----------------------'
  DO i=1,n
     PRINT *, i, x(i)
  END DO
  PRINT *, '-----------------------'

END PROGRAM MATSOLVE

! Create subroutine to make L and U matrices
SUBROUTINE lumake(LL,UU,AA,n1)
  IMPLICIT NONE
  INTEGER,PARAMETER :: n=3
  INTEGER :: i,j,k
  REAL :: LUSUM
  INTEGER,INTENT(IN) :: n1
  REAL,DIMENSION(n,n),INTENT(IN) :: AA
  REAL,DIMENSION(n,n),INTENT(OUT) :: LL,UU

! We first note that the diagonal in our UPPER matrix is
! going to be UU(j,j) = 1.0, this allows us to initialize
! the first set of expressions
  UU(1,1) = 1.

! Find first column of LL
  DO i = 1,n1
     LL(i,1) = AA(i,1)/UU(1,1)
  END DO

! Now find first row of UU
  DO j = 2,n1
     UU(1,j) = AA(1,j)/LL(1,1)
  END DO

! Now find middle LL elements
  DO j = 2,n1
     DO i = j,n1
        LUSUM = 0.
        DO k = 1,j-1
           LUSUM = LUSUM + LL(i,k)*UU(k,j)
        END DO
        LL(i,j) = AA(i,j) - LUSUM
     END DO

! Set Diagonal UU
     UU(j,j) = 1.

! Now find middle UU elements
     DO i = j+1,n1
        LUSUM = 0.
        DO k = 1,j-1
           LUSUM = LUSUM + LL(j,k)*UU(k,i)
        END DO
        UU(j,i) = (AA(j,i) - LUSUM)/LL(j,j)
     END DO
  END DO
END SUBROUTINE lumake

! Make subroutine to solve for x
SUBROUTINE lusolve(xx,L2,U2,bb,n2)
  IMPLICIT NONE
  INTEGER,PARAMETER :: n=3
  INTEGER :: i,j,k
  REAL :: LYSUM,UXSUM
  REAL,DIMENSION(n):: y
  INTEGER,INTENT(IN) :: n2
  REAL,DIMENSION(n),INTENT(IN) :: bb
  REAL,DIMENSION(n,n),INTENT(IN) :: L2,U2
  REAL,DIMENSION(n),INTENT(OUT) :: xx

! Initialize
  DO i=1,n2
     y(i)  = 0.
     xx(i) = 0.
  END DO

! Solve L.y = b
  y(1) = bb(1)/L2(1,1)
  DO i = 2,n2
     LYSUM = 0.
     DO k = 1,i-1
        LYSUM = LYSUM + L2(i,k)*y(k)
     END DO
     y(i) = (bb(i) - LYSUM)/L2(i,i)
  END DO

! Now do back subsitution for U.x = y
  xx(n2) = y(n2)/U2(n2,n2)
  DO j = n2-1,1,-1
     UXSUM = 0.
     DO k = j+1,n2
        UXSUM = UXSUM + U2(j,k)*xx(k)
     END DO
     xx(j) = y(j) - UXSUM
  END DO
END SUBROUTINE lusolve


-- 
Tyler Joseph Hayes
600 Talbot St. -- Apt. 812
London, Ontario
N6A 5L9

Tel   : 519.435.0967
Fax  : 519.661.3198
Cell  : 416.655.7897
email: thayes at uwo.ca

GPG Key ID# 0x3AE05130


From pearu at cens.ioc.ee  Wed Feb 28 09:25:28 2007
From: pearu at cens.ioc.ee (pearu at cens.ioc.ee)
Date: Wed, 28 Feb 2007 16:25:28 +0200 (EET)
Subject: [Numpy-discussion] f2py and Fortran90 gfortran_filename error
In-Reply-To: <92bcafc50702280616n4fc74500ic8750b96ca9a2ad3@mail.gmail.com>
References: <92bcafc50702280616n4fc74500ic8750b96ca9a2ad3@mail.gmail.com>
Message-ID: <57756.82.131.71.232.1172672728.squirrel@cens.ioc.ee>

> Hello All:
>
> I was suggested to post this question here with the f2py experts from
> comp.lang.python.
>
> I have been able to use the example F77 files suggested in the f2py
> User Manual, but the problems happened when I tried my own F90 file
> (attached below.
>
> I am able to create a <module_name>.so file using the following method:
>
> (1) Created a Fortran90 program matsolve.f90
>
> Note: The program compiles fine and prints the proper output for the
> simple matrix specified.
>
> (2) f2py matsolve.f90 -m matsolve2 -h matsolve2.pyf
>
> This created the matsolve2.pyf fine
>
> (3) f2py -c matsolve2.pyf --f90exec=/usr/bin/gfortran matsolve.f90

Use
 f2py -c  --fcompiler=gnu95 matsolve2.pyf matsolve.f90

HTH,
Pearu




From hayes.tyler at gmail.com  Wed Feb 28 10:00:13 2007
From: hayes.tyler at gmail.com (Tyler Hayes)
Date: Wed, 28 Feb 2007 10:00:13 -0500
Subject: [Numpy-discussion] f2py and Fortran90 gfortran_filename error
In-Reply-To: <57756.82.131.71.232.1172672728.squirrel@cens.ioc.ee>
References: <92bcafc50702280616n4fc74500ic8750b96ca9a2ad3@mail.gmail.com>
	<57756.82.131.71.232.1172672728.squirrel@cens.ioc.ee>
Message-ID: <92bcafc50702280700y248180b2v6e6a692bc73546cb@mail.gmail.com>

Thanks for helping out Pearu,

But it won't even create the .so file using that command. Using what
you suggested, seems to work as it goes through many of the processes
but then f2py stops at:

error: don't know how to compile Fortran code on platform 'posix' with
'gnu95' compiler. Supported compilers are:
compaq,absoft,intel,gnu,sun,f,vast,ibm,lahey,intelv,intele,pg,compaqv,mips,hpux,intelev,nag)

Even more weird is on my laptop the situation is even worse. Check out
this error which I seem to get regardless of whether I use the gnu95
line like you suggested on the --f90exec option.

thayes at seneca:~/fortran90/am562b/misc_finite$ f2py -c
--fcompiler=gnu95 matsolve2.pyf matsolve.f90
Unknown vendor: "gnu95"
numpy_info:
  FOUND:
    define_macros = [('NUMERIC_VERSION', '"\\"24.2\\""')]
    include_dirs = ['/usr/include/python2.4']

Traceback (most recent call last):
  File "/usr/bin/f2py", line 4, in ?
    f2py2e.main()
  File "/usr/lib/python2.4/site-packages/f2py2e/f2py2e.py", line 677, in main
    run_compile()
  File "/usr/lib/python2.4/site-packages/f2py2e/f2py2e.py", line 650,
in run_compile
    ext = Extension(**ext_args)
  File "/usr/lib/python2.4/site-packages/scipy_distutils/extension.py",
line 45, in __init__
    export_symbols)
  File "/usr/lib/python2.4/distutils/extension.py", line 106, in __init__
    assert type(name) is StringType, "'name' must be a string"
AssertionError: 'name' must be a string

Thanks for the help anyways.

Cheers,

t.

-- 
Tyler Joseph Hayes

GPG Key ID# 0x3AE05130


From pearu at cens.ioc.ee  Wed Feb 28 10:09:11 2007
From: pearu at cens.ioc.ee (pearu at cens.ioc.ee)
Date: Wed, 28 Feb 2007 17:09:11 +0200 (EET)
Subject: [Numpy-discussion] f2py and Fortran90 gfortran_filename error
In-Reply-To: <92bcafc50702280700y248180b2v6e6a692bc73546cb@mail.gmail.com>
References: <92bcafc50702280616n4fc74500ic8750b96ca9a2ad3@mail.gmail.com>
	<57756.82.131.71.232.1172672728.squirrel@cens.ioc.ee>
	<92bcafc50702280700y248180b2v6e6a692bc73546cb@mail.gmail.com>
Message-ID: <20163.82.131.71.232.1172675351.squirrel@cens.ioc.ee>

> Thanks for helping out Pearu,
>
> But it won't even create the .so file using that command. Using what
> you suggested, seems to work as it goes through many of the processes
> but then f2py stops at:
>
> error: don't know how to compile Fortran code on platform 'posix' with
> 'gnu95' compiler. Supported compilers are:
> compaq,absoft,intel,gnu,sun,f,vast,ibm,lahey,intelv,intele,pg,compaqv,mips,hpux,intelev,nag)
>
> Even more weird is on my laptop the situation is even worse. Check out
> this error which I seem to get regardless of whether I use the gnu95
> line like you suggested on the --f90exec option.
>
> thayes at seneca:~/fortran90/am562b/misc_finite$ f2py -c
> --fcompiler=gnu95 matsolve2.pyf matsolve.f90
> Unknown vendor: "gnu95"
> numpy_info:
>   FOUND:
>     define_macros = [('NUMERIC_VERSION', '"\\"24.2\\""')]
>     include_dirs = ['/usr/include/python2.4']

The problem is that you are using Numeric and f2py2e combination that do
not have gfortran support (in f2py2e, to be specific).
You can either switch to Numpy (that includes f2py with gfortran support)
or use your original command line with -lgfortran specified:

  f2py -c matsolve2.pyf --f90exec=/usr/bin/gfortran matsolve.f90 -lgfortran

If you get more undefined symbol errors then find out which libraries
define these symbols and include the corresponding -l<libname> switches.

Note that it is recommended to switch using to numpy as f2py has better
F90 support there.

Pearu





From hayes.tyler at gmail.com  Wed Feb 28 10:29:10 2007
From: hayes.tyler at gmail.com (Tyler Hayes)
Date: Wed, 28 Feb 2007 10:29:10 -0500
Subject: [Numpy-discussion] f2py and Fortran90 gfortran_filename error
In-Reply-To: <20163.82.131.71.232.1172675351.squirrel@cens.ioc.ee>
References: <92bcafc50702280616n4fc74500ic8750b96ca9a2ad3@mail.gmail.com>
	<57756.82.131.71.232.1172672728.squirrel@cens.ioc.ee>
	<92bcafc50702280700y248180b2v6e6a692bc73546cb@mail.gmail.com>
	<20163.82.131.71.232.1172675351.squirrel@cens.ioc.ee>
Message-ID: <92bcafc50702280729k5ce72643k1c8009eaeffa6df6@mail.gmail.com>

Hi Pearu:

Well, the -l option worked!!! (May you live to be one thousand.)

To be honest though, I had thought I was using NumPy (I even purchased
the NumPy manual). How do I switch f2py to always use NumPy?

Either way, it now works....

Cheers,

t.

On 28/02/07, pearu at cens.ioc.ee <pearu at cens.ioc.ee> wrote:
> > Thanks for helping out Pearu,
> >
> > But it won't even create the .so file using that command. Using what
> > you suggested, seems to work as it goes through many of the processes
> > but then f2py stops at:
> >
> > error: don't know how to compile Fortran code on platform 'posix' with
> > 'gnu95' compiler. Supported compilers are:
> > compaq,absoft,intel,gnu,sun,f,vast,ibm,lahey,intelv,intele,pg,compaqv,mips,hpux,intelev,nag)
> >
> > Even more weird is on my laptop the situation is even worse. Check out
> > this error which I seem to get regardless of whether I use the gnu95
> > line like you suggested on the --f90exec option.
> >
> > thayes at seneca:~/fortran90/am562b/misc_finite$ f2py -c
> > --fcompiler=gnu95 matsolve2.pyf matsolve.f90
> > Unknown vendor: "gnu95"
> > numpy_info:
> >   FOUND:
> >     define_macros = [('NUMERIC_VERSION', '"\\"24.2\\""')]
> >     include_dirs = ['/usr/include/python2.4']
>
> The problem is that you are using Numeric and f2py2e combination that do
> not have gfortran support (in f2py2e, to be specific).
> You can either switch to Numpy (that includes f2py with gfortran support)
> or use your original command line with -lgfortran specified:
>
>   f2py -c matsolve2.pyf --f90exec=/usr/bin/gfortran matsolve.f90 -lgfortran
>
> If you get more undefined symbol errors then find out which libraries
> define these symbols and include the corresponding -l<libname> switches.
>
> Note that it is recommended to switch using to numpy as f2py has better
> F90 support there.
>
> Pearu
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


-- 
Tyler Joseph Hayes

GPG Key ID# 0x3AE05130


From pearu at cens.ioc.ee  Wed Feb 28 11:36:34 2007
From: pearu at cens.ioc.ee (pearu at cens.ioc.ee)
Date: Wed, 28 Feb 2007 18:36:34 +0200 (EET)
Subject: [Numpy-discussion] f2py and Fortran90 gfortran_filename error
In-Reply-To: <92bcafc50702280729k5ce72643k1c8009eaeffa6df6@mail.gmail.com>
References: <92bcafc50702280616n4fc74500ic8750b96ca9a2ad3@mail.gmail.com>
	<57756.82.131.71.232.1172672728.squirrel@cens.ioc.ee>
	<92bcafc50702280700y248180b2v6e6a692bc73546cb@mail.gmail.com>
	<20163.82.131.71.232.1172675351.squirrel@cens.ioc.ee>
	<92bcafc50702280729k5ce72643k1c8009eaeffa6df6@mail.gmail.com>
Message-ID: <20600.82.131.71.232.1172680594.squirrel@cens.ioc.ee>

> Hi Pearu:
>
> Well, the -l option worked!!! (May you live to be one thousand.)
>
> To be honest though, I had thought I was using NumPy (I even purchased
> the NumPy manual). How do I switch f2py to always use NumPy?

Well, it depends how did you installed numpy, before or after installing
f2py2e. Both install f2py scripts. So, just make sure that f2py
script is from numpy installation, for example, reinstalling numpy
should be sufficient (you don't need to rebuild numpy for that).
Also, you can safely remove f2py2e from your system if you are going to
use numpy.

HTH,
Pearu



From lists.steve at arachnedesign.net  Wed Feb 28 11:49:47 2007
From: lists.steve at arachnedesign.net (Steve Lianoglou)
Date: Wed, 28 Feb 2007 11:49:47 -0500
Subject: [Numpy-discussion] [ANN] mlabrap-1.0b: a high level python to
	matlab
In-Reply-To: <yfsr6sbffly.fsf@oc.ex.ac.uk>
References: <yfsr6sbffly.fsf@oc.ex.ac.uk>
Message-ID: <B39DE6A7-E581-4358-9B0E-ACFB2974775A@arachnedesign.net>

Hi,

> Mlabwrap-1.0 is a high-level python to matlab(tm) bridge that makes  
> calling
> matlab functions from python almost as convenient as using a normal  
> python
> library. It is available under a very liberal license (BSD/MIT) and  
> should
> work on all major platforms and (non-ancient) python and matlab  
> versions and
> either numpy or Numeric (Numeric support will be dropped in the  
> future).

Although I haven't had time to play with this just yet, I just want  
to thank you for working on this and sharing it.

It looks to be pretty cool and it's much appreciated.

Cheers,
-steve



From hayes.tyler at gmail.com  Wed Feb 28 12:27:39 2007
From: hayes.tyler at gmail.com (Tyler Hayes)
Date: Wed, 28 Feb 2007 12:27:39 -0500
Subject: [Numpy-discussion] f2py and Fortran90 gfortran_filename error
In-Reply-To: <20600.82.131.71.232.1172680594.squirrel@cens.ioc.ee>
References: <92bcafc50702280616n4fc74500ic8750b96ca9a2ad3@mail.gmail.com>
	<57756.82.131.71.232.1172672728.squirrel@cens.ioc.ee>
	<92bcafc50702280700y248180b2v6e6a692bc73546cb@mail.gmail.com>
	<20163.82.131.71.232.1172675351.squirrel@cens.ioc.ee>
	<92bcafc50702280729k5ce72643k1c8009eaeffa6df6@mail.gmail.com>
	<20600.82.131.71.232.1172680594.squirrel@cens.ioc.ee>
Message-ID: <92bcafc50702280927j6f6db2delf482b97644f736b@mail.gmail.com>

Well, I removed the f2py2e directory and reinstalled numpy as you
suggested. Now, I still use the -lgfortran option and the --f90exec
option, but am no longer getting the "NUMERIC" version.

Thank you for the help. You have been more than patient with me :-)

Now all I have to do is get my laptop to also behave the same way.
f2py creates the .pyf file fine, but balks at the following command.




thayes at seneca$ f2py -c --f90exec=/usr/bin/gfortran matsolve2.pyf
-lgfortran matsolve.f90

/usr/lib/python2.4/site-packages/numpy/ctypeslib.py:12: UserWarning:
All features of ctypes interface may not work with ctypes < 1.0.1
  warnings.warn("All features of ctypes interface may not work with " \
Traceback (most recent call last):
  File "/usr/bin/f2py", line 26, in ?
    main()
  File "/usr/lib/python2.4/site-packages/numpy/f2py/f2py2e.py", line
552, in main
    run_compile()
  File "/usr/lib/python2.4/site-packages/numpy/f2py/f2py2e.py", line
528, in run_compile
    ext = Extension(**ext_args)
  File "/usr/lib/python2.4/site-packages/numpy/distutils/extension.py",
line 45, in __init__
    export_symbols)
  File "/usr/lib/python2.4/distutils/extension.py", line 106, in __init__
    assert type(name) is StringType, "'name' must be a string"
AssertionError: 'name' must be a string





However, I can save this for another day.....

Thanks again,

Tyler


-- 
Tyler Joseph Hayes
GPG Key ID# 0x3AE05130


From pearu at cens.ioc.ee  Wed Feb 28 12:37:42 2007
From: pearu at cens.ioc.ee (pearu at cens.ioc.ee)
Date: Wed, 28 Feb 2007 19:37:42 +0200 (EET)
Subject: [Numpy-discussion] f2py and Fortran90 gfortran_filename error
In-Reply-To: <92bcafc50702280927j6f6db2delf482b97644f736b@mail.gmail.com>
References: <92bcafc50702280616n4fc74500ic8750b96ca9a2ad3@mail.gmail.com>
	<57756.82.131.71.232.1172672728.squirrel@cens.ioc.ee>
	<92bcafc50702280700y248180b2v6e6a692bc73546cb@mail.gmail.com>
	<20163.82.131.71.232.1172675351.squirrel@cens.ioc.ee>
	<92bcafc50702280729k5ce72643k1c8009eaeffa6df6@mail.gmail.com>
	<20600.82.131.71.232.1172680594.squirrel@cens.ioc.ee>
	<92bcafc50702280927j6f6db2delf482b97644f736b@mail.gmail.com>
Message-ID: <20877.82.131.71.232.1172684262.squirrel@cens.ioc.ee>

> Well, I removed the f2py2e directory and reinstalled numpy as you
> suggested. Now, I still use the -lgfortran option and the --f90exec
> option, but am no longer getting the "NUMERIC" version.
>
> Thank you for the help. You have been more than patient with me :-)
>
> Now all I have to do is get my laptop to also behave the same way.
> f2py creates the .pyf file fine, but balks at the following command.
>
>
>
>
> thayes at seneca$ f2py -c --f90exec=/usr/bin/gfortran matsolve2.pyf
> -lgfortran matsolve.f90

You should use

  f2py -c --fcompiler=gnu95 matsolve2.pyf matsolve.f90

no need to specify --f90exec or -lgfortran if you specify the gnu95 compiler.

Pearu



From hayes.tyler at gmail.com  Wed Feb 28 13:42:32 2007
From: hayes.tyler at gmail.com (Tyler Hayes)
Date: Wed, 28 Feb 2007 13:42:32 -0500
Subject: [Numpy-discussion] f2py and Fortran90 gfortran_filename error
In-Reply-To: <92bcafc50702280927j6f6db2delf482b97644f736b@mail.gmail.com>
References: <92bcafc50702280616n4fc74500ic8750b96ca9a2ad3@mail.gmail.com>
	<57756.82.131.71.232.1172672728.squirrel@cens.ioc.ee>
	<92bcafc50702280700y248180b2v6e6a692bc73546cb@mail.gmail.com>
	<20163.82.131.71.232.1172675351.squirrel@cens.ioc.ee>
	<92bcafc50702280729k5ce72643k1c8009eaeffa6df6@mail.gmail.com>
	<20600.82.131.71.232.1172680594.squirrel@cens.ioc.ee>
	<92bcafc50702280927j6f6db2delf482b97644f736b@mail.gmail.com>
Message-ID: <92bcafc50702281042q40e226adk8dd46eb5183bb0c1@mail.gmail.com>

Hi again Pearu:

As you suggested, the gnu95 option works fine on my desktop without
calling the other options, but I think I may have to reinstall
everything from scratch on my laptop as I keep getting that "'name'
must be a string" error message (which doesn't show up on my desktop).

Like I said, I think I'll save that for another day. I really should
get back to coding my assignment now :-)

Anyways, you have been more than helpful, and if you are ever in
London, Ontario, I owe you a drink.

Thanks again.

Cheers,

t,


-- 
Tyler Joseph Hayes
GPG Key ID# 0x3AE05130


From mjanikas at esri.com  Wed Feb 28 14:19:33 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Wed, 28 Feb 2007 11:19:33 -0800
Subject: [Numpy-discussion] Source install
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFD4@hardwire.esri.com>

Hello all,

 

I have used numpy on both Mac and Windows.  The latter is easily
installed with the exe file.  The former required the gcc program from
XCode... but once installed, "python setup.py install" worked.  I cant
seem to get numpy to work on my linux machine.  Can someone point me to
a platform-independent doc on how to install from the source tar file?
Thanks ahead of time,

 

MJ

 

Mark Janikas

Product Engineer

ESRI, Geoprocessing

380 New York St.

Redlands, CA 92373

909-793-2853 (2563)

mjanikas at esri.com

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070228/67b4a46c/attachment.html>

From robert.kern at gmail.com  Wed Feb 28 14:26:27 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 28 Feb 2007 13:26:27 -0600
Subject: [Numpy-discussion] Source install
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFD4@hardwire.esri.com>
References: <627102C921CD9745B070C3B10CB8199B010EBFD4@hardwire.esri.com>
Message-ID: <45E5D763.9050507@gmail.com>

Mark Janikas wrote:
> Hello all,
> 
> I have used numpy on both Mac and Windows.  The latter is easily
> installed with the exe file.  The former required the gcc program from
> XCode? but once installed, ?python setup.py install? worked.  I cant
> seem to get numpy to work on my linux machine.  Can someone point me to
> a platform-independent doc on how to install from the source tar file?
>  Thanks ahead of time,

We need more information from you. There is no way one can make a
platform-independent doc that covers all of the cases. We need to know what you
tried and exactly how it failed (i.e., we need you to copy the exact error
messages and paste them into an email).

If I had to guess, though, since you succeeded doing an install from source on
OS X, the problem on Linux is likely that you do not have the appropriate Python
development package for your system. On RPM-based systems like Fedora Core, it
is usually named something like python-devel.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From oliphant at ee.byu.edu  Wed Feb 28 15:47:09 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed, 28 Feb 2007 13:47:09 -0700
Subject: [Numpy-discussion] Buffer PEP in NumPy SVN
Message-ID: <45E5EA4D.1010303@ee.byu.edu>


I just wanted to point people to the online version of the PEP.  I'm 
still looking for comments and suggestions.   The current version is here:

http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/pep_buffer.txt

-Travis



From oliphant at ee.byu.edu  Wed Feb 28 16:03:53 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed, 28 Feb 2007 14:03:53 -0700
Subject: [Numpy-discussion] PyCon 2007
Message-ID: <45E5EE39.2040905@ee.byu.edu>


I took the opportunity to go to PyCon this year and met several people 
there.  I had a really good time although I would have liked to stay 
longer.   If you want to see the slides for my talk they are here:

http://us.pycon.org/common/talkdata/PyCon2007/045/PythonTalk.pdf


It was a good opportunity to integrate better with the Python developers 
and keep our voices heard.  I think in the long term this is going to be 
important.  We are working actively to get something like the array 
interface into a re-vamped buffer interface for Python 3.0.   There are 
some issues that arise in this process that need addressing so if you 
have any understanding of the array interface (or want to) please dive 
in and contribute.

Hopefully in this process we will get tighter integration with the 
struct module, the array module, and ctypes as well.   Most of the 
changes are written for back-port to Python 2.6 as well.

Besides how to integrate the buffer interface with discontiguous arrays, 
the only other issue is how to manage the memory needed to return the 
shape information and the format information through the buffer interface.

I think using Python CObject's (like we do for __array_struct__) is a 
good idea and allows flexible and simple memory management as well as 
fast access to direct C-like information about the array.

Other solutions, however, may be better.   If you have any ideas, it 
would be good to hear them.   I would also like to get feedback on the 
proposed additions to the string-syntax that the struct module uses for 
describing data formats.  In particular, I'm interested in ideas about 
what kind of C-API functions for processing such strings might be useful.

These are some ideas that I have:

1) A function that produces a ctypes type from a string-struct description.
2) A function that produces a string-struct description from a ctypes type.
3) A function that extracts the string corresponding to a named 
sub-element.

-Travis








From lou_boog2000 at yahoo.com  Wed Feb 28 16:40:48 2007
From: lou_boog2000 at yahoo.com (Lou Pecora)
Date: Wed, 28 Feb 2007 13:40:48 -0800 (PST)
Subject: [Numpy-discussion] PyCon 2007
In-Reply-To: <45E5EE39.2040905@ee.byu.edu>
Message-ID: <2085.20042.qm@web34402.mail.mud.yahoo.com>


--- Travis Oliphant <oliphant at ee.byu.edu> wrote:

> 
> I took the opportunity to go to PyCon this year and
> met several people 
> there.  I had a really good time although I would
> have liked to stay 
> longer.   If you want to see the slides for my talk
> they are here:
> 
>
http://us.pycon.org/common/talkdata/PyCon2007/045/PythonTalk.pdf

This is a good talk, Travis.  It would go well at the
SIAM Dynamical Systems conferenc held at Snowbird,
near SLC in May this year.  Unfortunately, too late to
get a talk there.  But people in that area are
interested in using Python as their numeric language. 
NumPy/SciPy would get attention.

Thanks.



-- Lou Pecora,   my views are my own.
---------------
Three laws of thermodynamics:
  First law: "You can't win."
  Second law: "You can't break even."
  Third law: "You can't quit."
  -- Allen Ginsberg, beat poet


 
____________________________________________________________________________________
Sucker-punch spam with award-winning protection. 
Try the free Yahoo! Mail Beta.
http://advision.webevents.yahoo.com/mailbeta/features_spam.html


From haase at msg.ucsf.edu  Wed Feb 28 17:14:55 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed, 28 Feb 2007 14:14:55 -0800
Subject: [Numpy-discussion] PyCon 2007
In-Reply-To: <2085.20042.qm@web34402.mail.mud.yahoo.com>
References: <45E5EE39.2040905@ee.byu.edu>
	<2085.20042.qm@web34402.mail.mud.yahoo.com>
Message-ID: <bc657ead0702281414u1366ae4sc73e0ebd82d9394b@mail.gmail.com>

On 2/28/07, Lou Pecora <lou_boog2000 at yahoo.com> wrote:
>
> --- Travis Oliphant <oliphant at ee.byu.edu> wrote:
>
> >
> > I took the opportunity to go to PyCon this year and
> > met several people
> > there.  I had a really good time although I would
> > have liked to stay
> > longer.   If you want to see the slides for my talk
> > they are here:
> >
> >
> http://us.pycon.org/common/talkdata/PyCon2007/045/PythonTalk.pdf
>
Travis,
very nice overview !

Could the file be renamed to
NumpyTalk.pdf
?

Just a thought...

-Sebastian Haase


From mjanikas at esri.com  Wed Feb 28 17:46:19 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Wed, 28 Feb 2007 14:46:19 -0800
Subject: [Numpy-discussion] Source install
In-Reply-To: <45E5D763.9050507@gmail.com>
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFD5@hardwire.esri.com>

Thanks Robert,

Sorry for the incomplete request for help.  The install of numpy seems
to go fine, but when I import numpy it reports that it is running from
the source directory.  I assume this has to do with the BLAS/ATLAS stuff
I have been reading about.  What I am actually trying to do is get NumPy
wrapped in the install of our software program.  We currently wrap
Python2.4 as our scripting language and I need a way to get numpy in our
compiler.  The gui portions of our software runs on Windows but the
engine works on Unix flavors.  I am afraid I am not too knowledgeable
about what goes on under the hood of the NumPy install.  I assume I need
an appropriate C compiler (where gcc fit in for Mac OSX), but I was
wondering if there was an appropriate Doc I should closely examine that
would point me in the right direction.  I hope this clears my question
up a bit.  Again, thanks in advance....

MJ

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Robert Kern
Sent: Wednesday, February 28, 2007 11:26 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Source install

Mark Janikas wrote:
> Hello all,
> 
> I have used numpy on both Mac and Windows.  The latter is easily
> installed with the exe file.  The former required the gcc program from
> XCode... but once installed, "python setup.py install" worked.  I cant
> seem to get numpy to work on my linux machine.  Can someone point me
to
> a platform-independent doc on how to install from the source tar file?
>  Thanks ahead of time,

We need more information from you. There is no way one can make a
platform-independent doc that covers all of the cases. We need to know
what you
tried and exactly how it failed (i.e., we need you to copy the exact
error
messages and paste them into an email).

If I had to guess, though, since you succeeded doing an install from
source on
OS X, the problem on Linux is likely that you do not have the
appropriate Python
development package for your system. On RPM-based systems like Fedora
Core, it
is usually named something like python-devel.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma
 that is made terrible by our own mad attempt to interpret it as though
it had
 an underlying truth."
  -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion


From robert.kern at gmail.com  Wed Feb 28 18:32:23 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 28 Feb 2007 17:32:23 -0600
Subject: [Numpy-discussion] Source install
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBFD5@hardwire.esri.com>
References: <627102C921CD9745B070C3B10CB8199B010EBFD5@hardwire.esri.com>
Message-ID: <45E61107.2090707@gmail.com>

Mark Janikas wrote:
> Thanks Robert,
> 
> Sorry for the incomplete request for help.  The install of numpy seems
> to go fine, but when I import numpy it reports that it is running from
> the source directory. 

Oh, just cd to somewhere else. Otherwise, you will pick up the partial numpy
package in the source directory that we use to bootstrap the build.

> I assume this has to do with the BLAS/ATLAS stuff
> I have been reading about.  What I am actually trying to do is get NumPy
> wrapped in the install of our software program.  We currently wrap
> Python2.4 as our scripting language and I need a way to get numpy in our
> compiler.  The gui portions of our software runs on Windows but the
> engine works on Unix flavors.  I am afraid I am not too knowledgeable
> about what goes on under the hood of the NumPy install.  I assume I need
> an appropriate C compiler (where gcc fit in for Mac OSX), but I was
> wondering if there was an appropriate Doc I should closely examine that
> would point me in the right direction.  I hope this clears my question
> up a bit.  Again, thanks in advance....

The standard Python documentation "Install Python Modules" is useful to read. We
have extended the standard distutils package that is described there, but the
fundamentals are applicable.

  http://docs.python.org/inst/inst.html

Reading the site.cfg.example file in the numpy source is useful information for
configuring numpy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From jh at physics.ucf.edu  Wed Feb 28 19:32:41 2007
From: jh at physics.ucf.edu (Joe Harrington)
Date: Wed, 28 Feb 2007 19:32:41 -0500
Subject: [Numpy-discussion] NumPy in Teaching
Message-ID: <200703010032.l210WfhI005995@glup.physics.ucf.edu>

Hi Steve,

I have taught Astronomical Data Analysis twice at Cornell using IDL,
and I will be teaching it next Fall at UCF using NumPy.  Though I've
been active here in the recent past, I'm actually not a regular NumPy
user myself yet (I used Numeric experimentally for about 6 months in
1997), so I'm a bit nervous.  There isn't the kind of documentation
and how-to support for Numpy that there is for IDL, though our web
site is a start in that direction.  One thought I've had in making the
transition easier is to put up a syntax and function concordance,
similar to that available for MATLAB.  I thought this existed.  Maybe
Perry can point me to it.  Just adding a column to the MATLAB one
would be fine.

My syllabi (there are undergrad and grad versions) are at:

Cornell courses (undergrad only):
http://physics.ucf.edu/~jh/ast/ast234-2003/
http://physics.ucf.edu/~jh/ast/ast234-2004/

UCF course (4xxx is undergrad, 5xxx is grad, numbers not yet assigned):
http://physics.ucf.edu/~jh/ast/dacourse/

The goal of the course is for students to go out and do research with
faculty as soon as they're done, and be useful enough to be included
on papers.  Rather than the usual (and failing) "just do what I do"
model, in which physics students learn to program badly and in
FORTRAN77 from their professors, I teach programming from a CS point
of view, focusing on good top-down design and bottom-up construction
(indentation, documentation, sensible naming, testing, etc.).  I teach
error analysis by first teaching probability.  Then we go into the
physics of detectors and finally do an end-to-end analysis of some
simple spacecraft data sets (photometry and spectroscopy), the
programming of which make up most of their assignments.  There is a
project at the end, in which many in the class seem to get an epiphany
for how all this stuff fits together.  They write up the result in the
format of an Astrophysical Journal article, and while I don't teach
writing as a topic, I do demand that it is done well (and to my shock
it usually is!).

The first two times I taught it, it was way too much material (good
students spent 15+ hours on the class weekly), so I'm ripping out
about half the programming assignments for the undergrads, and giving
simpler project datasets.

My main lesson learned is that the old adage of "They know less than
you think they know but they can do more than you think they can do"
falls completely on its face here.  Many of them actually do know how
to program, and that ability, rather than their academic level, is
really the best predictor of course success.  A computer-savvy
freshman will just kill a computerphobic grad student, because the
rest of the class just isn't that hard.  What I wasn't prepared for
the first time I taught it is just how hard it is to teach debugging.
These kids will stare a simple twiddle-characters bug in the face for
hours and not see it.  It's been twenty-five years since I was at that
stage and it's hard to remember what it was like.  To teach debugging,
I'm emphasizing "fingers as the creators of error" (since you KNOW
your brain didn't do it!), and that they should test each small bit of
code before incorporating it in their function.  I'm also showing them
how to use a debugger, giving them a list of common bug types and how
to find them, and only having them do every other step in the
photometry pipeline.  I'll give them the other half of the steps and
that will teach them how to design to an API.

The other lesson is that it is a hell of a lot of work to grade
programming assignments from even four students, if you care about
grading for good practice and not just whether it runs.  I probably
spent 20 hours a week on the class the second year I taught it.  Since
I'll have 10 students next semester, I plan on doing something here
with peer evaluation.  Wish me luck...

I'm posting here because I'm interested in your results and any advice
you or your respondents have to share.  I hope other respondents will
post here rather than sending private email.  If we get enough people,
let's start a page on the wiki (a mailing list!  a conference!  a
movement!  ok, well, let's start with a wiki page).

--jh--
Prof. Joseph Harrington
Department of Physics
University of Central Florida


From mjanikas at esri.com  Wed Feb 28 19:39:01 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Wed, 28 Feb 2007 16:39:01 -0800
Subject: [Numpy-discussion] Source install
In-Reply-To: <45E61107.2090707@gmail.com>
Message-ID: <627102C921CD9745B070C3B10CB8199B010EBFD6@hardwire.esri.com>

Thanks Robert,

All good info as usual.  Best wishes,

MJ

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Robert Kern
Sent: Wednesday, February 28, 2007 3:32 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Source install

Mark Janikas wrote:
> Thanks Robert,
> 
> Sorry for the incomplete request for help.  The install of numpy seems
> to go fine, but when I import numpy it reports that it is running from
> the source directory. 

Oh, just cd to somewhere else. Otherwise, you will pick up the partial
numpy
package in the source directory that we use to bootstrap the build.

> I assume this has to do with the BLAS/ATLAS stuff
> I have been reading about.  What I am actually trying to do is get
NumPy
> wrapped in the install of our software program.  We currently wrap
> Python2.4 as our scripting language and I need a way to get numpy in
our
> compiler.  The gui portions of our software runs on Windows but the
> engine works on Unix flavors.  I am afraid I am not too knowledgeable
> about what goes on under the hood of the NumPy install.  I assume I
need
> an appropriate C compiler (where gcc fit in for Mac OSX), but I was
> wondering if there was an appropriate Doc I should closely examine
that
> would point me in the right direction.  I hope this clears my question
> up a bit.  Again, thanks in advance....

The standard Python documentation "Install Python Modules" is useful to
read. We
have extended the standard distutils package that is described there,
but the
fundamentals are applicable.

  http://docs.python.org/inst/inst.html

Reading the site.cfg.example file in the numpy source is useful
information for
configuring numpy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma
 that is made terrible by our own mad attempt to interpret it as though
it had
 an underlying truth."
  -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From hayes.tyler at gmail.com  Wed Feb 28 18:03:54 2007
From: hayes.tyler at gmail.com (Tyler J Hayes)
Date: Wed, 28 Feb 2007 23:03:54 +0000 (UTC)
Subject: [Numpy-discussion]
	=?utf-8?q?f2py_and_Fortran90_gfortran=5Ffilena?=
	=?utf-8?q?me_error?=
References: <92bcafc50702280616n4fc74500ic8750b96ca9a2ad3@mail.gmail.com>
	<57756.82.131.71.232.1172672728.squirrel@cens.ioc.ee>
	<92bcafc50702280700y248180b2v6e6a692bc73546cb@mail.gmail.com>
	<20163.82.131.71.232.1172675351.squirrel@cens.ioc.ee>
	<92bcafc50702280729k5ce72643k1c8009eaeffa6df6@mail.gmail.com>
	<20600.82.131.71.232.1172680594.squirrel@cens.ioc.ee>
	<92bcafc50702280927j6f6db2delf482b97644f736b@mail.gmail.com>
	<92bcafc50702281042q40e226adk8dd46eb5183bb0c1@mail.gmail.com>
Message-ID: <loom.20070301T000148-235@post.gmane.org>

Ok, so I'm a total nutcase and couldn't let the laptop thing go. So, I
reinstalled everything from source, and Voila!, it worked.

You have been a huge help Pearu. 

Cheers,

t.






From steve at shrogers.com  Wed Feb 28 23:14:48 2007
From: steve at shrogers.com (Steven H. Rogers)
Date: Wed, 28 Feb 2007 21:14:48 -0700
Subject: [Numpy-discussion] NumPy in Teaching
In-Reply-To: <20070228124928.8D9D2BA05AD@phaser.physics.ucf.edu>
References: <20070228124928.8D9D2BA05AD@phaser.physics.ucf.edu>
Message-ID: <45E65338.9060806@shrogers.com>

Hi Joe:

Thanks for the comprehensive response.  I'll post the results to the 
lists when I've compiled them. 

# Steve


From sb at csse.unimelb.edu.au  Wed Feb 28 16:17:18 2007
From: sb at csse.unimelb.edu.au (Steven Bird)
Date: Thu, 1 Mar 2007 08:17:18 +1100
Subject: [Numpy-discussion] mac binary distributions
Message-ID: <97e4e62e0702281317r6086008dja9a5b98d6a1a5826@mail.gmail.com>

Hi -- the last dmg distribution was for numpy version 0.9.6.  Is there
any chance of having dmg distributions posted for the current version
please?