From mklingberg at spray.se  Sun Aug  1 04:14:02 2004
From: mklingberg at spray.se (Mats Klingberg)
Date: Sun Aug  1 04:14:02 2004
Subject: [Numpy-discussion] Building Numeric with a native blas  ?
In-Reply-To: <410A7733.10408@noaa.gov>
References: <BD2ECF86.13DBC%perry@stsci.edu> <20040730025254.GA26933@arbutus.physics.mcmaster.ca> <410A7733.10408@noaa.gov>
Message-ID: <200408011313.06254.mklingberg@spray.se>

I recently did this under Gentoo. It seems that the atlas package doesn't 
build a complete lapack, which is necessary. Instead I emerged the newer 
blas-atlas and lapack-atlas packages. These build a complete lapack library 
with atlas optimizations. They are however masked with ~x86 so you have to 
add the following to your /etc/portage/package.keywords:

app-sci/lapack-atlas ~x86
app-sci/blas-atlas ~x86
app-sci/blas-config ~x86
app-sci/lapack-config ~x86

blas-config and lapack-config are packages that are needed by the other ones. 
They can be used to switch between reference lapack or atlas optimized 
libraries (if you install lapack-reference and blas-reference)

For more information see Gentoo bug #30453

Hope this helps,
Mats

On Friday 30 July 2004 18.28, Chris Barker wrote:
> David M. Cooke wrote:
> > Atlas might have installed a liblapack, with the (few) functions that it
> > overrides with faster ones. It's by no means a complete LAPACK
> > installation. Have a look at the difference in library sizes; a full
> > LAPACK is a few megs; Atlas's routines are a few hundred K.
>
> OK, I'm really confused now. I got it working, but it seems to have
> virtually identical performance to the Numeric-supplied lapack-lite.
>
> I'm guessing that the LAPACK package I emerged does NOT use the atlas BLAS.
>
> if the atlas liblapack doesn't have all of lapack, how in the world are
> you supposed to use it? I have no idea how I would get the linker to get
> what it can from the atlas lapack, and the rest from another one.
>
> Has anyone done this on Gentoo? If not how about another linux distro, I
> don't have to use portage for this after all.
>
> -Chris



From sdhyok at email.unc.edu  Sun Aug  1 21:22:13 2004
From: sdhyok at email.unc.edu (Shin)
Date: Sun Aug  1 21:22:13 2004
Subject: [Numpy-discussion] nan, inf?
Message-ID: <1091420510.2953.13.camel@dhcp8210.dhcp.unc.edu>

Following the document in ieeespacial.py,
I tried:

>>> from numarray import *
>>> nan
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
NameError: name 'nan' is not defined
>>> inf
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
NameError: name 'inf' is not defined

What did I miss? Thanks.

-- 
Daehyok Shin




From stephen.walton at csun.edu  Sun Aug  1 22:03:03 2004
From: stephen.walton at csun.edu (Stephen Walton)
Date: Sun Aug  1 22:03:03 2004
Subject: [Numpy-discussion] nan, inf?
In-Reply-To: <1091420510.2953.13.camel@dhcp8210.dhcp.unc.edu>
References: <1091420510.2953.13.camel@dhcp8210.dhcp.unc.edu>
Message-ID: <1091422776.15732.0.camel@localhost.localdomain>

On Sun, 2004-08-01 at 21:21, Shin wrote:

> Following the document in ieeespacial.py,
> I tried:
> 
> >>> from numarray import *
> >>> nan
> 

[swalton at localhost ~]$ python
Python 2.2.3 (#1, Jun 16 2003, 13:21:11)
[GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from numarray import *
>>> from numarray.ieeespecial import *
>>> nan
nan
>>> inf
inf

-- 
Stephen Walton <stephen.walton at csun.edu>
Dept. of Physics & Astronomy, CSU Northridge




From Fernando.Perez at colorado.edu  Mon Aug  2 11:50:05 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon Aug  2 11:50:05 2004
Subject: [Numpy-discussion] Numeric bug in array2str() for rank >1 arrays when using MA?
Message-ID: <410E8CB3.4060706@colorado.edu>

Hi all,

I think I've found a bug in Numeric's MA package.  The numeric docs say:

Finally, the last attribute, array_output, specifies whether to prepend the 
string "array(" and append either the string ")" or ", 'X')" where X is a 
typecode for non-default typecodes (in other words, the typecode will only be 
displayed if it is not that corresponding to Float, Complex or Int, which are 
the standard typecodes associated with floating point numbers, complex numbers 
and integers respectively). The array() is so that an eval of the returned 
string will return an array object (provided a comma separator is also used).

and they provide this example:

 >>> type(eval(array2string(arange(3), array_output=1, separator=',')))

<type 'array'>

This is all fine and dandy, but things seem to break down for any array of 
rank > 1 *** when MA has been imported only ***

In [234]: type(eval(array2string(RA.random((2,3)),separator=',',array_output=1)))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

/home/fperez/research/code/mwadap/<console>

/home/fperez/research/code/mwadap/<string>

TypeError: list indices must be integers


The problem is easily seen to be caused by MA:

In [1]: a=arange(4)

In [2]: a.shape= 2,2

In [3]: a
Out[3]:
array([[0, 1],
        [2, 3]])

In [4]: import MA

In [5]: a
Out[5]:
[[0,1,]
  [2,3,]]

Note the misplaced commas above, which make this an invalid nested list.

Once MA is imported it's impossible to get array2string() to return a valid 
representation of an array (something which survives an eval() call).  This is 
a problem for saving things to disk.

I'm using Numeric 23.1.

For now I'll work around this by avoiding MA, which is unfortunate because 
it's otherwise great for preventing huge machine-killing printouts of large 
arrays.

Best,

Fernando.



From sdhyok at email.unc.edu  Mon Aug  2 19:50:00 2004
From: sdhyok at email.unc.edu (Shin)
Date: Mon Aug  2 19:50:00 2004
Subject: [Numpy-discussion] nan, inf?
In-Reply-To: <1091422776.15732.0.camel@localhost.localdomain>
References: <1091420510.2953.13.camel@dhcp8210.dhcp.unc.edu>
	 <1091422776.15732.0.camel@localhost.localdomain>
Message-ID: <1091501313.2108.10.camel@localhost.localdomain>

Thanks. I thought importing all modules in numarray package also imports
nan and inf because the doc string of ieeespecial module has no mention
about "from numarray.ieeespecial import *"

>>> from numarray import *

>>> inf    # the repr() of inf may vary from platform to platform
inf
>>> nan    # the repr() of nan may vary from platform to platform
nan

Daehyok

On Mon, 2004-08-02 at 00:59, Stephen Walton wrote:
> On Sun, 2004-08-01 at 21:21, Shin wrote:
> 
> > Following the document in ieeespacial.py,
> > I tried:
> > 
> > >>> from numarray import *
> > >>> nan
> > 
> 
> [swalton at localhost ~]$ python
> Python 2.2.3 (#1, Jun 16 2003, 13:21:11)
> [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from numarray import *
> >>> from numarray.ieeespecial import *
> >>> nan
> nan
> >>> inf
> inf
-- 
Daehyok Shin (Peter)
Geography Department
Univ. of North Carolina-Chapel Hill




From sdhyok at email.unc.edu  Mon Aug  2 21:11:03 2004
From: sdhyok at email.unc.edu (Shin)
Date: Mon Aug  2 21:11:03 2004
Subject: [Numpy-discussion] segmentation fault
Message-ID: <1091506164.2108.25.camel@localhost.localdomain>

The followind code generates "Process Python segmentation fault".
Is it expected, or a bug?

from numarray import *
from numarray.ieeespecial import *

x = arange(10, typecode='d')
x[5] = nan
del x[getnan(x)]


-- 
Daehyok Shin




From perry at stsci.edu  Tue Aug  3 07:22:14 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Tue Aug  3 07:22:14 2004
Subject: [Numpy-discussion] segmentation fault
In-Reply-To: <1091506164.2108.25.camel@localhost.localdomain>
Message-ID: <BD3517BA.158C2%perry@stsci.edu>

On 8/3/04 12:09 AM, "Shin" <sdhyok at email.unc.edu> wrote:

> The followind code generates "Process Python segmentation fault".
> Is it expected, or a bug?
> 
I'd hazard that it is a bug since I'm not sure when a seg fault is really
desired, but maybe that's just me ;-)

> from numarray import *
> from numarray.ieeespecial import *
> 
> x = arange(10, typecode='d')
> x[5] = nan
> del x[getnan(x)]
> 
But note that this is an error of usage. You shouldn't be trying to delete a
part of an array. You can delete the array but here you appear to be trying
to delete a nan value. Arrays don't work like lists in this regard. What are
you trying to do?

Perry Greenfield




From sdhyok at catchlab.org  Tue Aug  3 08:39:14 2004
From: sdhyok at catchlab.org (Shin, Daehyok)
Date: Tue Aug  3 08:39:14 2004
Subject: [Numpy-discussion] segmentation fault
In-Reply-To: <BD3517BA.158C2%perry@stsci.edu>
Message-ID: <OAEOKPIGCLDDHAEMCAKIAEMLDBAA.sdhyok@catchlab.org>

I am just playing with ieeespecial.py  
to see what happens if I tried some intuitive things.
I know the command has an error.
But, I think "segmentation fault" is too harsh as a response to the error.
Am I right?

Daehyok Shin

> -----Original Message-----
> From: Perry Greenfield [mailto:perry at stsci.edu]
> Sent: Tuesday, August 03, 2004 AM 10:22
> To: Shin; numpy
> Subject: Re: [Numpy-discussion] segmentation fault
> 
> 
> On 8/3/04 12:09 AM, "Shin" <sdhyok at email.unc.edu> wrote:
> 
> > The followind code generates "Process Python segmentation fault".
> > Is it expected, or a bug?
> > 
> I'd hazard that it is a bug since I'm not sure when a seg fault is really
> desired, but maybe that's just me ;-)
> 
> > from numarray import *
> > from numarray.ieeespecial import *
> > 
> > x = arange(10, typecode='d')
> > x[5] = nan
> > del x[getnan(x)]
> > 
> But note that this is an error of usage. You shouldn't be trying 
> to delete a
> part of an array. You can delete the array but here you appear to 
> be trying
> to delete a nan value. Arrays don't work like lists in this 
> regard. What are
> you trying to do?
> 
> Perry Greenfield
> 
> 



From perry at stsci.edu  Tue Aug  3 08:42:37 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Tue Aug  3 08:42:37 2004
Subject: [Numpy-discussion] segmentation fault
In-Reply-To: <OAEOKPIGCLDDHAEMCAKIAEMLDBAA.sdhyok@catchlab.org>
Message-ID: <BD352A4D.158F1%perry@stsci.edu>

On 8/3/04 11:38 AM, "Shin, Daehyok" <sdhyok at catchlab.org> wrote:

> I am just playing with ieeespecial.py
> to see what happens if I tried some intuitive things.
> I know the command has an error.
> But, I think "segmentation fault" is too harsh as a response to the error.
> Am I right?
> 
Yes, of course. Seg faults are almost never the right response.

Perry Greenfield




From tim.hochberg at cox.net  Thu Aug  5 10:06:04 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Thu Aug  5 10:06:04 2004
Subject: [Numpy-discussion] argmax & py2exe
Message-ID: <411268CE.8050107@cox.net>

I just ran into a little snag using py2exe with CVS numarray: argmax 
breaks in py2exe packaged programs although it works fine when run 
normally. My believe is that py2exe is getting confused by the trick of 
using 'import numeric' inside the function argmax, which I believe is 
there to avoid circular imports. In any event, generic.argmax and 
numarraycore.NumArray.argmax mysteriously ending up calling each other 
recursively until the recursion limit is exceeded.

This is almost certainly a problem with py2exe, not with numarray, but 
since circular imports are sort of bad form and since numarray side of 
things is almost certainly easier to to fix, I though I'd try to 
persuade you, the numarray developers, to reorganize argmax so the 
circular imports go away.

I think that this is mostly a matter of moving the numeric.argmax/argmin 
code into NumArray.argmax/argmin and then numeric.argmax/argmin can just 
call a.argmax/a.argmin on the array a that it is passed./

-tim




From tim.hochberg at cox.net  Thu Aug  5 13:40:08 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Thu Aug  5 13:40:08 2004
Subject: [Numpy-discussion] argmax & py2exe
In-Reply-To: <411268CE.8050107@cox.net>
References: <411268CE.8050107@cox.net>
Message-ID: <41129AEB.3060507@cox.net>

Tim Hochberg wrote:

Please ignore this. The problem went away when I deleted numerics pyc 
files and started over. It's a mysterious phenomena, that I ran into 
once before with py2exe and stale pyc files. Very strange, but no time 
to dig into it now since things appear to be working again. Sorry for 
the confusion.

Although getting rid of those circular imports would still probably be 
good style.

-tim

>
> I just ran into a little snag using py2exe with CVS numarray: argmax 
> breaks in py2exe packaged programs although it works fine when run 
> normally. My believe is that py2exe is getting confused by the trick 
> of using 'import numeric' inside the function argmax, which I believe 
> is there to avoid circular imports. In any event, generic.argmax and 
> numarraycore.NumArray.argmax mysteriously ending up calling each other 
> recursively until the recursion limit is exceeded.
>
> This is almost certainly a problem with py2exe, not with numarray, but 
> since circular imports are sort of bad form and since numarray side of 
> things is almost certainly easier to to fix, I though I'd try to 
> persuade you, the numarray developers, to reorganize argmax so the 
> circular imports go away.
>
> I think that this is mostly a matter of moving the 
> numeric.argmax/argmin code into NumArray.argmax/argmin and then 
> numeric.argmax/argmin can just call a.argmax/a.argmin on the array a 
> that it is passed./
>
> -tim
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by OSTG. Have you noticed the changes on
> Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
> one more big change to announce. We are now OSTG- Open Source Technology
> Group. Come see the changes on the new OSTG site. www.ostg.com
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>





From jmiller at stsci.edu  Fri Aug  6 13:55:12 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Fri Aug  6 13:55:12 2004
Subject: [Numpy-discussion] argmax & py2exe
In-Reply-To: <41129AEB.3060507@cox.net>
References: <411268CE.8050107@cox.net>  <41129AEB.3060507@cox.net>
Message-ID: <1091825685.3761.8.camel@localhost.localdomain>

I was on leave this week so I'm getting to this later than usual,  but
Perry and I discussed it this morning.  We concluded that this is the
MODEL numarray problem report because:

1.  You reported the problem.

2.  You suggested a fix even I could implement in a half hour or less.

3.  You declared the problem a non-problem before I could waste a single
solitary second on it.  Well, except this one anyway.

Everyone take note because I think Tim is on to something.  We need more
bug reports exactly like this one...

Regards,
Todd



On Thu, 2004-08-05 at 16:39, Tim Hochberg wrote:
> Tim Hochberg wrote:
> 
> Please ignore this. The problem went away when I deleted numerics pyc 
> files and started over. It's a mysterious phenomena, that I ran into 
> once before with py2exe and stale pyc files. Very strange, but no time 
> to dig into it now since things appear to be working again. Sorry for 
> the confusion.
> 
> Although getting rid of those circular imports would still probably be 
> good style.
> 
> -tim
> 
> >
> > I just ran into a little snag using py2exe with CVS numarray: argmax 
> > breaks in py2exe packaged programs although it works fine when run 
> > normally. My believe is that py2exe is getting confused by the trick 
> > of using 'import numeric' inside the function argmax, which I believe 
> > is there to avoid circular imports. In any event, generic.argmax and 
> > numarraycore.NumArray.argmax mysteriously ending up calling each other 
> > recursively until the recursion limit is exceeded.
> >
> > This is almost certainly a problem with py2exe, not with numarray, but 
> > since circular imports are sort of bad form and since numarray side of 
> > things is almost certainly easier to to fix, I though I'd try to 
> > persuade you, the numarray developers, to reorganize argmax so the 
> > circular imports go away.
> >
> > I think that this is mostly a matter of moving the 
> > numeric.argmax/argmin code into NumArray.argmax/argmin and then 
> > numeric.argmax/argmin can just call a.argmax/a.argmin on the array a 
> > that it is passed./
> >
> > -tim
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by OSTG. Have you noticed the changes on
> > Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
> > one more big change to announce. We are now OSTG- Open Source Technology
> > Group. Come see the changes on the new OSTG site. www.ostg.com
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> >
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by OSTG. Have you noticed the changes on
> Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
> one more big change to announce. We are now OSTG- Open Source Technology
> Group. Come see the changes on the new OSTG site. www.ostg.com
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 




From rbastian at club-internet.fr  Sun Aug  8 12:58:12 2004
From: rbastian at club-internet.fr (=?iso-8859-15?q?Ren=E9=20Bastian?=)
Date: Sun Aug  8 12:58:12 2004
Subject: [Numpy-discussion] numarray, fromfile
Message-ID: <04080813422800.00759@rbastian>

Hello,

sometimes (not always) :

           w=NA.fromfile(self.f, NA.Float64, d)

delivers an error message :  

TypeError : NA_intSequenceProduct: object is not a sequence

Why not always ?

There is no  error if one writes 

           w=NA.fromfile(self.f, NA.Float64, (d,))



-- 
Ren? Bastian
http://www.musiques-rb.org : Musique en Python 



From jmiller at stsci.edu  Mon Aug  9 04:00:05 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Mon Aug  9 04:00:05 2004
Subject: [Numpy-discussion] numarray, fromfile
In-Reply-To: <04080813422800.00759@rbastian>
References: <04080813422800.00759@rbastian>
Message-ID: <1092049130.3748.3.camel@localhost.localdomain>

It's hard to tell without knowing the value of d,  but my guess is that
sometimes d is an integer and sometimes it's a float, and that perhaps
the "it works if (d,) is used" is a red herring.  Using a float for d
definitely produces the exception you saw.

Regards,
Todd

On Sun, 2004-08-08 at 07:57, Ren? Bastian wrote:
> Hello,
> 
> sometimes (not always) :
> 
>            w=NA.fromfile(self.f, NA.Float64, d)
> 
> delivers an error message :  
> 
> TypeError : NA_intSequenceProduct: object is not a sequence
> 
> Why not always ?
> 
> There is no  error if one writes 
> 
>            w=NA.fromfile(self.f, NA.Float64, (d,))
-- 




From eqdaqvdkyk at yahoo.com  Mon Aug  9 04:24:05 2004
From: eqdaqvdkyk at yahoo.com (Karl Donaldson)
Date: Mon Aug  9 04:24:05 2004
Subject: [Numpy-discussion] Fw: lozenge
Message-ID: <MPSPRMXSXZGPYBIHAIVLOD@yahoo.com>

I got this forwarded to me from a friend of mine, thought you might want to see it.  http://www.suncoastrewards.com/ref44.html
You can download dvd movies, full cd albums and stuff.  Even console video games.  I had found at least 74 CD's on there that I wanted.
Wait untill you see the selection of music and dvd's - it's pretty awsome.
Theres a few pages in there on how to get your selection on CD if you need the help.
I saw a few areas of the movie section, and found titles in there still in theaters - it's nutty.
Call me later if you get the chance.

Gregory

----327293974229964655----




From strawman at astraw.com  Tue Aug 10 23:27:00 2004
From: strawman at astraw.com (Andrew Straw)
Date: Tue Aug 10 23:27:00 2004
Subject: [Numpy-discussion] floating point exception weirdness
Message-ID: <4119BBFC.6020304@astraw.com>

Hi All,

I've isolated numarray.ieeespecial as the cause of a strange error which 
I posted on c.l.py:
http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&threadm=41198B29.7090600%40NOSPAM-DELETE-THIS.astraw.com&prev=/groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26group%3Dcomp.lang.python

My non-expert guess as to an explanation: some statements early in 
numarray.ieeespecial (e.g. _na.array(1.0)/_na.array(0.0)) result in a 
floating point exception being set which lies dormant until I call 
certain functions in the Intel IPP library (e.g. 
ippiAddWeighted_8u32f_C1IR), at which point the latent exception bit my 
Python program is killed while the console says "Floating point exception".

I've come to this tentative conclusion based on 3 minimal Pyrex programs:

---------------------------------------------------- Version 1:
import numarray.numarrayall as _na
plus_inf = inf = _na.array(1.0)/_na.array(0.0)

cimport avg
avg.work() # calls IPP, terminates with "Floating point exception".
---------------------------------------------------- Version 2:
import numarray.numarrayall as _na

# Define *ieee special values*
_na.Error.pushMode(all="ignore")
plus_inf = inf = _na.array(1.0)/_na.array(0.0)
_na.Error.popMode()

cimport avg
avg.work() # calls IPP, terminates with "Floating point exception".
---------------------------------------------------- Version 3:
import numarray.numarrayall as _na
cimport avg
avg.work() # calls IPP, terminates normally

In the short term, I'll try and work around this problem by not 
importing numarray.ieeespecial, but in the long term, I think we should 
try and fix this.  I'm happy to try to debug further if someone gives me 
advice on where to go from here.

Cheers!
Andrew





From jmiller at stsci.edu  Wed Aug 11 03:50:24 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Aug 11 03:50:24 2004
Subject: [Numpy-discussion] floating point exception weirdness
In-Reply-To: <4119BBFC.6020304@astraw.com>
References: <4119BBFC.6020304@astraw.com>
Message-ID: <1092221365.3752.32.camel@localhost.localdomain>

Hi Andrew,

So far you mentioned debian, linux-2.6.7, Python-2.3.4, and the Intel
IPP library.   What compiler are you using?

More comments below.

On Wed, 2004-08-11 at 02:26, Andrew Straw wrote:
> Hi All,
> 
> I've isolated numarray.ieeespecial as the cause of a strange error which 
> I posted on c.l.py:
> http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&threadm=41198B29.7090600%40NOSPAM-DELETE-THIS.astraw.com&prev=/groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26group%3Dcomp.lang.python
> 
> My non-expert guess as to an explanation: some statements early in 
> numarray.ieeespecial (e.g. _na.array(1.0)/_na.array(0.0)) result in a 
> floating point exception being set which lies dormant until I call 
> certain functions in the Intel IPP library (e.g. 
> ippiAddWeighted_8u32f_C1IR), at which point the latent exception bit my 
> Python program is killed while the console says "Floating point exception".

It sounds to me like the IPP library is aborting as a result of
previously irrelevant floating point error state.  My guess is that the
problem is not confined to ieeespecial,  but rather to floating point
exception handling in general in combination with the IPP library and/or
whatever compiler you're using.  

The key numarray code for clearing the floating point error state is
NA_checkFPErrors() in numarray/Src/newarray.ch.

Some things that occur to me are:

1. Verify that any array divide by zero results in the same failure.  
>>> x = arange(10,20)/zeros((10,)) 
>>> call_your_ipp_code()

2. Look in the IPP library for routines related to clearing floating
point exceptions.  Look in your compiler runtime library for routines
related to clearing floating point exceptions.  Modify NA_checkFPErrors
accordingly and re-build numarray with 

% python setup.py install --genapi

4. Walk through the Pyrex generated wrapper code and make sure there's
nothing going on there.  I'm doubtful that this has anything to do with
the problem.

More ideas will come as you let us know what your compiler is and verify
item 1.

Regards,
Todd




From nadavh at visionsense.com  Wed Aug 11 04:49:02 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Wed Aug 11 04:49:02 2004
Subject: [Numpy-discussion] Bug in built in SVD?
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDA1@exchange2k.envision.co.il>

When numarray is compiled with the buit-in linear-algebra routine the svd routine runs into an infinite loop.

Platform2: python 2.3.4 and 2.4a2; numarray 1.1; gcc 3.4.1; RH9 and gentoo 1.4

example:

>> from numarray.linear_algebra.mlab import *
>>> svd(arange(20.0, shape = (5,4)))

... infine loop ...


--------------------------------------------------------------------------

svd runs OK when compiled againts external LAPACK.

BTW: Why numarray used dgesdd routine for SVD, and not the svd routine that has the ATLAS equivalent?

  Nadav.




From strawman at astraw.com  Wed Aug 11 04:55:13 2004
From: strawman at astraw.com (Andrew Straw)
Date: Wed Aug 11 04:55:13 2004
Subject: [Numpy-discussion] floating point exception weirdness
In-Reply-To: <1092221365.3752.32.camel@localhost.localdomain>
References: <4119BBFC.6020304@astraw.com> <1092221365.3752.32.camel@localhost.localdomain>
Message-ID: <411A08FA.7000601@astraw.com>

Dear Todd,

I'll respond to your other comments when I get a chance to investigate 
further.

Todd Miller wrote:

>So far you mentioned debian, linux-2.6.7, Python-2.3.4, and the Intel
>IPP library.   What compiler are you using?
>  
>
astraw at flygate:~$ gcc -v
Reading specs from /usr/lib/gcc-lib/i486-linux/3.3.4/specs
Configured with: ../src/configure -v 
--enable-languages=c,c++,java,f77,pascal,objc,ada,treelang --prefix=/usr 
--mandir=/usr/share/man --infodir=/usr/share/info 
--with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared 
--with-system-zlib --enable-nls --without-included-gettext 
--enable-__cxa_atexit --enable-clocale=gnu --enable-debug 
--enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc i486-linux
Thread model: posix
gcc version 3.3.4 (Debian)

Geez, I hope that's enough info to tell you gcc 3.3.4. :)

(Also, I could mention this is with numarray-1.0 -- tested both with 
whatever Debian gives me in addition to a source build of the 1.0 tarball.)

Cheers!
Andrew



From curzio.basso at unibas.ch  Wed Aug 11 05:54:08 2004
From: curzio.basso at unibas.ch (Curzio Basso)
Date: Wed Aug 11 05:54:08 2004
Subject: [Numpy-discussion] masked arrays and nd_image
Message-ID: <411A16BD.6050902@unibas.ch>

Hi all.

I just tried to use a masked array with nd_image, namely to pass it to 
the convolve1d() function, and I got:

error: getShape: couldn't get sequence length.

Does someone know if it possible at all to use nd_image on masked 
arrays? Or if there is a possibility to do operations on masked images 
without using the ma module?

thanks. curzio



From verveer at embl-heidelberg.de  Wed Aug 11 06:05:02 2004
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Wed Aug 11 06:05:02 2004
Subject: [Numpy-discussion] masked arrays and nd_image
In-Reply-To: <411A16BD.6050902@unibas.ch>
References: <411A16BD.6050902@unibas.ch>
Message-ID: <EE004E79-EB96-11D8-BB04-000A95C92C8E@embl-heidelberg.de>

I did not write nd_image with masked arrays in mind, I imagine it won't 
work for most functions. I am not planning to support masked arrays

. If you just want to mask out some of the results, apply the mask 
afterwards. If you actually want to deal with masked values in the 
convolution itself, you are out of luck  with the convolve1d function, 
it does not support that. In that case you could try to write your own, 
using the generic filter functions (see the  manual).

Cheers, Peter

On 11 Aug 2004, at 14:53, Curzio Basso wrote:

>
> Hi all.
>
> I just tried to use a masked array with nd_image, namely to pass it to 
> the convolve1d() function, and I got:
>
> error: getShape: couldn't get sequence length.
>
> Does someone know if it possible at all to use nd_image on masked 
> arrays? Or if there is a possibility to do operations on masked images 
> without using the ma module?
>
> thanks. curzio
>
>
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion




From curzio.basso at unibas.ch  Wed Aug 11 06:17:10 2004
From: curzio.basso at unibas.ch (Curzio Basso)
Date: Wed Aug 11 06:17:10 2004
Subject: [Numpy-discussion] masked arrays and nd_image
In-Reply-To: <EE004E79-EB96-11D8-BB04-000A95C92C8E@embl-heidelberg.de>
References: <411A16BD.6050902@unibas.ch> <EE004E79-EB96-11D8-BB04-000A95C92C8E@embl-heidelberg.de>
Message-ID: <411A1C2D.9040303@unibas.ch>

Peter Verveer wrote:

 > I did not write nd_image with masked arrays in mind, I imagine it won't
 > work for most functions. I am not planning to support masked arrays

Yes, I expected this. I knew I was pretending too much ;o)

 > . If you just want to mask out some of the results, apply the mask
 > afterwards. If you actually want to deal with masked values in the
 > convolution itself, you are out of luck  with the convolve1d function,
 > it does not support that. In that case you could try to write your own,
 > using the generic filter functions (see the  manual).

Ok, this would imply writing a function with prototype FilterFunction1D, 
right? And the mask argument should be packed with the data or is there 
a way (which I don't see) to pass arguments to the filter other than the 
input image. Sorry if I'm annoying ;o)

thanks, curzio



From jmiller at stsci.edu  Wed Aug 11 06:25:11 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Aug 11 06:25:11 2004
Subject: [Numpy-discussion] masked arrays and nd_image
In-Reply-To: <411A16BD.6050902@unibas.ch>
References: <411A16BD.6050902@unibas.ch>
Message-ID: <1092230676.2633.15.camel@halloween.stsci.edu>

On Wed, 2004-08-11 at 08:53, Curzio Basso wrote:
> Hi all.
> 
> I just tried to use a masked array with nd_image, namely to pass it to 
> the convolve1d() function, and I got:
> 
> error: getShape: couldn't get sequence length.
> 
> Does someone know if it possible at all to use nd_image on masked 
> arrays? 

I don't believe it is possible to use masked arrays directly because
they are not NumArrays or NumArray subclasses.   Masked arrays only
contain NumArrays which store the data and mask.

> Or if there is a possibility to do operations on masked images 
> without using the ma module?

I think the key here may be the "filled()" method which lets you convert
a masked array into a NumArray with the masked values filled into some
fill value, say 0.  I'm not sure what the post convolution mask value
should be.

Regards,
Todd




From verveer at embl-heidelberg.de  Wed Aug 11 06:27:05 2004
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Wed Aug 11 06:27:05 2004
Subject: [Numpy-discussion] masked arrays and nd_image
In-Reply-To: <411A1C2D.9040303@unibas.ch>
References: <411A16BD.6050902@unibas.ch> <EE004E79-EB96-11D8-BB04-000A95C92C8E@embl-heidelberg.de> <411A1C2D.9040303@unibas.ch>
Message-ID: <0A575C55-EB9A-11D8-BB04-000A95C92C8E@embl-heidelberg.de>

> > I did not write nd_image with masked arrays in mind, I imagine it 
> won't
> > work for most functions. I am not planning to support masked arrays
>
> Yes, I expected this. I knew I was pretending too much ;o)

One can always try :-)

> > . If you just want to mask out some of the results, apply the mask
> > afterwards. If you actually want to deal with masked values in the
> > convolution itself, you are out of luck  with the convolve1d 
> function,
> > it does not support that. In that case you could try to write your 
> own,
> > using the generic filter functions (see the  manual).
>
> Ok, this would imply writing a function with prototype 
> FilterFunction1D, right? And the mask argument should be packed with 
> the data or is there a way (which I don't see) to pass arguments to 
> the filter other than the input image. Sorry if I'm annoying ;o)

If you do it in python, you feed generic_filter1d a function, if you 
make that a method of an object, your mask can be accessed as a 
attribute of your object.

If you want to do it in C, you can pass a CObject that holds the 
function pointer to a callback function with prototype 
FilterFunction1D. Additionally you can set the description pointer of 
the same CObject to an arbitrary pointer, which is passed at every call 
to the callback function. You can use that to store additional 
information for you filter, such as the mask. See the example in the 
manual which is written for the shift function, but the principle is 
the same.

Cheers, Peter




From Chris.Barker at noaa.gov  Wed Aug 11 11:34:00 2004
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Wed Aug 11 11:34:00 2004
Subject: [Numpy-discussion] Pure Python NumPy-like module?
In-Reply-To: <200408011313.06254.mklingberg@spray.se>
References: <BD2ECF86.13DBC%perry@stsci.edu> <20040730025254.GA26933@arbutus.physics.mcmaster.ca> <410A7733.10408@noaa.gov> <200408011313.06254.mklingberg@spray.se>
Message-ID: <411A60E8.8060600@noaa.gov>

Hi folks,

I've got a project that uses Numeric arrays a fair bit. Unfortunately, I 
have a fair number of users that would like to be able to use it without 
Numeric (frankly I don't get it. It's a wxPython widget, and once you've 
installed wxPython, Numeric should not be a show stopper, but whatever). 
I though one way to accommodate this was to create a pure python 
Numeric-lite package, that contained the functionality need for my app, 
and was written in pure python. A side benefit of this is that it would 
be an easy way to see how much a performance gain Numeric gives me. 
Frankly, I don't really care that much, I like using Numeric operations 
for the syntactical advantages, aside from performance issues.

Anyway, I recall a while back a couple folks did some testing with Psyco 
and a pure-python implementation of a small fraction of Numeric 
functionality. I'd love to see that code if it's still around. Or if any 
of you has done something similar, I'd love to see that too.

-thanks, Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From tim.hochberg at cox.net  Wed Aug 11 11:52:02 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Wed Aug 11 11:52:02 2004
Subject: [Numpy-discussion] Pure Python NumPy-like module?
In-Reply-To: <411A60E8.8060600@noaa.gov>
References: <BD2ECF86.13DBC%perry@stsci.edu> <20040730025254.GA26933@arbutus.physics.mcmaster.ca> <410A7733.10408@noaa.gov> <200408011313.06254.mklingberg@spray.se> <411A60E8.8060600@noaa.gov>
Message-ID: <411A6AA2.3090600@cox.net>

Chris Barker wrote:

> Hi folks,
>
> I've got a project that uses Numeric arrays a fair bit. Unfortunately, 
> I have a fair number of users that would like to be able to use it 
> without Numeric (frankly I don't get it. It's a wxPython widget, and 
> once you've installed wxPython, Numeric should not be a show stopper, 
> but whatever). I though one way to accommodate this was to create a 
> pure python Numeric-lite package, that contained the functionality 
> need for my app, and was written in pure python. A side benefit of 
> this is that it would be an easy way to see how much a performance 
> gain Numeric gives me. Frankly, I don't really care that much, I like 
> using Numeric operations for the syntactical advantages, aside from 
> performance issues.
>
> Anyway, I recall a while back a couple folks did some testing with 
> Psyco and a pure-python implementation of a small fraction of Numeric 
> functionality. I'd love to see that code if it's still around. Or if 
> any of you has done something similar, I'd love to see that too.


Hi Chris,

That was probably me. Your message prompted me to take a look at that 
code for the first time in a long time and it is indeed a *small* 
fraction of Numeric functionality.  That being said your welcome to try 
and do something useful with it. I'll go ahead and send you the file in 
private email to avoid cluttering the list.

If anyone else wants a copy let me know.

-tim

>
> -thanks, Chris
>
>
>
>
>





From stephen.walton at csun.edu  Wed Aug 11 12:02:08 2004
From: stephen.walton at csun.edu (Stephen Walton)
Date: Wed Aug 11 12:02:08 2004
Subject: [Numpy-discussion] masked arrays and nd_image
In-Reply-To: <1092230676.2633.15.camel@halloween.stsci.edu>
References: <411A16BD.6050902@unibas.ch>
	 <1092230676.2633.15.camel@halloween.stsci.edu>
Message-ID: <1092250881.6980.32.camel@freyer.sfo.csun.edu>

On Wed, 2004-08-11 at 06:24, Todd Miller wrote:

> I think the key here may be the "filled()" method which lets you convert
> a masked array into a NumArray with the masked values filled into some
> fill value, say 0.  I'm not sure what the post convolution mask value
> should be.

I hope I'm not jumping in where I don't belong, but here at SFO/CSUN
we've had quite a bit of experience with convolutions and correlations
of time series (serieses?) with missing data points.

I'm sure you all know this, but:  For the scaling to be correct, you
have to not only mask out the values you don't want, but normalize the
sum to reflect the fact that different numbers of values will appear in
the sum.  Our MATLAB code to convolve x and y, with bad points marked by
NaNs, is:

for i = 1 : xlen-ylen+1
   j(i)=i;
   x1=x(i:i+ylen-1);
   a=x1.*y;
   b=a(find(~isnan(a)));
   if isempty(b)
      z(i)= NaN;
   else
       z(i)=sum(b)/length(b)
   end
end

I'd be happy to know how to code up the equivalent in numarray.  In the
above, note that x1 is x padded on both ends with ylen-1 NaNs.

Unfortunately, and again I'm sure everyone knows this, you can't use
FFTs to speed up convolutions/correlations if you have missing data
points, so you have to use discrete techniques.  The numerical analysis
literature refers to this problem as Fourier analysis of unequally
spaced data.  The only publications and algorithms I could find went the
wrong way:  given an unequally spaced set of points in Fourier space,
find the most likely reconstruction in real space.

-- 
Stephen Walton <stephen.walton at csun.edu>
Dept. of Physics & Astronomy, Cal State Northridge
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040811/69e91579/attachment.sig>

From rlw at stsci.edu  Wed Aug 11 12:43:01 2004
From: rlw at stsci.edu (Rick White)
Date: Wed Aug 11 12:43:01 2004
Subject: [Numpy-discussion] masked arrays and nd_image
In-Reply-To: <1092250881.6980.32.camel@freyer.sfo.csun.edu>
Message-ID: <Pine.GSO.4.44.0408111522290.21604-100000@sundog.stsci.edu>

On Wed, 11 Aug 2004, Stephen Walton wrote:

> I hope I'm not jumping in where I don't belong, but here at SFO/CSUN
> we've had quite a bit of experience with convolutions and correlations
> of time series (serieses?) with missing data points.
>
> I'm sure you all know this, but:  For the scaling to be correct, you
> have to not only mask out the values you don't want, but normalize the
> sum to reflect the fact that different numbers of values will appear in
> the sum.  Our MATLAB code to convolve x and y, with bad points marked by
> NaNs, is:
>
> for i = 1 : xlen-ylen+1
>    j(i)=i;
>    x1=x(i:i+ylen-1);
>    a=x1.*y;
>    b=a(find(~isnan(a)));
>    if isempty(b)
>       z(i)= NaN;
>    else
>        z(i)=sum(b)/length(b)
>    end
> end
>
> I'd be happy to know how to code up the equivalent in numarray.  In the
> above, note that x1 is x padded on both ends with ylen-1 NaNs.

If you have a mask array with ones marking the good points and zeros the
points to ignore, here is a simpler version that does pretty much the same
calculation:

import numarray as na
kernel = na.ones(ylen)
z = nd.convolve1d(x*mask,kernel,mode='constant') / \
    nd.convolve1d(mask,kernel,mode='constant')

This would need some elaboration to work with your data -- e.g., NaN*0 is
not zero so you don't want the x array to have NaN values in it.  And it
may need an extra test to handle the case where a region larger than ylen
is masked out (this version would put in NaN values, which might be OK).
But the basic idea is the same.

I imagine this is possible in Matlab (which I don't use) -- I use it all
the time in IDL to do convolutions in the presence of missing data.

> Unfortunately, and again I'm sure everyone knows this, you can't use
> FFTs to speed up convolutions/correlations if you have missing data
> points, so you have to use discrete techniques.  The numerical analysis
> literature refers to this problem as Fourier analysis of unequally
> spaced data.  The only publications and algorithms I could find went the
> wrong way:  given an unequally spaced set of points in Fourier space,
> find the most likely reconstruction in real space.

This can in fact be done using FFTs.  You have to be careful about how the
boundary conditions are handled, adding padding cells to avoid wraparound
(that's what the mode='constant' is about).  The approach is very similar
though.  This works in 2-D as well and is vastly faster than doing a
direct convolution.

Incidentally, if you have truly unequally spaced data points (as opposed
to missing points out of an equally spaced series), this particular trick
doesn't work but there is another approach to doing DFTs in FFT time.
It's a bit lengthy to explain (and off the path for this group), but radio
astronomers figured out how to do it a long time ago.  It is probably the
algorithm you mention using unequally-spaced points in Fourier space; the
same algorithm works fine if the points are in real space since there is
hardly any difference between a forward and reverse Fourier transform.
					Rick




From geoffp at rochester.rr.com  Wed Aug 11 20:08:09 2004
From: geoffp at rochester.rr.com (Geoffrey Philbrick)
Date: Wed Aug 11 20:08:09 2004
Subject: [Numpy-discussion] matrixmultiply problem
Message-ID: <000801c48019$2c830be0$6601a8c0@Laptop>

Executing the following code has the side effect of modifying one of the arguments (matrices) I passed to matrixmultiply. Note that the matrix m2 changes after the call to matrixmultiply. This is on Windows with numarray 1.0. 

Why does this happen?

Code:

from numarray import array, matrixmultiply

m1 = array(
[[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]]
)

m2 = array(
[[ 0.500011, 0.86601905, 0. ],
[-0.86601905, 0.500011, 0. ],
[ 0., 0., 1. ]]
)

print "m1 before\n", m1
print "m2 before\n", m2
matrixmultiply(m1, m2)
print "m1 after\n", m1
print "m2 after\n", m2

Output:

m1 before
[[ 1. 0. 0.]
[ 0. 1. 0.]
[ 0. 0. 1.]]
m2 before
[[ 0.500011 0.86601905 0. ]
[-0.86601905 0.500011 0. ]
[ 0. 0. 1. ]]
m1 after
[[ 1. 0. 0.]
[ 0. 1. 0.]
[ 0. 0. 1.]]
m2 after
[[ 0.500011 -0.86601905 0. ]
[ 0.86601905 0.500011 0. ]
[ 0. 0. 1. ]]



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040811/ecea08a4/attachment.html>

From cookedm at physics.mcmaster.ca  Wed Aug 11 21:28:10 2004
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Wed Aug 11 21:28:10 2004
Subject: [Numpy-discussion] matrixmultiply problem
In-Reply-To: <000801c48019$2c830be0$6601a8c0@Laptop>
References: <000801c48019$2c830be0$6601a8c0@Laptop>
Message-ID: <20040812042736.GA14498@arbutus.physics.mcmaster.ca>

On Wed, Aug 11, 2004 at 11:05:07PM -0400, Geoffrey Philbrick wrote:
> Executing the following code has the side effect of modifying one of
> the arguments (matrices) I passed to matrixmultiply. Note that the
> matrix m2 changes after the call to matrixmultiply. This is on Windows
> with numarray 1.0. 

Wow, that's bad. Since I had nothing better to do [1], I tracked it down
and submitted a patch (#1007730) to SourceForge to fix this.

[1] actually, I do, but...

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From baud.ph at ifrance.com  Thu Aug 12 00:02:07 2004
From: baud.ph at ifrance.com (baud.ph at ifrance.com)
Date: Thu Aug 12 00:02:07 2004
Subject: [Numpy-discussion] Repondeur de <baud.ph@ifrance.com>
Message-ID: <0408120648.2c114d@b0508.idoo.com>

Cette adresse mail n'est plus utilis?e.


ifrance.com, l'email gratuit le plus complet de l'Internet!
En vacances ou absent du bureau... vous souhaitez en informer vos
correspondants? Facile avec l'email i france et son r?pondeur!
>>>>>> http://www.ifrance.com/_reloc/rep.email




From nadavh at visionsense.com  Thu Aug 12 00:15:07 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu Aug 12 00:15:07 2004
Subject: [Numpy-discussion] Bug in built in SVD?
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDAB@exchange2k.envision.co.il>

Sorry it is a gcc3.4.1 optimization bug. Either lowering to optimization to -O1 or stepping back to gcc3.3.4 (with all optimization guns on) fix it.

   Nadav.

-----Original Message-----
From:	Nadav Horesh
Sent:	Wed 11-Aug-04 15:48
To:	numpy-discussion
Cc:	
Subject:	[Numpy-discussion] Bug in built in SVD?
When numarray is compiled with the buit-in linear-algebra routine the svd routine runs into an infinite loop.

Platform2: python 2.3.4 and 2.4a2; numarray 1.1; gcc 3.4.1; RH9 and gentoo 1.4

example:

>> from numarray.linear_algebra.mlab import *
>>> svd(arange(20.0, shape = (5,4)))

... infine loop ...


--------------------------------------------------------------------------

svd runs OK when compiled againts external LAPACK.

BTW: Why numarray used dgesdd routine for SVD, and not the svd routine that has the ATLAS equivalent?

  Nadav.



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion






From jmiller at stsci.edu  Thu Aug 12 09:49:06 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu Aug 12 09:49:06 2004
Subject: [Numpy-discussion] matrixmultiply problem
In-Reply-To: <20040812042736.GA14498@arbutus.physics.mcmaster.ca>
References: <000801c48019$2c830be0$6601a8c0@Laptop>
	 <20040812042736.GA14498@arbutus.physics.mcmaster.ca>
Message-ID: <1092329006.6703.5.camel@halloween.stsci.edu>

On Thu, 2004-08-12 at 00:27, David M. Cooke wrote:
> On Wed, Aug 11, 2004 at 11:05:07PM -0400, Geoffrey Philbrick wrote:
> > Executing the following code has the side effect of modifying one of
> > the arguments (matrices) I passed to matrixmultiply. Note that the
> > matrix m2 changes after the call to matrixmultiply. This is on Windows
> > with numarray 1.0. 
> 
> Wow, that's bad. Since I had nothing better to do [1], I tracked it down
> and submitted a patch (#1007730) to SourceForge to fix this.
> 
> [1] actually, I do, but...

This bug has been reported so many times... and I thought it was already
fixed.   The patch is in CVS now.  Thanks for pitching in here guys!

Regards,
Todd




From nadavh at visionsense.com  Thu Aug 12 10:30:01 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu Aug 12 10:30:01 2004
Subject: [Numpy-discussion] Bug in built in SVD?
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDB0@exchange2k.envision.co.il>

It can be a bit more complicate, the story is this:
One my gentoo system I use ATLAS, and gentoo instaltion generates the ATLAS version of blas and lapack which omit some of the original blas/lapack routines. Till now I used the numarray/Numeric internal lapack_lite package, since every time I tried to compile against the external lapck, I could not load the linear-algebra package because I got an error complaining about a missing "dgesdd_".

 In order to solve it I downloaded the original lapack library from netlib and compiled it with gcc3.4.1 and full optimization (by tweaking the makefile). I copied the generated lapack.a to /usr/local/lob/lapack_original.a and updated the link list in addon.py/setup.py (numarray/Numeric). That worked fine --- I have numarray/Numeric linked against highly optimized lapack without this infinite loop problem.

That leads me to the other issue: Assuming ATLAS routines runs faster then their equivalent lapack's. Why not replace dgesdd with the one which has an ATLAS equivalent?

Nadav

-----Original Message-----
From:	Michiel Jan Laurens de Hoon [mailto:mdehoon at ims.u-tokyo.ac.jp]
Sent:	Thu 12-Aug-04 10:29
To:	Nadav Horesh
Cc:	
Subject:	Re: [Numpy-discussion] Bug in built in SVD?
This may also be related to patch 732520 for Numerical Python. Some parts of 
lapack need to be compiled without optimization. Numerical Python (and I assume 
numarray) use optimization anyway, and usually gets away with it. So gcc 3.4.1 
may be doing the right thing after all. The external Lapack that you mentioned 
may have been compiled correctly, i.e. without optimization for the specific 
Lapack files.

--Michiel, U Tokyo.

Nadav Horesh wrote:
> Sorry it is a gcc3.4.1 optimization bug. Either lowering to optimization to -O1 or stepping back to gcc3.3.4 (with all optimization guns on) fix it.
> 
>    Nadav.
> 
> -----Original Message-----
> From:	Nadav Horesh
> Sent:	Wed 11-Aug-04 15:48
> To:	numpy-discussion
> Cc:	
> Subject:	[Numpy-discussion] Bug in built in SVD?
> When numarray is compiled with the buit-in linear-algebra routine the svd routine runs into an infinite loop.
> 
> Platform2: python 2.3.4 and 2.4a2; numarray 1.1; gcc 3.4.1; RH9 and gentoo 1.4
> 
> example:
> 
> 
>>>from numarray.linear_algebra.mlab import *
>>>
>>>>svd(arange(20.0, shape = (5,4)))
> 
> 
> ... infine loop ...
> 
> 
> --------------------------------------------------------------------------
> 
> svd runs OK when compiled againts external LAPACK.
> 
> BTW: Why numarray used dgesdd routine for SVD, and not the svd routine that has the ATLAS equivalent?
> 
>   Nadav.
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 
> 
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 
> 

-- 
Michiel de Hoon, Assistant Professor
University of Tokyo, Institute of Medical Science
Human Genome Center
4-6-1 Shirokane-dai, Minato-ku
Tokyo 108-8639
Japan
http://bonsai.ims.u-tokyo.ac.jp/~mdehoon







From Chris.Barker at noaa.gov  Thu Aug 12 11:49:01 2004
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Thu Aug 12 11:49:01 2004
Subject: [Numpy-discussion] Bug in built in SVD?
In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DEDB0@exchange2k.envision.co.il>
References: <07C6A61102C94148B8104D42DE95F7E86DEDB0@exchange2k.envision.co.il>
Message-ID: <411BB198.9040902@noaa.gov>

Nadav Horesh wrote:
> It can be a bit more complicate, the story is this:
> One my gentoo system I use ATLAS, and gentoo instaltion generates the ATLAS version of blas and lapack which omit some of the original blas/lapack routines. Till now I used the numarray/Numeric internal lapack_lite package, since every time I tried to compile against the external lapck, I could not load the linear-algebra package because I got an error complaining about a missing "dgesdd_".

I had the same issue, and got this reply from this very list:

Mats Klingberg wrote:
> I recently did this under Gentoo. It seems that the atlas package doesn't 
> build a complete lapack, which is necessary. Instead I emerged the newer 
> blas-atlas and lapack-atlas packages. These build a complete lapack library 
> with atlas optimizations. They are however masked with ~x86 so you have to 
> add the following to your /etc/portage/package.keywords:
> 
> app-sci/lapack-atlas ~x86
> app-sci/blas-atlas ~x86
> app-sci/blas-config ~x86
> app-sci/lapack-config ~x86
> 
> blas-config and lapack-config are packages that are needed by the other ones. 
> They can be used to switch between reference lapack or atlas optimized 
> libraries (if you install lapack-reference and blas-reference)
> 
> For more information see Gentoo bug #30453

I haven't had a chance to try it yet, but I will soon.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From aisaac at american.edu  Thu Aug 12 13:04:43 2004
From: aisaac at american.edu (Alan G Isaac)
Date: Thu Aug 12 13:04:43 2004
Subject: [Numpy-discussion] Re: matrixmultiply problem
Message-ID: <Mahogany-0.66.0-1792-20040812-160544.00@american.edu>

> The patch is in CVS now.

Can the conditions under which this problem occurs
be simply stated?

Thank you,
Alan Isaac





From cookedm at physics.mcmaster.ca  Thu Aug 12 13:32:03 2004
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu Aug 12 13:32:03 2004
Subject: [Numpy-discussion] Re: matrixmultiply problem
In-Reply-To: <Mahogany-0.66.0-1792-20040812-160544.00@american.edu>
References: <Mahogany-0.66.0-1792-20040812-160544.00@american.edu>
Message-ID: <20040812203120.GA8465@arbutus.physics.mcmaster.ca>

On Thu, Aug 12, 2004 at 04:05:41PM -0400, Alan G Isaac wrote:
> > The patch is in CVS now.
> 
> Can the conditions under which this problem occurs
> be simply stated?
> 
> Thank you,
> Alan Isaac

a = some matrix
b = some other matrix
transpose(b) is not contiguous

The numarray 1.0 (and earlier behaviour) is this:
dot(a,b) transposes b internally, finds it's not contiguous, and
makes a contiguous copy. Unfortunately, when it finishes, it
untransposes the *copy* of b back, not b itself.

So after dot(a, b), b is now the transpose of the original matrix

A quick fix is to define your own dot:
def dot(a, b):
    return numarray.dot(a, b.view())

The original b is then not touched, and the screwed-up matrix is
discarded. Note that transpose happens by fiddling with the layout of
the dimensions and strides in the array object; the data itself is not
touched.

[This all applies to matrixmultiply also, as matrixmultiply == dot]

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From a_wilson at mit.edu  Thu Aug 12 13:55:12 2004
From: a_wilson at mit.edu (Andrew Wilson)
Date: Thu Aug 12 13:55:12 2004
Subject: [Numpy-discussion] clear a variable
Message-ID: <411BD90F.7050308@mit.edu>

Hello,
    I can't seem to find this in the documentation, is there a way to 
clear a variable.  For example I have an application that opens displays 
a bunch of images, they can be big memory hogs. When I'm not showing 
them I'd like to clear out those variables, instead of having to close 
down my program and start over.  I know matlab has a command "clear 
/var/" which will do exactly that.
    Thanks for the help!
       Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040812/ba85ea02/attachment.html>

From rowen at u.washington.edu  Thu Aug 12 14:19:09 2004
From: rowen at u.washington.edu (Russell E Owen)
Date: Thu Aug 12 14:19:09 2004
Subject: [Numpy-discussion] clear a variable
In-Reply-To: <411BD90F.7050308@mit.edu>
References: <411BD90F.7050308@mit.edu>
Message-ID: <p06110401bd418e25c146@[128.95.99.44]>

At 4:54 PM -0400 2004-08-12, Andrew Wilson wrote:
Hello,
     I can't seem to find this in the documentation, is there a way to 
clear a variable.  For example I have an application that opens 
displays a bunch of images, they can be big memory hogs. When I'm not 
showing them I'd like to clear out those variables, instead of having 
to close down my program and start over.  I know matlab has a command 
"clear var" which will do exactly that.

del(obj)

If you are the only user of "obj", this'll do the trick. (del tries 
to be polite and not pull the rug out from under other users).

obj can be nearly anything, such as an array or any other object you 
created. del can also be used to delete specific elements of lists 
and dictionaries, i.e.:
del(adict[akey])
del(alist[ind1:ind2])

I hope this help.

-- Russell



From curzio.basso at unibas.ch  Fri Aug 13 05:42:04 2004
From: curzio.basso at unibas.ch (Curzio Basso)
Date: Fri Aug 13 05:42:04 2004
Subject: [Numpy-discussion] invalid numeric result(s) in divide
Message-ID: <411CB6D9.1010906@unibas.ch>

Hi all.

I have a matrix tmp a matrix w, both Float32, with

 >>> tmp.shape
(3,512,512)
 >>> w.shape
(512,512)
 >>> w.max()
1.0
 >>> w.min()
0.0
 >>> NA.where(NA.and_(w > 0.0, w < 1.0e-2))
(array([]), array([]))

What is for me surprising is the following:

 >>> tmp = NA.where(w > 0.0, tmp / w, 0.0)
Warning: Encountered invalid numeric result(s)  in divide

Ah, both matrices have no 'nan' or 'inf' elements. So, where could the 
invalid result come from?

thanks, curzio



From focke at slac.stanford.edu  Fri Aug 13 08:40:07 2004
From: focke at slac.stanford.edu (Warren Focke)
Date: Fri Aug 13 08:40:07 2004
Subject: [Numpy-discussion] invalid numeric result(s) in divide
In-Reply-To: <411CB6D9.1010906@unibas.ch>
References: <411CB6D9.1010906@unibas.ch>
Message-ID: <Pine.LNX.4.58.0408130814250.26978@auriga.slac.stanford.edu>


On Fri, 13 Aug 2004, Curzio Basso wrote:

>  >>> tmp = NA.where(w > 0.0, tmp / w, 0.0)
> Warning: Encountered invalid numeric result(s)  in divide
>
> Ah, both matrices have no 'nan' or 'inf' elements. So, where could the
> invalid result come from?

All of the arguments to NA.where are evaluated before the function is
called, so tmp/w results in divide-by-zero.  The array that gets passed to
NA.where contains some inf or nan (I suspect the latter based on the
warning message), but you don't see them in the return value because
NA.where replaces them w/ 0.

Warren Focke




From jimcser at pacifier.com  Sat Aug 14 14:05:03 2004
From: jimcser at pacifier.com (Jim Cser)
Date: Sat Aug 14 14:05:03 2004
Subject: [Numpy-discussion] Using sum() within a function
Message-ID: <411E7E49.2020709@pacifier.com>

Hello-
I have a function to generate a multi-dimensional array, which then
gets summed over one axis.  The problem is that the dimensions
are large, and I run out of memory when I create the entire array,
so I'm trying to do the sum *within* the function.

Example-- variables x,y,z,t; dimensions numX, numY, numZ, numT;
functions f1(x,y,z,t), f2(y,z,t);  want to calculate f1*f2 and
sum over t to get out[x,y,z].

With loops, I could do it like--
out = zeros((numX,numY,numZ))
for x in range(numX):
   for y in range(numY):
     for z in range(numZ):
        for t in range(numT):
           tempval = f1(x,y,z,t) * f2(y,z,t)
           out[x,y,z] = out[x,y,z] + tempval

With numarray, if I had enough memory, I could just do--
temp1 = fromfunction(f1,(numX,numY,numZ,numT))
temp2 = resize(fromfunction(f2,(numY,numZ,numT)),(numX,numY,numZ,numT))
out = sum(temp1 * temp2, axis = 3)

Instead, I'm trying to do something like--
def f3(x,y,z):
   for t in range(numT):
     tempval = f1(x,y,z,t) * f2(y,z,t)

   outval = sum(tempval,axis = 3)
   return outval

out = fromfunction(f3,(numX,numY,numZ))

I've been trying various slicing and indexing, but I can't seem to
get that *extra* dimension within the 3-D function. I've scoured the
documentation and list archives, but haven't found exactly what I need.
Any suggestions?  Am I stuck with generating the entire 4-D array?

Thanks in advance,
Jim Cser




From perry at stsci.edu  Sat Aug 14 14:44:32 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Sat Aug 14 14:44:32 2004
Subject: [Numpy-discussion] Using sum() within a function
In-Reply-To: <411E7E49.2020709@pacifier.com>
Message-ID: <NEBBIJKBMLDBLNCEEFOCCEEMFHAA.perry@stsci.edu>

Jim Cser wrote:
> 
> Instead, I'm trying to do something like--
> def f3(x,y,z):
>    for t in range(numT):
>      tempval = f1(x,y,z,t) * f2(y,z,t)
> 
>    outval = sum(tempval,axis = 3)
>    return outval
>
Perhaps I'm confused but it seems to me the for loop is wrong.

Don't you want something like:

def f3(x,y,z):
   tempval = 0.*x
   for t in range(numT):
     tempval += f1(x,y,z,t) * f2(y,z,t)
   return tempval

Instead?

Not that I've tried this, but so long as the function will
work given 3-d arrays for x,y,z, it should. (though the dependence
on numT as a global is a bit worrisome)

Perry GReenfield




From jimcser at pacifier.com  Mon Aug 16 09:59:07 2004
From: jimcser at pacifier.com (Jim Cser)
Date: Mon Aug 16 09:59:07 2004
Subject: [Numpy-discussion] Using sum() within a function
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCCEEMFHAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCCEEMFHAA.perry@stsci.edu>
Message-ID: <4120E7B9.5070001@pacifier.com>

Perry Greenfield wrote:

> Jim Cser wrote:
> 
>>Instead, I'm trying to do something like--
>>def f3(x,y,z):
>>   for t in range(numT):
>>     tempval = f1(x,y,z,t) * f2(y,z,t)
>>
>>   outval = sum(tempval,axis = 3)
>>   return outval
>>
> 
> Perhaps I'm confused but it seems to me the for loop is wrong.
> 
> Don't you want something like:
> 
> def f3(x,y,z):
>    tempval = 0.*x
>    for t in range(numT):
>      tempval += f1(x,y,z,t) * f2(y,z,t)
>    return tempval
> 
> Instead?
> 
> Not that I've tried this, but so long as the function will
> work given 3-d arrays for x,y,z, it should. (though the dependence
> on numT as a global is a bit worrisome)
> 
> Perry GReenfield


The loop was indeed wrong, and "tempval = 0.*x" was the trick I needed, 
thanks.  Unfortunately, iterating over t is still extremely slow. 
Ideally, there would be a way to use fromfunction() with slices as 
arguments.

-Jim



From rob at hooft.net  Mon Aug 16 10:06:05 2004
From: rob at hooft.net (Rob Hooft)
Date: Mon Aug 16 10:06:05 2004
Subject: [Numpy-discussion] Is there a better way to do this?
In-Reply-To: <16A7C641-DB7D-11D8-A37A-000393479EE8@earthlink.net>
References: <16A7C641-DB7D-11D8-A37A-000393479EE8@earthlink.net>
Message-ID: <4120E925.70501@hooft.net>

Hee-Seng Kye wrote:
> My question is not directly related to NumPy, but since many people here 
> deal with numbers, I was wondering if I could get some help; it would be 
> even better if there is a NumPy (or Numarray) function that takes care 
> of what I want!
> 
> I'm trying to write a program that computes six-digit numbers, in which 
> the left digit is always smaller than its following digit (i.e., it's 
> always ascending).  The best I could do was to have many embedded 'for' 
> statement:
> 
> c = 1
> for p0 in range(0, 7):
>   for p1 in range(1, 12):
>     for p2 in range(2, 12):
>       for p3 in range(3, 12):
>         for p4 in range(4, 12):
>           for p5 in range(5, 12):
>             if p0 < p1 < p2 < p3 < p4 < p5:
>               print repr(c).rjust(3), "\t",
>               print "%X %X %X %X %X %X" % (p0, p1, p2, p3, p4, p5)
>               c += 1
> print "...Done"
> 
> This works, except that it's very slow.  I need to get it up to 
> nine-digit numbers, in which case it's significantly slow.  I was 
> wondering if there is a more efficient way to do this.
> 
> I would highly appreciate it if anyone could help.

Sorry for taking a month. My solution is not faster than any of the ones 
that have already been proposed, but it is more elegant because it does 
not require the "n" nested for loops.

Another hint is to use a generator for this purpose.

Advantage is that the base and the number of digits can both be 
externally defined!

Regards,

Rob

# As a reference, the general case where digits are not created in
# order
def gen(ndigits, base):
   dig = [0]*ndigits
   yield dig
   while 1:
     for num in range(ndigits-1,-1,-1):
       if dig[num] < base - 1:
         dig[num] += 1
         break
       else:
         dig[num] = 0
     else:
       return
     yield dig

def genordered(ndigits, base):
   if ndigits > base:
     return
   dig = range(ndigits)
   yield dig
   while 1:
     for num in range(ndigits-1,-1,-1):
       if dig[num] < base-ndigits+num:
         dig[num] += 1
         for num2 in range(num+1,ndigits):
           dig[num2] = dig[num2-1] + 1
         break
     else:
       return
     yield dig

c = 1
for dig in genordered(ndigits = 6, base = 12):
   # This is the same as your print statement, but again independent of
   # the number of digits
   print "%3d\t"%c,' '.join(map(lambda x: "%X"%x, dig))
   c += 1
print "...Done"



From perry at stsci.edu  Mon Aug 16 10:10:05 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Mon Aug 16 10:10:05 2004
Subject: [Numpy-discussion] Using sum() within a function
In-Reply-To: <4120E7B9.5070001@pacifier.com>
Message-ID: <CEELJPECNGEGKFNDLHFFGEHGCEAA.perry@stsci.edu>

Jim Cser wrote:
> 
> The loop was indeed wrong, and "tempval = 0.*x" was the trick I needed, 
> thanks.  Unfortunately, iterating over t is still extremely slow. 
> Ideally, there would be a way to use fromfunction() with slices as 
> arguments.
If the other 3 dimensions are small (e.g., each time sample 
involves an array whose size is less than 1000 elements total), 
then the slowness is due to the small arrays. In that event
you may want iterate over a different dimension than time, or
construct the for loop to iterate over ranges of times rather
than one at a time. 

Perry



From eric at enthought.com  Mon Aug 16 10:42:12 2004
From: eric at enthought.com (eric jones)
Date: Mon Aug 16 10:42:12 2004
Subject: [Numpy-discussion] ANN: Python/Scientific Job openings at Enthought
Message-ID: <4120F1F3.4020006@enthought.com>

Hey Folks,

Enthought continues to grow, and we're hiring again.  There are several 
positions open, and most of them involve python+scientific computing.  A 
couple of the positions also involve support of SciPy.  I'm very 
interested in finding a person for the "scientist" job posting that is 
both able to work well with our customers and also has the personality 
and interest to work with the SciPy community to help push the library 
forward.

More information is available on our web site.

    http://www.enthought.com/careers.htm

Our current development efforts are in the areas of electromagnetics, 
geophysics, and graphics (2D and 3D). 

If you have any questions, please feel free to contact me.  Please 
forward the link to anyone you feel might be interested.

thanks,
eric




From jimcser at pacifier.com  Mon Aug 16 13:29:03 2004
From: jimcser at pacifier.com (jimcser at pacifier.com)
Date: Mon Aug 16 13:29:03 2004
Subject: [Numpy-discussion] Using sum() within a function
In-Reply-To: <CEELJPECNGEGKFNDLHFFGEHGCEAA.perry@stsci.edu>
References: <4120E7B9.5070001@pacifier.com>
    <CEELJPECNGEGKFNDLHFFGEHGCEAA.perry@stsci.edu>
Message-ID: <16683.66.192.179.130.1092688071.squirrel@webmail.pacifier.com>

>
> Jim Cser wrote:
>>
>> The loop was indeed wrong, and "tempval = 0.*x" was the trick I needed,
>> thanks.  Unfortunately, iterating over t is still extremely slow.
>> Ideally, there would be a way to use fromfunction() with slices as
>> arguments.
> If the other 3 dimensions are small (e.g., each time sample
> involves an array whose size is less than 1000 elements total),
> then the slowness is due to the small arrays. In that event
> you may want iterate over a different dimension than time, or
> construct the for loop to iterate over ranges of times rather
> than one at a time.
>
> Perry

I was just using T as a sample name, so I am not actually doing
a time series of anything.  My dimesions are on the order of
numT = 500, numX = 500, numY = 100, numZ = 10, so smallness is not
the problem.  Unfortunately, I don't need to sum over Z, the short
dimension.  If my system didn't hang, I would have done it the
easier way in the first place.

Perhaps I need to learn more about memory management-- I'm sure
that many Python users are crunching bigger even arrays than this.
For what it's worth, I'm using Python 2.3 / Numarray 1.0 with
PythonWin on an XP box, 512M RAM.

-Jim




From skip at pobox.com  Tue Aug 17 12:07:01 2004
From: skip at pobox.com (Skip Montanaro)
Date: Tue Aug 17 12:07:01 2004
Subject: [Numpy-discussion] Re: core dump during import
Message-ID: <16674.17638.251233.868612@montanaro.dyndns.org>

A few days ago on scipy-user and earlier today on numpy-discussion I wrote:

    If I execute

        from scipy.stats import distribution

    at the interpreter prompt I get a segfault.

Stepping through the suspect code yielded very strange results, so I tried
recompiling cephesmodule.c with -O0.  That seems to have solved the problem
for now.  I guess that means it's a GCC optimizer bug.

-- 
Skip Montanaro
Got spam? http://www.spambayes.org/
skip at pobox.com



From jimcser at pacifier.com  Tue Aug 17 13:54:01 2004
From: jimcser at pacifier.com (Jim Cser)
Date: Tue Aug 17 13:54:01 2004
Subject: [Numpy-discussion] Re: Using sum() within a function
In-Reply-To: <411E7E49.2020709@pacifier.com>
References: <411E7E49.2020709@pacifier.com>
Message-ID: <41225FDC.8080000@pacifier.com>

> Example-- variables x,y,z,t; dimensions numX, numY, numZ, numT;
> functions f1(x,y,z,t), f2(y,z,t);  want to calculate f1*f2 and
> sum over t to get out[x,y,z].

Cobbling together a number of suggestions, what finally worked was--

def f3(x, y, z, t_range=arange(numT)):
   tempval = 0.* x
   for t in t_range:
     tempval += f1(x,y,z,t) + f2(y,z,t)
   return tempval

out = fromfunction(f3,(numX,numY,numZ,1))


I couldn't quite get sum() to work inside the function, but this
is definitely good enough for now.  Thanks to all for your help.
-Jim Cser




From skip at pobox.com  Tue Aug 17 15:57:41 2004
From: skip at pobox.com (Skip Montanaro)
Date: Tue Aug 17 15:57:41 2004
Subject: [Numpy-discussion] core dump during import
Message-ID: <16674.3703.733907.166344@montanaro.dyndns.org>

(I posted this last week to scipy-user but received no responses.  Since it
seems to involve Numeric I thought I'd try here.)

------------------------------------------------------------------------------

I'm the "python guy" at work which by default makes me the "scipy
installation guy" as well, though I'm not using it for any of my own
projects.  Awhile ago I performed a very basic installation:

    install latest f2py
    install Numeric 23.1
    install SciPy_complete-0.3

The platform is Solaris 9 on Intel.  I'm using Python 2.3.4 (though
originally installed using 2.3.3).  GCC and G77 are both at 3.3.2.

The installation goes fine with the one caveat that I needed to point
LD_RUN_PATH at the g77 lib dir so libg2c.so.0 could be found at runtime.

If I execute

    from scipy.stats import distribution

at the interpreter prompt I get a segfault.  Here's the beginning of the gdb
traceback:

    #0  0x20ec8356 in ?? ()
    #1  0xdd2d7e24 in Cephes_InitOperators (dictionary=0x8486dfc)
        at Lib/special/cephesmodule.c:413
    #2  0xdd2d7a7a in initcephes () at Lib/special/cephesmodule.c:1105
    #3  0x080d082e in _PyImport_LoadDynamicModule (
        name=0x8040500 "scipy.special.cephes", 
        pathname=0x8040070 "/opt/lang/python/lib/python2.3/site-packages/scipy/special/cephes.so", fp=0x8148780) at ../Python/importdl.c:53
    #4  0x080cd637 in load_module (name=0x8040500 "scipy.special.cephes", 
        fp=0x8148780, 
        buf=0x8040070 "/opt/lang/python/lib/python2.3/site-packages/scipy/special/cephes.so", type=3, loader=0x0) at ../Python/import.c:1708
    #5  0x080ce345 in import_submodule (mod=0x8149fa4, subname=0x804050e "cephes", 
        fullname=0x8040500 "scipy.special.cephes") at ../Python/import.c:2291
    #6  0x080cdea4 in load_next (mod=0x8149fa4, altmod=0x81265e0, 
        p_name=0x8040930, buf=0x8040500 "scipy.special.cephes", p_buflen=0x80404fc)
        at ../Python/import.c:2111
    #7  0x080cfbf1 in import_module_ex (name=0x0, globals=0x8486c64, 
        locals=0x8486c64, fromlist=0x83e256c) at ../Python/import.c:1957
    #8  0x080ced40 in PyImport_ImportModuleEx (name=0x8369c34 "cephes", 
        globals=0x8486c64, locals=0x8486c64, fromlist=0x83e256c)
        at ../Python/import.c:1998
    #9  0x080a1bde in builtin___import__ (self=0x0, args=0x8483d4c)
        at ../Python/bltinmodule.c:45
    #10 0x080fac78 in PyCFunction_Call (func=0x8151bec, arg=0x8483d4c, kw=0x0)
        at ../Objects/methodobject.c:108
    #11 0x08062e90 in PyObject_Call (func=0x8151bec, arg=0x8483d4c, kw=0x0)
        at ../Objects/abstract.c:1755

Since I don't use SciPy, Cephes or f2py outside of installing SciPy I don't
really know where to start poking.  Frame 1 is at this line:

    413    f = PyUFunc_FromFuncAndData(cephes3a_functions, bdtr_data, cephes_4_types, 2, 3, 1, PyUFunc_None, "bdtr", bdtr_doc, 0);

and cephes3a_functions looks suspicious:

    (gdb) p cephes3a_functions
    $2 = {0x5, 0x6}

I presume considering the variable name those are supposed to be valid
function pointers.  Stepping through initcephes leads me to this:

    (gdb) n
    393         cephes3a_functions[0] = PyUFunc_fff_f_As_iid_d;
    (gdb) n
    391         cephes3_functions[2] = PyUFunc_ffF_F_As_ddD_D;
    (gdb) n
    392         cephes3_functions[3] = PyUFunc_ddD_D;
    (gdb) n
    394         cephes3a_functions[1] = PyUFunc_ddd_d_As_iid_d;
    (gdb) n
    395         cephes3_2_functions[0] = PyUFunc_fff_ff_As_ddd_dd;
    (gdb) p cephes3a_functions
    $9 = {0x5, 0x6}
    (gdb) p PyUFunc_ddD_D
    $10 = {void (char **, int *, int *, void *)} 0xdd2e4720 <PyUFunc_ddD_D>
    (gdb) p PyUFunc_ddd_d_As_iid_d
    $11 = {void (char **, int *, int *, 
        void *)} 0xdd2e4a20 <PyUFunc_ddd_d_As_iid_d>

It seems there's something amiss with the cephes3a_functions initialization.
In fact, other cephes*_functions variables appear bogus as well:

    (gdb) p cephes3_functions
    $12 = {0, 0, 0x3, 0x4}
    (gdb) p cephes1_functions
    $13 = {0x464c457f, 0x10101}
    (gdb) p cephes2_4_functions
    $14 = {0x7e34, 0x7}

Does this ring a bell with anybody?  Any clues about how to proceed would be
appreciated.

Thanks,

-- 
Skip Montanaro
Got spam? http://www.spambayes.org/
skip at pobox.com



From nadavh at visionsense.com  Wed Aug 18 03:32:05 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Wed Aug 18 03:32:05 2004
Subject: [Numpy-discussion] Bool array missing sum function.
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDB7@exchange2k.envision.co.il>

Why Bool arrays are missing the sum method?

>>> b = array((1,0,1), type=Bool)
>>> b
array([1, 0, 1], type=Bool)
>>> b.sum()

Traceback (most recent call last):
  File "<pyshell#129>", line 1, in -toplevel-
    b.sum()
  File "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", line 1125, in sum
    return ufunc.add.reduce(ufunc.add.areduce(self, type=type).flat, type=type)
  File "/usr/local/lib/python2.4/site-packages/numarray/ufunc.py", line 1165, in _cum_cache_miss
    mode, win1, wout, cfunc, ufargs = \
  File "/usr/local/lib/python2.4/site-packages/numarray/ufunc.py", line 1225, in _cum_setup
    otype, cfunc = ufdict[(optype,)]
KeyError: Bool

   Nadav.



From nadavh at visionsense.com  Wed Aug 18 03:36:10 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Wed Aug 18 03:36:10 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDB8@exchange2k.envision.co.il>

>>> a = arange(20, shape=(4,5))
>>> a
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])
>>> a[2:4,5:7]   # Shouldn't it raise index error?
array([])

>>> a[2:4, 5:7].iscontiguous()
0    # ????
>>> a[2:4, 5:7].copy().iscontiguous()
0    # ????

Shouldn't iscontiguous method of empty arrays and array.copy() return 1?

  Nadav.



From jmiller at stsci.edu  Wed Aug 18 03:48:04 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Aug 18 03:48:04 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DEDB8@exchange2k.envision.co.il>
References: 	 <07C6A61102C94148B8104D42DE95F7E86DEDB8@exchange2k.envision.co.il>
Message-ID: <1092825988.3752.37.camel@localhost.localdomain>

On Wed, 2004-08-18 at 07:36, Nadav Horesh wrote:
> >>> a = arange(20, shape=(4,5))
> >>> a
> array([[ 0,  1,  2,  3,  4],
>        [ 5,  6,  7,  8,  9],
>        [10, 11, 12, 13, 14],
>        [15, 16, 17, 18, 19]])
> >>> a[2:4,5:7]   # Shouldn't it raise index error?
> array([])

I wasn't sure about this,  but what we're doing is faithful to what
Numeric does.

> >>> a[2:4, 5:7].iscontiguous()
> 0    # ????

Likewise here.

> >>> a[2:4, 5:7].copy().iscontiguous()
> 0    # ????

We're not compatible here.

> Shouldn't iscontiguous method of empty arrays and array.copy() return 1?

I'd say we should go with compatibility.

Todd




From jmiller at stsci.edu  Wed Aug 18 04:09:30 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Aug 18 04:09:30 2004
Subject: [Numpy-discussion] Bool array missing sum function.
In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DEDB7@exchange2k.envision.co.il>
References: 	 <07C6A61102C94148B8104D42DE95F7E86DEDB7@exchange2k.envision.co.il>
Message-ID: <1092827284.3752.58.camel@localhost.localdomain>

I logged this as a bug on Source Forge and I'll look into it.

Todd

On Wed, 2004-08-18 at 07:32, Nadav Horesh wrote:
> Why Bool arrays are missing the sum method?
> 
> >>> b = array((1,0,1), type=Bool)
> >>> b
> array([1, 0, 1], type=Bool)
> >>> b.sum()
> 
> Traceback (most recent call last):
>   File "<pyshell#129>", line 1, in -toplevel-
>     b.sum()
>   File "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", line 1125, in sum
>     return ufunc.add.reduce(ufunc.add.areduce(self, type=type).flat, type=type)
>   File "/usr/local/lib/python2.4/site-packages/numarray/ufunc.py", line 1165, in _cum_cache_miss
>     mode, win1, wout, cfunc, ufargs = \
>   File "/usr/local/lib/python2.4/site-packages/numarray/ufunc.py", line 1225, in _cum_setup
>     otype, cfunc = ufdict[(optype,)]
> KeyError: Bool
> 
>    Nadav.
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 




From nadavh at visionsense.com  Wed Aug 18 04:51:18 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Wed Aug 18 04:51:18 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDB9@exchange2k.envision.co.il>



-----Original Message-----
From:	Todd Miller [mailto:jmiller at stsci.edu]
Sent:	Wed 18-Aug-04 13:46
To:	Nadav Horesh
Cc:	numpy-discussion
Subject:	Re: [Numpy-discussion] Empty arrays strange behaviour
On Wed, 2004-08-18 at 07:36, Nadav Horesh wrote:
> >>> a = arange(20, shape=(4,5))
> >>> a
> array([[ 0,  1,  2,  3,  4],
>        [ 5,  6,  7,  8,  9],
>        [10, 11, 12, 13, 14],
>        [15, 16, 17, 18, 19]])
> >>> a[2:4,5:7]   # Shouldn't it raise index error?
> array([])

I wasn't sure about this,  but what we're doing is faithful to what
Numeric does.

> >>> a[2:4, 5:7].iscontiguous()
> 0    # ????

Likewise here.

> >>> a[2:4, 5:7].copy().iscontiguous()
> 0    # ????

We're not compatible here.

> Shouldn't iscontiguous method of empty arrays and array.copy() return 1?

I'd say we should go with compatibility.

Todd



It isn't a major flaw, so compatibility is OK.
I got to this issue since I had a bug. If there was an option to get an IndexError exception it would ease the debugging.

Thank you,
  Nadav.



From tim.hochberg at cox.net  Wed Aug 18 07:24:02 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Wed Aug 18 07:24:02 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DEDB9@exchange2k.envision.co.il>
References: <07C6A61102C94148B8104D42DE95F7E86DEDB9@exchange2k.envision.co.il>
Message-ID: <4123662C.2090807@cox.net>

Nadav Horesh wrote:

>
>-----Original Message-----
>
>On Wed, 2004-08-18 at 07:36, Nadav Horesh wrote:
>  
>
>>>>>a = arange(20, shape=(4,5))
>>>>>a
>>>>>          
>>>>>
>>array([[ 0,  1,  2,  3,  4],
>>       [ 5,  6,  7,  8,  9],
>>       [10, 11, 12, 13, 14],
>>       [15, 16, 17, 18, 19]])
>>    
>>
>>>>>a[2:4,5:7]   # Shouldn't it raise index error?
>>>>>          
>>>>>
>>array([])
>>    
>>
>
>I wasn't sure about this,  but what we're doing is faithful to what
>Numeric does.
>
>[SNIP]
>
>
>I'd say we should go with compatibility.
>
>Todd
>
>  
>
Definately.

>It isn't a major flaw, so compatibility is OK.
>I got to this issue since I had a bug. If there was an option to get an IndexError exception it would ease the debugging.
>  
>
Please, no. In addition to breaking everybody's code, this change would 
make numarray's behaviour gratuitously different from the python list 
type as well as the numeric array type. In addition, the current 
behaviour does the Right Thing (TM) more often than not in the corner 
cases of algorithms.

-tim





From aisaac at american.edu  Wed Aug 18 08:15:11 2004
From: aisaac at american.edu (Alan G Isaac)
Date: Wed Aug 18 08:15:11 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
In-Reply-To: <4123662C.2090807@cox.net>
References: <07C6A61102C94148B8104D42DE95F7E86DEDB9@exchange2k.envision.co.il><4123662C.2090807@cox.net>
Message-ID: <Mahogany-0.66.0-588-20040818-111626.00@american.edu>

>> I'd say we should go with compatibility.
>> Todd


Script:
a=[[i+i*j for i in range(5)]for j in range(4)]
print a[2:4][5:7]
import Numeric
b=Numeric.array(a)
print b[2:4,5:7]
import numarray
c=numarray.array(a)
print c[2:4,5:7]

Output:
[]
zeros((2, 0), 'l')
[]

fwiw,
Alan Isaac





From jmiller at stsci.edu  Wed Aug 18 08:31:05 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Aug 18 08:31:05 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
In-Reply-To: <Mahogany-0.66.0-588-20040818-111626.00@american.edu>
References: 	 <07C6A61102C94148B8104D42DE95F7E86DEDB9@exchange2k.envision.co.il>
	 <4123662C.2090807@cox.net>
	 <Mahogany-0.66.0-588-20040818-111626.00@american.edu>
Message-ID: <1092843024.2260.17.camel@halloween.stsci.edu>

On Wed, 2004-08-18 at 11:16, Alan G Isaac wrote:
> >> I'd say we should go with compatibility.
> >> Todd
> 
> 
> Script:
> a=[[i+i*j for i in range(5)]for j in range(4)]
> print a[2:4][5:7]
> import Numeric
> b=Numeric.array(a)
> print b[2:4,5:7]
> import numarray
> c=numarray.array(a)
> print c[2:4,5:7]
> 
> Output:
> []
> zeros((2, 0), 'l')
> []
> 

You've got something of a point here,  if you're trying to say that
compatibility isn't perfect anyway.

Here's what I meant:

>>> from numarray import * 
>>> a = arange(20)
>>> a.shape = (4,5)
>>> a[2:4,5:7]
array([])
>>> a[2:4,5:7].info()
class: <class 'numarray.numarraycore.NumArray'>
shape: (2, 0)
strides: (20, 4)
byteoffset: 60
bytestride: 4
itemsize: 4
aligned: 1
contiguous: 0
data: <memory at 0x404c0298 with size:0x00000050 held by object 0x401839c8 aliasing object 0x00000000>
byteorder: little
byteswap: 0
type: Int32

The key points for me are:

1.  The slice operation doesn't raise an exception even though it has
"invalid" indexes.  As Tim H. pointed out,  those are standard Python
semantics.

2.  The shape of the resultant array is the same.

I don't know anything about the purpose of "zeros()" in this context. It
looks inconsistent to me.

Regards,
Todd





From jmiller at stsci.edu  Wed Aug 18 10:26:12 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Aug 18 10:26:12 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
In-Reply-To: <412378FE.5080203@cox.net>
References: 	 <07C6A61102C94148B8104D42DE95F7E86DEDB9@exchange2k.envision.co.il>
	 <4123662C.2090807@cox.net>
	 <Mahogany-0.66.0-588-20040818-111626.00@american.edu>
	 <1092843024.2260.17.camel@halloween.stsci.edu>  <412378FE.5080203@cox.net>
Message-ID: <1092849929.3418.41.camel@halloween.stsci.edu>

On Wed, 2004-08-18 at 11:42, Tim Hochberg wrote:
> Todd Miller wrote:
> 
> 
> [CHOP]
> 
> >The key points for me are:
> >
> >1.  The slice operation doesn't raise an exception even though it has
> >"invalid" indexes.  As Tim H. pointed out,  those are standard Python
> >semantics.
> >
> >2.  The shape of the resultant array is the same.
> >
> >I don't know anything about the purpose of "zeros()" in this context. It
> >looks inconsistent to me.
> >
> >  
> >
> I believe it's to disambiguate the various possible arrays of total size 
> zero. I like the Numeric behaviour better here, although it's not a big 
> deal. In the following, both the size and type of the numarrays are 
> ambiguous from their repr(). This means that among other things 
> eval(repr(arg)) won't return arg even for nice short arrays (I realize 
> the numarray truncates the reprs of long arrays by default and I think 
> that's fine -- I imagine there's some way to disable it, but I haven't 
> looked for it yet).
> 
> -tim
> 
>  >>> import Numeric as np
>  >>> np.zeros([0])
> zeros((0,), 'l')
>  >>> np.zeros([0,2])
> zeros((0, 2), 'l')
>  >>> import numarray as na
>  >>> na.zeros([0])
> array([])
>  >>> na.zeros([0,2])
> array([])
>  >>> na.arange(100)
> 
> 
OK, I logged this on SF.   Thanks for the explanation.

Todd




From southey at uiuc.edu  Fri Aug 20 07:52:09 2004
From: southey at uiuc.edu (Bruce Southey)
Date: Fri Aug 20 07:52:09 2004
Subject: [Numpy-discussion] Accessing a C library from numarray
Message-ID: <7644556.72de3142.81fd500@expms6.cites.uiuc.edu>

Hi,
I need to access a GLP'ed C library in numarray. The library functions are
scalar ints and doubles as inputs and usually doubles as outputs. 

Is there a recommended (and simple) way to achieve this? There are a few
different  C API's that exist from the Python to numarrays.

Would it be sufficient to first write C code that uses the library and use that
code for a C API in Python or numarray?

Thanks,

Bruce Southey



From smpitts at ou.edu  Mon Aug 23 09:25:16 2004
From: smpitts at ou.edu (smpitts at ou.edu)
Date: Mon Aug 23 09:25:16 2004
Subject: [Numpy-discussion] MA.zeros problem
Message-ID: <88f1be9567c6.4129d431@ou.edu>

Hi all,
I have some code that uses Numeric, and I'm trying to change it so that it supports MA as well. My code uses Numeric.zeros to create a matrix and then populates it, but I'm having trouble figuring out how to do something similar in MA.

Python 2.2.3 (#1, Oct 15 2003, 23:33:35) 
[GCC 3.3.1 20030930 (Red Hat Linux 3.3.1-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MA
>>> MA.__version__
'11.1.0'
>>> a = MA.zeros((2,2))
>>> a
array(
 [[0,0,]
 [0,0,]])
>>> a[1][1] = 2
>>> a
array(
 [[0,0,]
 [0,0,]])

Is there some function to automatically create a masked array of arbitrary dimensions filled with zeroes? Thanks.
--
Stephen Pitts
smpitts at ou.edu 




From reggie at merfinllc.com  Mon Aug 23 10:27:38 2004
From: reggie at merfinllc.com (Reggie Dugard)
Date: Mon Aug 23 10:27:38 2004
Subject: [Numpy-discussion] MA.zeros problem
In-Reply-To: <88f1be9567c6.4129d431@ou.edu>
References: <88f1be9567c6.4129d431@ou.edu>
Message-ID: <1093281989.31515.21.camel@pika>

Stephen,

You might try something like:

>>> a = MA.zeros((2,2))
>>> a
array(
 [[0,0,]
 [0,0,]])
>>> a[1,1] = 2
>>> a
array(
 [[0,0,]
 [0,2,]])

Because MA uses copy semantics, when you specify a[1][1], the a[1] makes
a copy of the first row of the original array, and then the second [1]
index does a setitem on that.  If you use the a[1,1] syntax, the setitem
is done on 'a' itself.

Hope this is of some help.

Reggie Dugard
Merfin, LLC


On Mon, 2004-08-23 at 09:25, smpitts at ou.edu wrote:
> Hi all,
> I have some code that uses Numeric, and I'm trying to change it so that it supports MA as well. My code uses Numeric.zeros to create a matrix and then populates it, but I'm having trouble figuring out how to do something similar in MA.
> 
> Python 2.2.3 (#1, Oct 15 2003, 23:33:35) 
> [GCC 3.3.1 20030930 (Red Hat Linux 3.3.1-6)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import MA
> >>> MA.__version__
> '11.1.0'
> >>> a = MA.zeros((2,2))
> >>> a
> array(
>  [[0,0,]
>  [0,0,]])
> >>> a[1][1] = 2
> >>> a
> array(
>  [[0,0,]
>  [0,0,]])
> 
> Is there some function to automatically create a masked array of arbitrary dimensions filled with zeroes? Thanks.
> --
> Stephen Pitts
> smpitts at ou.edu 
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion





From simon at arrowtheory.com  Wed Aug 25 17:28:08 2004
From: simon at arrowtheory.com (Simon Burton)
Date: Wed Aug 25 17:28:08 2004
Subject: [Numpy-discussion] Accessing a C library from numarray
In-Reply-To: <7644556.72de3142.81fd500@expms6.cites.uiuc.edu>
References: <7644556.72de3142.81fd500@expms6.cites.uiuc.edu>
Message-ID: <20040826100722.7278abdf.simon@arrowtheory.com>

This sounds like a job for pyrex. Here is a recent message from the
pyrex list.

cheers,
Simon.

On Wed, 25 Aug 2004 15:37:58 +0100 (BST)
Michael Hoffman <hoffman at ebi.ac.uk> wrote:

> I wanted to use Pyrex with Numarray, which is the supported
> implementation of Numeric, which is no longer supported. Changing the
> demo that comes with Pyrex to use Numarray instead of Numeric was
> instructive. I thought the results of this might be useful to anyone
> else trying to do the same thing. Here are diffs:
> 
> --- numeric_demo.pyx	2003-07-09 11:37:15.000000000 +0100
> +++ numarray_demo.pyx	2004-08-25 15:30:12.000000000 +0100
> @@ -1,33 +1,92 @@
>  #
>  #  This example demonstrates how to access the internals
> -#  of a Numeric array object.
> +#  of a Numarray object.
>  #
> 
> -cdef extern from "Numeric/arrayobject.h":
> +cdef extern from "numarray/libnumarray.h":
> +  ctypedef int maybelong
> 
> -  struct PyArray_Descr:
> -    int type_num, elsize
> -    char type
> +  cdef struct PyArray_Descr:
> +    int type_num # PyArray_TYPES
> +    int elsize   # bytes for 1 element
> +    char type    # One of "cb1silfdFD "  Object array not supported
> +    # function pointers omitted
> 
> -  ctypedef class Numeric.ArrayType [object PyArrayObject]:
> +  ctypedef class numarray._numarray._numarray [object PyArrayObject]:
>      cdef char *data
>      cdef int nd
> -    cdef int *dimensions, *strides
> +    cdef maybelong *dimensions
> +    cdef maybelong *strides
>      cdef object base
>      cdef PyArray_Descr *descr
>      cdef int flags
> +
> +    # numarray extras
> +    cdef maybelong *_dimensions
> +    cdef maybelong *_strides
> 
> -def print_2d_array(ArrayType a):
> -  print "Type:", chr(a.descr.type)
> -  if chr(a.descr.type) <> "f":
> +    cdef object _data         # object must meet buffer API
> +    cdef object _shadows      # ill-behaved original array.
> +    cdef int    nstrides      # elements in strides array
> +    cdef long   byteoffset    # offset into buffer where array data begins
> +    cdef long   bytestride    # basic seperation of elements in bytes
> +    cdef long   itemsize      # length of 1 element in bytes
> +
> +    cdef char   byteorder     # NUM_BIG_ENDIAN, NUM_LITTLE_ENDIAN
> +
> +    cdef char   _aligned      # test override flag
> +    cdef char   _contiguous   # test override flag
> +
> +  ctypedef enum:
> +    NUM_UNCONVERTED # 0
> +    NUM_CONTIGUOUS  # 1
> +    NUM_NOTSWAPPED  # 2
> +    NUM_ALIGNED     # 4
> +    NUM_WRITABLE    # 8
> +    NUM_COPY        # 16
> +    NUM_C_ARRAY     #  = (NUM_CONTIGUOUS | NUM_ALIGNED | NUM_NOTSWAPPED)
> +
> +  ctypedef enum NumarrayType:
> +    tAny
> +    tBool
> +    tInt8
> +    tUInt8
> +    tInt16
> +    tUInt16
> +    tInt32
> +    tUInt32
> +    tInt64
> +    tUInt64
> +    tFloat32
> +    tFloat64
> +    tComplex32
> +    tComplex64
> +    tObject                   # placeholder... does nothing
> +    tDefault = tFloat64
> +    tLong = tInt32,
> +    tMaxType
> +
> +  void import_libnumarray()
> +  _numarray NA_InputArray (object, NumarrayType, int)
> +  void *NA_OFFSETDATA(_numarray)
> +
> +import_libnumarray()
> +
> +def print_2d_array(_numarray _a):
> +  print "Type:", chr(_a.descr.type)
> +  if chr(_a.descr.type) <> "f":
>      raise TypeError("Float array required")
> -  if a.nd <> 2:
> +  if _a.nd <> 2:
>      raise ValueError("2 dimensional array required")
> +
> +  cdef _numarray a
> +  a = NA_InputArray(_a, tFloat32, NUM_C_ARRAY)
> +
>    cdef int nrows, ncols
>    cdef float *elems, x
>    nrows = a.dimensions[0]
>    ncols = a.dimensions[1]
> -  elems = <float *>a.data
> +  elems = <float *>NA_OFFSETDATA(a)
>    hyphen = "-"
>    divider = ("+" + 10 * hyphen) * ncols + "+"
>    print divider
> 
> --- run_numeric_demo.py	2003-07-09 11:37:15.000000000 +0100
> +++ run_numarray_demo.py	2004-08-25 12:06:47.000000000 +0100
> @@ -1,5 +1,5 @@
> -import Numeric
> -import numeric_demo
> +import numarray
> +import numarray_demo
> 
> -a = Numeric.array([[1.0, 3.5, 8.4], [2.3, 6.6, 4.1]], "f")
> -numeric_demo.print_2d_array(a)
> +a = numarray.array([[1.0, 3.5, 8.4], [2.3, 6.6, 4.1]], "f")
> +numarray_demo.print_2d_array(a)
> -- 
> Michael Hoffman <hoffman at ebi.ac.uk>
> European Bioinformatics Institute
> 
> P.S. Thanks for Pyrex! I just coded a Python implementation of a
> bioinformatics algorithm with the inner loop in Pyrex and it runs as
> fast as the pure C implementation we use in my group.
> 
> _______________________________________________
> Pyrex mailing list
> Pyrex at lists.copyleft.no
> http://lists.copyleft.no/mailman/listinfo/pyrex



On Fri, 20 Aug 2004 09:51:10 -0500
Bruce Southey <southey at uiuc.edu> wrote:

> Hi,
> I need to access a GLP'ed C library in numarray. The library functions are
> scalar ints and doubles as inputs and usually doubles as outputs. 
> 
> Is there a recommended (and simple) way to achieve this? There are a few
> different  C API's that exist from the Python to numarrays.
> 
> Would it be sufficient to first write C code that uses the library and use that
> code for a C API in Python or numarray?
> 
> Thanks,
> 
> Bruce Southey
> 
> 

-- 
Simon Burton, B.Sc.
Licensed PO Box 8066
ANU Canberra 2601
Australia
Ph. 61 02 6249 6940
http://arrowtheory.com 



From nadavh at visionsense.com  Thu Aug 26 05:28:31 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu Aug 26 05:28:31 2004
Subject: [Numpy-discussion] fromstring doesn't accept buffer object
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDCD@exchange2k.envision.co.il>

Numerc accepts a buffer object instead of a string:

  arr = Numeric.fromstring(buffer(.some_string), ..)
while numarray doesn't.

Is it intentional?

  Nadav.



From jmiller at stsci.edu  Thu Aug 26 05:53:28 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu Aug 26 05:53:28 2004
Subject: [Numpy-discussion] fromstring doesn't accept buffer object
In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DEDCD@exchange2k.envision.co.il>
References: 	 <07C6A61102C94148B8104D42DE95F7E86DEDCD@exchange2k.envision.co.il>
Message-ID: <1093524678.12979.1.camel@halloween.stsci.edu>

This is what I see this morning using CVS:

>>> from numarray import *
>>> s = "thisthis"
>>> fromstring(buffer(s), shape=(2,), type=Int32)
array([1936287860, 1936287860])

What kind of failure are you seeing?

Regards,
Todd

On Thu, 2004-08-26 at 09:28, Nadav Horesh wrote:
> Numerc accepts a buffer object instead of a string:
> 
>   arr = Numeric.fromstring(buffer(.some_string), ..)
> while numarray doesn't.
> 
> Is it intentional?
> 
>   Nadav.
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 




From nadavh at visionsense.com  Thu Aug 26 07:07:03 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu Aug 26 07:07:03 2004
Subject: [Numpy-discussion] fromstring doesn't accept buffer object
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDD0@exchange2k.envision.co.il>

Sorry for mixing test and production systems: There is no problem here with python 2.3, I got this failure with python 2.4a2:

==========================================
import numarray as n
import Numeric as N

s = '12341234'
b = buffer(s)

print 'numarray: fromstring (string ..) :'
print n.fromstring(s, type=n.Float64)

print 'Numeric: fromstring (string ..) :'
print N.fromstring(s, typecode=N.Float64)

print 'Numeric: fromstring (buffer ..) :'
print N.fromstring(b, typecode=N.Float64)

print 'numarray: fromstring (buffer ..) :'
print n.fromstring(b, type=n.Float64)

=== Results: ====


numarray: fromstring (string ..) :
[  3.05810932e-57]
Numeric: fromstring (string ..) :
[  3.05810932e-57]
Numeric: fromstring (buffer ..) :
[  3.05810932e-57]

=== This error happens only with Python 2.4.a2:

numarray: fromstring (buffer ..) :
Traceback (most recent call last):
  File "test.py", line 17, in ?
    print n.fromstring(b, type=n.Float64)
  File "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", line 378, in fromstring
    arr._data, 0, (type.bytes,), type.bytes)
libnumarray.error: copyNbytes: access beyond buffer. offset=7 buffersize=0


  Nadav.

-----Original Message-----
From:	Todd Miller [mailto:jmiller at stsci.edu]
Sent:	Thu 26-Aug-04 15:51
To:	Nadav Horesh
Cc:	numpy-discussion
Subject:	Re: [Numpy-discussion] fromstring doesn't accept buffer object
This is what I see this morning using CVS:

>>> from numarray import *
>>> s = "thisthis"
>>> fromstring(buffer(s), shape=(2,), type=Int32)
array([1936287860, 1936287860])

What kind of failure are you seeing?

Regards,
Todd

On Thu, 2004-08-26 at 09:28, Nadav Horesh wrote:
> Numerc accepts a buffer object instead of a string:
> 
>   arr = Numeric.fromstring(buffer(.some_string), ..)
> while numarray doesn't.
> 
> Is it intentional?
> 
>   Nadav.
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 







From exarkun at divmod.com  Thu Aug 26 19:20:05 2004
From: exarkun at divmod.com (Jp Calderone)
Date: Thu Aug 26 19:20:05 2004
Subject: [Numpy-discussion] Confusion regarding Numeric array resizing
Message-ID: <412E9A23.2000608@divmod.com>

   I'm confused by the restriction on resizing arrays demonstrated below:

 >>> from Numeric import array
 >>> a = array([0])
 >>> a.resize((2,))
 >>> b = a
 >>> a.resize((3,))
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
ValueError: cannot resize an array that has been referenced or is 
referencing another array in this way.  Use the resize function.
 >>> del b
 >>> a.resize((3,))
 >>> a
array([0, 0, 0])

   This seems like an unnecessary restriction.  Is there a reason I'm 
missing for it to be in place?

   Jp




From cookedm at physics.mcmaster.ca  Thu Aug 26 21:52:09 2004
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu Aug 26 21:52:09 2004
Subject: [Numpy-discussion] Confusion regarding Numeric array resizing
In-Reply-To: <412E9A23.2000608@divmod.com>
References: <412E9A23.2000608@divmod.com>
Message-ID: <20040827045208.GA11380@arbutus.physics.mcmaster.ca>

On Thu, Aug 26, 2004 at 10:19:15PM -0400, Jp Calderone wrote:
> 
>   I'm confused by the restriction on resizing arrays demonstrated below:
> 
> >>> from Numeric import array
> >>> a = array([0])
> >>> a.resize((2,))
> >>> b = a
> >>> a.resize((3,))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> ValueError: cannot resize an array that has been referenced or is 
> referencing another array in this way.  Use the resize function.
> >>> del b
> >>> a.resize((3,))
> >>> a
> array([0, 0, 0])
> 
>   This seems like an unnecessary restriction.  Is there a reason I'm 
> missing for it to be in place?

In this case it's obvious that b is not a copy of a (because that's how
Python assignment works), but if your statement was b = a[1:], it's a
little different.  Then, b is a view of a: a[1] = 1 will make b[0] == 1.
So, the question is, how should resizing a change b? The Numeric
developers decided to punt on this, I guess, and tell you to explictly
make a new copy.

Now, this restriction doesn't occur in numarray. Resizing an array will
(maybe) make a new copy of the underlying memory. Then b in this case
will no longer point to a's data, but will be the owner of the old data.

[I say "maybe" as it depends on where the memory for a came from.]

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From hsu at stsci.edu  Fri Aug 27 13:30:08 2004
From: hsu at stsci.edu (Jin-chung Hsu)
Date: Fri Aug 27 13:30:08 2004
Subject: [Numpy-discussion] marker edge color
Message-ID: <200408272029.ASF18358@donner.stsci.edu>

In 0.61.0, the marker edge color is still black, instead of being the same as 
the marker face color.  Personally, I think they should be the same.  What do 
people think?

One reason for the same color argument is that if the markersize is set to a 
small value, it will show mostly the edge color.

One possible reason against it is that if the marker color is white (or whatever 
the background color is), then you can't see the marker.  But we can default 
this case to black (or whatever).

JC Hsu




From hsu at stsci.edu  Fri Aug 27 13:43:14 2004
From: hsu at stsci.edu (Jin-chung Hsu)
Date: Fri Aug 27 13:43:14 2004
Subject: [Numpy-discussion] error bar default color
Message-ID: <200408272042.ASF18919@donner.stsci.edu>

In 0.61.0, when plotting a simple array with error bars, the default color of 
the error bars is black, instead of being the same as the line/markers color, 
e.g.:

>>> errorbar([1,2,3,4,5],[3,4,5,6,7],fmt='ro',yerr=[1,1,1,1,1])

I prefer them to be the same, especially since the default color for marker/line 
is blue and a beginner may be surprised to see the different color.

This may be related to my last posting regarding the marker edge color.

JC




From owen at astro.washington.edu  Fri Aug 27 17:38:31 2004
From: owen at astro.washington.edu (Russell E. Owen)
Date: Fri Aug 27 17:38:31 2004
Subject: [Numpy-discussion] rebin
In-Reply-To: <200408272042.ASF18919@donner.stsci.edu>
References: <200408272042.ASF18919@donner.stsci.edu>
Message-ID: <20040827203447.71bo3cw0c4oggcg8@webmail.spamcop.net>

Any suggestions on an efficient means to bin a 2-d array? REBIN is the IDL
function I'm trying to mimic. Binning allows one to combine sets of pixels from
one array to form a new array that is smaller by a given factor along each
dimension.

To nxm bin a 2-dimensional array, one averages (or sums or ?) each nxm block of
entries from the input image to form the corresponding entry of the output
image.

For example, to 2x2 bin a two-dimensional image, one would:
average (0,0), (0,1), (1,0), (1,1) to form (0,0)
average (0,2), (0,3), (1,2), (1,3) to form (0,1)
...

In case it helps, in my immediate case I'm binning a boolean array (a mask) and
thus can live with almost any kind of combination.

-- Russell



From nadavh at visionsense.com  Sat Aug 28 13:54:05 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Sat Aug 28 13:54:05 2004
Subject: [Numpy-discussion] rebin
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDD2@exchange2k.envision.co.il>

For the most general form of binning I use a convolution (by a 2D mask) followed by a subsmapling.

For example for a 3x3 binning:
mask = ones((3,3))
binned = convolve2d(data,mask,'same')[1::3,1::3]

  Nadav.
-----Original Message-----
From:	Russell E. Owen [mailto:owen at astro.washington.edu]
Sent:	Sat 28-Aug-04 03:34
To:	numpy-discussion at lists.sourceforge.net
Cc:	
Subject:	[Numpy-discussion] rebin
Any suggestions on an efficient means to bin a 2-d array? REBIN is the IDL
function I'm trying to mimic. Binning allows one to combine sets of pixels from
one array to form a new array that is smaller by a given factor along each
dimension.

To nxm bin a 2-dimensional array, one averages (or sums or ?) each nxm block of
entries from the input image to form the corresponding entry of the output
image.

For example, to 2x2 bin a two-dimensional image, one would:
average (0,0), (0,1), (1,0), (1,1) to form (0,0)
average (0,2), (0,3), (1,2), (1,3) to form (0,1)
...

In case it helps, in my immediate case I'm binning a boolean array (a mask) and
thus can live with almost any kind of combination.

-- Russell


-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion






From perry at stsci.edu  Mon Aug 30 06:15:08 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Mon Aug 30 06:15:08 2004
Subject: [Numpy-discussion] rebin
In-Reply-To: <20040827203447.71bo3cw0c4oggcg8@webmail.spamcop.net>
References: <200408272042.ASF18919@donner.stsci.edu> <20040827203447.71bo3cw0c4oggcg8@webmail.spamcop.net>
Message-ID: <7A026B97-FA86-11D8-B467-000A95B68E50@stsci.edu>

On Aug 27, 2004, at 8:34 PM, Russell E. Owen wrote:

> Any suggestions on an efficient means to bin a 2-d array? REBIN is the 
> IDL
> function I'm trying to mimic. Binning allows one to combine sets of 
> pixels from
> one array to form a new array that is smaller by a given factor along 
> each
> dimension.
>
> To nxm bin a 2-dimensional array, one averages (or sums or ?) each nxm 
> block of
> entries from the input image to form the corresponding entry of the 
> output
> image.
>
> For example, to 2x2 bin a two-dimensional image, one would:
> average (0,0), (0,1), (1,0), (1,1) to form (0,0)
> average (0,2), (0,3), (1,2), (1,3) to form (0,1)
> ...
>
> In case it helps, in my immediate case I'm binning a boolean array (a 
> mask) and
> thus can live with almost any kind of combination.
>
Note that a boxcar smoothing costs no more than doing the above 
averaging.
So in numarray, you could do the following:

from numarray.convolve import boxcar
b = boxcar(a, (n,n))
rebinnedarray = b[::n,::n].copy()

or something like this (I haven't tried to figure out the correct offset
for the slice) where one wants to rebin by a factor of n in both 
dimensions.
We should probably add a built in function to do this.

Perry




From rowen at u.washington.edu  Mon Aug 30 09:07:07 2004
From: rowen at u.washington.edu (Russell E Owen)
Date: Mon Aug 30 09:07:07 2004
Subject: [Numpy-discussion] rebin
In-Reply-To: <7A026B97-FA86-11D8-B467-000A95B68E50@stsci.edu>
References: <200408272042.ASF18919@donner.stsci.edu>
 <20040827203447.71bo3cw0c4oggcg8@webmail.spamcop.net>
 <7A026B97-FA86-11D8-B467-000A95B68E50@stsci.edu>
Message-ID: <p06110406bd58fe1b0893@[128.95.99.44]>

At 9:14 AM -0400 2004-08-30, Perry Greenfield wrote:
>On Aug 27, 2004, at 8:34 PM, Russell E. Owen wrote:
>
>>  Any suggestions on an efficient means to bin a 2-d array?
>>  ...
>>  For example, to 2x2 bin a two-dimensional image, one would:
>>  average (0,0), (0,1), (1,0), (1,1) to form (0,0)
>>  average (0,2), (0,3), (1,2), (1,3) to form (0,1)
>>  ...
>>
>Note that a boxcar smoothing costs no more than doing the above averaging.
>So in numarray, you could do the following:
>
>from numarray.convolve import boxcar
>b = boxcar(a, (n,n))
>rebinnedarray = b[::n,::n].copy()
>
>or something like this (I haven't tried to figure out the correct offset
>for the slice) where one wants to rebin by a factor of n in both dimensions.
>We should probably add a built in function to do this.

Thanks! Great suggestion!

I think the polished version (for nxm binning) is:

from numarray.convolve import boxcar
b = boxcar(a, (n,m), mode='constant', cval=0)
rebinnedarray = b[n//2::n,m//2::m].copy()

A rebin function would be handy since using boxcar is a bit tricky.

-- Russell



From rowen at u.washington.edu  Mon Aug 30 09:44:06 2004
From: rowen at u.washington.edu (Russell E Owen)
Date: Mon Aug 30 09:44:06 2004
Subject: [Numpy-discussion] rebin (corrected)
In-Reply-To: <p06110406bd58fe1b0893@[128.95.99.44]>
References: <200408272042.ASF18919@donner.stsci.edu>
 <20040827203447.71bo3cw0c4oggcg8@webmail.spamcop.net>
 <7A026B97-FA86-11D8-B467-000A95B68E50@stsci.edu>
 <p06110406bd58fe1b0893@[128.95.99.44]>
Message-ID: <p0611040abd59042171e6@[128.95.99.44]>

At 9:06 AM -0700 2004-08-30, Russell E Owen wrote:
>At 9:14 AM -0400 2004-08-30, Perry Greenfield wrote:
>>...
>>Note that a boxcar smoothing costs no more than doing the above averaging.
>>So in numarray, you could do the following:
>>
>>from numarray.convolve import boxcar
>>b = boxcar(a, (n,n))
>>rebinnedarray = b[::n,::n].copy()
>>
>>or something like this (I haven't tried to figure out the correct offset
>>for the slice) where one wants to rebin by a factor of n in both dimensions.
>>We should probably add a built in function to do this.
>...
>I think the polished version (for nxm binning) is:
>
>from numarray.convolve import boxcar
>b = boxcar(a, (n,m), mode='constant', cval=0)
>rebinnedarray = b[n//2::n,m//2::m].copy()
>
>A rebin function would be handy since using boxcar is a bit tricky.

I made several mistakes, one of them very serious: the convolve 
boxcar cannot do the job unless the array size is an exact multiple 
of the bin factor. The problem is that boxcar starts in the "wrong" 
place. Here's an example:

Problem: solve rebin [0, 1, 2, 3, 4] by 2 to yield: [0.5, 2.5, 4.0]
where the last point (value 4) is averaged with next-off-the-end, 
which we approximate by extending the data
(note: my propsed "polished" version messed that up; Perry had it right).

Using boxcar almost works:

>>>  import numarray as num
>>>  from numarray.convolve import boxcar
>>>  a = num.arange(5)
>>>  a
array([0, 1, 2, 3, 4])
>>>  boxcar(a, (2,))
array([ 0. ,  0.5,  1.5,  2.5,  3.5])
>>>  b = boxcar(a, (2,))
>>>  b
array([ 0. ,  0.5,  1.5,  2.5,  3.5])
>>>  b[1::2]
array([ 0.5,  2.5])

but oops: the last point is missing!!!

What is needed is some way to make the boxcar average start later, so 
it finishes by averaging 4 plus the next value off the edge of the 
array, e.g. a hypothetical version of boxcar with a start argument:

>>>  b2 = nonexistent_boxcar(a, (2,), start=1)
>>>  b2
[0.5, 1.5, 2.5, 3.5, 4.0]
b[0::2]
[0.5, 2.5, 4.0]

nd_image.boxcar_filter has an origin argument that *might* be for 
this purpose. Unfortunately, it is not documented and my attempt to 
use it as desired failed. I have no idea if this is a bug in nd_image 
or a misunderstanding on my part:
>>>  from numarray.nd_image import boxcar_filter
>>>  # first the usual answer; omitting the origin argument yields the 
>>>same answer
>>>  b = boxcar_filter(a, (2,), origin=(0,), output_type=num.Float32)
array([ 0. ,  0.5,  1.5,  2.5,  3.5], type=Float32)
>>>  # now try the origin argument and get a traceback; origin=1 gives 
>>>the same error:
>>>  b = boxcar_filter(a, (2,), origin=(1,), output_type=num.Float32)
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
   File 
"/usr/local/lib/python2.3/site-packages/numarray/nd_image/filters.py", 
line 339, in boxcar_filter
     output_type = output_type)
   File 
"/usr/local/lib/python2.3/site-packages/numarray/nd_image/filters.py", 
line 280, in boxcar_filter1d
     cval, origin, output_type)
RuntimeError: shift not within filter extent

So, yes, a rebin function that actually worked would be a real godsend!

Meanwhile, any other suggestions? Fortunately in our application we 
*can* call out to IDL, but it seems a shame to have to do that.

-- Russell



From rlw at stsci.edu  Mon Aug 30 10:16:10 2004
From: rlw at stsci.edu (Rick White)
Date: Mon Aug 30 10:16:10 2004
Subject: [Numpy-discussion] rebin (corrected)
In-Reply-To: <p0611040abd59042171e6@[128.95.99.44]>
Message-ID: <Pine.GSO.4.44.0408301300130.8067-100000@sundog.stsci.edu>

On Mon, 30 Aug 2004, Russell E Owen wrote:

> nd_image.boxcar_filter has an origin argument that *might* be for
> this purpose. Unfortunately, it is not documented and my attempt to
> use it as desired failed. I have no idea if this is a bug in nd_image
> or a misunderstanding on my part:
> >>>  from numarray.nd_image import boxcar_filter
> >>>  b = boxcar_filter(a, (2,), origin=(1,), output_type=num.Float32)
> Traceback (most recent call last):
>    File "<stdin>", line 1, in ?
>    File
> "/usr/local/lib/python2.3/site-packages/numarray/nd_image/filters.py",
> line 339, in boxcar_filter
>      output_type = output_type)
>    File
> "/usr/local/lib/python2.3/site-packages/numarray/nd_image/filters.py",
> line 280, in boxcar_filter1d
>      cval, origin, output_type)
> RuntimeError: shift not within filter extent

Seems like you got close to the answer.  This gives the answer you want:

>>> boxcar_filter(a, (2,), output_type=num.Float32,origin=-1)
array([ 0.5,  1.5,  2.5,  3.5,  4. ], type=Float32)

And so this works for rebin:

>>> boxcar_filter(a, (2,), output_type=num.Float32,origin=-1)[::2].copy()
array([ 0.5,  2.5,  4. ], type=Float32)

But I still agree with Perry that we ought to provide a built-in rebin
function.  It is particularly useful for large multi-dimensional arrays
where it is wasteful (in both CPU and memory) to create a full-size
copy of the array before resampling it down to the desired rebinned
size.  I appended the .copy() so that at least the big array is not
still hanging around in memory (remember that the slice creates a
view rather than a copy.)
				Rick




From tim.hochberg at cox.net  Mon Aug 30 10:58:05 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Mon Aug 30 10:58:05 2004
Subject: [Numpy-discussion] rebin (corrected)
In-Reply-To: <Pine.GSO.4.44.0408301300130.8067-100000@sundog.stsci.edu>
References: <Pine.GSO.4.44.0408301300130.8067-100000@sundog.stsci.edu>
Message-ID: <41336A6A.90208@cox.net>

[SNIP]

>
>But I still agree with Perry that we ought to provide a built-in rebin
>function.  It is particularly useful for large multi-dimensional arrays
>where it is wasteful (in both CPU and memory) to create a full-size
>copy of the array before resampling it down to the desired rebinned
>size.  I appended the .copy() so that at least the big array is not
>still hanging around in memory (remember that the slice creates a
>view rather than a copy.)
>				Rick
>  
>
A reasonable facsimile of this should  be doable without dropping into 
C. Something like:

def rebin_sum(a, (m, n)):
    M, N = a.shape
    a = na.reshape(a, (M/m,m,N/n,n))
    return na.sum(na.sum(a, 3), 1) / float(m*n)

This does create some temps, but they're smaller than in the boxcar case 
and it doesn't do all the extra calculation. This doesn't handle the 
case where a.shape isn't an exact multiple of (m,n). However, I don't 
think that would be all that hard to implement, if there is a consensus 
on what should happen then.
    I can think of at least two different ways this might be done: 
tacking on values that match the last value as already proposed and 
tacking on zeros. There may be others as well. It should probably get a 
boundary condition argument like convolve and friends.
    Personally, I'd be find rebin a little suprising if it resulted in 
an average, as all the implementations thus far have done, rather than a 
simple sum over the stencil.  When I think of rebinning I'm thinking of 
number of occurences per bin, and rebinning should keep the totals 
occurences the same, not change them by the inverse of the stencil size.

My 2.3 cents anyway

-tim

>
>
>-------------------------------------------------------
>This SF.Net email is sponsored by BEA Weblogic Workshop
>FREE Java Enterprise J2EE developer tools!
>Get your free copy of BEA WebLogic Workshop 8.1 today.
>http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
>_______________________________________________
>Numpy-discussion mailing list
>Numpy-discussion at lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>  
>





From rowen at u.washington.edu  Mon Aug 30 11:28:16 2004
From: rowen at u.washington.edu (Russell E Owen)
Date: Mon Aug 30 11:28:16 2004
Subject: [Numpy-discussion] rebin (corrected)
In-Reply-To: <41336A6A.90208@cox.net>
References: <Pine.GSO.4.44.0408301300130.8067-100000@sundog.stsci.edu>
 <41336A6A.90208@cox.net>
Message-ID: <p06110413bd591f28c7b0@[128.95.99.44]>

At 10:56 AM -0700 2004-08-30, Tim Hochberg wrote:
>[SNIP]
>
>>
>>But I still agree with Perry that we ought to provide a built-in rebin
>>function.  It is particularly useful for large multi-dimensional arrays
>>where it is wasteful (in both CPU and memory) to create a full-size
>>copy of the array before resampling it down to the desired rebinned
>>size.  I appended the .copy() so that at least the big array is not
>>still hanging around in memory (remember that the slice creates a
>>view rather than a copy.)
>>				Rick
>>
>A reasonable facsimile of this should  be doable without dropping 
>into C. Something like:
>
>def rebin_sum(a, (m, n)):
>    M, N = a.shape
>    a = na.reshape(a, (M/m,m,N/n,n))
>    return na.sum(na.sum(a, 3), 1) / float(m*n)
>
>This does create some temps, but they're smaller than in the boxcar 
>case and it doesn't do all the extra calculation. This doesn't 
>handle the case where a.shape isn't an exact multiple of (m,n). 
>However, I don't think that would be all that hard to implement, if 
>there is a consensus on what should happen then.
>    I can think of at least two different ways this might be done: 
>tacking on values that match the last value as already proposed and 
>tacking on zeros. There may be others as well. It should probably 
>get a boundary condition argument like convolve and friends.
>    Personally, I'd be find rebin a little suprising if it resulted 
>in an average, as all the implementations thus far have done, rather 
>than a simple sum over the stencil.  When I think of rebinning I'm 
>thinking of number of occurences per bin, and rebinning should keep 
>the totals occurences the same, not change them by the inverse of 
>the stencil size.
>
>My 2.3 cents anyway

I agree that it would be nice to avoid the extra calculation involved 
in convolution or boxcar averaging, and the extra temp storage.

Your algorithm certainly looks promising, but I'm not sure there's 
any space saving when the array shape is not an exact multiple of the 
bin factor. Duplicating the last value is probably the most 
reasonable alternative for my own applications (imaging). To use your 
algorithm, I guess one has to increase the array first, creating a 
new temporary array that is the same as the original except expanded 
to an even mutiple of the bin factor. In theory one could avoid 
duplication, but I suspect to do this efficiently one really needs to 
use C code.

I personally have no strong opinion on averaging vs summing. Summing 
retains precision but risks overflow. Averaging potentially has the 
opposite advantages, though avoiding overflow is tricky. Note that 
Nadav Horesh's suggested solution (convolution with a mask of 1s 
instead of boxcar averaging) computed the sum.

-- Russell



From tim.hochberg at cox.net  Mon Aug 30 12:06:07 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Mon Aug 30 12:06:07 2004
Subject: [Numpy-discussion] rebin (corrected)
In-Reply-To: <p06110413bd591f28c7b0@[128.95.99.44]>
References: <Pine.GSO.4.44.0408301300130.8067-100000@sundog.stsci.edu> <41336A6A.90208@cox.net> <p06110413bd591f28c7b0@[128.95.99.44]>
Message-ID: <41337A61.2090400@cox.net>

Russell E Owen wrote:

> At 10:56 AM -0700 2004-08-30, Tim Hochberg wrote:
>
>> [SNIP]
>>
>>>
>>> But I still agree with Perry that we ought to provide a built-in rebin
>>> function.  It is particularly useful for large multi-dimensional arrays
>>> where it is wasteful (in both CPU and memory) to create a full-size
>>> copy of the array before resampling it down to the desired rebinned
>>> size.  I appended the .copy() so that at least the big array is not
>>> still hanging around in memory (remember that the slice creates a
>>> view rather than a copy.)
>>>                 Rick
>>>
>> A reasonable facsimile of this should  be doable without dropping 
>> into C. Something like:
>>
>> def rebin_sum(a, (m, n)):
>>    M, N = a.shape
>>    a = na.reshape(a, (M/m,m,N/n,n))
>>    return na.sum(na.sum(a, 3), 1) / float(m*n)
>>
>> This does create some temps, but they're smaller than in the boxcar 
>> case and it doesn't do all the extra calculation. This doesn't handle 
>> the case where a.shape isn't an exact multiple of (m,n). However, I 
>> don't think that would be all that hard to implement, if there is a 
>> consensus on what should happen then.
>>    I can think of at least two different ways this might be done: 
>> tacking on values that match the last value as already proposed and 
>> tacking on zeros. There may be others as well. It should probably get 
>> a boundary condition argument like convolve and friends.
>>    Personally, I'd be find rebin a little suprising if it resulted in 
>> an average, as all the implementations thus far have done, rather 
>> than a simple sum over the stencil.  When I think of rebinning I'm 
>> thinking of number of occurences per bin, and rebinning should keep 
>> the totals occurences the same, not change them by the inverse of the 
>> stencil size.
>>
>> My 2.3 cents anyway
>
>
> I agree that it would be nice to avoid the extra calculation involved 
> in convolution or boxcar averaging, and the extra temp storage.
>
> Your algorithm certainly looks promising, but I'm not sure there's any 
> space saving when the array shape is not an exact multiple of the bin 
> factor. Duplicating the last value is probably the most reasonable 
> alternative for my own applications (imaging). To use your algorithm, 
> I guess one has to increase the array first, creating a new temporary 
> array that is the same as the original except expanded to an even 
> mutiple of the bin factor. In theory one could avoid duplication, but 
> I suspect to do this efficiently one really needs to use C code.

I think you could probably do considerably better than the boxcar code, 
but it it looks like it would get fairly messy once you start worrying 
about odd number of bins. It might end up being simpler to implement it 
C, so that's probably a better idea in the long run.

> I personally have no strong opinion on averaging vs summing. Summing 
> retains precision but risks overflow. Averaging potentially has the 
> opposite advantages, though avoiding overflow is tricky. Note that 
> Nadav Horesh's suggested solution (convolution with a mask of 1s 
> instead of boxcar averaging) computed the sum.

I really have no strong feelings since I have no use for rebinning at 
the moment. Back when I did, it would have been for rebinning data from 
particle detectors. So for instance, you would change the bin size so 
that you had enough data in each bin that you could attempt to do 
statistics on it or plot it or whatever. In that domain it would make no 
sense to average on rebinning. However, I can see how it makes sense for 
imaging applications.

In the absence of any compelling reason to do otherwise, I imagine the 
thing to do is copy what every one else is doing as long as they're 
consistent. Do you know what Matlab and friends do?

-tim




From curzio.basso at unibas.ch  Tue Aug 31 06:04:00 2004
From: curzio.basso at unibas.ch (Curzio Basso)
Date: Tue Aug 31 06:04:00 2004
Subject: [Numpy-discussion] extracting a random subset of a vector
Message-ID: <4134753A.7070503@unibas.ch>

Hi all, I have an optimization problem.

I currently use the following code to select a random subset of a rank-1 
array:

----------------------------------------------------------

import numarray as NA
import numarray.random_array as RA

N = 1000
M = 100
full = NA.arange(N)
subset = full[RA.permutation(N)][:M]

---------------------------------------------------------

However, it's quite slow (at least with N~40k), and from the hotshot 
output is looks like it's the indexing, not the permutation, which takes 
time.
Does anyone have a suggestion on a faster pure-python implementation?

thanks



From rays at san.rr.com  Tue Aug 31 06:51:04 2004
From: rays at san.rr.com (RJ)
Date: Tue Aug 31 06:51:04 2004
Subject: [Numpy-discussion] Help req: IDL --> Numeric/numarray
Message-ID: <5.2.1.1.2.20040831064859.1db44500@blue-cove.com>

Hi all,

The current IDL REBIN() thread reminded me to post this here...
I'm trying to port Hongjie Xie's IDL/ENVI Implementation of the FFT Based 
Algorithm for Automatic Image Registration
http://www.nmt.edu/%7Ehjxie/xie-paper.pdf
ftp://ftp.iamg.org/VOL29/v29-08-10.zip
to Python/numarray/Numeric and running into some subtle problems.
For instance, the IDL ROT() requires some additional steps to emulate, and 
the MAX(array, position) as well.
And, I'm not at all familiar with the correspondence of the FFT 
libraries... even the simple shift-only routine
http://rjs.org/astro/1004x/Python/register/shift_idl.py
produces different (incorrect) x shift values than the IDL.

My current code
http://rjs.org/astro/1004x/Python/register/
conforms to Xie's flow as much as possible for now, but should and will be 
re-factored when functional. His LogPolar function, to convert rectangular 
coordinate arrays to polar, is particularly slow.

I was recently informed that IDL has a usable demo version which I will try 
this week, but I would love to hear if anyone else has interest in this.

I think that the algorithm would make a nice standalone module, or perhaps 
something for nd_image, as it calculates shift, scale and rotation for 2d 
arrays.
Xie's main examples are registration of satellite Earth imagery, but it 
should work just as well looking up ;-)

Thank you,
Ray


Secret anti-spam filter-passing text. Include with reply:
qwertyuiop 




From rays at san.rr.com  Tue Aug 31 07:05:10 2004
From: rays at san.rr.com (RJ)
Date: Tue Aug 31 07:05:10 2004
Subject: [Numpy-discussion] Help req: IDL --> Numeric/numarray
Message-ID: <5.2.1.1.2.20040831070138.1dba8e30@pop-server.san.rr.com>

Hi all,

The current IDL REBIN() thread reminded me to post this here...

I'm trying to port Dr. Hongjie Xie's IDL/ENVI Implementation of the FFT 
Based Algorithm for Automatic Image Registration (it calculates shift, 
scale and rotation for 2d arrays)
http://www.nmt.edu/%7Ehjxie/xie-paper.pdf
ftp://ftp.iamg.org/VOL29/v29-08-10.zip
to Python/numarray/Numeric and running into some subtle problems.
For instance, the IDL ROT() requires some additional steps to emulate, and 
the MAX(array, position) as well.
And, I'm not at all familiar with the correspondence of the FFT 
libraries... even the simple shift-only routine I translated
http://rjs.org/astro/1004x/Python/register/shift_idl.py
produces different (incorrect) x shift values than the IDL.

My current code
http://rjs.org/astro/1004x/Python/register/
conforms to Xie's flow as much as possible for now, but should and will be 
re-factored when functional; his LogPolar function, to convert rectangular 
coordinate arrays to polar, is particularly slow.

I was recently informed that IDL has a usable demo version which I will try 
this week, but I would love to hear if anyone else has interest in this.

I think that the algorithm would make a nice standalone module, or perhaps 
something for nd_image.
Xie's main examples are registration of satellite Earth imagery, but it 
should work just as well looking up ;-)

Thank you,
Ray


Secret anti-spam filter-passing text. Include with reply:
qwertyuiop  




From Chris.Barker at noaa.gov  Tue Aug 31 10:41:11 2004
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Tue Aug 31 10:41:11 2004
Subject: [Numpy-discussion] extracting a random subset of a vector
In-Reply-To: <4134753A.7070503@unibas.ch>
References: <4134753A.7070503@unibas.ch>
Message-ID: <4134B6CE.4080807@noaa.gov>

Curzio Basso wrote:

> import numarray as NA
> import numarray.random_array as RA
> 
> N = 1000
> M = 100
> full = NA.arange(N)
> subset = full[RA.permutation(N)][:M]
> 
> ---------------------------------------------------------
> 
> However, it's quite slow (at least with N~40k), 

you can speed it up a tiny bit my subsetting the permutation array first:
subset = full[ RA.permutation(N)[:M] ]

> and from the hotshot 
> output is looks like it's the indexing, not the permutation, which takes 
> time.

not from my tests:

import numarray.random_array as RA
import numarray as NA
import time

N = 1000000
M = 100
full = NA.arange(N)

start = time.clock()
P = RA.permutation(N)
print "permutation took %F seconds"%(time.clock() - start)
start = time.clock()
subset = full[P[:M]]
print "subsetting took %F seconds"%(time.clock() - start)

which results in:
permutation took 1.640000 seconds
subsetting took 0.000000 seconds

so it's the permutation that takes the time, as I suspected. What would 
really speed this up is a random_array.non-repeat-randint() function, 
written in C. That way you wouldn't have to permute the entire N values, 
when you really just need M of them.

Does anyone else think this would be a useful function? I can't imagine 
it wouldn't be that hard to write.

If M <<< N, then you could probably write a little function in Python 
that called randint, and removed the repeats. If M is only a little 
smaller than N, this would be slow.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From stephen.walton at csun.edu  Tue Aug 31 11:49:09 2004
From: stephen.walton at csun.edu (Stephen Walton)
Date: Tue Aug 31 11:49:09 2004
Subject: [Numpy-discussion] rebin (corrected)
In-Reply-To: <41337A61.2090400@cox.net>
References: <Pine.GSO.4.44.0408301300130.8067-100000@sundog.stsci.edu>
	 <41336A6A.90208@cox.net> <p06110413bd591f28c7b0@[128.95.99.44]>
	 <41337A61.2090400@cox.net>
Message-ID: <1093978080.5706.24.camel@freyer.sfo.csun.edu>

On Mon, 2004-08-30 at 12:05, Tim Hochberg wrote:
> Russell E Owen wrote:
> 
> > I personally have no strong opinion on averaging vs summing
> 
> I really have no strong feelings since I have no use for rebinning at 
> the moment. Back when I did, it would have been for rebinning data from 
> particle detectors.

In the IRAF image tools, one gets summing by setting
"fluxconserve=yes".  The name of this option is nicely descriptive of
what an astronomer would mean by summing as opposed to averaging.

Many of the images I work with are ratio images;  for example, a solar
contrast map.  When I reshape a ratio image I want average, not sum. 
So, I would have to say that I need to have both average and sum
available with an option to switch between them.

By the way, has anyone else read

http://adsabs.harvard.edu/cgi-bin/nph-data_query?bibcode=2004SoPh..219....3D&db_key=AST&link_type=ABSTRACT&high=400734345713289

Craig has implemented the algorithm described therein in PDL (Perl Data
Language, http://pdl.perl.org), and a Python based implementation would
be awfully nice.

Hope to see a few of you in Pasadena.

-- 
Stephen Walton <stephen.walton at csun.edu>
Dept. of Physics & Astronomy, Cal State Northridge
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040831/ad6daed5/attachment.sig>

From rlw at stsci.edu  Tue Aug 31 12:49:06 2004
From: rlw at stsci.edu (Rick White)
Date: Tue Aug 31 12:49:06 2004
Subject: [Numpy-discussion] extracting a random subset of a vector
In-Reply-To: <4134753A.7070503@unibas.ch>
Message-ID: <Pine.GSO.4.44.0408311519050.19185-100000@sundog.stsci.edu>

On Tue, 31 Aug 2004, Curzio Basso wrote:

> Hi all, I have an optimization problem.
>
> I currently use the following code to select a random subset of a rank-1
> array:

Here's a slightly faster version.  It's about 3x faster than Chris Barker's
version (4x faster than your original version) for N=1000000, M=100:

import numarray as NA
import numarray.random_array as RA
from math import sqrt

N = 1000000
M = 100
full = NA.arange(N)

r = RA.random(N)
thresh = (M+3*sqrt(M))/N
subset = NA.compress(r<thresh, full)
while len(subset) < M:
    # rarely executed
    thresh = thresh+3*sqrt(M)/N
    subset = NA.compress(r<thresh, full)
subset = subset[RA.permutation(len(subset))[:M]]

By the way, I also find that most of the time gets spent in the
permutation computation.  That's why this is faster -- it gets do a
smaller permutation.
					Rick




From mklingberg at spray.se  Sun Aug  1 04:14:02 2004
From: mklingberg at spray.se (Mats Klingberg)
Date: Sun Aug  1 04:14:02 2004
Subject: [Numpy-discussion] Building Numeric with a native blas  ?
In-Reply-To: <410A7733.10408@noaa.gov>
References: <BD2ECF86.13DBC%perry@stsci.edu> <20040730025254.GA26933@arbutus.physics.mcmaster.ca> <410A7733.10408@noaa.gov>
Message-ID: <200408011313.06254.mklingberg@spray.se>

I recently did this under Gentoo. It seems that the atlas package doesn't 
build a complete lapack, which is necessary. Instead I emerged the newer 
blas-atlas and lapack-atlas packages. These build a complete lapack library 
with atlas optimizations. They are however masked with ~x86 so you have to 
add the following to your /etc/portage/package.keywords:

app-sci/lapack-atlas ~x86
app-sci/blas-atlas ~x86
app-sci/blas-config ~x86
app-sci/lapack-config ~x86

blas-config and lapack-config are packages that are needed by the other ones. 
They can be used to switch between reference lapack or atlas optimized 
libraries (if you install lapack-reference and blas-reference)

For more information see Gentoo bug #30453

Hope this helps,
Mats

On Friday 30 July 2004 18.28, Chris Barker wrote:
> David M. Cooke wrote:
> > Atlas might have installed a liblapack, with the (few) functions that it
> > overrides with faster ones. It's by no means a complete LAPACK
> > installation. Have a look at the difference in library sizes; a full
> > LAPACK is a few megs; Atlas's routines are a few hundred K.
>
> OK, I'm really confused now. I got it working, but it seems to have
> virtually identical performance to the Numeric-supplied lapack-lite.
>
> I'm guessing that the LAPACK package I emerged does NOT use the atlas BLAS.
>
> if the atlas liblapack doesn't have all of lapack, how in the world are
> you supposed to use it? I have no idea how I would get the linker to get
> what it can from the atlas lapack, and the rest from another one.
>
> Has anyone done this on Gentoo? If not how about another linux distro, I
> don't have to use portage for this after all.
>
> -Chris



From sdhyok at email.unc.edu  Sun Aug  1 21:22:13 2004
From: sdhyok at email.unc.edu (Shin)
Date: Sun Aug  1 21:22:13 2004
Subject: [Numpy-discussion] nan, inf?
Message-ID: <1091420510.2953.13.camel@dhcp8210.dhcp.unc.edu>

Following the document in ieeespacial.py,
I tried:

>>> from numarray import *
>>> nan
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
NameError: name 'nan' is not defined
>>> inf
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
NameError: name 'inf' is not defined

What did I miss? Thanks.

-- 
Daehyok Shin




From stephen.walton at csun.edu  Sun Aug  1 22:03:03 2004
From: stephen.walton at csun.edu (Stephen Walton)
Date: Sun Aug  1 22:03:03 2004
Subject: [Numpy-discussion] nan, inf?
In-Reply-To: <1091420510.2953.13.camel@dhcp8210.dhcp.unc.edu>
References: <1091420510.2953.13.camel@dhcp8210.dhcp.unc.edu>
Message-ID: <1091422776.15732.0.camel@localhost.localdomain>

On Sun, 2004-08-01 at 21:21, Shin wrote:

> Following the document in ieeespacial.py,
> I tried:
> 
> >>> from numarray import *
> >>> nan
> 

[swalton at localhost ~]$ python
Python 2.2.3 (#1, Jun 16 2003, 13:21:11)
[GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from numarray import *
>>> from numarray.ieeespecial import *
>>> nan
nan
>>> inf
inf

-- 
Stephen Walton <stephen.walton at csun.edu>
Dept. of Physics & Astronomy, CSU Northridge




From Fernando.Perez at colorado.edu  Mon Aug  2 11:50:05 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon Aug  2 11:50:05 2004
Subject: [Numpy-discussion] Numeric bug in array2str() for rank >1 arrays when using MA?
Message-ID: <410E8CB3.4060706@colorado.edu>

Hi all,

I think I've found a bug in Numeric's MA package.  The numeric docs say:

Finally, the last attribute, array_output, specifies whether to prepend the 
string "array(" and append either the string ")" or ", 'X')" where X is a 
typecode for non-default typecodes (in other words, the typecode will only be 
displayed if it is not that corresponding to Float, Complex or Int, which are 
the standard typecodes associated with floating point numbers, complex numbers 
and integers respectively). The array() is so that an eval of the returned 
string will return an array object (provided a comma separator is also used).

and they provide this example:

 >>> type(eval(array2string(arange(3), array_output=1, separator=',')))

<type 'array'>

This is all fine and dandy, but things seem to break down for any array of 
rank > 1 *** when MA has been imported only ***

In [234]: type(eval(array2string(RA.random((2,3)),separator=',',array_output=1)))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

/home/fperez/research/code/mwadap/<console>

/home/fperez/research/code/mwadap/<string>

TypeError: list indices must be integers


The problem is easily seen to be caused by MA:

In [1]: a=arange(4)

In [2]: a.shape= 2,2

In [3]: a
Out[3]:
array([[0, 1],
        [2, 3]])

In [4]: import MA

In [5]: a
Out[5]:
[[0,1,]
  [2,3,]]

Note the misplaced commas above, which make this an invalid nested list.

Once MA is imported it's impossible to get array2string() to return a valid 
representation of an array (something which survives an eval() call).  This is 
a problem for saving things to disk.

I'm using Numeric 23.1.

For now I'll work around this by avoiding MA, which is unfortunate because 
it's otherwise great for preventing huge machine-killing printouts of large 
arrays.

Best,

Fernando.



From sdhyok at email.unc.edu  Mon Aug  2 19:50:00 2004
From: sdhyok at email.unc.edu (Shin)
Date: Mon Aug  2 19:50:00 2004
Subject: [Numpy-discussion] nan, inf?
In-Reply-To: <1091422776.15732.0.camel@localhost.localdomain>
References: <1091420510.2953.13.camel@dhcp8210.dhcp.unc.edu>
	 <1091422776.15732.0.camel@localhost.localdomain>
Message-ID: <1091501313.2108.10.camel@localhost.localdomain>

Thanks. I thought importing all modules in numarray package also imports
nan and inf because the doc string of ieeespecial module has no mention
about "from numarray.ieeespecial import *"

>>> from numarray import *

>>> inf    # the repr() of inf may vary from platform to platform
inf
>>> nan    # the repr() of nan may vary from platform to platform
nan

Daehyok

On Mon, 2004-08-02 at 00:59, Stephen Walton wrote:
> On Sun, 2004-08-01 at 21:21, Shin wrote:
> 
> > Following the document in ieeespacial.py,
> > I tried:
> > 
> > >>> from numarray import *
> > >>> nan
> > 
> 
> [swalton at localhost ~]$ python
> Python 2.2.3 (#1, Jun 16 2003, 13:21:11)
> [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from numarray import *
> >>> from numarray.ieeespecial import *
> >>> nan
> nan
> >>> inf
> inf
-- 
Daehyok Shin (Peter)
Geography Department
Univ. of North Carolina-Chapel Hill




From sdhyok at email.unc.edu  Mon Aug  2 21:11:03 2004
From: sdhyok at email.unc.edu (Shin)
Date: Mon Aug  2 21:11:03 2004
Subject: [Numpy-discussion] segmentation fault
Message-ID: <1091506164.2108.25.camel@localhost.localdomain>

The followind code generates "Process Python segmentation fault".
Is it expected, or a bug?

from numarray import *
from numarray.ieeespecial import *

x = arange(10, typecode='d')
x[5] = nan
del x[getnan(x)]


-- 
Daehyok Shin




From perry at stsci.edu  Tue Aug  3 07:22:14 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Tue Aug  3 07:22:14 2004
Subject: [Numpy-discussion] segmentation fault
In-Reply-To: <1091506164.2108.25.camel@localhost.localdomain>
Message-ID: <BD3517BA.158C2%perry@stsci.edu>

On 8/3/04 12:09 AM, "Shin" <sdhyok at email.unc.edu> wrote:

> The followind code generates "Process Python segmentation fault".
> Is it expected, or a bug?
> 
I'd hazard that it is a bug since I'm not sure when a seg fault is really
desired, but maybe that's just me ;-)

> from numarray import *
> from numarray.ieeespecial import *
> 
> x = arange(10, typecode='d')
> x[5] = nan
> del x[getnan(x)]
> 
But note that this is an error of usage. You shouldn't be trying to delete a
part of an array. You can delete the array but here you appear to be trying
to delete a nan value. Arrays don't work like lists in this regard. What are
you trying to do?

Perry Greenfield




From sdhyok at catchlab.org  Tue Aug  3 08:39:14 2004
From: sdhyok at catchlab.org (Shin, Daehyok)
Date: Tue Aug  3 08:39:14 2004
Subject: [Numpy-discussion] segmentation fault
In-Reply-To: <BD3517BA.158C2%perry@stsci.edu>
Message-ID: <OAEOKPIGCLDDHAEMCAKIAEMLDBAA.sdhyok@catchlab.org>

I am just playing with ieeespecial.py  
to see what happens if I tried some intuitive things.
I know the command has an error.
But, I think "segmentation fault" is too harsh as a response to the error.
Am I right?

Daehyok Shin

> -----Original Message-----
> From: Perry Greenfield [mailto:perry at stsci.edu]
> Sent: Tuesday, August 03, 2004 AM 10:22
> To: Shin; numpy
> Subject: Re: [Numpy-discussion] segmentation fault
> 
> 
> On 8/3/04 12:09 AM, "Shin" <sdhyok at email.unc.edu> wrote:
> 
> > The followind code generates "Process Python segmentation fault".
> > Is it expected, or a bug?
> > 
> I'd hazard that it is a bug since I'm not sure when a seg fault is really
> desired, but maybe that's just me ;-)
> 
> > from numarray import *
> > from numarray.ieeespecial import *
> > 
> > x = arange(10, typecode='d')
> > x[5] = nan
> > del x[getnan(x)]
> > 
> But note that this is an error of usage. You shouldn't be trying 
> to delete a
> part of an array. You can delete the array but here you appear to 
> be trying
> to delete a nan value. Arrays don't work like lists in this 
> regard. What are
> you trying to do?
> 
> Perry Greenfield
> 
> 



From perry at stsci.edu  Tue Aug  3 08:42:37 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Tue Aug  3 08:42:37 2004
Subject: [Numpy-discussion] segmentation fault
In-Reply-To: <OAEOKPIGCLDDHAEMCAKIAEMLDBAA.sdhyok@catchlab.org>
Message-ID: <BD352A4D.158F1%perry@stsci.edu>

On 8/3/04 11:38 AM, "Shin, Daehyok" <sdhyok at catchlab.org> wrote:

> I am just playing with ieeespecial.py
> to see what happens if I tried some intuitive things.
> I know the command has an error.
> But, I think "segmentation fault" is too harsh as a response to the error.
> Am I right?
> 
Yes, of course. Seg faults are almost never the right response.

Perry Greenfield




From tim.hochberg at cox.net  Thu Aug  5 10:06:04 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Thu Aug  5 10:06:04 2004
Subject: [Numpy-discussion] argmax & py2exe
Message-ID: <411268CE.8050107@cox.net>

I just ran into a little snag using py2exe with CVS numarray: argmax 
breaks in py2exe packaged programs although it works fine when run 
normally. My believe is that py2exe is getting confused by the trick of 
using 'import numeric' inside the function argmax, which I believe is 
there to avoid circular imports. In any event, generic.argmax and 
numarraycore.NumArray.argmax mysteriously ending up calling each other 
recursively until the recursion limit is exceeded.

This is almost certainly a problem with py2exe, not with numarray, but 
since circular imports are sort of bad form and since numarray side of 
things is almost certainly easier to to fix, I though I'd try to 
persuade you, the numarray developers, to reorganize argmax so the 
circular imports go away.

I think that this is mostly a matter of moving the numeric.argmax/argmin 
code into NumArray.argmax/argmin and then numeric.argmax/argmin can just 
call a.argmax/a.argmin on the array a that it is passed./

-tim




From tim.hochberg at cox.net  Thu Aug  5 13:40:08 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Thu Aug  5 13:40:08 2004
Subject: [Numpy-discussion] argmax & py2exe
In-Reply-To: <411268CE.8050107@cox.net>
References: <411268CE.8050107@cox.net>
Message-ID: <41129AEB.3060507@cox.net>

Tim Hochberg wrote:

Please ignore this. The problem went away when I deleted numerics pyc 
files and started over. It's a mysterious phenomena, that I ran into 
once before with py2exe and stale pyc files. Very strange, but no time 
to dig into it now since things appear to be working again. Sorry for 
the confusion.

Although getting rid of those circular imports would still probably be 
good style.

-tim

>
> I just ran into a little snag using py2exe with CVS numarray: argmax 
> breaks in py2exe packaged programs although it works fine when run 
> normally. My believe is that py2exe is getting confused by the trick 
> of using 'import numeric' inside the function argmax, which I believe 
> is there to avoid circular imports. In any event, generic.argmax and 
> numarraycore.NumArray.argmax mysteriously ending up calling each other 
> recursively until the recursion limit is exceeded.
>
> This is almost certainly a problem with py2exe, not with numarray, but 
> since circular imports are sort of bad form and since numarray side of 
> things is almost certainly easier to to fix, I though I'd try to 
> persuade you, the numarray developers, to reorganize argmax so the 
> circular imports go away.
>
> I think that this is mostly a matter of moving the 
> numeric.argmax/argmin code into NumArray.argmax/argmin and then 
> numeric.argmax/argmin can just call a.argmax/a.argmin on the array a 
> that it is passed./
>
> -tim
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by OSTG. Have you noticed the changes on
> Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
> one more big change to announce. We are now OSTG- Open Source Technology
> Group. Come see the changes on the new OSTG site. www.ostg.com
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>





From jmiller at stsci.edu  Fri Aug  6 13:55:12 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Fri Aug  6 13:55:12 2004
Subject: [Numpy-discussion] argmax & py2exe
In-Reply-To: <41129AEB.3060507@cox.net>
References: <411268CE.8050107@cox.net>  <41129AEB.3060507@cox.net>
Message-ID: <1091825685.3761.8.camel@localhost.localdomain>

I was on leave this week so I'm getting to this later than usual,  but
Perry and I discussed it this morning.  We concluded that this is the
MODEL numarray problem report because:

1.  You reported the problem.

2.  You suggested a fix even I could implement in a half hour or less.

3.  You declared the problem a non-problem before I could waste a single
solitary second on it.  Well, except this one anyway.

Everyone take note because I think Tim is on to something.  We need more
bug reports exactly like this one...

Regards,
Todd



On Thu, 2004-08-05 at 16:39, Tim Hochberg wrote:
> Tim Hochberg wrote:
> 
> Please ignore this. The problem went away when I deleted numerics pyc 
> files and started over. It's a mysterious phenomena, that I ran into 
> once before with py2exe and stale pyc files. Very strange, but no time 
> to dig into it now since things appear to be working again. Sorry for 
> the confusion.
> 
> Although getting rid of those circular imports would still probably be 
> good style.
> 
> -tim
> 
> >
> > I just ran into a little snag using py2exe with CVS numarray: argmax 
> > breaks in py2exe packaged programs although it works fine when run 
> > normally. My believe is that py2exe is getting confused by the trick 
> > of using 'import numeric' inside the function argmax, which I believe 
> > is there to avoid circular imports. In any event, generic.argmax and 
> > numarraycore.NumArray.argmax mysteriously ending up calling each other 
> > recursively until the recursion limit is exceeded.
> >
> > This is almost certainly a problem with py2exe, not with numarray, but 
> > since circular imports are sort of bad form and since numarray side of 
> > things is almost certainly easier to to fix, I though I'd try to 
> > persuade you, the numarray developers, to reorganize argmax so the 
> > circular imports go away.
> >
> > I think that this is mostly a matter of moving the 
> > numeric.argmax/argmin code into NumArray.argmax/argmin and then 
> > numeric.argmax/argmin can just call a.argmax/a.argmin on the array a 
> > that it is passed./
> >
> > -tim
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by OSTG. Have you noticed the changes on
> > Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
> > one more big change to announce. We are now OSTG- Open Source Technology
> > Group. Come see the changes on the new OSTG site. www.ostg.com
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> >
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by OSTG. Have you noticed the changes on
> Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
> one more big change to announce. We are now OSTG- Open Source Technology
> Group. Come see the changes on the new OSTG site. www.ostg.com
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 




From rbastian at club-internet.fr  Sun Aug  8 12:58:12 2004
From: rbastian at club-internet.fr (=?iso-8859-15?q?Ren=E9=20Bastian?=)
Date: Sun Aug  8 12:58:12 2004
Subject: [Numpy-discussion] numarray, fromfile
Message-ID: <04080813422800.00759@rbastian>

Hello,

sometimes (not always) :

           w=NA.fromfile(self.f, NA.Float64, d)

delivers an error message :  

TypeError : NA_intSequenceProduct: object is not a sequence

Why not always ?

There is no  error if one writes 

           w=NA.fromfile(self.f, NA.Float64, (d,))



-- 
Ren? Bastian
http://www.musiques-rb.org : Musique en Python 



From jmiller at stsci.edu  Mon Aug  9 04:00:05 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Mon Aug  9 04:00:05 2004
Subject: [Numpy-discussion] numarray, fromfile
In-Reply-To: <04080813422800.00759@rbastian>
References: <04080813422800.00759@rbastian>
Message-ID: <1092049130.3748.3.camel@localhost.localdomain>

It's hard to tell without knowing the value of d,  but my guess is that
sometimes d is an integer and sometimes it's a float, and that perhaps
the "it works if (d,) is used" is a red herring.  Using a float for d
definitely produces the exception you saw.

Regards,
Todd

On Sun, 2004-08-08 at 07:57, Ren? Bastian wrote:
> Hello,
> 
> sometimes (not always) :
> 
>            w=NA.fromfile(self.f, NA.Float64, d)
> 
> delivers an error message :  
> 
> TypeError : NA_intSequenceProduct: object is not a sequence
> 
> Why not always ?
> 
> There is no  error if one writes 
> 
>            w=NA.fromfile(self.f, NA.Float64, (d,))
-- 




From eqdaqvdkyk at yahoo.com  Mon Aug  9 04:24:05 2004
From: eqdaqvdkyk at yahoo.com (Karl Donaldson)
Date: Mon Aug  9 04:24:05 2004
Subject: [Numpy-discussion] Fw: lozenge
Message-ID: <MPSPRMXSXZGPYBIHAIVLOD@yahoo.com>

I got this forwarded to me from a friend of mine, thought you might want to see it.  http://www.suncoastrewards.com/ref44.html
You can download dvd movies, full cd albums and stuff.  Even console video games.  I had found at least 74 CD's on there that I wanted.
Wait untill you see the selection of music and dvd's - it's pretty awsome.
Theres a few pages in there on how to get your selection on CD if you need the help.
I saw a few areas of the movie section, and found titles in there still in theaters - it's nutty.
Call me later if you get the chance.

Gregory

----327293974229964655----




From strawman at astraw.com  Tue Aug 10 23:27:00 2004
From: strawman at astraw.com (Andrew Straw)
Date: Tue Aug 10 23:27:00 2004
Subject: [Numpy-discussion] floating point exception weirdness
Message-ID: <4119BBFC.6020304@astraw.com>

Hi All,

I've isolated numarray.ieeespecial as the cause of a strange error which 
I posted on c.l.py:
http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&threadm=41198B29.7090600%40NOSPAM-DELETE-THIS.astraw.com&prev=/groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26group%3Dcomp.lang.python

My non-expert guess as to an explanation: some statements early in 
numarray.ieeespecial (e.g. _na.array(1.0)/_na.array(0.0)) result in a 
floating point exception being set which lies dormant until I call 
certain functions in the Intel IPP library (e.g. 
ippiAddWeighted_8u32f_C1IR), at which point the latent exception bit my 
Python program is killed while the console says "Floating point exception".

I've come to this tentative conclusion based on 3 minimal Pyrex programs:

---------------------------------------------------- Version 1:
import numarray.numarrayall as _na
plus_inf = inf = _na.array(1.0)/_na.array(0.0)

cimport avg
avg.work() # calls IPP, terminates with "Floating point exception".
---------------------------------------------------- Version 2:
import numarray.numarrayall as _na

# Define *ieee special values*
_na.Error.pushMode(all="ignore")
plus_inf = inf = _na.array(1.0)/_na.array(0.0)
_na.Error.popMode()

cimport avg
avg.work() # calls IPP, terminates with "Floating point exception".
---------------------------------------------------- Version 3:
import numarray.numarrayall as _na
cimport avg
avg.work() # calls IPP, terminates normally

In the short term, I'll try and work around this problem by not 
importing numarray.ieeespecial, but in the long term, I think we should 
try and fix this.  I'm happy to try to debug further if someone gives me 
advice on where to go from here.

Cheers!
Andrew





From jmiller at stsci.edu  Wed Aug 11 03:50:24 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Aug 11 03:50:24 2004
Subject: [Numpy-discussion] floating point exception weirdness
In-Reply-To: <4119BBFC.6020304@astraw.com>
References: <4119BBFC.6020304@astraw.com>
Message-ID: <1092221365.3752.32.camel@localhost.localdomain>

Hi Andrew,

So far you mentioned debian, linux-2.6.7, Python-2.3.4, and the Intel
IPP library.   What compiler are you using?

More comments below.

On Wed, 2004-08-11 at 02:26, Andrew Straw wrote:
> Hi All,
> 
> I've isolated numarray.ieeespecial as the cause of a strange error which 
> I posted on c.l.py:
> http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&threadm=41198B29.7090600%40NOSPAM-DELETE-THIS.astraw.com&prev=/groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26group%3Dcomp.lang.python
> 
> My non-expert guess as to an explanation: some statements early in 
> numarray.ieeespecial (e.g. _na.array(1.0)/_na.array(0.0)) result in a 
> floating point exception being set which lies dormant until I call 
> certain functions in the Intel IPP library (e.g. 
> ippiAddWeighted_8u32f_C1IR), at which point the latent exception bit my 
> Python program is killed while the console says "Floating point exception".

It sounds to me like the IPP library is aborting as a result of
previously irrelevant floating point error state.  My guess is that the
problem is not confined to ieeespecial,  but rather to floating point
exception handling in general in combination with the IPP library and/or
whatever compiler you're using.  

The key numarray code for clearing the floating point error state is
NA_checkFPErrors() in numarray/Src/newarray.ch.

Some things that occur to me are:

1. Verify that any array divide by zero results in the same failure.  
>>> x = arange(10,20)/zeros((10,)) 
>>> call_your_ipp_code()

2. Look in the IPP library for routines related to clearing floating
point exceptions.  Look in your compiler runtime library for routines
related to clearing floating point exceptions.  Modify NA_checkFPErrors
accordingly and re-build numarray with 

% python setup.py install --genapi

4. Walk through the Pyrex generated wrapper code and make sure there's
nothing going on there.  I'm doubtful that this has anything to do with
the problem.

More ideas will come as you let us know what your compiler is and verify
item 1.

Regards,
Todd




From nadavh at visionsense.com  Wed Aug 11 04:49:02 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Wed Aug 11 04:49:02 2004
Subject: [Numpy-discussion] Bug in built in SVD?
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDA1@exchange2k.envision.co.il>

When numarray is compiled with the buit-in linear-algebra routine the svd routine runs into an infinite loop.

Platform2: python 2.3.4 and 2.4a2; numarray 1.1; gcc 3.4.1; RH9 and gentoo 1.4

example:

>> from numarray.linear_algebra.mlab import *
>>> svd(arange(20.0, shape = (5,4)))

... infine loop ...


--------------------------------------------------------------------------

svd runs OK when compiled againts external LAPACK.

BTW: Why numarray used dgesdd routine for SVD, and not the svd routine that has the ATLAS equivalent?

  Nadav.




From strawman at astraw.com  Wed Aug 11 04:55:13 2004
From: strawman at astraw.com (Andrew Straw)
Date: Wed Aug 11 04:55:13 2004
Subject: [Numpy-discussion] floating point exception weirdness
In-Reply-To: <1092221365.3752.32.camel@localhost.localdomain>
References: <4119BBFC.6020304@astraw.com> <1092221365.3752.32.camel@localhost.localdomain>
Message-ID: <411A08FA.7000601@astraw.com>

Dear Todd,

I'll respond to your other comments when I get a chance to investigate 
further.

Todd Miller wrote:

>So far you mentioned debian, linux-2.6.7, Python-2.3.4, and the Intel
>IPP library.   What compiler are you using?
>  
>
astraw at flygate:~$ gcc -v
Reading specs from /usr/lib/gcc-lib/i486-linux/3.3.4/specs
Configured with: ../src/configure -v 
--enable-languages=c,c++,java,f77,pascal,objc,ada,treelang --prefix=/usr 
--mandir=/usr/share/man --infodir=/usr/share/info 
--with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared 
--with-system-zlib --enable-nls --without-included-gettext 
--enable-__cxa_atexit --enable-clocale=gnu --enable-debug 
--enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc i486-linux
Thread model: posix
gcc version 3.3.4 (Debian)

Geez, I hope that's enough info to tell you gcc 3.3.4. :)

(Also, I could mention this is with numarray-1.0 -- tested both with 
whatever Debian gives me in addition to a source build of the 1.0 tarball.)

Cheers!
Andrew



From curzio.basso at unibas.ch  Wed Aug 11 05:54:08 2004
From: curzio.basso at unibas.ch (Curzio Basso)
Date: Wed Aug 11 05:54:08 2004
Subject: [Numpy-discussion] masked arrays and nd_image
Message-ID: <411A16BD.6050902@unibas.ch>

Hi all.

I just tried to use a masked array with nd_image, namely to pass it to 
the convolve1d() function, and I got:

error: getShape: couldn't get sequence length.

Does someone know if it possible at all to use nd_image on masked 
arrays? Or if there is a possibility to do operations on masked images 
without using the ma module?

thanks. curzio



From verveer at embl-heidelberg.de  Wed Aug 11 06:05:02 2004
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Wed Aug 11 06:05:02 2004
Subject: [Numpy-discussion] masked arrays and nd_image
In-Reply-To: <411A16BD.6050902@unibas.ch>
References: <411A16BD.6050902@unibas.ch>
Message-ID: <EE004E79-EB96-11D8-BB04-000A95C92C8E@embl-heidelberg.de>

I did not write nd_image with masked arrays in mind, I imagine it won't 
work for most functions. I am not planning to support masked arrays

. If you just want to mask out some of the results, apply the mask 
afterwards. If you actually want to deal with masked values in the 
convolution itself, you are out of luck  with the convolve1d function, 
it does not support that. In that case you could try to write your own, 
using the generic filter functions (see the  manual).

Cheers, Peter

On 11 Aug 2004, at 14:53, Curzio Basso wrote:

>
> Hi all.
>
> I just tried to use a masked array with nd_image, namely to pass it to 
> the convolve1d() function, and I got:
>
> error: getShape: couldn't get sequence length.
>
> Does someone know if it possible at all to use nd_image on masked 
> arrays? Or if there is a possibility to do operations on masked images 
> without using the ma module?
>
> thanks. curzio
>
>
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion




From curzio.basso at unibas.ch  Wed Aug 11 06:17:10 2004
From: curzio.basso at unibas.ch (Curzio Basso)
Date: Wed Aug 11 06:17:10 2004
Subject: [Numpy-discussion] masked arrays and nd_image
In-Reply-To: <EE004E79-EB96-11D8-BB04-000A95C92C8E@embl-heidelberg.de>
References: <411A16BD.6050902@unibas.ch> <EE004E79-EB96-11D8-BB04-000A95C92C8E@embl-heidelberg.de>
Message-ID: <411A1C2D.9040303@unibas.ch>

Peter Verveer wrote:

 > I did not write nd_image with masked arrays in mind, I imagine it won't
 > work for most functions. I am not planning to support masked arrays

Yes, I expected this. I knew I was pretending too much ;o)

 > . If you just want to mask out some of the results, apply the mask
 > afterwards. If you actually want to deal with masked values in the
 > convolution itself, you are out of luck  with the convolve1d function,
 > it does not support that. In that case you could try to write your own,
 > using the generic filter functions (see the  manual).

Ok, this would imply writing a function with prototype FilterFunction1D, 
right? And the mask argument should be packed with the data or is there 
a way (which I don't see) to pass arguments to the filter other than the 
input image. Sorry if I'm annoying ;o)

thanks, curzio



From jmiller at stsci.edu  Wed Aug 11 06:25:11 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Aug 11 06:25:11 2004
Subject: [Numpy-discussion] masked arrays and nd_image
In-Reply-To: <411A16BD.6050902@unibas.ch>
References: <411A16BD.6050902@unibas.ch>
Message-ID: <1092230676.2633.15.camel@halloween.stsci.edu>

On Wed, 2004-08-11 at 08:53, Curzio Basso wrote:
> Hi all.
> 
> I just tried to use a masked array with nd_image, namely to pass it to 
> the convolve1d() function, and I got:
> 
> error: getShape: couldn't get sequence length.
> 
> Does someone know if it possible at all to use nd_image on masked 
> arrays? 

I don't believe it is possible to use masked arrays directly because
they are not NumArrays or NumArray subclasses.   Masked arrays only
contain NumArrays which store the data and mask.

> Or if there is a possibility to do operations on masked images 
> without using the ma module?

I think the key here may be the "filled()" method which lets you convert
a masked array into a NumArray with the masked values filled into some
fill value, say 0.  I'm not sure what the post convolution mask value
should be.

Regards,
Todd




From verveer at embl-heidelberg.de  Wed Aug 11 06:27:05 2004
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Wed Aug 11 06:27:05 2004
Subject: [Numpy-discussion] masked arrays and nd_image
In-Reply-To: <411A1C2D.9040303@unibas.ch>
References: <411A16BD.6050902@unibas.ch> <EE004E79-EB96-11D8-BB04-000A95C92C8E@embl-heidelberg.de> <411A1C2D.9040303@unibas.ch>
Message-ID: <0A575C55-EB9A-11D8-BB04-000A95C92C8E@embl-heidelberg.de>

> > I did not write nd_image with masked arrays in mind, I imagine it 
> won't
> > work for most functions. I am not planning to support masked arrays
>
> Yes, I expected this. I knew I was pretending too much ;o)

One can always try :-)

> > . If you just want to mask out some of the results, apply the mask
> > afterwards. If you actually want to deal with masked values in the
> > convolution itself, you are out of luck  with the convolve1d 
> function,
> > it does not support that. In that case you could try to write your 
> own,
> > using the generic filter functions (see the  manual).
>
> Ok, this would imply writing a function with prototype 
> FilterFunction1D, right? And the mask argument should be packed with 
> the data or is there a way (which I don't see) to pass arguments to 
> the filter other than the input image. Sorry if I'm annoying ;o)

If you do it in python, you feed generic_filter1d a function, if you 
make that a method of an object, your mask can be accessed as a 
attribute of your object.

If you want to do it in C, you can pass a CObject that holds the 
function pointer to a callback function with prototype 
FilterFunction1D. Additionally you can set the description pointer of 
the same CObject to an arbitrary pointer, which is passed at every call 
to the callback function. You can use that to store additional 
information for you filter, such as the mask. See the example in the 
manual which is written for the shift function, but the principle is 
the same.

Cheers, Peter




From Chris.Barker at noaa.gov  Wed Aug 11 11:34:00 2004
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Wed Aug 11 11:34:00 2004
Subject: [Numpy-discussion] Pure Python NumPy-like module?
In-Reply-To: <200408011313.06254.mklingberg@spray.se>
References: <BD2ECF86.13DBC%perry@stsci.edu> <20040730025254.GA26933@arbutus.physics.mcmaster.ca> <410A7733.10408@noaa.gov> <200408011313.06254.mklingberg@spray.se>
Message-ID: <411A60E8.8060600@noaa.gov>

Hi folks,

I've got a project that uses Numeric arrays a fair bit. Unfortunately, I 
have a fair number of users that would like to be able to use it without 
Numeric (frankly I don't get it. It's a wxPython widget, and once you've 
installed wxPython, Numeric should not be a show stopper, but whatever). 
I though one way to accommodate this was to create a pure python 
Numeric-lite package, that contained the functionality need for my app, 
and was written in pure python. A side benefit of this is that it would 
be an easy way to see how much a performance gain Numeric gives me. 
Frankly, I don't really care that much, I like using Numeric operations 
for the syntactical advantages, aside from performance issues.

Anyway, I recall a while back a couple folks did some testing with Psyco 
and a pure-python implementation of a small fraction of Numeric 
functionality. I'd love to see that code if it's still around. Or if any 
of you has done something similar, I'd love to see that too.

-thanks, Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From tim.hochberg at cox.net  Wed Aug 11 11:52:02 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Wed Aug 11 11:52:02 2004
Subject: [Numpy-discussion] Pure Python NumPy-like module?
In-Reply-To: <411A60E8.8060600@noaa.gov>
References: <BD2ECF86.13DBC%perry@stsci.edu> <20040730025254.GA26933@arbutus.physics.mcmaster.ca> <410A7733.10408@noaa.gov> <200408011313.06254.mklingberg@spray.se> <411A60E8.8060600@noaa.gov>
Message-ID: <411A6AA2.3090600@cox.net>

Chris Barker wrote:

> Hi folks,
>
> I've got a project that uses Numeric arrays a fair bit. Unfortunately, 
> I have a fair number of users that would like to be able to use it 
> without Numeric (frankly I don't get it. It's a wxPython widget, and 
> once you've installed wxPython, Numeric should not be a show stopper, 
> but whatever). I though one way to accommodate this was to create a 
> pure python Numeric-lite package, that contained the functionality 
> need for my app, and was written in pure python. A side benefit of 
> this is that it would be an easy way to see how much a performance 
> gain Numeric gives me. Frankly, I don't really care that much, I like 
> using Numeric operations for the syntactical advantages, aside from 
> performance issues.
>
> Anyway, I recall a while back a couple folks did some testing with 
> Psyco and a pure-python implementation of a small fraction of Numeric 
> functionality. I'd love to see that code if it's still around. Or if 
> any of you has done something similar, I'd love to see that too.


Hi Chris,

That was probably me. Your message prompted me to take a look at that 
code for the first time in a long time and it is indeed a *small* 
fraction of Numeric functionality.  That being said your welcome to try 
and do something useful with it. I'll go ahead and send you the file in 
private email to avoid cluttering the list.

If anyone else wants a copy let me know.

-tim

>
> -thanks, Chris
>
>
>
>
>





From stephen.walton at csun.edu  Wed Aug 11 12:02:08 2004
From: stephen.walton at csun.edu (Stephen Walton)
Date: Wed Aug 11 12:02:08 2004
Subject: [Numpy-discussion] masked arrays and nd_image
In-Reply-To: <1092230676.2633.15.camel@halloween.stsci.edu>
References: <411A16BD.6050902@unibas.ch>
	 <1092230676.2633.15.camel@halloween.stsci.edu>
Message-ID: <1092250881.6980.32.camel@freyer.sfo.csun.edu>

On Wed, 2004-08-11 at 06:24, Todd Miller wrote:

> I think the key here may be the "filled()" method which lets you convert
> a masked array into a NumArray with the masked values filled into some
> fill value, say 0.  I'm not sure what the post convolution mask value
> should be.

I hope I'm not jumping in where I don't belong, but here at SFO/CSUN
we've had quite a bit of experience with convolutions and correlations
of time series (serieses?) with missing data points.

I'm sure you all know this, but:  For the scaling to be correct, you
have to not only mask out the values you don't want, but normalize the
sum to reflect the fact that different numbers of values will appear in
the sum.  Our MATLAB code to convolve x and y, with bad points marked by
NaNs, is:

for i = 1 : xlen-ylen+1
   j(i)=i;
   x1=x(i:i+ylen-1);
   a=x1.*y;
   b=a(find(~isnan(a)));
   if isempty(b)
      z(i)= NaN;
   else
       z(i)=sum(b)/length(b)
   end
end

I'd be happy to know how to code up the equivalent in numarray.  In the
above, note that x1 is x padded on both ends with ylen-1 NaNs.

Unfortunately, and again I'm sure everyone knows this, you can't use
FFTs to speed up convolutions/correlations if you have missing data
points, so you have to use discrete techniques.  The numerical analysis
literature refers to this problem as Fourier analysis of unequally
spaced data.  The only publications and algorithms I could find went the
wrong way:  given an unequally spaced set of points in Fourier space,
find the most likely reconstruction in real space.

-- 
Stephen Walton <stephen.walton at csun.edu>
Dept. of Physics & Astronomy, Cal State Northridge
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040811/69e91579/attachment-0001.sig>

From rlw at stsci.edu  Wed Aug 11 12:43:01 2004
From: rlw at stsci.edu (Rick White)
Date: Wed Aug 11 12:43:01 2004
Subject: [Numpy-discussion] masked arrays and nd_image
In-Reply-To: <1092250881.6980.32.camel@freyer.sfo.csun.edu>
Message-ID: <Pine.GSO.4.44.0408111522290.21604-100000@sundog.stsci.edu>

On Wed, 11 Aug 2004, Stephen Walton wrote:

> I hope I'm not jumping in where I don't belong, but here at SFO/CSUN
> we've had quite a bit of experience with convolutions and correlations
> of time series (serieses?) with missing data points.
>
> I'm sure you all know this, but:  For the scaling to be correct, you
> have to not only mask out the values you don't want, but normalize the
> sum to reflect the fact that different numbers of values will appear in
> the sum.  Our MATLAB code to convolve x and y, with bad points marked by
> NaNs, is:
>
> for i = 1 : xlen-ylen+1
>    j(i)=i;
>    x1=x(i:i+ylen-1);
>    a=x1.*y;
>    b=a(find(~isnan(a)));
>    if isempty(b)
>       z(i)= NaN;
>    else
>        z(i)=sum(b)/length(b)
>    end
> end
>
> I'd be happy to know how to code up the equivalent in numarray.  In the
> above, note that x1 is x padded on both ends with ylen-1 NaNs.

If you have a mask array with ones marking the good points and zeros the
points to ignore, here is a simpler version that does pretty much the same
calculation:

import numarray as na
kernel = na.ones(ylen)
z = nd.convolve1d(x*mask,kernel,mode='constant') / \
    nd.convolve1d(mask,kernel,mode='constant')

This would need some elaboration to work with your data -- e.g., NaN*0 is
not zero so you don't want the x array to have NaN values in it.  And it
may need an extra test to handle the case where a region larger than ylen
is masked out (this version would put in NaN values, which might be OK).
But the basic idea is the same.

I imagine this is possible in Matlab (which I don't use) -- I use it all
the time in IDL to do convolutions in the presence of missing data.

> Unfortunately, and again I'm sure everyone knows this, you can't use
> FFTs to speed up convolutions/correlations if you have missing data
> points, so you have to use discrete techniques.  The numerical analysis
> literature refers to this problem as Fourier analysis of unequally
> spaced data.  The only publications and algorithms I could find went the
> wrong way:  given an unequally spaced set of points in Fourier space,
> find the most likely reconstruction in real space.

This can in fact be done using FFTs.  You have to be careful about how the
boundary conditions are handled, adding padding cells to avoid wraparound
(that's what the mode='constant' is about).  The approach is very similar
though.  This works in 2-D as well and is vastly faster than doing a
direct convolution.

Incidentally, if you have truly unequally spaced data points (as opposed
to missing points out of an equally spaced series), this particular trick
doesn't work but there is another approach to doing DFTs in FFT time.
It's a bit lengthy to explain (and off the path for this group), but radio
astronomers figured out how to do it a long time ago.  It is probably the
algorithm you mention using unequally-spaced points in Fourier space; the
same algorithm works fine if the points are in real space since there is
hardly any difference between a forward and reverse Fourier transform.
					Rick




From geoffp at rochester.rr.com  Wed Aug 11 20:08:09 2004
From: geoffp at rochester.rr.com (Geoffrey Philbrick)
Date: Wed Aug 11 20:08:09 2004
Subject: [Numpy-discussion] matrixmultiply problem
Message-ID: <000801c48019$2c830be0$6601a8c0@Laptop>

Executing the following code has the side effect of modifying one of the arguments (matrices) I passed to matrixmultiply. Note that the matrix m2 changes after the call to matrixmultiply. This is on Windows with numarray 1.0. 

Why does this happen?

Code:

from numarray import array, matrixmultiply

m1 = array(
[[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]]
)

m2 = array(
[[ 0.500011, 0.86601905, 0. ],
[-0.86601905, 0.500011, 0. ],
[ 0., 0., 1. ]]
)

print "m1 before\n", m1
print "m2 before\n", m2
matrixmultiply(m1, m2)
print "m1 after\n", m1
print "m2 after\n", m2

Output:

m1 before
[[ 1. 0. 0.]
[ 0. 1. 0.]
[ 0. 0. 1.]]
m2 before
[[ 0.500011 0.86601905 0. ]
[-0.86601905 0.500011 0. ]
[ 0. 0. 1. ]]
m1 after
[[ 1. 0. 0.]
[ 0. 1. 0.]
[ 0. 0. 1.]]
m2 after
[[ 0.500011 -0.86601905 0. ]
[ 0.86601905 0.500011 0. ]
[ 0. 0. 1. ]]



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040811/ecea08a4/attachment-0001.html>

From cookedm at physics.mcmaster.ca  Wed Aug 11 21:28:10 2004
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Wed Aug 11 21:28:10 2004
Subject: [Numpy-discussion] matrixmultiply problem
In-Reply-To: <000801c48019$2c830be0$6601a8c0@Laptop>
References: <000801c48019$2c830be0$6601a8c0@Laptop>
Message-ID: <20040812042736.GA14498@arbutus.physics.mcmaster.ca>

On Wed, Aug 11, 2004 at 11:05:07PM -0400, Geoffrey Philbrick wrote:
> Executing the following code has the side effect of modifying one of
> the arguments (matrices) I passed to matrixmultiply. Note that the
> matrix m2 changes after the call to matrixmultiply. This is on Windows
> with numarray 1.0. 

Wow, that's bad. Since I had nothing better to do [1], I tracked it down
and submitted a patch (#1007730) to SourceForge to fix this.

[1] actually, I do, but...

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From baud.ph at ifrance.com  Thu Aug 12 00:02:07 2004
From: baud.ph at ifrance.com (baud.ph at ifrance.com)
Date: Thu Aug 12 00:02:07 2004
Subject: [Numpy-discussion] Repondeur de <baud.ph@ifrance.com>
Message-ID: <0408120648.2c114d@b0508.idoo.com>

Cette adresse mail n'est plus utilis?e.


ifrance.com, l'email gratuit le plus complet de l'Internet!
En vacances ou absent du bureau... vous souhaitez en informer vos
correspondants? Facile avec l'email i france et son r?pondeur!
>>>>>> http://www.ifrance.com/_reloc/rep.email




From nadavh at visionsense.com  Thu Aug 12 00:15:07 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu Aug 12 00:15:07 2004
Subject: [Numpy-discussion] Bug in built in SVD?
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDAB@exchange2k.envision.co.il>

Sorry it is a gcc3.4.1 optimization bug. Either lowering to optimization to -O1 or stepping back to gcc3.3.4 (with all optimization guns on) fix it.

   Nadav.

-----Original Message-----
From:	Nadav Horesh
Sent:	Wed 11-Aug-04 15:48
To:	numpy-discussion
Cc:	
Subject:	[Numpy-discussion] Bug in built in SVD?
When numarray is compiled with the buit-in linear-algebra routine the svd routine runs into an infinite loop.

Platform2: python 2.3.4 and 2.4a2; numarray 1.1; gcc 3.4.1; RH9 and gentoo 1.4

example:

>> from numarray.linear_algebra.mlab import *
>>> svd(arange(20.0, shape = (5,4)))

... infine loop ...


--------------------------------------------------------------------------

svd runs OK when compiled againts external LAPACK.

BTW: Why numarray used dgesdd routine for SVD, and not the svd routine that has the ATLAS equivalent?

  Nadav.



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion






From jmiller at stsci.edu  Thu Aug 12 09:49:06 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu Aug 12 09:49:06 2004
Subject: [Numpy-discussion] matrixmultiply problem
In-Reply-To: <20040812042736.GA14498@arbutus.physics.mcmaster.ca>
References: <000801c48019$2c830be0$6601a8c0@Laptop>
	 <20040812042736.GA14498@arbutus.physics.mcmaster.ca>
Message-ID: <1092329006.6703.5.camel@halloween.stsci.edu>

On Thu, 2004-08-12 at 00:27, David M. Cooke wrote:
> On Wed, Aug 11, 2004 at 11:05:07PM -0400, Geoffrey Philbrick wrote:
> > Executing the following code has the side effect of modifying one of
> > the arguments (matrices) I passed to matrixmultiply. Note that the
> > matrix m2 changes after the call to matrixmultiply. This is on Windows
> > with numarray 1.0. 
> 
> Wow, that's bad. Since I had nothing better to do [1], I tracked it down
> and submitted a patch (#1007730) to SourceForge to fix this.
> 
> [1] actually, I do, but...

This bug has been reported so many times... and I thought it was already
fixed.   The patch is in CVS now.  Thanks for pitching in here guys!

Regards,
Todd




From nadavh at visionsense.com  Thu Aug 12 10:30:01 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu Aug 12 10:30:01 2004
Subject: [Numpy-discussion] Bug in built in SVD?
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDB0@exchange2k.envision.co.il>

It can be a bit more complicate, the story is this:
One my gentoo system I use ATLAS, and gentoo instaltion generates the ATLAS version of blas and lapack which omit some of the original blas/lapack routines. Till now I used the numarray/Numeric internal lapack_lite package, since every time I tried to compile against the external lapck, I could not load the linear-algebra package because I got an error complaining about a missing "dgesdd_".

 In order to solve it I downloaded the original lapack library from netlib and compiled it with gcc3.4.1 and full optimization (by tweaking the makefile). I copied the generated lapack.a to /usr/local/lob/lapack_original.a and updated the link list in addon.py/setup.py (numarray/Numeric). That worked fine --- I have numarray/Numeric linked against highly optimized lapack without this infinite loop problem.

That leads me to the other issue: Assuming ATLAS routines runs faster then their equivalent lapack's. Why not replace dgesdd with the one which has an ATLAS equivalent?

Nadav

-----Original Message-----
From:	Michiel Jan Laurens de Hoon [mailto:mdehoon at ims.u-tokyo.ac.jp]
Sent:	Thu 12-Aug-04 10:29
To:	Nadav Horesh
Cc:	
Subject:	Re: [Numpy-discussion] Bug in built in SVD?
This may also be related to patch 732520 for Numerical Python. Some parts of 
lapack need to be compiled without optimization. Numerical Python (and I assume 
numarray) use optimization anyway, and usually gets away with it. So gcc 3.4.1 
may be doing the right thing after all. The external Lapack that you mentioned 
may have been compiled correctly, i.e. without optimization for the specific 
Lapack files.

--Michiel, U Tokyo.

Nadav Horesh wrote:
> Sorry it is a gcc3.4.1 optimization bug. Either lowering to optimization to -O1 or stepping back to gcc3.3.4 (with all optimization guns on) fix it.
> 
>    Nadav.
> 
> -----Original Message-----
> From:	Nadav Horesh
> Sent:	Wed 11-Aug-04 15:48
> To:	numpy-discussion
> Cc:	
> Subject:	[Numpy-discussion] Bug in built in SVD?
> When numarray is compiled with the buit-in linear-algebra routine the svd routine runs into an infinite loop.
> 
> Platform2: python 2.3.4 and 2.4a2; numarray 1.1; gcc 3.4.1; RH9 and gentoo 1.4
> 
> example:
> 
> 
>>>from numarray.linear_algebra.mlab import *
>>>
>>>>svd(arange(20.0, shape = (5,4)))
> 
> 
> ... infine loop ...
> 
> 
> --------------------------------------------------------------------------
> 
> svd runs OK when compiled againts external LAPACK.
> 
> BTW: Why numarray used dgesdd routine for SVD, and not the svd routine that has the ATLAS equivalent?
> 
>   Nadav.
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 
> 
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 
> 

-- 
Michiel de Hoon, Assistant Professor
University of Tokyo, Institute of Medical Science
Human Genome Center
4-6-1 Shirokane-dai, Minato-ku
Tokyo 108-8639
Japan
http://bonsai.ims.u-tokyo.ac.jp/~mdehoon







From Chris.Barker at noaa.gov  Thu Aug 12 11:49:01 2004
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Thu Aug 12 11:49:01 2004
Subject: [Numpy-discussion] Bug in built in SVD?
In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DEDB0@exchange2k.envision.co.il>
References: <07C6A61102C94148B8104D42DE95F7E86DEDB0@exchange2k.envision.co.il>
Message-ID: <411BB198.9040902@noaa.gov>

Nadav Horesh wrote:
> It can be a bit more complicate, the story is this:
> One my gentoo system I use ATLAS, and gentoo instaltion generates the ATLAS version of blas and lapack which omit some of the original blas/lapack routines. Till now I used the numarray/Numeric internal lapack_lite package, since every time I tried to compile against the external lapck, I could not load the linear-algebra package because I got an error complaining about a missing "dgesdd_".

I had the same issue, and got this reply from this very list:

Mats Klingberg wrote:
> I recently did this under Gentoo. It seems that the atlas package doesn't 
> build a complete lapack, which is necessary. Instead I emerged the newer 
> blas-atlas and lapack-atlas packages. These build a complete lapack library 
> with atlas optimizations. They are however masked with ~x86 so you have to 
> add the following to your /etc/portage/package.keywords:
> 
> app-sci/lapack-atlas ~x86
> app-sci/blas-atlas ~x86
> app-sci/blas-config ~x86
> app-sci/lapack-config ~x86
> 
> blas-config and lapack-config are packages that are needed by the other ones. 
> They can be used to switch between reference lapack or atlas optimized 
> libraries (if you install lapack-reference and blas-reference)
> 
> For more information see Gentoo bug #30453

I haven't had a chance to try it yet, but I will soon.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From aisaac at american.edu  Thu Aug 12 13:04:43 2004
From: aisaac at american.edu (Alan G Isaac)
Date: Thu Aug 12 13:04:43 2004
Subject: [Numpy-discussion] Re: matrixmultiply problem
Message-ID: <Mahogany-0.66.0-1792-20040812-160544.00@american.edu>

> The patch is in CVS now.

Can the conditions under which this problem occurs
be simply stated?

Thank you,
Alan Isaac





From cookedm at physics.mcmaster.ca  Thu Aug 12 13:32:03 2004
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu Aug 12 13:32:03 2004
Subject: [Numpy-discussion] Re: matrixmultiply problem
In-Reply-To: <Mahogany-0.66.0-1792-20040812-160544.00@american.edu>
References: <Mahogany-0.66.0-1792-20040812-160544.00@american.edu>
Message-ID: <20040812203120.GA8465@arbutus.physics.mcmaster.ca>

On Thu, Aug 12, 2004 at 04:05:41PM -0400, Alan G Isaac wrote:
> > The patch is in CVS now.
> 
> Can the conditions under which this problem occurs
> be simply stated?
> 
> Thank you,
> Alan Isaac

a = some matrix
b = some other matrix
transpose(b) is not contiguous

The numarray 1.0 (and earlier behaviour) is this:
dot(a,b) transposes b internally, finds it's not contiguous, and
makes a contiguous copy. Unfortunately, when it finishes, it
untransposes the *copy* of b back, not b itself.

So after dot(a, b), b is now the transpose of the original matrix

A quick fix is to define your own dot:
def dot(a, b):
    return numarray.dot(a, b.view())

The original b is then not touched, and the screwed-up matrix is
discarded. Note that transpose happens by fiddling with the layout of
the dimensions and strides in the array object; the data itself is not
touched.

[This all applies to matrixmultiply also, as matrixmultiply == dot]

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From a_wilson at mit.edu  Thu Aug 12 13:55:12 2004
From: a_wilson at mit.edu (Andrew Wilson)
Date: Thu Aug 12 13:55:12 2004
Subject: [Numpy-discussion] clear a variable
Message-ID: <411BD90F.7050308@mit.edu>

Hello,
    I can't seem to find this in the documentation, is there a way to 
clear a variable.  For example I have an application that opens displays 
a bunch of images, they can be big memory hogs. When I'm not showing 
them I'd like to clear out those variables, instead of having to close 
down my program and start over.  I know matlab has a command "clear 
/var/" which will do exactly that.
    Thanks for the help!
       Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040812/ba85ea02/attachment-0001.html>

From rowen at u.washington.edu  Thu Aug 12 14:19:09 2004
From: rowen at u.washington.edu (Russell E Owen)
Date: Thu Aug 12 14:19:09 2004
Subject: [Numpy-discussion] clear a variable
In-Reply-To: <411BD90F.7050308@mit.edu>
References: <411BD90F.7050308@mit.edu>
Message-ID: <p06110401bd418e25c146@[128.95.99.44]>

At 4:54 PM -0400 2004-08-12, Andrew Wilson wrote:
Hello,
     I can't seem to find this in the documentation, is there a way to 
clear a variable.  For example I have an application that opens 
displays a bunch of images, they can be big memory hogs. When I'm not 
showing them I'd like to clear out those variables, instead of having 
to close down my program and start over.  I know matlab has a command 
"clear var" which will do exactly that.

del(obj)

If you are the only user of "obj", this'll do the trick. (del tries 
to be polite and not pull the rug out from under other users).

obj can be nearly anything, such as an array or any other object you 
created. del can also be used to delete specific elements of lists 
and dictionaries, i.e.:
del(adict[akey])
del(alist[ind1:ind2])

I hope this help.

-- Russell



From curzio.basso at unibas.ch  Fri Aug 13 05:42:04 2004
From: curzio.basso at unibas.ch (Curzio Basso)
Date: Fri Aug 13 05:42:04 2004
Subject: [Numpy-discussion] invalid numeric result(s) in divide
Message-ID: <411CB6D9.1010906@unibas.ch>

Hi all.

I have a matrix tmp a matrix w, both Float32, with

 >>> tmp.shape
(3,512,512)
 >>> w.shape
(512,512)
 >>> w.max()
1.0
 >>> w.min()
0.0
 >>> NA.where(NA.and_(w > 0.0, w < 1.0e-2))
(array([]), array([]))

What is for me surprising is the following:

 >>> tmp = NA.where(w > 0.0, tmp / w, 0.0)
Warning: Encountered invalid numeric result(s)  in divide

Ah, both matrices have no 'nan' or 'inf' elements. So, where could the 
invalid result come from?

thanks, curzio



From focke at slac.stanford.edu  Fri Aug 13 08:40:07 2004
From: focke at slac.stanford.edu (Warren Focke)
Date: Fri Aug 13 08:40:07 2004
Subject: [Numpy-discussion] invalid numeric result(s) in divide
In-Reply-To: <411CB6D9.1010906@unibas.ch>
References: <411CB6D9.1010906@unibas.ch>
Message-ID: <Pine.LNX.4.58.0408130814250.26978@auriga.slac.stanford.edu>


On Fri, 13 Aug 2004, Curzio Basso wrote:

>  >>> tmp = NA.where(w > 0.0, tmp / w, 0.0)
> Warning: Encountered invalid numeric result(s)  in divide
>
> Ah, both matrices have no 'nan' or 'inf' elements. So, where could the
> invalid result come from?

All of the arguments to NA.where are evaluated before the function is
called, so tmp/w results in divide-by-zero.  The array that gets passed to
NA.where contains some inf or nan (I suspect the latter based on the
warning message), but you don't see them in the return value because
NA.where replaces them w/ 0.

Warren Focke




From jimcser at pacifier.com  Sat Aug 14 14:05:03 2004
From: jimcser at pacifier.com (Jim Cser)
Date: Sat Aug 14 14:05:03 2004
Subject: [Numpy-discussion] Using sum() within a function
Message-ID: <411E7E49.2020709@pacifier.com>

Hello-
I have a function to generate a multi-dimensional array, which then
gets summed over one axis.  The problem is that the dimensions
are large, and I run out of memory when I create the entire array,
so I'm trying to do the sum *within* the function.

Example-- variables x,y,z,t; dimensions numX, numY, numZ, numT;
functions f1(x,y,z,t), f2(y,z,t);  want to calculate f1*f2 and
sum over t to get out[x,y,z].

With loops, I could do it like--
out = zeros((numX,numY,numZ))
for x in range(numX):
   for y in range(numY):
     for z in range(numZ):
        for t in range(numT):
           tempval = f1(x,y,z,t) * f2(y,z,t)
           out[x,y,z] = out[x,y,z] + tempval

With numarray, if I had enough memory, I could just do--
temp1 = fromfunction(f1,(numX,numY,numZ,numT))
temp2 = resize(fromfunction(f2,(numY,numZ,numT)),(numX,numY,numZ,numT))
out = sum(temp1 * temp2, axis = 3)

Instead, I'm trying to do something like--
def f3(x,y,z):
   for t in range(numT):
     tempval = f1(x,y,z,t) * f2(y,z,t)

   outval = sum(tempval,axis = 3)
   return outval

out = fromfunction(f3,(numX,numY,numZ))

I've been trying various slicing and indexing, but I can't seem to
get that *extra* dimension within the 3-D function. I've scoured the
documentation and list archives, but haven't found exactly what I need.
Any suggestions?  Am I stuck with generating the entire 4-D array?

Thanks in advance,
Jim Cser




From perry at stsci.edu  Sat Aug 14 14:44:32 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Sat Aug 14 14:44:32 2004
Subject: [Numpy-discussion] Using sum() within a function
In-Reply-To: <411E7E49.2020709@pacifier.com>
Message-ID: <NEBBIJKBMLDBLNCEEFOCCEEMFHAA.perry@stsci.edu>

Jim Cser wrote:
> 
> Instead, I'm trying to do something like--
> def f3(x,y,z):
>    for t in range(numT):
>      tempval = f1(x,y,z,t) * f2(y,z,t)
> 
>    outval = sum(tempval,axis = 3)
>    return outval
>
Perhaps I'm confused but it seems to me the for loop is wrong.

Don't you want something like:

def f3(x,y,z):
   tempval = 0.*x
   for t in range(numT):
     tempval += f1(x,y,z,t) * f2(y,z,t)
   return tempval

Instead?

Not that I've tried this, but so long as the function will
work given 3-d arrays for x,y,z, it should. (though the dependence
on numT as a global is a bit worrisome)

Perry GReenfield




From jimcser at pacifier.com  Mon Aug 16 09:59:07 2004
From: jimcser at pacifier.com (Jim Cser)
Date: Mon Aug 16 09:59:07 2004
Subject: [Numpy-discussion] Using sum() within a function
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCCEEMFHAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCCEEMFHAA.perry@stsci.edu>
Message-ID: <4120E7B9.5070001@pacifier.com>

Perry Greenfield wrote:

> Jim Cser wrote:
> 
>>Instead, I'm trying to do something like--
>>def f3(x,y,z):
>>   for t in range(numT):
>>     tempval = f1(x,y,z,t) * f2(y,z,t)
>>
>>   outval = sum(tempval,axis = 3)
>>   return outval
>>
> 
> Perhaps I'm confused but it seems to me the for loop is wrong.
> 
> Don't you want something like:
> 
> def f3(x,y,z):
>    tempval = 0.*x
>    for t in range(numT):
>      tempval += f1(x,y,z,t) * f2(y,z,t)
>    return tempval
> 
> Instead?
> 
> Not that I've tried this, but so long as the function will
> work given 3-d arrays for x,y,z, it should. (though the dependence
> on numT as a global is a bit worrisome)
> 
> Perry GReenfield


The loop was indeed wrong, and "tempval = 0.*x" was the trick I needed, 
thanks.  Unfortunately, iterating over t is still extremely slow. 
Ideally, there would be a way to use fromfunction() with slices as 
arguments.

-Jim



From rob at hooft.net  Mon Aug 16 10:06:05 2004
From: rob at hooft.net (Rob Hooft)
Date: Mon Aug 16 10:06:05 2004
Subject: [Numpy-discussion] Is there a better way to do this?
In-Reply-To: <16A7C641-DB7D-11D8-A37A-000393479EE8@earthlink.net>
References: <16A7C641-DB7D-11D8-A37A-000393479EE8@earthlink.net>
Message-ID: <4120E925.70501@hooft.net>

Hee-Seng Kye wrote:
> My question is not directly related to NumPy, but since many people here 
> deal with numbers, I was wondering if I could get some help; it would be 
> even better if there is a NumPy (or Numarray) function that takes care 
> of what I want!
> 
> I'm trying to write a program that computes six-digit numbers, in which 
> the left digit is always smaller than its following digit (i.e., it's 
> always ascending).  The best I could do was to have many embedded 'for' 
> statement:
> 
> c = 1
> for p0 in range(0, 7):
>   for p1 in range(1, 12):
>     for p2 in range(2, 12):
>       for p3 in range(3, 12):
>         for p4 in range(4, 12):
>           for p5 in range(5, 12):
>             if p0 < p1 < p2 < p3 < p4 < p5:
>               print repr(c).rjust(3), "\t",
>               print "%X %X %X %X %X %X" % (p0, p1, p2, p3, p4, p5)
>               c += 1
> print "...Done"
> 
> This works, except that it's very slow.  I need to get it up to 
> nine-digit numbers, in which case it's significantly slow.  I was 
> wondering if there is a more efficient way to do this.
> 
> I would highly appreciate it if anyone could help.

Sorry for taking a month. My solution is not faster than any of the ones 
that have already been proposed, but it is more elegant because it does 
not require the "n" nested for loops.

Another hint is to use a generator for this purpose.

Advantage is that the base and the number of digits can both be 
externally defined!

Regards,

Rob

# As a reference, the general case where digits are not created in
# order
def gen(ndigits, base):
   dig = [0]*ndigits
   yield dig
   while 1:
     for num in range(ndigits-1,-1,-1):
       if dig[num] < base - 1:
         dig[num] += 1
         break
       else:
         dig[num] = 0
     else:
       return
     yield dig

def genordered(ndigits, base):
   if ndigits > base:
     return
   dig = range(ndigits)
   yield dig
   while 1:
     for num in range(ndigits-1,-1,-1):
       if dig[num] < base-ndigits+num:
         dig[num] += 1
         for num2 in range(num+1,ndigits):
           dig[num2] = dig[num2-1] + 1
         break
     else:
       return
     yield dig

c = 1
for dig in genordered(ndigits = 6, base = 12):
   # This is the same as your print statement, but again independent of
   # the number of digits
   print "%3d\t"%c,' '.join(map(lambda x: "%X"%x, dig))
   c += 1
print "...Done"



From perry at stsci.edu  Mon Aug 16 10:10:05 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Mon Aug 16 10:10:05 2004
Subject: [Numpy-discussion] Using sum() within a function
In-Reply-To: <4120E7B9.5070001@pacifier.com>
Message-ID: <CEELJPECNGEGKFNDLHFFGEHGCEAA.perry@stsci.edu>

Jim Cser wrote:
> 
> The loop was indeed wrong, and "tempval = 0.*x" was the trick I needed, 
> thanks.  Unfortunately, iterating over t is still extremely slow. 
> Ideally, there would be a way to use fromfunction() with slices as 
> arguments.
If the other 3 dimensions are small (e.g., each time sample 
involves an array whose size is less than 1000 elements total), 
then the slowness is due to the small arrays. In that event
you may want iterate over a different dimension than time, or
construct the for loop to iterate over ranges of times rather
than one at a time. 

Perry



From eric at enthought.com  Mon Aug 16 10:42:12 2004
From: eric at enthought.com (eric jones)
Date: Mon Aug 16 10:42:12 2004
Subject: [Numpy-discussion] ANN: Python/Scientific Job openings at Enthought
Message-ID: <4120F1F3.4020006@enthought.com>

Hey Folks,

Enthought continues to grow, and we're hiring again.  There are several 
positions open, and most of them involve python+scientific computing.  A 
couple of the positions also involve support of SciPy.  I'm very 
interested in finding a person for the "scientist" job posting that is 
both able to work well with our customers and also has the personality 
and interest to work with the SciPy community to help push the library 
forward.

More information is available on our web site.

    http://www.enthought.com/careers.htm

Our current development efforts are in the areas of electromagnetics, 
geophysics, and graphics (2D and 3D). 

If you have any questions, please feel free to contact me.  Please 
forward the link to anyone you feel might be interested.

thanks,
eric




From jimcser at pacifier.com  Mon Aug 16 13:29:03 2004
From: jimcser at pacifier.com (jimcser at pacifier.com)
Date: Mon Aug 16 13:29:03 2004
Subject: [Numpy-discussion] Using sum() within a function
In-Reply-To: <CEELJPECNGEGKFNDLHFFGEHGCEAA.perry@stsci.edu>
References: <4120E7B9.5070001@pacifier.com>
    <CEELJPECNGEGKFNDLHFFGEHGCEAA.perry@stsci.edu>
Message-ID: <16683.66.192.179.130.1092688071.squirrel@webmail.pacifier.com>

>
> Jim Cser wrote:
>>
>> The loop was indeed wrong, and "tempval = 0.*x" was the trick I needed,
>> thanks.  Unfortunately, iterating over t is still extremely slow.
>> Ideally, there would be a way to use fromfunction() with slices as
>> arguments.
> If the other 3 dimensions are small (e.g., each time sample
> involves an array whose size is less than 1000 elements total),
> then the slowness is due to the small arrays. In that event
> you may want iterate over a different dimension than time, or
> construct the for loop to iterate over ranges of times rather
> than one at a time.
>
> Perry

I was just using T as a sample name, so I am not actually doing
a time series of anything.  My dimesions are on the order of
numT = 500, numX = 500, numY = 100, numZ = 10, so smallness is not
the problem.  Unfortunately, I don't need to sum over Z, the short
dimension.  If my system didn't hang, I would have done it the
easier way in the first place.

Perhaps I need to learn more about memory management-- I'm sure
that many Python users are crunching bigger even arrays than this.
For what it's worth, I'm using Python 2.3 / Numarray 1.0 with
PythonWin on an XP box, 512M RAM.

-Jim




From skip at pobox.com  Tue Aug 17 12:07:01 2004
From: skip at pobox.com (Skip Montanaro)
Date: Tue Aug 17 12:07:01 2004
Subject: [Numpy-discussion] Re: core dump during import
Message-ID: <16674.17638.251233.868612@montanaro.dyndns.org>

A few days ago on scipy-user and earlier today on numpy-discussion I wrote:

    If I execute

        from scipy.stats import distribution

    at the interpreter prompt I get a segfault.

Stepping through the suspect code yielded very strange results, so I tried
recompiling cephesmodule.c with -O0.  That seems to have solved the problem
for now.  I guess that means it's a GCC optimizer bug.

-- 
Skip Montanaro
Got spam? http://www.spambayes.org/
skip at pobox.com



From jimcser at pacifier.com  Tue Aug 17 13:54:01 2004
From: jimcser at pacifier.com (Jim Cser)
Date: Tue Aug 17 13:54:01 2004
Subject: [Numpy-discussion] Re: Using sum() within a function
In-Reply-To: <411E7E49.2020709@pacifier.com>
References: <411E7E49.2020709@pacifier.com>
Message-ID: <41225FDC.8080000@pacifier.com>

> Example-- variables x,y,z,t; dimensions numX, numY, numZ, numT;
> functions f1(x,y,z,t), f2(y,z,t);  want to calculate f1*f2 and
> sum over t to get out[x,y,z].

Cobbling together a number of suggestions, what finally worked was--

def f3(x, y, z, t_range=arange(numT)):
   tempval = 0.* x
   for t in t_range:
     tempval += f1(x,y,z,t) + f2(y,z,t)
   return tempval

out = fromfunction(f3,(numX,numY,numZ,1))


I couldn't quite get sum() to work inside the function, but this
is definitely good enough for now.  Thanks to all for your help.
-Jim Cser




From skip at pobox.com  Tue Aug 17 15:57:41 2004
From: skip at pobox.com (Skip Montanaro)
Date: Tue Aug 17 15:57:41 2004
Subject: [Numpy-discussion] core dump during import
Message-ID: <16674.3703.733907.166344@montanaro.dyndns.org>

(I posted this last week to scipy-user but received no responses.  Since it
seems to involve Numeric I thought I'd try here.)

------------------------------------------------------------------------------

I'm the "python guy" at work which by default makes me the "scipy
installation guy" as well, though I'm not using it for any of my own
projects.  Awhile ago I performed a very basic installation:

    install latest f2py
    install Numeric 23.1
    install SciPy_complete-0.3

The platform is Solaris 9 on Intel.  I'm using Python 2.3.4 (though
originally installed using 2.3.3).  GCC and G77 are both at 3.3.2.

The installation goes fine with the one caveat that I needed to point
LD_RUN_PATH at the g77 lib dir so libg2c.so.0 could be found at runtime.

If I execute

    from scipy.stats import distribution

at the interpreter prompt I get a segfault.  Here's the beginning of the gdb
traceback:

    #0  0x20ec8356 in ?? ()
    #1  0xdd2d7e24 in Cephes_InitOperators (dictionary=0x8486dfc)
        at Lib/special/cephesmodule.c:413
    #2  0xdd2d7a7a in initcephes () at Lib/special/cephesmodule.c:1105
    #3  0x080d082e in _PyImport_LoadDynamicModule (
        name=0x8040500 "scipy.special.cephes", 
        pathname=0x8040070 "/opt/lang/python/lib/python2.3/site-packages/scipy/special/cephes.so", fp=0x8148780) at ../Python/importdl.c:53
    #4  0x080cd637 in load_module (name=0x8040500 "scipy.special.cephes", 
        fp=0x8148780, 
        buf=0x8040070 "/opt/lang/python/lib/python2.3/site-packages/scipy/special/cephes.so", type=3, loader=0x0) at ../Python/import.c:1708
    #5  0x080ce345 in import_submodule (mod=0x8149fa4, subname=0x804050e "cephes", 
        fullname=0x8040500 "scipy.special.cephes") at ../Python/import.c:2291
    #6  0x080cdea4 in load_next (mod=0x8149fa4, altmod=0x81265e0, 
        p_name=0x8040930, buf=0x8040500 "scipy.special.cephes", p_buflen=0x80404fc)
        at ../Python/import.c:2111
    #7  0x080cfbf1 in import_module_ex (name=0x0, globals=0x8486c64, 
        locals=0x8486c64, fromlist=0x83e256c) at ../Python/import.c:1957
    #8  0x080ced40 in PyImport_ImportModuleEx (name=0x8369c34 "cephes", 
        globals=0x8486c64, locals=0x8486c64, fromlist=0x83e256c)
        at ../Python/import.c:1998
    #9  0x080a1bde in builtin___import__ (self=0x0, args=0x8483d4c)
        at ../Python/bltinmodule.c:45
    #10 0x080fac78 in PyCFunction_Call (func=0x8151bec, arg=0x8483d4c, kw=0x0)
        at ../Objects/methodobject.c:108
    #11 0x08062e90 in PyObject_Call (func=0x8151bec, arg=0x8483d4c, kw=0x0)
        at ../Objects/abstract.c:1755

Since I don't use SciPy, Cephes or f2py outside of installing SciPy I don't
really know where to start poking.  Frame 1 is at this line:

    413    f = PyUFunc_FromFuncAndData(cephes3a_functions, bdtr_data, cephes_4_types, 2, 3, 1, PyUFunc_None, "bdtr", bdtr_doc, 0);

and cephes3a_functions looks suspicious:

    (gdb) p cephes3a_functions
    $2 = {0x5, 0x6}

I presume considering the variable name those are supposed to be valid
function pointers.  Stepping through initcephes leads me to this:

    (gdb) n
    393         cephes3a_functions[0] = PyUFunc_fff_f_As_iid_d;
    (gdb) n
    391         cephes3_functions[2] = PyUFunc_ffF_F_As_ddD_D;
    (gdb) n
    392         cephes3_functions[3] = PyUFunc_ddD_D;
    (gdb) n
    394         cephes3a_functions[1] = PyUFunc_ddd_d_As_iid_d;
    (gdb) n
    395         cephes3_2_functions[0] = PyUFunc_fff_ff_As_ddd_dd;
    (gdb) p cephes3a_functions
    $9 = {0x5, 0x6}
    (gdb) p PyUFunc_ddD_D
    $10 = {void (char **, int *, int *, void *)} 0xdd2e4720 <PyUFunc_ddD_D>
    (gdb) p PyUFunc_ddd_d_As_iid_d
    $11 = {void (char **, int *, int *, 
        void *)} 0xdd2e4a20 <PyUFunc_ddd_d_As_iid_d>

It seems there's something amiss with the cephes3a_functions initialization.
In fact, other cephes*_functions variables appear bogus as well:

    (gdb) p cephes3_functions
    $12 = {0, 0, 0x3, 0x4}
    (gdb) p cephes1_functions
    $13 = {0x464c457f, 0x10101}
    (gdb) p cephes2_4_functions
    $14 = {0x7e34, 0x7}

Does this ring a bell with anybody?  Any clues about how to proceed would be
appreciated.

Thanks,

-- 
Skip Montanaro
Got spam? http://www.spambayes.org/
skip at pobox.com



From nadavh at visionsense.com  Wed Aug 18 03:32:05 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Wed Aug 18 03:32:05 2004
Subject: [Numpy-discussion] Bool array missing sum function.
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDB7@exchange2k.envision.co.il>

Why Bool arrays are missing the sum method?

>>> b = array((1,0,1), type=Bool)
>>> b
array([1, 0, 1], type=Bool)
>>> b.sum()

Traceback (most recent call last):
  File "<pyshell#129>", line 1, in -toplevel-
    b.sum()
  File "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", line 1125, in sum
    return ufunc.add.reduce(ufunc.add.areduce(self, type=type).flat, type=type)
  File "/usr/local/lib/python2.4/site-packages/numarray/ufunc.py", line 1165, in _cum_cache_miss
    mode, win1, wout, cfunc, ufargs = \
  File "/usr/local/lib/python2.4/site-packages/numarray/ufunc.py", line 1225, in _cum_setup
    otype, cfunc = ufdict[(optype,)]
KeyError: Bool

   Nadav.



From nadavh at visionsense.com  Wed Aug 18 03:36:10 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Wed Aug 18 03:36:10 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDB8@exchange2k.envision.co.il>

>>> a = arange(20, shape=(4,5))
>>> a
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])
>>> a[2:4,5:7]   # Shouldn't it raise index error?
array([])

>>> a[2:4, 5:7].iscontiguous()
0    # ????
>>> a[2:4, 5:7].copy().iscontiguous()
0    # ????

Shouldn't iscontiguous method of empty arrays and array.copy() return 1?

  Nadav.



From jmiller at stsci.edu  Wed Aug 18 03:48:04 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Aug 18 03:48:04 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DEDB8@exchange2k.envision.co.il>
References: 	 <07C6A61102C94148B8104D42DE95F7E86DEDB8@exchange2k.envision.co.il>
Message-ID: <1092825988.3752.37.camel@localhost.localdomain>

On Wed, 2004-08-18 at 07:36, Nadav Horesh wrote:
> >>> a = arange(20, shape=(4,5))
> >>> a
> array([[ 0,  1,  2,  3,  4],
>        [ 5,  6,  7,  8,  9],
>        [10, 11, 12, 13, 14],
>        [15, 16, 17, 18, 19]])
> >>> a[2:4,5:7]   # Shouldn't it raise index error?
> array([])

I wasn't sure about this,  but what we're doing is faithful to what
Numeric does.

> >>> a[2:4, 5:7].iscontiguous()
> 0    # ????

Likewise here.

> >>> a[2:4, 5:7].copy().iscontiguous()
> 0    # ????

We're not compatible here.

> Shouldn't iscontiguous method of empty arrays and array.copy() return 1?

I'd say we should go with compatibility.

Todd




From jmiller at stsci.edu  Wed Aug 18 04:09:30 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Aug 18 04:09:30 2004
Subject: [Numpy-discussion] Bool array missing sum function.
In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DEDB7@exchange2k.envision.co.il>
References: 	 <07C6A61102C94148B8104D42DE95F7E86DEDB7@exchange2k.envision.co.il>
Message-ID: <1092827284.3752.58.camel@localhost.localdomain>

I logged this as a bug on Source Forge and I'll look into it.

Todd

On Wed, 2004-08-18 at 07:32, Nadav Horesh wrote:
> Why Bool arrays are missing the sum method?
> 
> >>> b = array((1,0,1), type=Bool)
> >>> b
> array([1, 0, 1], type=Bool)
> >>> b.sum()
> 
> Traceback (most recent call last):
>   File "<pyshell#129>", line 1, in -toplevel-
>     b.sum()
>   File "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", line 1125, in sum
>     return ufunc.add.reduce(ufunc.add.areduce(self, type=type).flat, type=type)
>   File "/usr/local/lib/python2.4/site-packages/numarray/ufunc.py", line 1165, in _cum_cache_miss
>     mode, win1, wout, cfunc, ufargs = \
>   File "/usr/local/lib/python2.4/site-packages/numarray/ufunc.py", line 1225, in _cum_setup
>     otype, cfunc = ufdict[(optype,)]
> KeyError: Bool
> 
>    Nadav.
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 




From nadavh at visionsense.com  Wed Aug 18 04:51:18 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Wed Aug 18 04:51:18 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDB9@exchange2k.envision.co.il>



-----Original Message-----
From:	Todd Miller [mailto:jmiller at stsci.edu]
Sent:	Wed 18-Aug-04 13:46
To:	Nadav Horesh
Cc:	numpy-discussion
Subject:	Re: [Numpy-discussion] Empty arrays strange behaviour
On Wed, 2004-08-18 at 07:36, Nadav Horesh wrote:
> >>> a = arange(20, shape=(4,5))
> >>> a
> array([[ 0,  1,  2,  3,  4],
>        [ 5,  6,  7,  8,  9],
>        [10, 11, 12, 13, 14],
>        [15, 16, 17, 18, 19]])
> >>> a[2:4,5:7]   # Shouldn't it raise index error?
> array([])

I wasn't sure about this,  but what we're doing is faithful to what
Numeric does.

> >>> a[2:4, 5:7].iscontiguous()
> 0    # ????

Likewise here.

> >>> a[2:4, 5:7].copy().iscontiguous()
> 0    # ????

We're not compatible here.

> Shouldn't iscontiguous method of empty arrays and array.copy() return 1?

I'd say we should go with compatibility.

Todd



It isn't a major flaw, so compatibility is OK.
I got to this issue since I had a bug. If there was an option to get an IndexError exception it would ease the debugging.

Thank you,
  Nadav.



From tim.hochberg at cox.net  Wed Aug 18 07:24:02 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Wed Aug 18 07:24:02 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DEDB9@exchange2k.envision.co.il>
References: <07C6A61102C94148B8104D42DE95F7E86DEDB9@exchange2k.envision.co.il>
Message-ID: <4123662C.2090807@cox.net>

Nadav Horesh wrote:

>
>-----Original Message-----
>
>On Wed, 2004-08-18 at 07:36, Nadav Horesh wrote:
>  
>
>>>>>a = arange(20, shape=(4,5))
>>>>>a
>>>>>          
>>>>>
>>array([[ 0,  1,  2,  3,  4],
>>       [ 5,  6,  7,  8,  9],
>>       [10, 11, 12, 13, 14],
>>       [15, 16, 17, 18, 19]])
>>    
>>
>>>>>a[2:4,5:7]   # Shouldn't it raise index error?
>>>>>          
>>>>>
>>array([])
>>    
>>
>
>I wasn't sure about this,  but what we're doing is faithful to what
>Numeric does.
>
>[SNIP]
>
>
>I'd say we should go with compatibility.
>
>Todd
>
>  
>
Definately.

>It isn't a major flaw, so compatibility is OK.
>I got to this issue since I had a bug. If there was an option to get an IndexError exception it would ease the debugging.
>  
>
Please, no. In addition to breaking everybody's code, this change would 
make numarray's behaviour gratuitously different from the python list 
type as well as the numeric array type. In addition, the current 
behaviour does the Right Thing (TM) more often than not in the corner 
cases of algorithms.

-tim





From aisaac at american.edu  Wed Aug 18 08:15:11 2004
From: aisaac at american.edu (Alan G Isaac)
Date: Wed Aug 18 08:15:11 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
In-Reply-To: <4123662C.2090807@cox.net>
References: <07C6A61102C94148B8104D42DE95F7E86DEDB9@exchange2k.envision.co.il><4123662C.2090807@cox.net>
Message-ID: <Mahogany-0.66.0-588-20040818-111626.00@american.edu>

>> I'd say we should go with compatibility.
>> Todd


Script:
a=[[i+i*j for i in range(5)]for j in range(4)]
print a[2:4][5:7]
import Numeric
b=Numeric.array(a)
print b[2:4,5:7]
import numarray
c=numarray.array(a)
print c[2:4,5:7]

Output:
[]
zeros((2, 0), 'l')
[]

fwiw,
Alan Isaac





From jmiller at stsci.edu  Wed Aug 18 08:31:05 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Aug 18 08:31:05 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
In-Reply-To: <Mahogany-0.66.0-588-20040818-111626.00@american.edu>
References: 	 <07C6A61102C94148B8104D42DE95F7E86DEDB9@exchange2k.envision.co.il>
	 <4123662C.2090807@cox.net>
	 <Mahogany-0.66.0-588-20040818-111626.00@american.edu>
Message-ID: <1092843024.2260.17.camel@halloween.stsci.edu>

On Wed, 2004-08-18 at 11:16, Alan G Isaac wrote:
> >> I'd say we should go with compatibility.
> >> Todd
> 
> 
> Script:
> a=[[i+i*j for i in range(5)]for j in range(4)]
> print a[2:4][5:7]
> import Numeric
> b=Numeric.array(a)
> print b[2:4,5:7]
> import numarray
> c=numarray.array(a)
> print c[2:4,5:7]
> 
> Output:
> []
> zeros((2, 0), 'l')
> []
> 

You've got something of a point here,  if you're trying to say that
compatibility isn't perfect anyway.

Here's what I meant:

>>> from numarray import * 
>>> a = arange(20)
>>> a.shape = (4,5)
>>> a[2:4,5:7]
array([])
>>> a[2:4,5:7].info()
class: <class 'numarray.numarraycore.NumArray'>
shape: (2, 0)
strides: (20, 4)
byteoffset: 60
bytestride: 4
itemsize: 4
aligned: 1
contiguous: 0
data: <memory at 0x404c0298 with size:0x00000050 held by object 0x401839c8 aliasing object 0x00000000>
byteorder: little
byteswap: 0
type: Int32

The key points for me are:

1.  The slice operation doesn't raise an exception even though it has
"invalid" indexes.  As Tim H. pointed out,  those are standard Python
semantics.

2.  The shape of the resultant array is the same.

I don't know anything about the purpose of "zeros()" in this context. It
looks inconsistent to me.

Regards,
Todd





From jmiller at stsci.edu  Wed Aug 18 10:26:12 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Aug 18 10:26:12 2004
Subject: [Numpy-discussion] Empty arrays strange behaviour
In-Reply-To: <412378FE.5080203@cox.net>
References: 	 <07C6A61102C94148B8104D42DE95F7E86DEDB9@exchange2k.envision.co.il>
	 <4123662C.2090807@cox.net>
	 <Mahogany-0.66.0-588-20040818-111626.00@american.edu>
	 <1092843024.2260.17.camel@halloween.stsci.edu>  <412378FE.5080203@cox.net>
Message-ID: <1092849929.3418.41.camel@halloween.stsci.edu>

On Wed, 2004-08-18 at 11:42, Tim Hochberg wrote:
> Todd Miller wrote:
> 
> 
> [CHOP]
> 
> >The key points for me are:
> >
> >1.  The slice operation doesn't raise an exception even though it has
> >"invalid" indexes.  As Tim H. pointed out,  those are standard Python
> >semantics.
> >
> >2.  The shape of the resultant array is the same.
> >
> >I don't know anything about the purpose of "zeros()" in this context. It
> >looks inconsistent to me.
> >
> >  
> >
> I believe it's to disambiguate the various possible arrays of total size 
> zero. I like the Numeric behaviour better here, although it's not a big 
> deal. In the following, both the size and type of the numarrays are 
> ambiguous from their repr(). This means that among other things 
> eval(repr(arg)) won't return arg even for nice short arrays (I realize 
> the numarray truncates the reprs of long arrays by default and I think 
> that's fine -- I imagine there's some way to disable it, but I haven't 
> looked for it yet).
> 
> -tim
> 
>  >>> import Numeric as np
>  >>> np.zeros([0])
> zeros((0,), 'l')
>  >>> np.zeros([0,2])
> zeros((0, 2), 'l')
>  >>> import numarray as na
>  >>> na.zeros([0])
> array([])
>  >>> na.zeros([0,2])
> array([])
>  >>> na.arange(100)
> 
> 
OK, I logged this on SF.   Thanks for the explanation.

Todd




From southey at uiuc.edu  Fri Aug 20 07:52:09 2004
From: southey at uiuc.edu (Bruce Southey)
Date: Fri Aug 20 07:52:09 2004
Subject: [Numpy-discussion] Accessing a C library from numarray
Message-ID: <7644556.72de3142.81fd500@expms6.cites.uiuc.edu>

Hi,
I need to access a GLP'ed C library in numarray. The library functions are
scalar ints and doubles as inputs and usually doubles as outputs. 

Is there a recommended (and simple) way to achieve this? There are a few
different  C API's that exist from the Python to numarrays.

Would it be sufficient to first write C code that uses the library and use that
code for a C API in Python or numarray?

Thanks,

Bruce Southey



From smpitts at ou.edu  Mon Aug 23 09:25:16 2004
From: smpitts at ou.edu (smpitts at ou.edu)
Date: Mon Aug 23 09:25:16 2004
Subject: [Numpy-discussion] MA.zeros problem
Message-ID: <88f1be9567c6.4129d431@ou.edu>

Hi all,
I have some code that uses Numeric, and I'm trying to change it so that it supports MA as well. My code uses Numeric.zeros to create a matrix and then populates it, but I'm having trouble figuring out how to do something similar in MA.

Python 2.2.3 (#1, Oct 15 2003, 23:33:35) 
[GCC 3.3.1 20030930 (Red Hat Linux 3.3.1-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MA
>>> MA.__version__
'11.1.0'
>>> a = MA.zeros((2,2))
>>> a
array(
 [[0,0,]
 [0,0,]])
>>> a[1][1] = 2
>>> a
array(
 [[0,0,]
 [0,0,]])

Is there some function to automatically create a masked array of arbitrary dimensions filled with zeroes? Thanks.
--
Stephen Pitts
smpitts at ou.edu 




From reggie at merfinllc.com  Mon Aug 23 10:27:38 2004
From: reggie at merfinllc.com (Reggie Dugard)
Date: Mon Aug 23 10:27:38 2004
Subject: [Numpy-discussion] MA.zeros problem
In-Reply-To: <88f1be9567c6.4129d431@ou.edu>
References: <88f1be9567c6.4129d431@ou.edu>
Message-ID: <1093281989.31515.21.camel@pika>

Stephen,

You might try something like:

>>> a = MA.zeros((2,2))
>>> a
array(
 [[0,0,]
 [0,0,]])
>>> a[1,1] = 2
>>> a
array(
 [[0,0,]
 [0,2,]])

Because MA uses copy semantics, when you specify a[1][1], the a[1] makes
a copy of the first row of the original array, and then the second [1]
index does a setitem on that.  If you use the a[1,1] syntax, the setitem
is done on 'a' itself.

Hope this is of some help.

Reggie Dugard
Merfin, LLC


On Mon, 2004-08-23 at 09:25, smpitts at ou.edu wrote:
> Hi all,
> I have some code that uses Numeric, and I'm trying to change it so that it supports MA as well. My code uses Numeric.zeros to create a matrix and then populates it, but I'm having trouble figuring out how to do something similar in MA.
> 
> Python 2.2.3 (#1, Oct 15 2003, 23:33:35) 
> [GCC 3.3.1 20030930 (Red Hat Linux 3.3.1-6)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import MA
> >>> MA.__version__
> '11.1.0'
> >>> a = MA.zeros((2,2))
> >>> a
> array(
>  [[0,0,]
>  [0,0,]])
> >>> a[1][1] = 2
> >>> a
> array(
>  [[0,0,]
>  [0,0,]])
> 
> Is there some function to automatically create a masked array of arbitrary dimensions filled with zeroes? Thanks.
> --
> Stephen Pitts
> smpitts at ou.edu 
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion





From simon at arrowtheory.com  Wed Aug 25 17:28:08 2004
From: simon at arrowtheory.com (Simon Burton)
Date: Wed Aug 25 17:28:08 2004
Subject: [Numpy-discussion] Accessing a C library from numarray
In-Reply-To: <7644556.72de3142.81fd500@expms6.cites.uiuc.edu>
References: <7644556.72de3142.81fd500@expms6.cites.uiuc.edu>
Message-ID: <20040826100722.7278abdf.simon@arrowtheory.com>

This sounds like a job for pyrex. Here is a recent message from the
pyrex list.

cheers,
Simon.

On Wed, 25 Aug 2004 15:37:58 +0100 (BST)
Michael Hoffman <hoffman at ebi.ac.uk> wrote:

> I wanted to use Pyrex with Numarray, which is the supported
> implementation of Numeric, which is no longer supported. Changing the
> demo that comes with Pyrex to use Numarray instead of Numeric was
> instructive. I thought the results of this might be useful to anyone
> else trying to do the same thing. Here are diffs:
> 
> --- numeric_demo.pyx	2003-07-09 11:37:15.000000000 +0100
> +++ numarray_demo.pyx	2004-08-25 15:30:12.000000000 +0100
> @@ -1,33 +1,92 @@
>  #
>  #  This example demonstrates how to access the internals
> -#  of a Numeric array object.
> +#  of a Numarray object.
>  #
> 
> -cdef extern from "Numeric/arrayobject.h":
> +cdef extern from "numarray/libnumarray.h":
> +  ctypedef int maybelong
> 
> -  struct PyArray_Descr:
> -    int type_num, elsize
> -    char type
> +  cdef struct PyArray_Descr:
> +    int type_num # PyArray_TYPES
> +    int elsize   # bytes for 1 element
> +    char type    # One of "cb1silfdFD "  Object array not supported
> +    # function pointers omitted
> 
> -  ctypedef class Numeric.ArrayType [object PyArrayObject]:
> +  ctypedef class numarray._numarray._numarray [object PyArrayObject]:
>      cdef char *data
>      cdef int nd
> -    cdef int *dimensions, *strides
> +    cdef maybelong *dimensions
> +    cdef maybelong *strides
>      cdef object base
>      cdef PyArray_Descr *descr
>      cdef int flags
> +
> +    # numarray extras
> +    cdef maybelong *_dimensions
> +    cdef maybelong *_strides
> 
> -def print_2d_array(ArrayType a):
> -  print "Type:", chr(a.descr.type)
> -  if chr(a.descr.type) <> "f":
> +    cdef object _data         # object must meet buffer API
> +    cdef object _shadows      # ill-behaved original array.
> +    cdef int    nstrides      # elements in strides array
> +    cdef long   byteoffset    # offset into buffer where array data begins
> +    cdef long   bytestride    # basic seperation of elements in bytes
> +    cdef long   itemsize      # length of 1 element in bytes
> +
> +    cdef char   byteorder     # NUM_BIG_ENDIAN, NUM_LITTLE_ENDIAN
> +
> +    cdef char   _aligned      # test override flag
> +    cdef char   _contiguous   # test override flag
> +
> +  ctypedef enum:
> +    NUM_UNCONVERTED # 0
> +    NUM_CONTIGUOUS  # 1
> +    NUM_NOTSWAPPED  # 2
> +    NUM_ALIGNED     # 4
> +    NUM_WRITABLE    # 8
> +    NUM_COPY        # 16
> +    NUM_C_ARRAY     #  = (NUM_CONTIGUOUS | NUM_ALIGNED | NUM_NOTSWAPPED)
> +
> +  ctypedef enum NumarrayType:
> +    tAny
> +    tBool
> +    tInt8
> +    tUInt8
> +    tInt16
> +    tUInt16
> +    tInt32
> +    tUInt32
> +    tInt64
> +    tUInt64
> +    tFloat32
> +    tFloat64
> +    tComplex32
> +    tComplex64
> +    tObject                   # placeholder... does nothing
> +    tDefault = tFloat64
> +    tLong = tInt32,
> +    tMaxType
> +
> +  void import_libnumarray()
> +  _numarray NA_InputArray (object, NumarrayType, int)
> +  void *NA_OFFSETDATA(_numarray)
> +
> +import_libnumarray()
> +
> +def print_2d_array(_numarray _a):
> +  print "Type:", chr(_a.descr.type)
> +  if chr(_a.descr.type) <> "f":
>      raise TypeError("Float array required")
> -  if a.nd <> 2:
> +  if _a.nd <> 2:
>      raise ValueError("2 dimensional array required")
> +
> +  cdef _numarray a
> +  a = NA_InputArray(_a, tFloat32, NUM_C_ARRAY)
> +
>    cdef int nrows, ncols
>    cdef float *elems, x
>    nrows = a.dimensions[0]
>    ncols = a.dimensions[1]
> -  elems = <float *>a.data
> +  elems = <float *>NA_OFFSETDATA(a)
>    hyphen = "-"
>    divider = ("+" + 10 * hyphen) * ncols + "+"
>    print divider
> 
> --- run_numeric_demo.py	2003-07-09 11:37:15.000000000 +0100
> +++ run_numarray_demo.py	2004-08-25 12:06:47.000000000 +0100
> @@ -1,5 +1,5 @@
> -import Numeric
> -import numeric_demo
> +import numarray
> +import numarray_demo
> 
> -a = Numeric.array([[1.0, 3.5, 8.4], [2.3, 6.6, 4.1]], "f")
> -numeric_demo.print_2d_array(a)
> +a = numarray.array([[1.0, 3.5, 8.4], [2.3, 6.6, 4.1]], "f")
> +numarray_demo.print_2d_array(a)
> -- 
> Michael Hoffman <hoffman at ebi.ac.uk>
> European Bioinformatics Institute
> 
> P.S. Thanks for Pyrex! I just coded a Python implementation of a
> bioinformatics algorithm with the inner loop in Pyrex and it runs as
> fast as the pure C implementation we use in my group.
> 
> _______________________________________________
> Pyrex mailing list
> Pyrex at lists.copyleft.no
> http://lists.copyleft.no/mailman/listinfo/pyrex



On Fri, 20 Aug 2004 09:51:10 -0500
Bruce Southey <southey at uiuc.edu> wrote:

> Hi,
> I need to access a GLP'ed C library in numarray. The library functions are
> scalar ints and doubles as inputs and usually doubles as outputs. 
> 
> Is there a recommended (and simple) way to achieve this? There are a few
> different  C API's that exist from the Python to numarrays.
> 
> Would it be sufficient to first write C code that uses the library and use that
> code for a C API in Python or numarray?
> 
> Thanks,
> 
> Bruce Southey
> 
> 

-- 
Simon Burton, B.Sc.
Licensed PO Box 8066
ANU Canberra 2601
Australia
Ph. 61 02 6249 6940
http://arrowtheory.com 



From nadavh at visionsense.com  Thu Aug 26 05:28:31 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu Aug 26 05:28:31 2004
Subject: [Numpy-discussion] fromstring doesn't accept buffer object
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDCD@exchange2k.envision.co.il>

Numerc accepts a buffer object instead of a string:

  arr = Numeric.fromstring(buffer(.some_string), ..)
while numarray doesn't.

Is it intentional?

  Nadav.



From jmiller at stsci.edu  Thu Aug 26 05:53:28 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu Aug 26 05:53:28 2004
Subject: [Numpy-discussion] fromstring doesn't accept buffer object
In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DEDCD@exchange2k.envision.co.il>
References: 	 <07C6A61102C94148B8104D42DE95F7E86DEDCD@exchange2k.envision.co.il>
Message-ID: <1093524678.12979.1.camel@halloween.stsci.edu>

This is what I see this morning using CVS:

>>> from numarray import *
>>> s = "thisthis"
>>> fromstring(buffer(s), shape=(2,), type=Int32)
array([1936287860, 1936287860])

What kind of failure are you seeing?

Regards,
Todd

On Thu, 2004-08-26 at 09:28, Nadav Horesh wrote:
> Numerc accepts a buffer object instead of a string:
> 
>   arr = Numeric.fromstring(buffer(.some_string), ..)
> while numarray doesn't.
> 
> Is it intentional?
> 
>   Nadav.
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 




From nadavh at visionsense.com  Thu Aug 26 07:07:03 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu Aug 26 07:07:03 2004
Subject: [Numpy-discussion] fromstring doesn't accept buffer object
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDD0@exchange2k.envision.co.il>

Sorry for mixing test and production systems: There is no problem here with python 2.3, I got this failure with python 2.4a2:

==========================================
import numarray as n
import Numeric as N

s = '12341234'
b = buffer(s)

print 'numarray: fromstring (string ..) :'
print n.fromstring(s, type=n.Float64)

print 'Numeric: fromstring (string ..) :'
print N.fromstring(s, typecode=N.Float64)

print 'Numeric: fromstring (buffer ..) :'
print N.fromstring(b, typecode=N.Float64)

print 'numarray: fromstring (buffer ..) :'
print n.fromstring(b, type=n.Float64)

=== Results: ====


numarray: fromstring (string ..) :
[  3.05810932e-57]
Numeric: fromstring (string ..) :
[  3.05810932e-57]
Numeric: fromstring (buffer ..) :
[  3.05810932e-57]

=== This error happens only with Python 2.4.a2:

numarray: fromstring (buffer ..) :
Traceback (most recent call last):
  File "test.py", line 17, in ?
    print n.fromstring(b, type=n.Float64)
  File "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", line 378, in fromstring
    arr._data, 0, (type.bytes,), type.bytes)
libnumarray.error: copyNbytes: access beyond buffer. offset=7 buffersize=0


  Nadav.

-----Original Message-----
From:	Todd Miller [mailto:jmiller at stsci.edu]
Sent:	Thu 26-Aug-04 15:51
To:	Nadav Horesh
Cc:	numpy-discussion
Subject:	Re: [Numpy-discussion] fromstring doesn't accept buffer object
This is what I see this morning using CVS:

>>> from numarray import *
>>> s = "thisthis"
>>> fromstring(buffer(s), shape=(2,), type=Int32)
array([1936287860, 1936287860])

What kind of failure are you seeing?

Regards,
Todd

On Thu, 2004-08-26 at 09:28, Nadav Horesh wrote:
> Numerc accepts a buffer object instead of a string:
> 
>   arr = Numeric.fromstring(buffer(.some_string), ..)
> while numarray doesn't.
> 
> Is it intentional?
> 
>   Nadav.
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 







From exarkun at divmod.com  Thu Aug 26 19:20:05 2004
From: exarkun at divmod.com (Jp Calderone)
Date: Thu Aug 26 19:20:05 2004
Subject: [Numpy-discussion] Confusion regarding Numeric array resizing
Message-ID: <412E9A23.2000608@divmod.com>

   I'm confused by the restriction on resizing arrays demonstrated below:

 >>> from Numeric import array
 >>> a = array([0])
 >>> a.resize((2,))
 >>> b = a
 >>> a.resize((3,))
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
ValueError: cannot resize an array that has been referenced or is 
referencing another array in this way.  Use the resize function.
 >>> del b
 >>> a.resize((3,))
 >>> a
array([0, 0, 0])

   This seems like an unnecessary restriction.  Is there a reason I'm 
missing for it to be in place?

   Jp




From cookedm at physics.mcmaster.ca  Thu Aug 26 21:52:09 2004
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu Aug 26 21:52:09 2004
Subject: [Numpy-discussion] Confusion regarding Numeric array resizing
In-Reply-To: <412E9A23.2000608@divmod.com>
References: <412E9A23.2000608@divmod.com>
Message-ID: <20040827045208.GA11380@arbutus.physics.mcmaster.ca>

On Thu, Aug 26, 2004 at 10:19:15PM -0400, Jp Calderone wrote:
> 
>   I'm confused by the restriction on resizing arrays demonstrated below:
> 
> >>> from Numeric import array
> >>> a = array([0])
> >>> a.resize((2,))
> >>> b = a
> >>> a.resize((3,))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> ValueError: cannot resize an array that has been referenced or is 
> referencing another array in this way.  Use the resize function.
> >>> del b
> >>> a.resize((3,))
> >>> a
> array([0, 0, 0])
> 
>   This seems like an unnecessary restriction.  Is there a reason I'm 
> missing for it to be in place?

In this case it's obvious that b is not a copy of a (because that's how
Python assignment works), but if your statement was b = a[1:], it's a
little different.  Then, b is a view of a: a[1] = 1 will make b[0] == 1.
So, the question is, how should resizing a change b? The Numeric
developers decided to punt on this, I guess, and tell you to explictly
make a new copy.

Now, this restriction doesn't occur in numarray. Resizing an array will
(maybe) make a new copy of the underlying memory. Then b in this case
will no longer point to a's data, but will be the owner of the old data.

[I say "maybe" as it depends on where the memory for a came from.]

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From hsu at stsci.edu  Fri Aug 27 13:30:08 2004
From: hsu at stsci.edu (Jin-chung Hsu)
Date: Fri Aug 27 13:30:08 2004
Subject: [Numpy-discussion] marker edge color
Message-ID: <200408272029.ASF18358@donner.stsci.edu>

In 0.61.0, the marker edge color is still black, instead of being the same as 
the marker face color.  Personally, I think they should be the same.  What do 
people think?

One reason for the same color argument is that if the markersize is set to a 
small value, it will show mostly the edge color.

One possible reason against it is that if the marker color is white (or whatever 
the background color is), then you can't see the marker.  But we can default 
this case to black (or whatever).

JC Hsu




From hsu at stsci.edu  Fri Aug 27 13:43:14 2004
From: hsu at stsci.edu (Jin-chung Hsu)
Date: Fri Aug 27 13:43:14 2004
Subject: [Numpy-discussion] error bar default color
Message-ID: <200408272042.ASF18919@donner.stsci.edu>

In 0.61.0, when plotting a simple array with error bars, the default color of 
the error bars is black, instead of being the same as the line/markers color, 
e.g.:

>>> errorbar([1,2,3,4,5],[3,4,5,6,7],fmt='ro',yerr=[1,1,1,1,1])

I prefer them to be the same, especially since the default color for marker/line 
is blue and a beginner may be surprised to see the different color.

This may be related to my last posting regarding the marker edge color.

JC




From owen at astro.washington.edu  Fri Aug 27 17:38:31 2004
From: owen at astro.washington.edu (Russell E. Owen)
Date: Fri Aug 27 17:38:31 2004
Subject: [Numpy-discussion] rebin
In-Reply-To: <200408272042.ASF18919@donner.stsci.edu>
References: <200408272042.ASF18919@donner.stsci.edu>
Message-ID: <20040827203447.71bo3cw0c4oggcg8@webmail.spamcop.net>

Any suggestions on an efficient means to bin a 2-d array? REBIN is the IDL
function I'm trying to mimic. Binning allows one to combine sets of pixels from
one array to form a new array that is smaller by a given factor along each
dimension.

To nxm bin a 2-dimensional array, one averages (or sums or ?) each nxm block of
entries from the input image to form the corresponding entry of the output
image.

For example, to 2x2 bin a two-dimensional image, one would:
average (0,0), (0,1), (1,0), (1,1) to form (0,0)
average (0,2), (0,3), (1,2), (1,3) to form (0,1)
...

In case it helps, in my immediate case I'm binning a boolean array (a mask) and
thus can live with almost any kind of combination.

-- Russell



From nadavh at visionsense.com  Sat Aug 28 13:54:05 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Sat Aug 28 13:54:05 2004
Subject: [Numpy-discussion] rebin
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DEDD2@exchange2k.envision.co.il>

For the most general form of binning I use a convolution (by a 2D mask) followed by a subsmapling.

For example for a 3x3 binning:
mask = ones((3,3))
binned = convolve2d(data,mask,'same')[1::3,1::3]

  Nadav.
-----Original Message-----
From:	Russell E. Owen [mailto:owen at astro.washington.edu]
Sent:	Sat 28-Aug-04 03:34
To:	numpy-discussion at lists.sourceforge.net
Cc:	
Subject:	[Numpy-discussion] rebin
Any suggestions on an efficient means to bin a 2-d array? REBIN is the IDL
function I'm trying to mimic. Binning allows one to combine sets of pixels from
one array to form a new array that is smaller by a given factor along each
dimension.

To nxm bin a 2-dimensional array, one averages (or sums or ?) each nxm block of
entries from the input image to form the corresponding entry of the output
image.

For example, to 2x2 bin a two-dimensional image, one would:
average (0,0), (0,1), (1,0), (1,1) to form (0,0)
average (0,2), (0,3), (1,2), (1,3) to form (0,1)
...

In case it helps, in my immediate case I'm binning a boolean array (a mask) and
thus can live with almost any kind of combination.

-- Russell


-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion






From perry at stsci.edu  Mon Aug 30 06:15:08 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Mon Aug 30 06:15:08 2004
Subject: [Numpy-discussion] rebin
In-Reply-To: <20040827203447.71bo3cw0c4oggcg8@webmail.spamcop.net>
References: <200408272042.ASF18919@donner.stsci.edu> <20040827203447.71bo3cw0c4oggcg8@webmail.spamcop.net>
Message-ID: <7A026B97-FA86-11D8-B467-000A95B68E50@stsci.edu>

On Aug 27, 2004, at 8:34 PM, Russell E. Owen wrote:

> Any suggestions on an efficient means to bin a 2-d array? REBIN is the 
> IDL
> function I'm trying to mimic. Binning allows one to combine sets of 
> pixels from
> one array to form a new array that is smaller by a given factor along 
> each
> dimension.
>
> To nxm bin a 2-dimensional array, one averages (or sums or ?) each nxm 
> block of
> entries from the input image to form the corresponding entry of the 
> output
> image.
>
> For example, to 2x2 bin a two-dimensional image, one would:
> average (0,0), (0,1), (1,0), (1,1) to form (0,0)
> average (0,2), (0,3), (1,2), (1,3) to form (0,1)
> ...
>
> In case it helps, in my immediate case I'm binning a boolean array (a 
> mask) and
> thus can live with almost any kind of combination.
>
Note that a boxcar smoothing costs no more than doing the above 
averaging.
So in numarray, you could do the following:

from numarray.convolve import boxcar
b = boxcar(a, (n,n))
rebinnedarray = b[::n,::n].copy()

or something like this (I haven't tried to figure out the correct offset
for the slice) where one wants to rebin by a factor of n in both 
dimensions.
We should probably add a built in function to do this.

Perry




From rowen at u.washington.edu  Mon Aug 30 09:07:07 2004
From: rowen at u.washington.edu (Russell E Owen)
Date: Mon Aug 30 09:07:07 2004
Subject: [Numpy-discussion] rebin
In-Reply-To: <7A026B97-FA86-11D8-B467-000A95B68E50@stsci.edu>
References: <200408272042.ASF18919@donner.stsci.edu>
 <20040827203447.71bo3cw0c4oggcg8@webmail.spamcop.net>
 <7A026B97-FA86-11D8-B467-000A95B68E50@stsci.edu>
Message-ID: <p06110406bd58fe1b0893@[128.95.99.44]>

At 9:14 AM -0400 2004-08-30, Perry Greenfield wrote:
>On Aug 27, 2004, at 8:34 PM, Russell E. Owen wrote:
>
>>  Any suggestions on an efficient means to bin a 2-d array?
>>  ...
>>  For example, to 2x2 bin a two-dimensional image, one would:
>>  average (0,0), (0,1), (1,0), (1,1) to form (0,0)
>>  average (0,2), (0,3), (1,2), (1,3) to form (0,1)
>>  ...
>>
>Note that a boxcar smoothing costs no more than doing the above averaging.
>So in numarray, you could do the following:
>
>from numarray.convolve import boxcar
>b = boxcar(a, (n,n))
>rebinnedarray = b[::n,::n].copy()
>
>or something like this (I haven't tried to figure out the correct offset
>for the slice) where one wants to rebin by a factor of n in both dimensions.
>We should probably add a built in function to do this.

Thanks! Great suggestion!

I think the polished version (for nxm binning) is:

from numarray.convolve import boxcar
b = boxcar(a, (n,m), mode='constant', cval=0)
rebinnedarray = b[n//2::n,m//2::m].copy()

A rebin function would be handy since using boxcar is a bit tricky.

-- Russell



From rowen at u.washington.edu  Mon Aug 30 09:44:06 2004
From: rowen at u.washington.edu (Russell E Owen)
Date: Mon Aug 30 09:44:06 2004
Subject: [Numpy-discussion] rebin (corrected)
In-Reply-To: <p06110406bd58fe1b0893@[128.95.99.44]>
References: <200408272042.ASF18919@donner.stsci.edu>
 <20040827203447.71bo3cw0c4oggcg8@webmail.spamcop.net>
 <7A026B97-FA86-11D8-B467-000A95B68E50@stsci.edu>
 <p06110406bd58fe1b0893@[128.95.99.44]>
Message-ID: <p0611040abd59042171e6@[128.95.99.44]>

At 9:06 AM -0700 2004-08-30, Russell E Owen wrote:
>At 9:14 AM -0400 2004-08-30, Perry Greenfield wrote:
>>...
>>Note that a boxcar smoothing costs no more than doing the above averaging.
>>So in numarray, you could do the following:
>>
>>from numarray.convolve import boxcar
>>b = boxcar(a, (n,n))
>>rebinnedarray = b[::n,::n].copy()
>>
>>or something like this (I haven't tried to figure out the correct offset
>>for the slice) where one wants to rebin by a factor of n in both dimensions.
>>We should probably add a built in function to do this.
>...
>I think the polished version (for nxm binning) is:
>
>from numarray.convolve import boxcar
>b = boxcar(a, (n,m), mode='constant', cval=0)
>rebinnedarray = b[n//2::n,m//2::m].copy()
>
>A rebin function would be handy since using boxcar is a bit tricky.

I made several mistakes, one of them very serious: the convolve 
boxcar cannot do the job unless the array size is an exact multiple 
of the bin factor. The problem is that boxcar starts in the "wrong" 
place. Here's an example:

Problem: solve rebin [0, 1, 2, 3, 4] by 2 to yield: [0.5, 2.5, 4.0]
where the last point (value 4) is averaged with next-off-the-end, 
which we approximate by extending the data
(note: my propsed "polished" version messed that up; Perry had it right).

Using boxcar almost works:

>>>  import numarray as num
>>>  from numarray.convolve import boxcar
>>>  a = num.arange(5)
>>>  a
array([0, 1, 2, 3, 4])
>>>  boxcar(a, (2,))
array([ 0. ,  0.5,  1.5,  2.5,  3.5])
>>>  b = boxcar(a, (2,))
>>>  b
array([ 0. ,  0.5,  1.5,  2.5,  3.5])
>>>  b[1::2]
array([ 0.5,  2.5])

but oops: the last point is missing!!!

What is needed is some way to make the boxcar average start later, so 
it finishes by averaging 4 plus the next value off the edge of the 
array, e.g. a hypothetical version of boxcar with a start argument:

>>>  b2 = nonexistent_boxcar(a, (2,), start=1)
>>>  b2
[0.5, 1.5, 2.5, 3.5, 4.0]
b[0::2]
[0.5, 2.5, 4.0]

nd_image.boxcar_filter has an origin argument that *might* be for 
this purpose. Unfortunately, it is not documented and my attempt to 
use it as desired failed. I have no idea if this is a bug in nd_image 
or a misunderstanding on my part:
>>>  from numarray.nd_image import boxcar_filter
>>>  # first the usual answer; omitting the origin argument yields the 
>>>same answer
>>>  b = boxcar_filter(a, (2,), origin=(0,), output_type=num.Float32)
array([ 0. ,  0.5,  1.5,  2.5,  3.5], type=Float32)
>>>  # now try the origin argument and get a traceback; origin=1 gives 
>>>the same error:
>>>  b = boxcar_filter(a, (2,), origin=(1,), output_type=num.Float32)
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
   File 
"/usr/local/lib/python2.3/site-packages/numarray/nd_image/filters.py", 
line 339, in boxcar_filter
     output_type = output_type)
   File 
"/usr/local/lib/python2.3/site-packages/numarray/nd_image/filters.py", 
line 280, in boxcar_filter1d
     cval, origin, output_type)
RuntimeError: shift not within filter extent

So, yes, a rebin function that actually worked would be a real godsend!

Meanwhile, any other suggestions? Fortunately in our application we 
*can* call out to IDL, but it seems a shame to have to do that.

-- Russell



From rlw at stsci.edu  Mon Aug 30 10:16:10 2004
From: rlw at stsci.edu (Rick White)
Date: Mon Aug 30 10:16:10 2004
Subject: [Numpy-discussion] rebin (corrected)
In-Reply-To: <p0611040abd59042171e6@[128.95.99.44]>
Message-ID: <Pine.GSO.4.44.0408301300130.8067-100000@sundog.stsci.edu>

On Mon, 30 Aug 2004, Russell E Owen wrote:

> nd_image.boxcar_filter has an origin argument that *might* be for
> this purpose. Unfortunately, it is not documented and my attempt to
> use it as desired failed. I have no idea if this is a bug in nd_image
> or a misunderstanding on my part:
> >>>  from numarray.nd_image import boxcar_filter
> >>>  b = boxcar_filter(a, (2,), origin=(1,), output_type=num.Float32)
> Traceback (most recent call last):
>    File "<stdin>", line 1, in ?
>    File
> "/usr/local/lib/python2.3/site-packages/numarray/nd_image/filters.py",
> line 339, in boxcar_filter
>      output_type = output_type)
>    File
> "/usr/local/lib/python2.3/site-packages/numarray/nd_image/filters.py",
> line 280, in boxcar_filter1d
>      cval, origin, output_type)
> RuntimeError: shift not within filter extent

Seems like you got close to the answer.  This gives the answer you want:

>>> boxcar_filter(a, (2,), output_type=num.Float32,origin=-1)
array([ 0.5,  1.5,  2.5,  3.5,  4. ], type=Float32)

And so this works for rebin:

>>> boxcar_filter(a, (2,), output_type=num.Float32,origin=-1)[::2].copy()
array([ 0.5,  2.5,  4. ], type=Float32)

But I still agree with Perry that we ought to provide a built-in rebin
function.  It is particularly useful for large multi-dimensional arrays
where it is wasteful (in both CPU and memory) to create a full-size
copy of the array before resampling it down to the desired rebinned
size.  I appended the .copy() so that at least the big array is not
still hanging around in memory (remember that the slice creates a
view rather than a copy.)
				Rick




From tim.hochberg at cox.net  Mon Aug 30 10:58:05 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Mon Aug 30 10:58:05 2004
Subject: [Numpy-discussion] rebin (corrected)
In-Reply-To: <Pine.GSO.4.44.0408301300130.8067-100000@sundog.stsci.edu>
References: <Pine.GSO.4.44.0408301300130.8067-100000@sundog.stsci.edu>
Message-ID: <41336A6A.90208@cox.net>

[SNIP]

>
>But I still agree with Perry that we ought to provide a built-in rebin
>function.  It is particularly useful for large multi-dimensional arrays
>where it is wasteful (in both CPU and memory) to create a full-size
>copy of the array before resampling it down to the desired rebinned
>size.  I appended the .copy() so that at least the big array is not
>still hanging around in memory (remember that the slice creates a
>view rather than a copy.)
>				Rick
>  
>
A reasonable facsimile of this should  be doable without dropping into 
C. Something like:

def rebin_sum(a, (m, n)):
    M, N = a.shape
    a = na.reshape(a, (M/m,m,N/n,n))
    return na.sum(na.sum(a, 3), 1) / float(m*n)

This does create some temps, but they're smaller than in the boxcar case 
and it doesn't do all the extra calculation. This doesn't handle the 
case where a.shape isn't an exact multiple of (m,n). However, I don't 
think that would be all that hard to implement, if there is a consensus 
on what should happen then.
    I can think of at least two different ways this might be done: 
tacking on values that match the last value as already proposed and 
tacking on zeros. There may be others as well. It should probably get a 
boundary condition argument like convolve and friends.
    Personally, I'd be find rebin a little suprising if it resulted in 
an average, as all the implementations thus far have done, rather than a 
simple sum over the stencil.  When I think of rebinning I'm thinking of 
number of occurences per bin, and rebinning should keep the totals 
occurences the same, not change them by the inverse of the stencil size.

My 2.3 cents anyway

-tim

>
>
>-------------------------------------------------------
>This SF.Net email is sponsored by BEA Weblogic Workshop
>FREE Java Enterprise J2EE developer tools!
>Get your free copy of BEA WebLogic Workshop 8.1 today.
>http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
>_______________________________________________
>Numpy-discussion mailing list
>Numpy-discussion at lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>  
>





From rowen at u.washington.edu  Mon Aug 30 11:28:16 2004
From: rowen at u.washington.edu (Russell E Owen)
Date: Mon Aug 30 11:28:16 2004
Subject: [Numpy-discussion] rebin (corrected)
In-Reply-To: <41336A6A.90208@cox.net>
References: <Pine.GSO.4.44.0408301300130.8067-100000@sundog.stsci.edu>
 <41336A6A.90208@cox.net>
Message-ID: <p06110413bd591f28c7b0@[128.95.99.44]>

At 10:56 AM -0700 2004-08-30, Tim Hochberg wrote:
>[SNIP]
>
>>
>>But I still agree with Perry that we ought to provide a built-in rebin
>>function.  It is particularly useful for large multi-dimensional arrays
>>where it is wasteful (in both CPU and memory) to create a full-size
>>copy of the array before resampling it down to the desired rebinned
>>size.  I appended the .copy() so that at least the big array is not
>>still hanging around in memory (remember that the slice creates a
>>view rather than a copy.)
>>				Rick
>>
>A reasonable facsimile of this should  be doable without dropping 
>into C. Something like:
>
>def rebin_sum(a, (m, n)):
>    M, N = a.shape
>    a = na.reshape(a, (M/m,m,N/n,n))
>    return na.sum(na.sum(a, 3), 1) / float(m*n)
>
>This does create some temps, but they're smaller than in the boxcar 
>case and it doesn't do all the extra calculation. This doesn't 
>handle the case where a.shape isn't an exact multiple of (m,n). 
>However, I don't think that would be all that hard to implement, if 
>there is a consensus on what should happen then.
>    I can think of at least two different ways this might be done: 
>tacking on values that match the last value as already proposed and 
>tacking on zeros. There may be others as well. It should probably 
>get a boundary condition argument like convolve and friends.
>    Personally, I'd be find rebin a little suprising if it resulted 
>in an average, as all the implementations thus far have done, rather 
>than a simple sum over the stencil.  When I think of rebinning I'm 
>thinking of number of occurences per bin, and rebinning should keep 
>the totals occurences the same, not change them by the inverse of 
>the stencil size.
>
>My 2.3 cents anyway

I agree that it would be nice to avoid the extra calculation involved 
in convolution or boxcar averaging, and the extra temp storage.

Your algorithm certainly looks promising, but I'm not sure there's 
any space saving when the array shape is not an exact multiple of the 
bin factor. Duplicating the last value is probably the most 
reasonable alternative for my own applications (imaging). To use your 
algorithm, I guess one has to increase the array first, creating a 
new temporary array that is the same as the original except expanded 
to an even mutiple of the bin factor. In theory one could avoid 
duplication, but I suspect to do this efficiently one really needs to 
use C code.

I personally have no strong opinion on averaging vs summing. Summing 
retains precision but risks overflow. Averaging potentially has the 
opposite advantages, though avoiding overflow is tricky. Note that 
Nadav Horesh's suggested solution (convolution with a mask of 1s 
instead of boxcar averaging) computed the sum.

-- Russell



From tim.hochberg at cox.net  Mon Aug 30 12:06:07 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Mon Aug 30 12:06:07 2004
Subject: [Numpy-discussion] rebin (corrected)
In-Reply-To: <p06110413bd591f28c7b0@[128.95.99.44]>
References: <Pine.GSO.4.44.0408301300130.8067-100000@sundog.stsci.edu> <41336A6A.90208@cox.net> <p06110413bd591f28c7b0@[128.95.99.44]>
Message-ID: <41337A61.2090400@cox.net>

Russell E Owen wrote:

> At 10:56 AM -0700 2004-08-30, Tim Hochberg wrote:
>
>> [SNIP]
>>
>>>
>>> But I still agree with Perry that we ought to provide a built-in rebin
>>> function.  It is particularly useful for large multi-dimensional arrays
>>> where it is wasteful (in both CPU and memory) to create a full-size
>>> copy of the array before resampling it down to the desired rebinned
>>> size.  I appended the .copy() so that at least the big array is not
>>> still hanging around in memory (remember that the slice creates a
>>> view rather than a copy.)
>>>                 Rick
>>>
>> A reasonable facsimile of this should  be doable without dropping 
>> into C. Something like:
>>
>> def rebin_sum(a, (m, n)):
>>    M, N = a.shape
>>    a = na.reshape(a, (M/m,m,N/n,n))
>>    return na.sum(na.sum(a, 3), 1) / float(m*n)
>>
>> This does create some temps, but they're smaller than in the boxcar 
>> case and it doesn't do all the extra calculation. This doesn't handle 
>> the case where a.shape isn't an exact multiple of (m,n). However, I 
>> don't think that would be all that hard to implement, if there is a 
>> consensus on what should happen then.
>>    I can think of at least two different ways this might be done: 
>> tacking on values that match the last value as already proposed and 
>> tacking on zeros. There may be others as well. It should probably get 
>> a boundary condition argument like convolve and friends.
>>    Personally, I'd be find rebin a little suprising if it resulted in 
>> an average, as all the implementations thus far have done, rather 
>> than a simple sum over the stencil.  When I think of rebinning I'm 
>> thinking of number of occurences per bin, and rebinning should keep 
>> the totals occurences the same, not change them by the inverse of the 
>> stencil size.
>>
>> My 2.3 cents anyway
>
>
> I agree that it would be nice to avoid the extra calculation involved 
> in convolution or boxcar averaging, and the extra temp storage.
>
> Your algorithm certainly looks promising, but I'm not sure there's any 
> space saving when the array shape is not an exact multiple of the bin 
> factor. Duplicating the last value is probably the most reasonable 
> alternative for my own applications (imaging). To use your algorithm, 
> I guess one has to increase the array first, creating a new temporary 
> array that is the same as the original except expanded to an even 
> mutiple of the bin factor. In theory one could avoid duplication, but 
> I suspect to do this efficiently one really needs to use C code.

I think you could probably do considerably better than the boxcar code, 
but it it looks like it would get fairly messy once you start worrying 
about odd number of bins. It might end up being simpler to implement it 
C, so that's probably a better idea in the long run.

> I personally have no strong opinion on averaging vs summing. Summing 
> retains precision but risks overflow. Averaging potentially has the 
> opposite advantages, though avoiding overflow is tricky. Note that 
> Nadav Horesh's suggested solution (convolution with a mask of 1s 
> instead of boxcar averaging) computed the sum.

I really have no strong feelings since I have no use for rebinning at 
the moment. Back when I did, it would have been for rebinning data from 
particle detectors. So for instance, you would change the bin size so 
that you had enough data in each bin that you could attempt to do 
statistics on it or plot it or whatever. In that domain it would make no 
sense to average on rebinning. However, I can see how it makes sense for 
imaging applications.

In the absence of any compelling reason to do otherwise, I imagine the 
thing to do is copy what every one else is doing as long as they're 
consistent. Do you know what Matlab and friends do?

-tim




From curzio.basso at unibas.ch  Tue Aug 31 06:04:00 2004
From: curzio.basso at unibas.ch (Curzio Basso)
Date: Tue Aug 31 06:04:00 2004
Subject: [Numpy-discussion] extracting a random subset of a vector
Message-ID: <4134753A.7070503@unibas.ch>

Hi all, I have an optimization problem.

I currently use the following code to select a random subset of a rank-1 
array:

----------------------------------------------------------

import numarray as NA
import numarray.random_array as RA

N = 1000
M = 100
full = NA.arange(N)
subset = full[RA.permutation(N)][:M]

---------------------------------------------------------

However, it's quite slow (at least with N~40k), and from the hotshot 
output is looks like it's the indexing, not the permutation, which takes 
time.
Does anyone have a suggestion on a faster pure-python implementation?

thanks



From rays at san.rr.com  Tue Aug 31 06:51:04 2004
From: rays at san.rr.com (RJ)
Date: Tue Aug 31 06:51:04 2004
Subject: [Numpy-discussion] Help req: IDL --> Numeric/numarray
Message-ID: <5.2.1.1.2.20040831064859.1db44500@blue-cove.com>

Hi all,

The current IDL REBIN() thread reminded me to post this here...
I'm trying to port Hongjie Xie's IDL/ENVI Implementation of the FFT Based 
Algorithm for Automatic Image Registration
http://www.nmt.edu/%7Ehjxie/xie-paper.pdf
ftp://ftp.iamg.org/VOL29/v29-08-10.zip
to Python/numarray/Numeric and running into some subtle problems.
For instance, the IDL ROT() requires some additional steps to emulate, and 
the MAX(array, position) as well.
And, I'm not at all familiar with the correspondence of the FFT 
libraries... even the simple shift-only routine
http://rjs.org/astro/1004x/Python/register/shift_idl.py
produces different (incorrect) x shift values than the IDL.

My current code
http://rjs.org/astro/1004x/Python/register/
conforms to Xie's flow as much as possible for now, but should and will be 
re-factored when functional. His LogPolar function, to convert rectangular 
coordinate arrays to polar, is particularly slow.

I was recently informed that IDL has a usable demo version which I will try 
this week, but I would love to hear if anyone else has interest in this.

I think that the algorithm would make a nice standalone module, or perhaps 
something for nd_image, as it calculates shift, scale and rotation for 2d 
arrays.
Xie's main examples are registration of satellite Earth imagery, but it 
should work just as well looking up ;-)

Thank you,
Ray


Secret anti-spam filter-passing text. Include with reply:
qwertyuiop 




From rays at san.rr.com  Tue Aug 31 07:05:10 2004
From: rays at san.rr.com (RJ)
Date: Tue Aug 31 07:05:10 2004
Subject: [Numpy-discussion] Help req: IDL --> Numeric/numarray
Message-ID: <5.2.1.1.2.20040831070138.1dba8e30@pop-server.san.rr.com>

Hi all,

The current IDL REBIN() thread reminded me to post this here...

I'm trying to port Dr. Hongjie Xie's IDL/ENVI Implementation of the FFT 
Based Algorithm for Automatic Image Registration (it calculates shift, 
scale and rotation for 2d arrays)
http://www.nmt.edu/%7Ehjxie/xie-paper.pdf
ftp://ftp.iamg.org/VOL29/v29-08-10.zip
to Python/numarray/Numeric and running into some subtle problems.
For instance, the IDL ROT() requires some additional steps to emulate, and 
the MAX(array, position) as well.
And, I'm not at all familiar with the correspondence of the FFT 
libraries... even the simple shift-only routine I translated
http://rjs.org/astro/1004x/Python/register/shift_idl.py
produces different (incorrect) x shift values than the IDL.

My current code
http://rjs.org/astro/1004x/Python/register/
conforms to Xie's flow as much as possible for now, but should and will be 
re-factored when functional; his LogPolar function, to convert rectangular 
coordinate arrays to polar, is particularly slow.

I was recently informed that IDL has a usable demo version which I will try 
this week, but I would love to hear if anyone else has interest in this.

I think that the algorithm would make a nice standalone module, or perhaps 
something for nd_image.
Xie's main examples are registration of satellite Earth imagery, but it 
should work just as well looking up ;-)

Thank you,
Ray


Secret anti-spam filter-passing text. Include with reply:
qwertyuiop  




From Chris.Barker at noaa.gov  Tue Aug 31 10:41:11 2004
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Tue Aug 31 10:41:11 2004
Subject: [Numpy-discussion] extracting a random subset of a vector
In-Reply-To: <4134753A.7070503@unibas.ch>
References: <4134753A.7070503@unibas.ch>
Message-ID: <4134B6CE.4080807@noaa.gov>

Curzio Basso wrote:

> import numarray as NA
> import numarray.random_array as RA
> 
> N = 1000
> M = 100
> full = NA.arange(N)
> subset = full[RA.permutation(N)][:M]
> 
> ---------------------------------------------------------
> 
> However, it's quite slow (at least with N~40k), 

you can speed it up a tiny bit my subsetting the permutation array first:
subset = full[ RA.permutation(N)[:M] ]

> and from the hotshot 
> output is looks like it's the indexing, not the permutation, which takes 
> time.

not from my tests:

import numarray.random_array as RA
import numarray as NA
import time

N = 1000000
M = 100
full = NA.arange(N)

start = time.clock()
P = RA.permutation(N)
print "permutation took %F seconds"%(time.clock() - start)
start = time.clock()
subset = full[P[:M]]
print "subsetting took %F seconds"%(time.clock() - start)

which results in:
permutation took 1.640000 seconds
subsetting took 0.000000 seconds

so it's the permutation that takes the time, as I suspected. What would 
really speed this up is a random_array.non-repeat-randint() function, 
written in C. That way you wouldn't have to permute the entire N values, 
when you really just need M of them.

Does anyone else think this would be a useful function? I can't imagine 
it wouldn't be that hard to write.

If M <<< N, then you could probably write a little function in Python 
that called randint, and removed the repeats. If M is only a little 
smaller than N, this would be slow.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From stephen.walton at csun.edu  Tue Aug 31 11:49:09 2004
From: stephen.walton at csun.edu (Stephen Walton)
Date: Tue Aug 31 11:49:09 2004
Subject: [Numpy-discussion] rebin (corrected)
In-Reply-To: <41337A61.2090400@cox.net>
References: <Pine.GSO.4.44.0408301300130.8067-100000@sundog.stsci.edu>
	 <41336A6A.90208@cox.net> <p06110413bd591f28c7b0@[128.95.99.44]>
	 <41337A61.2090400@cox.net>
Message-ID: <1093978080.5706.24.camel@freyer.sfo.csun.edu>

On Mon, 2004-08-30 at 12:05, Tim Hochberg wrote:
> Russell E Owen wrote:
> 
> > I personally have no strong opinion on averaging vs summing
> 
> I really have no strong feelings since I have no use for rebinning at 
> the moment. Back when I did, it would have been for rebinning data from 
> particle detectors.

In the IRAF image tools, one gets summing by setting
"fluxconserve=yes".  The name of this option is nicely descriptive of
what an astronomer would mean by summing as opposed to averaging.

Many of the images I work with are ratio images;  for example, a solar
contrast map.  When I reshape a ratio image I want average, not sum. 
So, I would have to say that I need to have both average and sum
available with an option to switch between them.

By the way, has anyone else read

http://adsabs.harvard.edu/cgi-bin/nph-data_query?bibcode=2004SoPh..219....3D&db_key=AST&link_type=ABSTRACT&high=400734345713289

Craig has implemented the algorithm described therein in PDL (Perl Data
Language, http://pdl.perl.org), and a Python based implementation would
be awfully nice.

Hope to see a few of you in Pasadena.

-- 
Stephen Walton <stephen.walton at csun.edu>
Dept. of Physics & Astronomy, Cal State Northridge
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040831/ad6daed5/attachment-0001.sig>

From rlw at stsci.edu  Tue Aug 31 12:49:06 2004
From: rlw at stsci.edu (Rick White)
Date: Tue Aug 31 12:49:06 2004
Subject: [Numpy-discussion] extracting a random subset of a vector
In-Reply-To: <4134753A.7070503@unibas.ch>
Message-ID: <Pine.GSO.4.44.0408311519050.19185-100000@sundog.stsci.edu>

On Tue, 31 Aug 2004, Curzio Basso wrote:

> Hi all, I have an optimization problem.
>
> I currently use the following code to select a random subset of a rank-1
> array:

Here's a slightly faster version.  It's about 3x faster than Chris Barker's
version (4x faster than your original version) for N=1000000, M=100:

import numarray as NA
import numarray.random_array as RA
from math import sqrt

N = 1000000
M = 100
full = NA.arange(N)

r = RA.random(N)
thresh = (M+3*sqrt(M))/N
subset = NA.compress(r<thresh, full)
while len(subset) < M:
    # rarely executed
    thresh = thresh+3*sqrt(M)/N
    subset = NA.compress(r<thresh, full)
subset = subset[RA.permutation(len(subset))[:M]]

By the way, I also find that most of the time gets spent in the
permutation computation.  That's why this is faster -- it gets do a
smaller permutation.
					Rick